When you ask an AI assistant to audit a piece of code, you usually get a long, well-written list with a file and line for each finding and, very often, sorted from most to least critical. It is very tempting to treat that list as a result. It is not: it is raw material.
Same Question, Same Code, Different List
In a hands-on session with around ten developers, exactly the same prompt was run on the same project. Each person got a different list: between 10 and 27 findings, in a different order and, in some cases, with findings that appeared separately in one run and were absorbed into another finding in the next. Nobody got it wrong. Every ordering was defensible on its own.
The conclusion is not that the model fails, but something more uncomfortable: the severity a model returns is not a property of the system being audited. It is an output of that particular run, just like the wording of a headline would be. If your work plan comes from that order, you will have a different plan every morning.
Why It Cannot Know What Is Serious
Deciding what is critical requires knowing things that are not in the code: who uses the system, which data is sensitive for that business, how many customers would notice, whether external integrations depend on a specific behavior. The model only sees the repository, so it fills that gap with what is usually serious in general, not with what is serious in your case.
How to Rank Without Delegating the Judgment
The practical way out is to define your own criteria with axes that can be answered with a verifiable fact rather than a subjective score. Damage: what breaks and who notices. Radius: how many places must be touched at once, counted and listed, never "several". Reversibility: whether a clean revert undoes it or it leaves traces in data that is already stored. Precedence: whether fixing one makes fixing another cheaper or more expensive.
With those four axes, the order stops being an opinion: two different people should reach the same result, and if they do not, it is almost always because someone estimated a radius instead of counting it or slipped in an opinion as if it were a fact. They are not the only possible axes: depending on context you can add regulatory compliance, effort, or the type of customer affected. What matters is that every axis is measurable.
What to Do With the Model's List
Use it for what it is good at: finding candidates quickly and cheaply. Discard its severity column without arguing about it and rank with your own criteria. And remember that the bottleneck is not finding problems but verifying them: opening the file, looking at the line and deciding whether the finding tells the truth. That is human time and it cannot be delegated without losing the guarantee.
Quick Summary
An AI audit list is raw material, not a result. Its order changes between runs and does not reflect your business. Define measurable axes, do the ranking yourself and spend human time on verification.