How do software professionals really judge the code generated by AI?

An Unregistered Report on arXiv tackles the question we were avoiding: what criteria, what biases, do developers use when they accept - or refuse - the code of an LLM. This is the empirical foundation that was missing from the debate.

In plain terms

A paper published on arXiv on July 13, 2026 (arXiv:2607.09434) formalizes, as a Registered Report, a study on how professional developers evaluate code generated by tools like Copilot, ChatGPT, or Claude. In other words: the first rigorous attempt to measure what "accepting AI code" really means in practice.

What the approach brings

A Registered Report publishes the protocol (question, hypotheses, analysis plan) BEFORE data collection - peer-reviewed methodology in advance, results published regardless of their sign. This format, imported from experimental psychology, cuts p-hacking and post-hoc storytelling. Its presence in Software Engineering is in itself a signal: the field is finally demanding built evidence, not demo anecdotes. The arXiv abstract states it clearly: several years after Copilot, the literature lacks empirical foundations on the central act - human review of AI code.

Analysis - why it matters for the profession

1. The gap in the racket. We measure generation speed, acceptance in the editor, billed tokens. We do not measure - seriously - the quality of the criteria that devs use when they click "accept". This paper aims right at this blind spot.

2. The link with the "hype-fatigue" thread. Another arXiv paper published the same day ("Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions", arXiv:2607.08885) suggests that devs overestimate their ability to judge LLM outputs. Cross-referenced, the two paint an uncomfortable picture: we judge quickly, we judge poorly, we are confident. This forces us to rethink workflows - more automated safeguards downstream, less faith in the human eye upstream.

3. What the craft can take from it, right away. Two concrete actions: (a) make the review of AI code explicit (short checklist: intent, invariants, edge cases) rather than implicit; (b) measure at home the post-merge incidents related to AI code "accepted without discussion".

So what

For a technical director: don't wait for the final results to act. The demand for empirical foundations on "how we judge AI code" is already a strategic demand. Instrument your own acceptance flows - organizations that have data on their devs will have a real advantage over those that drive the review by intuition.