Feeling Faster Is Not Evidence

The uncomfortable study

Experienced developers were asked to work on real tasks in their own codebases. Half with AI tooling, half without. Randomised.

They expected to be faster with AI. Afterwards, they reported having been faster with AI.

The measured times said they had been slower.

The gap between what they felt and what happened was roughly forty percentage points. These were not naive users. They were experienced people, on their own code, sincerely reporting their experience.

Why the feeling misleads

The sensation is real. Something genuinely did get easier.

Staring at a blank page is unpleasant, and AI removes that. Having a draft in ten seconds feels like enormous progress, because the unpleasant part finished.

What the feeling does not include is the time that comes afterwards: reading carefully, correcting the confidently wrong bit, re-prompting because the tone was off, and reconciling the parts that contradict each other.

Effort saved at the start is vivid. Effort added later is diffuse. So the ledger feels positive even when it is not.

This is not an argument against AI

It would be easy to read that study as "AI does not help". That is not the lesson.

The lesson is that you cannot tell which case you are in by introspection. Sometimes it helps enormously. Sometimes it quietly costs you. Both feel the same from the inside.

So if you want to know, you have to look. And looking turns out to be much cheaper than people expect.

Ten cases in a document

Here is the entire apparatus.

Pick a task you do repeatedly. Collect ten to thirty real examples of it. Real ones, from your actual work, including the awkward ones. Save them somewhere boring, like a document.

That is it. That is the test set.

The power is not in the size. It is in the fact that the cases are fixed. When you change something, you run the old and new versions over the same inputs, and the comparison means something.

Include the awkward ones

The instinct is to collect clean, representative examples. Resist it.

Deliberately include:

• the case that went wrong last month
• the unusually long one
• the one with missing information
• the one where the right answer is "I need to ask a human"

A test set of easy cases tells you your prompt handles easy cases. You already knew that. The awkward ones are where changes actually show up, and where regressions hide.