Guides for engineering managers

Measuring AI coding tools: what to track and what to ignore

A short, honest set of team-level signals for judging whether your AI coding tools are paying off.

Keel team · 15 October 2026 · 6 min read

Your team has been using AI coding tools for a while. The bill is real, the enthusiasm is uneven, and someone wants to know whether it's worth it. You don't need a data team to answer that. You need a few honest signals, a baseline, and the discipline to ignore the numbers the tools are keenest to show you.

This guide covers what to track, what to ignore, how to run a simple comparison, and how to have the cost conversation.

Measure the team, not the individual

The first decision matters most. If you measure AI usage per engineer, you will get more usage and less trust. People notice when a dashboard ranks them, and they adjust to it.

DX is clear on this: research shows developers don't like metrics aimed at measuring individual engineers, so keep these metrics at the team level and out of individual performance reviews. That fits what we see. The question you're answering is "is this tool helping the team deliver?", and that's a team-level question.

Say this out loud when you start measuring: "We're looking at how the team works, not at who uses AI the most. None of this goes into reviews." Then stick to it.

Signals worth tracking

Pick four or five. More than that and nobody will look at them.

• Cycle time. From first commit (or ticket start) to production. If AI is helping, this should trend down over months, not days.
• Time to first review and review wait time. How long PRs sit before someone looks at them. This is often where AI gains disappear.
• PR size. Median lines changed or files touched per PR. Rising size is an early warning, not a win.
• Change failure rate. The share of deploys that cause an incident, rollback or hotfix. Speed only counts if this holds steady.
• Customer-reported bugs. A lagging but honest quality signal.
• A short team survey. Every month or quarter, three questions: Do AI tools save you time? Where do they slow you down? What should we stop paying for? Perception isn't everything, but it catches problems the numbers miss.

Alongside the numbers, keep a running list of concrete examples: work that shipped because of AI, or that took days instead of weeks. Leadership remembers stories far longer than charts.

Signals to ignore

These are the numbers vendor dashboards show first. They measure activity, not value.

• Tokens used or requests made. High usage can mean high leverage or a lot of flailing. You can't tell which.
• Suggestion acceptance rate. Accepting a suggestion and then rewriting it counts as a win. So does accepting a bad one.
• Lines of code written or "AI-generated". AI makes code cheap to produce. More of it is not more value, and often means more to review and maintain.
• "Percent of code written by AI". Hard to measure honestly, and a target that invites people to route everything through a tool whether it helps or not.
• Per-person usage leaderboards. See above. They tell you who likes the tool, not who ships good work.

Usage data has one legitimate use: spotting seats nobody uses, so you can stop paying for them.

A simple before/after comparison

You won't get a controlled experiment in a startup. You can get something honest enough to make decisions with.

1. Pick your baseline. Pull the signals above for the three months before a meaningful change: rolling out a tool, moving to agents, or a big jump in adoption. If you've used AI tools for a while already, use the last quarter as the baseline and compare against the next.
2. Write down what else changed. New hires, people leaving, a big launch, a re-org, a holiday month. These move cycle time as much as any tool, and you'll want the context later.
3. Compare medians and trends, not single weeks. One big PR or one bad incident can swing a week. Look at month-over-month movement.
4. Look at the pairs. Cycle time with change failure rate. PR size with review wait time. A gain in one that comes with a loss in the other is a trade-off, not a win.
5. Add the survey and the examples. Numbers tell you what moved; the team tells you why.
6. Write a one-paragraph conclusion. What improved, what got worse, what you'll change. Revisit it next quarter.

The cost versus value conversation

AI tool spend is now a visible line in many engineering budgets. Gergely Orosz notes in The Pragmatic Engineer article "Headed for the Exit: the Great Engineering Leader Career Break" (August 2026) that CTOs face pressure on business results "as AI coding bills rack up", often while being asked to cut costs elsewhere. You want to have this conversation on your terms, before finance starts it for you.

• Know your spend per month and its trend. Seats, usage-based charges, and any agent or API costs, in one number.
• Put it next to outcomes. "We spent this, and in the same period we shipped these things, with cycle time and quality looking like this."
• Cut visibly. Unused seats, overlapping tools, experiments that didn't work. A managed budget is easier to defend.
• Avoid the headcount trade. Framing AI spend as "cheaper than a hire" invites the obvious follow-up. Frame it as what the current team can now deliver.

Pitfalls to watch for

• Bigger PRs, slower reviews. AI makes it easy to write a lot of code quickly. If PR size climbs, review becomes the bottleneck, reviewers skim, and defects slip through. The fix is process, not more tools: smaller PRs as a team norm, clear review ownership, and a cap on how long PRs wait.
• Speed now, maintenance later. Code nobody fully understands gets expensive the first time it breaks. Watch incidents and "who owns this?" questions over a few quarters, not weeks.
• Measuring too early. The first month of a new tool is mostly learning. Give it a quarter before drawing conclusions.
• Letting metrics leak into reviews. Once usage numbers show up in a performance conversation, the team will never trust the dashboard again. If you're working out how to review engineers fairly in this new setup, judge outcomes and judgement, not tool usage.
• Chasing one number. Any single metric gets gamed. A small set, read together, is much harder to fool.

Once you have the signals, the next step is reporting them upward; our guide on how to show leadership what AI is doing for your team includes a one-page monthly template. And for the people side, see how to write an engineering performance review grounded in outcomes rather than activity.