Before You Open the Req, What Can Your Team Already Do With AI?

Why won't a survey tell you what your team can do with AI?

Because the answer is a self-assessment with a career attached to it. The person most excited about AI reports capability nobody has watched work; the colleague who quietly rebuilt a reconciliation script ticks "basic". In Microsoft and LinkedIn's 2024 Work Trend Index, a survey of 31,000 knowledge workers across 31 markets, 52% of people who use AI at work are reluctant to admit using it for their most important tasks ¹.

The honest respondent is wrong too, which is the part better wording cannot fix. METR ran 16 experienced open-source developers through 246 real issues, in repositories they had contributed to for years that average over 22,000 stars and a million lines of code. With AI tools allowed, the issues took 19% longer. The developers had forecast a 24% speedup, and after living through the slowdown they still believed the tools had made them 20% faster ². People measured that closely misread their own direction of travel, and a five-point scale on a form will not recover it.

Usage telemetry answers a different question. A seat-license report or an assistant's activity dashboard tells you which accounts were open, never what happened to the output, and it is looking at a fraction of the tools in play: in the same Work Trend Index, 78% of AI users bring their own AI tools to work, rising to 80% at small and medium companies ¹. Somebody who has been running an unapproved assistant for six months also has an obvious reason to keep the answer vague on a form carrying their name.

So stop asking what people can do and go read what they shipped. The artifacts are already sitting in a repository, a shared drive or a ticket queue. They are dated, they had a recipient, and nobody had to characterise themselves to produce one. An org-readiness quiz misses for the same reason from the other end: five questions about strategy and data maturity return a verdict about the company, and this req is about nine people.

Which three deliverables should you pull from each person?

Three things they shipped in the last six weeks, each with a real recipient: one routine deliverable, one that took a full day or more, and one where they were working past the edge of what they usually do. Three is the smallest number that separates a habit from a good week. Ask for the file, not for a description of the file.

What counts, by the work your team actually does:

• Software engineering. A merged pull request with its review thread attached, not a snippet.
• Financial analysis. A memo or model that carried a recommendation to someone who then acted on it.
• Marketing. A brief, a launch page or a deck with a market claim inside it.
• Legal operations. A redline or a position memo where the playbook and the counterparty disagreed.
• Revenue cycle. A worked denial or an appeal letter, with the outcome attached.
• Data and analytics. A result somebody made a decision on, plus the query behind it.
• Support and operations. An escalation reply that went to a customer under a person's name.

Ask for the working trail alongside each one where it still exists: the intermediate outline, the branch history, the draft with comments on it, the chat transcript if they kept it. Most people will not have kept the transcript, and that is fine. The marks below can be read off the artifact plus a short conversation, which is the whole reason this is an afternoon and not a project.

One instruction changes the quality of everything you get back: say why you are asking, in one sentence, before anyone sends a file. This is sizing a req, not building a performance file. A team that learns afterward that it was appraised will answer the next round of anything the way people answer surveys. If the map later tells you what AI is actually doing inside the role, that came from a room that was told the truth first.

Mark three things on each artifact: touched, checked, kept

Three columns, one row per deliverable, filled in a ten-minute conversation with the file open: where AI touched this, what was checked before it shipped and against what, and what the person kept for themselves on purpose. There is no rating. A cell is either filled with something specific and checkable, or it is empty, and empty is the finding.

The difference between a filled cell and an empty one is entirely concreteness:

• Touched. Filled: "generated the first draft of the appeal, then rewrote the clinical rationale by hand because the payer policy it cited was the wrong version." Empty: "used it a bit for wording."
• Checked. Filled: "re-added segment revenue against the 10-K and found the model had it a quarter out." Empty: "reviewed it carefully."
• Kept. Filled: "wrote the recommendation paragraph without the assistant, because the client reads that paragraph and nothing else." Empty: "I review everything myself."

Collect examples rather than ratings, for a reason the research spells out. A CHI 2025 study by Lee and colleagues at Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers and gathered 936 first-hand examples of using generative AI in work tasks, and reports that it shifts the nature of critical thinking toward information verification, response integration and task stewardship ³. The same study associates higher confidence in the tool with less critical thinking, and higher self-confidence with more ³. That is the second reason a self-rating misleads: the people surest of the output are checking it least.

Keep the marks per artifact rather than per person. Somebody can run a disciplined check on a client deliverable and none at all on an internal one, and that difference is useful. It is also the difference between an inventory you can act on and a judgment about a colleague. If the exercise pushes you toward a competency model afterward, give AI its own competency only where the deliverable actually changed.

What does the finished map usually show about verification?

Production everywhere, checking thin and uneven. Expect the touched column filled on nearly every row, the kept column on one or two, and the checked column filled only where a past mistake had a name and an owner. Treat that as a pattern to test against your own map rather than a measurement. What the team is short of is people who catch a near miss.

The near miss is the costly one, and the people closest to it say so. In Stack Overflow's 2025 developer survey, the most-cited frustration with AI tools was "AI solutions that are almost right, but not quite," named by 66% of the developers who answered that question, with roughly 45% also naming that debugging AI-generated code is more time-consuming ⁴. A visibly wrong answer costs a minute. One that is wrong about which quarter a figure came from survives a read-through and fails at the point where somebody spends money on it.

Read the map at task grain, because that is the grain AI arrived at. Anthropic's Economic Index, built from roughly a million Claude.ai conversations matched onto the Labor Department's O*NET task list, reports that around 36% of jobs had some use of AI for at least 25% of their tasks while only about 4% used it for at least 75%, with 57% of tasks augmented against 43% automated ⁵. One person's "uses AI daily" can mean two tasks out of nine, and the seven it does not cover are where your req is really pointed.

Three patterns are worth naming when you read the finished grid. A person whose checked column is filled every time, undocumented, is the one to ask to write the check down, and nobody has yet. A team where nobody's checked column is filled on the highest-consequence work does not have a hiring problem yet; it has a review problem, and a new hire lands inside it. And a person with one thin artifact may simply not have been given consequential work in the window, which is a staffing fact rather than a skill fact. Hiring against the second and third of those is how a req solves the wrong thing. See how Olive measures this.

Write the req against the gap, not against a tool list

Turn each empty column into a line the job post and the interview can both use. An empty checked column on high-consequence work becomes a requirement written as an act: reconciles a figure to its source before it ships. An empty kept column becomes a judgment requirement. If both columns are full across the team, the gap is capacity, and you are hiring for volume rather than for skill.

Three rewrites, in the order the map produces them:

• Before: "Experience with ChatGPT or Copilot required." After: "You will draft the quarterly variance memo with an assistant and be accountable for every figure in it."
• Before: "Strong attention to detail." After: "You will be the person who reconciles a generated number to the filing before it reaches the partner."
• Before: "AI-first mindset." After: "You will decide which parts of this workflow stay manual, and defend that split at review."

The map also decides which of the two gaps you are looking at, which is the question hiring versus training for AI skills assumes you have already answered. A team with filled touched columns and empty checked ones does not need a new headcount to teach it prompting; it needs one person who models the check, or a review step that makes the check mandatory. A team with empty columns across three functions has a triage problem instead, and which roles need AI skills first is the order to run it in.

Then carry the same three columns into the round you run. If checking is what the map says is missing, the exercise has to make checking observable rather than describable: hand the candidate source material that settles nothing until somebody opens it, and grade what they opened and what changed as a result. Hiring for verification rather than production looks different in every field, and it is the round your map has just justified.

The inventory is a sizing instrument with a short memory. Six weeks of shipped work is the whole window, so it misses skill that had no occasion, judgment exercised in a meeting, and anything a person learned in a job before this one. Use it to write a sharper req, keep it out of anyone's file, and re-run it after the hire lands, because the only real test of the req is whether the next map looks different.

See the benchmarks >