The Admin Console Counts Prompts, Not Judgment

What is actually in the usage report?

Counts, dates and rates. Microsoft's documentation for the Copilot usage report in the Microsoft 365 admin center lists enabled users, active users, an active users rate, total prompts submitted and average prompts submitted per user across the organization. A per-user table adds prompts submitted, active days and a last activity date for each licensed person ¹. The whole table exports to a spreadsheet. Every column measures contact with the product.

The definition of active is the line worth reading twice. A user counts as active in an app when they take an intentional action for an AI-powered capability: opening the Copilot pane in Word does not count, and submitting a prompt does ¹.

The unit is the submitted prompt.

Microsoft's is the report documented here, and what it does not hold is the part you want. The console can say how many prompts a person sent and on how many days, but not what any prompt asked for, whether the answer was right, or what the person did with it afterwards.

Adoption reporting was built for license administration. The report exists so an administrator can see which seats are in use, and its own recommendation card points at driving adoption ¹.

Nothing in that design is hostile to hiring. The report was simply never asked the hiring question.

A promotion decision asks something different of the same rows, and the rows do not change to answer it. Telling what separates heavy AI use from good AI use takes evidence the export does not carry.

Why doesn't a high prompt count mean high skill?

Because in the two field studies below, the measured gains from these assistants concentrated in the people who had the least to start with. Across a staggered rollout to 5,179 customer support agents at one software firm, access to a generative assistant raised issues resolved per hour by 14% on average. The improvement was 34% for novice and low-skilled agents and almost nothing for experienced, highly skilled ones ².

Read the setting before the number travels anywhere. One firm, one occupation, an assistant trained on that firm's own transcripts and rolled out in 2020 and 2021, with issues resolved per hour as the outcome.

Problems with correct answers, not judgment on open work.

Adoption splits the same way. In three company-run randomized trials at Microsoft, Accenture and an anonymous Fortune 100 firm, pooled across 4,867 developers, an AI coding assistant raised completed tasks by 26.08%, and less experienced developers both adopted it more and gained more ³.

The standard error on that estimate is 10.3%, so the honest range runs from roughly 6% to 46%.

The outcome counted is weekly completed pull requests. Volume, not defect rates or review burden.

Put the two together and the order a usage report produces looks less like a ranking of skill than a map of who leaned hardest on the tool.

A count also cannot see a wrong answer.

In a field experiment with Boston Consulting Group consultants, two groups were given the model and a control group was not, so a usage count could only have put the two AI groups ahead. On the one task chosen to sit outside its capability, they found the right answer 60% and 70% of the time. The control group, with no usage to report, found it 84.5% of the time, 19 percentage points ahead on average ⁵.

A single task and a single sample, on a 2023 model. The group given a prompt-engineering overview did worse, down 24 points, than the group given none, down 13.

What the result shows is that people could not tell which side of the capability line the task sat on. The console cannot tell either.

When does a usage report become a selection procedure?

The moment it moves a decision. A selection procedure, in the US federal Uniform Guidelines of 1978, is any measure, combination of measures, or procedure used as a basis for any employment decision. The term reaches the full range of assessment techniques, down to informal or casual interviews and unscored application forms ⁶. Read by analogy, since the Guidelines name no software, a spreadsheet of prompt counts used to choose between applicants fits that definition's words.

The definition is not an audit mandate. The validation burden in the Guidelines attaches where adverse impact appears, and the Guidelines say they do not require validity studies of selection procedures where no adverse impact results ⁶.

Reading the definition the other way is the useful part: calling the file a report does not put it outside.

Consistency is the nearer problem, and it needs no lawyer to see. Where seat licenses are not spread evenly across functions, one internal applicant may carry six months of telemetry and another, in a department that never bought the license, none at all.

A criterion that exists for some applicants and not others is not one criterion.

Where GDPR applies, the reuse has a name in the text. Article 5(1)(b) requires personal data to be collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes ⁷. Data gathered to administer licenses, then read to choose between applicants, is a second purpose.

Whether the second purpose is compatible belongs to employment counsel rather than to the administrator who can export the file. None of this is legal advice.

The same definition catches the exercise a hiring manager runs on the side, and when that becomes a test you have to validate is the same question asked about a different artifact.

Ask for one piece of work, and ask everyone for it

Replace the export with a request. Ask every internal applicant to bring one thing they shipped with the tool in the last quarter. Then fifteen minutes on it: what the model produced first, what they changed, what they checked it against, and what they decided not to hand over. The same request goes to applicants who hold no license, on work made with whatever tool they did have.

Fifteen minutes of that reaches what a count cannot. Someone who edited the output can name the sentence they rewrote and the reason for it. Someone who shipped it unread has no such sentence to name.

Three things are worth writing down while they talk:

• what the first draft got wrong, in the applicant's own words
• the one claim they went outside the model to confirm, and what they confirmed it against
• the decision they kept, and why that one

Self-report will not close the gap on its own, and one trial measured how far off it ran. In a randomized trial, 16 experienced open-source developers completed 246 real tasks on repositories they had worked on for about five years.

The developers forecast that AI tools would cut completion time by 24%, believed afterwards that the cut had been 20%, and were measured 19% slower ⁴.

That result rests on sixteen developers working on code they knew intimately, which is not a general finding about productivity. What travels is the direction of the error: they thought the tool sped them up while it slowed them down.

So ask what changed in the artifact, which is checkable, rather than how much time the tool saved, which is not.

The judgment underneath, whether an internal candidate can really move into an AI-heavy role, was never going to come out of a telemetry column, and it does not come out of a self-assessment either.

Write the rule before an applicant is pending

Settle the question while nothing is open. Write one line into the tool's governance page: usage reporting is for license administration and enablement, and it does not enter selection, promotion or performance records. Name who may export per-user data and what has to be true for an exception.

A rule written after an applicant has been named is not a rule.

The console does hold one fact worth having. If the role's whole content is running an assistant every day, an internal applicant with no recorded activity across the report's whole window is telling you something. Nowhere else is that fact written down.

Treat that silence as a question, then, rather than a finding.

Zero activity has dull explanations: a license assigned last week, a team standardized on a different tool, a leave of absence, a department policy that barred the assistant until spring.

The comparison problem does not disappear either. Three years of watching against one hour of evidence is the shape of every shortlist that mixes internal and external applicants, and telemetry makes one side look measured without making it better measured.

Expect the file to be asked for again at the next performance cycle, where how AI use belongs in a performance review, and how it doesn't is the same argument in different clothes.

IT can still pull the report tomorrow, and it will still show active days, prompts submitted and a last activity date for every licensed applicant. Set it beside the fifteen-minute walkthrough and the columns read as what they were built to be, a record of seats in use.

The decision was never in the export.

See the benchmarks >