What Does 'AI Fluency' Mean on a Job Description, and How Do You Test It?

What does 'AI fluency' mean on a job description?

On its own, nothing you can test. In practice the phrase carries one of three meanings: an assistant will be available, the person is expected to use one on named tasks, or the person can be trusted with what it produces. The third is the actual skill, and it turns into a standard only when the job's own work is named.

The frameworks on offer stop one step short of that, by design. Anthropic's AI Fluency course, built with two academics, names four behaviors (Delegation, Description, Discernment and Diligence) and teaches practical skills for effective, efficient, ethical and safe AI interaction ¹. It travels across every discipline because it is a curriculum, and a curriculum cannot say what strong delegation looks like for an underwriter. Four good columns; no rows.

So a requirement written at framework altitude reproduces the problem it was meant to solve. "Fluent with AI tools" tells a candidate nothing about what to prepare and tells an interviewer nothing about what to mark. It also ages badly: a tool named in a requirement this quarter is a different product in two. The fix is to write the line from the tasks, which is worked through in how to write AI skills into a job requirement.

The search results for this phrase make the gap visible. Most of what ranks is course material (module summaries, quiz answers, certificate curricula), which answers what fluency is called and never what it looks like in the job you are filling.

What does it mean in six different functions?

Six different things, because the artifact that has to survive is different in each. In marketing the claim in a deck has to hold; in finance the number that moves a recommendation has to be re-derived; in software the change has to pass a test somebody ran. Each occupation's O*NET task list already names that work, which makes it the shortest route to a definition you can defend.

Read down the last column and the same shape appears six times: something the assistant asserted was tested against something outside the assistant. What does not appear is volume. A candidate who decided the model was the wrong instrument for a step and did that step by hand has demonstrated the thing being measured, and a rubric that rewards heavy usage marks them down for it.

The six also differ in what a wrong answer costs, which should set how hard the exercise pushes. A coding error surfaces in a test run; a miscoded claim surfaces in a payer denial weeks later. For the functions where nobody on the panel writes code, the translation is worked through in screening for AI judgment in a finance or marketing role.

Write the requirement so it can be tested

Three parts, one sentence each: the task the person will do with an assistant, the part of that task the assistant may do, and the thing they must be able to defend without one. Written that way, the requirement doubles as the test specification: anything you cannot picture a candidate doing in front of you does not belong in it.

A financial analyst opening reads: "You will draft diligence memos with an assistant available. It can summarize the packet. You are accountable for every figure in the recommendation, re-derived from the filing it came from." A revenue cycle opening reads: "You will work a denial queue with an assistant drafting appeals. You decide which denials to concede, and the code on every appeal is yours."

Four things to cut from the version you have now:

• Tool names as requirements. Name the task and let the tool be the candidate's business. A named product in a requirement screens for the last employer's licensing, not for judgment.
• Years of experience with a two-year-old category. "Three years of generative AI" excludes people who started when the tool did.
• "Proficiency," "fluency," "savvy" with nothing after them. Each of these is a rating with no object. Put the object in.
• Anything you will not assess. A requirement nobody tests is a filter that runs on how confidently a candidate claims things, which is the trait AI-assisted applications inflate first.

One requirement per function, not one for the company. The same sentence cannot cover a paralegal and a data scientist without becoming untestable, and whether one assessment can cover every role is the version of this question that decides your build.

How do you test for it?

Hand the candidate a real deliverable from the job, an assistant that will do all of it if nobody stops it, and source material whose claims only settle when someone opens them. Then read the record of the work rather than the finished artifact. The output of a strong candidate and a passive one look similar; the paths do not.

What to build, in order:

1. Take the task from the occupation, not from a framework. Pick one deliverable the person would produce in their first month, and take its shape from the job's own task list rather than from a generic prompt exercise. 2. Plant something that only fails on inspection. A figure the packet does not support, a precedent that says the opposite of the summary, a passing test that tests nothing. Fluency is only visible when there is something to catch. 3. Write the answer key before the first candidate. Name the two or three moments where a competent person diverges from a passive one, and what each looks like in the record. 4. Score behaviors separately, and never total them. Framing, evidence demanded, what was kept, what was refused, what was verified. A single number buys comparability you do not have and hides the shape that matters. 5. Two scorers, independently, reconciled on evidence. Each cites the moment behind the mark. Getting two people to the same mark is its own build. See writing a rubric two reviewers score the same way.

The legal frame points the same direction as the design one. A procedure that decides who advances is a selection procedure, and one that screens out a protected group has to be shown job-related for the position and consistent with business necessity ⁸. Under the Uniform Guidelines, a content validity argument rests on a job analysis of the important work behaviors, and the behavior demonstrated in the procedure has to be a representative sample of the behavior of the job ⁹. A generic AI-fluency quiz samples no particular job. One task from this occupation, with the assistant open, samples exactly one.

Keep it to the length people finish. An exercise built around one deliverable runs 45 to 70 minutes; past that, completion falls and the sample skews toward candidates with unclaimed evenings. The redesign path from a take-home you already run is in rebuilding a work sample test for AI-assisted work.

Don't test speed, and don't trust the self-report

Both fail on the same evidence. In a randomized trial, 16 experienced open-source developers took 19% longer to complete real issues when AI tools were allowed, and still believed afterwards that the tools had sped them up by 20% ¹⁰. If practitioners cannot report the effect on their own work correctly, an interview answer about how well someone works with AI is not a measurement.

That rules out most of what gets used as an AI fluency test today. A multiple-choice literacy quiz measures recall of tool vocabulary, which is the part that goes stale fastest. A certificate records attendance at a course. A four-D questionnaire returns an account of behavior rather than the behavior. Discernment and Diligence are acts, and asking about an act gets you a story about one.

Speed is the other trap, because it is the easiest thing to record. Time-to-completion rewards the candidate who accepted the first draft over the one who opened the source and found it did not say what the summary claimed. If the exercise is timed at all, cap it and ignore the time inside the cap.

What survives all three problems is an act somebody can point at afterward: a source opened, a figure recomputed, a direction refused with the reason stated. Build the exercise so those acts are possible and so their absence is visible, which is the design behind testing whether a candidate catches what the AI got wrong.

See what gets scored >