How Do You Screen for AI Judgment in Finance or Marketing?

What does AI judgment look like outside engineering?

The same as inside it. Framing the problem before generating, demanding a source for the claim that matters, keeping the judgment you should not hand over, building something between the brief and the deliverable, refusing an answer on stated grounds, and testing a claim against something outside the conversation. None of those is about code. All six are visible in a memo, a model, a brief or a spec.

The coding-adjacent answer is everywhere because engineering hands you three free graders. A diff shows what the assistant wrote and what the candidate changed. A test suite says whether the thing works without anyone's opinion. A compiler refuses to be talked around. Finance, marketing, consulting and product have none of that: a memo compiles no matter what it says, and a deck passes every test in the world.

So the instinct to skip the assessment for those requisitions is understandable and backwards. Those functions are where a confident wrong answer travels furthest, because nothing downstream catches it. A growth figure lands in a board deck. A competitor claim lands in positioning. An assumption lands in a spec that three teams then build against. The check happens at the desk or it never happens.

Decide first which roles genuinely need this, because not every opening does. Then keep one rubric across all of them and let only the task change per function. That split is the thing worth building around, and it is what lets a marketing result and a finance result mean the same words.

What do the six behaviors look like in a memo or a model?

Name the artifact first, then the observable. Marketing's deliverable is a report translating complex findings into written text, a forecast of sales and marketing trends, an analysis of competitors' prices and methods ⁴. Finance's is a memo or a model over a source packet. Consulting's is a sizing and a recommendation. Product's is a spec. Each has one moment where a claim either gets opened or gets copied.

That is a translation, not a menu. The row is identical everywhere; the anchor sentence under it is local, and it has to be written by someone who does that job. Get the anchors wrong and you have four rubrics wearing one name.

Two things get scored by accident in non-technical functions, and neither should. Prose quality, because the assistant is good at prose and the candidate's sentence rhythm is not what is being tested. And volume of AI use, because a candidate who decided the model was the wrong instrument for the pricing step and did it by hand has demonstrated the exact thing you are looking for. Grading a polished take-home breaks on precisely that confusion.

How do you build the task without a test suite?

Put the failure in the material rather than in the brief. Build a source packet where one number the deliverable depends on is contradicted by something you have to open: a footnote, a second tab, a methodology page. The assistant will summarize the packet fluently and carry the wrong number with it. Whether the candidate leaves the chat and opens the file is the assessment.

The packet has to come from the job. A selection procedure is supported by content validity only to the extent it is a representative sample of the content of the job, and where it samples a work behavior, its manner, setting, level and complexity should closely approximate the work situation ³. A marketing candidate sizing a semiconductor market has been tested on nothing they will ever do, and you would struggle to say otherwise if asked.

Forty-five to sixty minutes is enough, and the packet should be big enough that nobody reads all of it. Scarcity of time is what forces the delegation decision you want to watch: something has to be handed over, and which thing tells you more than the finished memo does. Cap it, say the cap in the brief, and let the assistant be genuinely willing to write the whole deliverable.

The same design sits behind hiring for verification rather than production. Production is now free, so the exercise has to be built around the step that is not, and the step that is not free is the one where somebody leaves the conversation to find out whether a sentence is true.

Which behavior carries the most signal?

Verification, by a distance. It is the behavior most often skipped, the one that separates two candidates whose deliverables read identically, and the one that gets rarer as trust in the assistant grows: a survey of 319 knowledge workers found that higher confidence in generative AI went with less critical thinking, while higher confidence in one's own skill went with more ¹.

That same study describes the shift underneath the hiring problem. Generative AI moves the nature of critical thinking toward information verification, response integration and task stewardship ¹, which is to say that the part of the job that survives is the part most assessments never look at, and the part a fluent draft makes easy to skip.

Retrieval does not fix it, and buying a specialist tool does not either. The first preregistered evaluation of the AI legal research products sold to a document profession (systems built on retrieval-augmented generation and marketed as avoiding or eliminating hallucination) found they hallucinated between 17% and 33% of the time ². Those tools have a curated corpus behind them. A general assistant reading your source packet has less.

Verification is an act, never a claim. "I would double-check that" in a debrief is worth nothing; a recomputed figure, an opened page, or a number that changed because of what was found is worth everything. If you plant one error, plant it where the correct answer changes the recommendation, and watch whether the candidate catches it rather than whether they say they would.

How do you grade a memo two reviewers score the same?

Write the anchor before the first candidate and keep the outcome words coarse. Three words per behavior (demonstrated, partly demonstrated, not demonstrated), plus one sentence per function saying what satisfying it looks like there. Two reviewers reading the same session should land on the same three words. Where they do not, the anchor is ambiguous, and the anchor is what gets rewritten.

Resist the total. Six behaviors averaged into one figure is a claim about a person that none of the six supports, and it is the first thing anyone challenging a rejection will ask about. Report the six separately, each with the moment it rests on: the timestamp, the transcript turn, the version of the memo where the number changed. That record is also the only honest answer to why one candidate was preferred.

The reviewer has to be someone who does the work. A marketing lead cannot tell whether an analyst's multiple is wrong, and a finance lead cannot tell whether a positioning claim is unsupported by the survey it cites. Pair each function's case with a reviewer from that function, and give them the answer key rather than asking them to improvise one at 9pm.

Agreement is the part that decays, and it decays faster with every function added. Two reviewers scoring the same rubric the same way needs a double-marked sample and a short conversation about the disagreements, monthly. When two finalists hand in work of the same quality, the record of how each of them got there is the only thing left to read.

See what gets scored >