Weight the AI Line by What the Level Is Accountable For

Why does one fixed weight fail at both ends?

Because a single number can only be right for one level, and an AI line inherited from a template was set for no level at all. It arrives late, takes a weight chosen because the subject is loud rather than because anyone measured it, and then copies onto every requisition from intern to vice president.

A weight that never varies is a claim that the job never varies.

The organizations SHRM surveyed barely varied their method by level either.

From April to November 2021, SHRM fielded a benchmarking survey to a random sample of its member organizations, and the answers were self-reported. Of the organizations that responded, 34% used structured interviews to assess executive candidates, 37% to assess middle management and 36% to assess individual contributors ¹.

Three levels, and three figures within three points of each other.

A weight is a filter whether or not anyone called it one. The 1978 US federal Uniform Guidelines define a selection procedure as any measure, combination of measures, or procedure used as a basis for an employment decision. They say the term reaches from paper-and-pencil tests through informal or casual interviews and unscored application forms ².

Validation is owed only where adverse impact shows up, so the definition is scope rather than an audit mandate.

Choosing which lines about AI use belong on the scorecard is one job. Deciding what those lines count for is another, and it is the easier one to skip.

The line goes on. The number behind it stays whatever the template said.

What should the AI line weigh for a new grad?

Less than it probably does now. At entry the failure that actually shows up is not a candidate who cannot drive the tool. It is a candidate who is fast and cannot tell when the output is wrong, and speed is the half that assistance supplies free.

Weight verification, fundamentals and the willingness to say the answer looked off above any evidence of fluency.

Researchers at MIT, Princeton, Wharton and Microsoft pooled three company-run randomized trials at Microsoft, Accenture and an anonymous Fortune 100 firm, covering 4,867 developers. Using the assistant, a 2023-era code-completion tool, raised completed tasks by 26.08%. The standard error of 10.3% puts the honest range somewhere near 6% to 46% ³.

Less experienced developers both adopted the tool more and gained more from it.

The outcome counted was weekly completed pull requests, which says nothing about defect rates or about whether the work was worth doing.

The consulting evidence shows a similar compression, split by baseline performance rather than by experience. A pre-registered 2023 experiment put 385 Boston Consulting Group consultants on 18 tasks chosen to sit inside the model's capability, each assigned at random to GPT-4, to GPT-4 with a prompt-engineering overview, or to no AI. Consultants below the average performance threshold gained 43% against their own baseline, and those above it gained 17% ⁴.

Assistance compresses the field. It does not tell you who was better before it arrived.

But the same study ran a second experiment, with a separate 373 consultants, on one task deliberately placed outside the model's capability, and there the direction reversed.

The unassisted group came out ahead: 84.5% correct in the control group, 60% and 70% correct in the two AI conditions. Averaged across the two AI conditions, using GPT-4 cost 19 percentage points on that task ⁴.

The consultants in that experiment could not tell which side of the line the task sat on. A graduating student has had the least practice at seeing it.

Tool description is trainable. The discernment to reject a plausible answer is the part worth paying for.

Where should the weight peak, and why there?

In the middle, at the senior individual contributor and the first-line manager. That level is where AI-assisted work leaves the building with nobody standing behind it: no second reader, no partner signing the memo, no reviewer whose name goes on the file.

A junior's output gets read by somebody. A director's calls get argued over in a room.

The senior's draft ships.

Set the number before the loop opens, because a weight chosen after the finalists arrive is a preference with arithmetic attached.

Kuncel, Klieger, Connelly and Ones put the average correlation with job performance at .44 when applicant data were combined mechanically by a formula. The same kinds of data, combined by expert judgment, gave .28 ⁵.

The job-performance estimate in that meta-analysis rests on nine samples, with 1,392 people behind the mechanical estimate and 1,156 behind the judgment one.

The comparison is about how evidence gets combined at the end, not about which evidence to gather.

The same paper classifies a group consensus meeting as holistic combination, alongside individual expert judgment. What makes a method holistic is that data are combined by judgment rather than by a rule applied the same way every time ⁵.

Keep the debrief. Move the arithmetic out of it.

The bar this number multiplies also moves with the level: what a good AI answer sounds like at junior, mid and senior.

Whether the line acts as a hurdle or trades against the domain scores is a separate call, and setting it as a hurdle or a trade, never both, comes before the number does.

Ask a director for the standard, not the demo

At director level, personal fluency is the wrong thing to weight. What the level is accountable for is the standard: which work runs through a model, which does not, who reviews the output, and what happens when a team ships something nobody checked.

Weight the standard, and let the demo go.

A director who has never opened the tool is a gap a laptop and four weeks will close. A director who cannot name one decision that stays off a model is a gap that reaches every requisition under them.

Ask what they stopped, what they refused to automate, what they sent a team back to redo. Asking an executive candidate what they stopped rather than what they can describe is the version of this interview that produces evidence instead of a position.

A director with a standard names the work that goes nowhere near a model, and can say who decided that and when.

Without one, the answer is enthusiasm, at length, and enthusiasm is not a standard.

Audit the weight against what it removed

Audit at the end of the round rather than at the start of the next one. Pull the candidates the AI line moved. Ask what else was true of them: what their domain scores said, how many came from employers where the tools were banned, how many were over fifty.

A weight that changed nobody's rank order is decoration, and a weight that rewrote the shortlist is the round's real screen.

One number on every requisition is easier to explain to a candidate, and immune to the manager who decides after meeting somebody that this role needed more AI than the last one. Consistency is the argument for leaving the weight alone, and it holds.

A moving weight needs a job analysis written before the requisition opens, which is the same ground a defensible AI bar for a specific role stands on.

Watch what the line correlates with, too. When the AI scores line up almost exactly with writing quality, the line is measuring polish, and the weight is buying polish at whatever price the card set.

The line on the card stays where it is.

What changes is the number beside it, which now carries a level, a date, and one sentence saying what that level is accountable for. Write the sentence next to the number.

That sentence is the only part of the scorecard that will still make sense a year from now.

See what gets scored >