Answers
Assessment design
Work samples, take-homes and rubrics, and what an exercise still measures once an assistant can produce the deliverable.
124 articles
Everything on assessment design
- What an AI Interview Score Predicts, and Who Measured It A peer-reviewed job-performance figure for machine-scored video interviews is an uncorrected .24, from a study four of whose five authors worked for the vendor.
- What an AI Actually Scores in a Spoken English Test Two vendors say their AI scores an automatic transcript of your speech. What that measures, where the error sits, and how to set up an accommodation first.
- An AI-Use Score That Tracks Polish Is Measuring Polish If the AI-use line agrees with the presentation rating on nearly every candidate, one impression likely produced both. The re-rating test, and the anchors that fix it.
- Public Questions Are Real Practice and a Bad Prediction Company-tagged question lists come from candidate recall, age without saying so, and sit in a corpus models solve at high rates. What that changes about preparing.
- When an Assessment Is Silent on AI, Ask Before You Start An assessment that says nothing about AI is common, not a trap. One message to the recruiter is the cheapest way to ask, and here is what to do when nobody replies.
- Use Their AI Account If They Provide One, Ask If They Don't Three setups: an account the employer provides, your own tools, or a brief that says nothing. What each one means for what gets read, and what to ask first.
- Why a Game-Based Assessment Has No High Score to Chase What a game-based assessment records, why your tier depends on employees you cannot see, and what a practice pack buys when it cannot buy the target.
- Which Roles Absorb a Career Changer Who Doesn't Know Your Field Sort the role's tasks by how long a wrong answer survives before something outside the person catches it. That column, not tool fluency, decides the fit.
- A Review Round Tests What to Check, Not Whether They Can Build Reviewing model-written code measures what to check and when to stop. The round only grades anything if you author the diff and write the key first.
- Your Coding Assessment Can Be Replayed, Not Just Read HackerRank and CodeSignal publish session replay as a feature employers switch on. What a reviewer sees in it, and why a corrected first attempt is worth leaving in.
- What a Paste Looks Like to a Coding Assessment, and What a Flag Does Coding platforms say they can record paste activity, tab switches and code-writing patterns. What that produces is a flag for a person to read, not a verdict.
- What Missing Empty and Error States Tell You About a Designer Missing empty, error and loading states in AI-made screens are what the request produced. What separates designers is the list they make before you point at anything.
- Finishing Fast Rarely Helps, and Your Time Can Feed a Flag Where vendors document it, the clock bounds the task rather than scoring it. What the timer records, who sees the number, and when speed is the measure.
- Hand a Candidate Five Documents and Ask Which One Cannot Go In A policy question everyone passes tests nothing. Hand a candidate five work items and two tools, ask what each would need before it could be used, and grade the reason.
- Grade a Data Analyst on What the Model Cannot See The model writes the SQL. Build the analyst exercise on a table with a defect in it, ask for the figure and the one thing that would move it, then grade the caveat.
- Reperform Three Items, and Interview the Disagreements Testing an accountant whose AI-built reconciliation you cannot read: pick three items, redo them yourself, and interview every gap against a source document.
- Keep the Algorithm Practice, Cut the Volume Models clear routine algorithm problems, and vendors and employers changed the round rather than drop it. What practice still buys, and what to drill instead.
- Give a Legal Candidate a Memo to Check, Not One to Write Swap the drafting exercise for a keyed verification pass: a two-page memo carrying eight cited propositions, some of them misgrounded, marked in forty minutes.
- Move a Cut Score for Evidence, Never for Yield A fail rate measures your exercise, your bar and your applicant pool at once. Diagnose which one before moving the line, and never adjust a cutoff by group.
- Hand a Manager Candidate One Flawed Document and Twenty Minutes The exercise is a review, not a conversation about reviewing: one document the team could have produced, one planted error, twenty minutes, three scores.
- Eight of Ten Is Not Agreement Until You Subtract Chance Raw percent agreement counts every match chance would have produced. Convert eight of ten to a kappa or an intraclass correlation, per rubric line, before relying on it.
- Ask Who Fixed It When It Broke, Not Who Built It Shipping an internal tool with AI evidences initiative and a finished delegation loop, not engineering judgment. Ask what broke, who fixed it, and who uses it now.
- Hand an Ops Candidate a Runbook That Fails Quietly Hand every ops candidate the same one-page runbook with the confirmation, the rollback or the escalation threshold cut out. What to grade, and what to ignore.
- What a Timed Assessment Does When You Close the Tab Whether you can stop and come back depends on the platform and how it is set up. What a countdown, a counting-up timer and a save-and-return link each do.
- Answer as Yourself at Work, Not the Ideal Employee Faking moves scores, which is documented, and it is still the weaker move. What the research shows and what an engineered profile costs you later.
- A PRD Is Free to Draft. Its Refusals Are Not. Clarity is free now. What separates two PM writing samples is the non-goal, the metric with its denominator, the option that lost, and a twenty-minute walkthrough.
- What a Proctoring Flag Actually Triggers What the platforms publish about flagged sessions, why one vendor's own precision figure puts roughly one flag in seven in the wrong, and what to ask for.
- An Agent Run Records the Model's Decisions, Not the Candidate's Twenty minutes of autonomous work is mostly the model's decisions. The candidate shows up at the brief, the interruptions, the checks afterwards and the discards.
- Grade a Security Candidate on Triage, Not on the Top Ten A security round built on recall tests what a model recites. Seed a real diff instead, then grade the order the candidate works in and what they decline to fix.
- Make the Repo Wrong, Not the Task Harder What to plant in an AI-open starter repo so finishing correctly requires checking: stale docs, a fixture that encodes the bug, and a dependency with an advisory.
- A Situational Judgement Test Has a Key, Not One Right Answer Scenario questions are scored against a key you never see, and the measured validity is the same whether the test asks what you would do or what you should do.
- Build the Support Hiring Test From the Tickets Your Bot Got Wrong Deflection deleted the ticket your old support exercise graded. Take three the bot answered wrong, hand over the whole thread, and grade the diagnosis first.
- Which Failure Does the Job Survive, Missing Work or Wrong Work? Which submission scores higher depends on the failure the role absorbs. Score coverage and correctness on separate lines, weights written before either file opens.
- Run the Exercise on Your Team. The Number It Returns Will Read Low. Correlating your team's AI exercise scores with job performance is a concurrent validation study. Expect a low correlation, and don't correct it for range restriction.
- A Missed Error Is a Verification Failure When Nothing Was Checked A miss has two causes: nothing was checked, or the candidate checked without the domain knowledge to see it. What the exercise recorded separates them.
- The AI Line Is a Hurdle or a Trade, and Most Teams Never Chose Decide before the loop opens whether the AI line is a hurdle nobody can trade against or a weight domain strength can offset, then write which one on the scorecard.
- Weight the AI Line by What the Level Is Accountable For Set the weight from what the level is accountable for: light at entry, heaviest where AI-assisted work ships unreviewed, and a standard-setting line for a director.
- What a Proctored Assessment Reaches on Your Machine Proctoring runs in tiers, from tab and paste tracking to full video, audio and screen recording with a photo ID matched on camera. Ask which tier you face.
- When the Reference and the Work Sample Disagree, Neither One Wins A reference summary and a one-session work sample are not two readings of the same thing. What each one observed, and the question that breaks the tie.
- Does Asking for an Accommodation on an AI Test Count Against You? Rarely in a process run well, and the law bans retaliation regardless. Where the request lands, what the panel is told, and the honest residual risk at a small employer.
- Make the Flexible Version the Default So Nobody Has to Disclose An accommodation request rate near zero measures the invitation, not the pool. Give the commonly requested adjustments to everyone and keep a named route for the rest.
- When AI Is Allowed in the Coding Round, Judgment Is the Test When AI is allowed in a coding interview, finishing stops being the bar. Interviewers watch how you frame, what you test, and what you refuse to ship unchecked.
- Outside Engineering, AI Skill Is Judgment About Your Own Work Outside engineering, the AI bar is recognizing when a confident output is wrong about your own field. Show it with an example, not a tool list.
- An AI Work Sample Tests Whether You Catch the Bad Answer An AI-collaboration exercise is usually built to include a wrong answer somewhere in the material. What earns the score is visible checking, not speed or polish.
- Asked to Redo the Exercise Live Without AI: Why, and What Passes A live, AI-free redo is usually a verification stage run on every finalist, not an accusation. Here is why it got added and what actually passes it.
- Send the Session Where You Corrected the Model Asked to share your AI chat log or screen recording? Send the real session, corrections included, and ask what's captured, who sees it, and how long it's kept.
- Before You Let an Assessment Record Your Screen or Your Face Screen capture and a face template are legally different things, and the second carries its own consent rules in a few states. What to ask before you agree.
- Ask for the Instance First, the Hypothetical Only Where There Is None Ask for the instance first, keep the hypothetical for candidates with no track record in it, and give a technical question a written pass bar or cut the slot.
- Same Model, Different Configuration, Different Impact A vendor audit describes an instrument under the vendor's conditions. The cutoff, the stage, the tuning target and the override path are yours, and each moves impact.
- Extra Time Has No Statutory Multiplier. Judge the Work, Not the Rate No US statute or agency document sets a time multiplier. The obligation is individual, and comparability only breaks if the clock was part of the measurement.
- Borrow the Bar, Not the Verdict, When You Cannot Judge the Work Two practitioners, half an hour each, give you the standard. The verdict stays yours if the exercise asks for a decision you can check without the domain.
- Two Hours Is the Ceiling, and It No Longer Bounds Effort Two hours is the working ceiling for a take-home, but the clock stopped bounding effort. Set the length from what you can grade, and bound the deliverable.
- A Question Bank Row Is Four Fields, and One Person Can Delete Keep the bank, shrink it, and give every row four fields: the question, the evidence a strong answer must contain, one follow-up, and the date it was last reviewed.
- Every Number on the Scale Needs an Observable Behavior Each point on an interview rating scale should name a behavior an interviewer could have watched, with a real quote beside it. Three anchored points beat five vague ones.
- Learn AI Is Not Advice Until It Names a Task You Already Do Every course platform and creator repeats learn AI because the vague version sells. Turn it into a two-week task on work you already do, and it becomes advice.
- The Less Discriminatory Alternative Is Usually a Stage, Not a Model Most of the disparity sits in settings you already control. The search order, the four cheap swaps, and the record worth keeping a year later.
- A Portfolio Piece Counts When You Can Defend Its Decisions AI could have built the artifact, so reviewers now grade the decisions behind it. Keep fewer projects and keep the record of why you made each choice.
- Replace the Five-Point Scale With a Call and Its Reason A rating out of five is optional. The call and the evidence behind it are not. Keep a scale only if a stated rule consumes it, and never average it with a tool's value.
- Ask a Reference for One Episode, Not an Overall Impression Rating questions come back as policy language. Ask a reference to walk through one bounded episode with a date on it, ask the same three of everyone, keep the words.
- Do Not Require a Skill Nobody Can Score: Borrow the Evaluator First A requirement nobody on the team can evaluate is one you cannot defend. Get a practitioner to write the answer key first, or post the task instead of the skill.
- Every Requirement Needs a Stage That Tests It Put the requirement list in two columns: the requirement, and the stage that will test it. Any line with an empty right column gets a stage or comes off.
- A Fixed Rule Beats the Same Manager's Holistic Read On the direct comparison the rule predicts better, but only if the weights are fixed before candidates are seen. Allow overrides, require them in writing, and count them.
- Send the Criteria, Withhold the Weights and the Key Candidates are owed the criteria and the intended time. The weights and the worked answer are what a model optimizes to, so those stay with the reviewer.
- The Take-Home Says No AI: What Happens If You Use It Anyway The take-home says no AI. Detection doesn't work reliably, but the rule itself is often part of what gets scored, and breaking it risks more than one bad grade.
- Diagnose a Thin Pipeline Before Widening It, Then Assess Everyone Nine applicants can mean a posting nobody saw or a skill few people have. Check three comparable postings, then spend the unused screening budget on assessing everyone.
- Vendor Validity Evidence Transfers Only If the Jobs Match Borrowing someone else's validation study has conditions attached, and the job analysis is the one that rarely arrives. What to ask a vendor before signing.
- Use a Real Task You Already Solved, and Keep the Outcome Source the exercise from a real problem your team closed six to twelve months ago. It resists an off-the-shelf answer and arrives with a known outcome.
- Pick One Load-Bearing Claim Before the Call, Then Stay on It Choose the claim that makes the rest of the application irrelevant if it is false, name the detail only a real owner would know, and write both down before dialling.
- What Survives Is Knowing Whether the Output Is Right The standard list of durable human skills is unfalsifiable. Here is the one skill employers are actually testing for now, and how to practice it directly.
- Do ADA Accommodations Apply to an AI-Based Assessment, and What Do You Offer? Yes, the ADA covers an AI-based assessment the same as a paper test. The rule that settles every request: change the conditions, never the skill being measured.
- How Do You Write an AI-Use Rubric Two Reviewers Score the Same Way? Anchor every level in the role's own evidence, rate one dimension at a time, calibrate both reviewers on three real sessions, then measure kappa per dimension.
- Is AI Training Enough, or Do You Have to Check the Work? Completion records attendance, not capability. Transfer depends on how much of a role's AI use is judgment. Run the training, then check the work with a work sample.
- The Algorithm Screen Now Measures Prep Time, Not Programming A model clears the standard format in seconds, so the screen now sorts on prep time. What to keep it for, and the multi-file exercise that replaces the ranking round.
- How Do You Assess a Candidate Who Mostly Supervises AI Agents? Skip the build exercise. Hand over a finished-looking agent run with real defects seeded in it, then score what gets checked, what gets rejected, what gets refused.
- How to Run an AI Skills Assessment on the Team You Have Give everyone the same hour of real occupational work with an assistant available, then read four behaviors off the record instead of collecting self-ratings.
- An Assessment Does Not Need to Live in Your ATS Work out what an assessment integration saves at your volume, then ask what it deposits in the candidate record. A link out and a report back is not the lesser option.
- Per-Candidate Pricing Buys Volume When You Need Depth The pricing model is a product argument in disguise. How to read per-candidate, per-seat and credit plans, and what the price buys in human reading time.
- A Timer Measures Speed, and Speed Is Now the Cheap Part A tight clock mostly separates tooling and quiet rooms. Keep a generous cap so an exercise cannot swallow a weekend, and put the clock on the part an assistant cannot do.
- Should You Build Your Own AI Exercise or Buy an Assessment? Building wins at one role and loses by the third. The cost runs per occupation: a realistic task, seeded errors, rubric anchors and reading hours for each one.
- A Multiple-Choice AI Fluency Test Measures Recall, Not Fluency A quiz measures whether someone can name four competencies. Measuring fluency takes an occupational task, an assistant, a deadline, and a planted error.
- What You May Collect and Keep From a Candidate's AI Session Capture the prompt-and-revision trail rather than the screen, say what happens to it before the candidate starts, and file it with the hiring record.
- Whose AI Account the Candidate Uses Changes What You Can Read Supply the tool when submissions must be comparable or the exercise turns on one model's behavior. Otherwise let candidates bring their own, and name the version.
- Should You Disqualify a Candidate for Using AI on a Take-Home? Disqualify only if the brief said no before they started. Otherwise grade the submission again for what was checked, and price the risk your field can't absorb.
- How Do You Run a Coding or Case Interview With AI Open? Hand the candidate AI output with a real flaw in it and score the audit: what they tested, what they refused to ship, and which option they killed.
- How Do You Tell Which Finalist Did the Thinking? Output converges when both finalists use the same model. Ask each the same three questions: how they framed it, which claim they checked, and what the check returned.
- How Do You Set a Defensible AI Bar for a Specific Role? A defensible bar comes from the occupation's own task list, sits at the minimum the job requires, and is documented before anyone is measured against it.
- Detector, Interview, or Work Sample: Which Defends the Decision? The AI detector loses on validity evidence, adverse impact, candidate experience and cost. Which of the other two defends your decision depends on the occupation.
- Two Graders on the First Ten, One After That Two reviewers on the first ten submissions, marked blind. Compare criterion by criterion, repair the wording where they diverged, then grade single with audits.
- How Do You Evaluate an AI-Skills Assessment Before Buying? Ask what the assessment is benchmarked against, what the candidate sees, whether it outputs a single number, and what evidence sits behind a divergent result.
- How Do You Evaluate a Portfolio When AI Made Half of It? Assume AI made the surface. Ask for the decision record each craft keeps (the cut list, the constraint set, the review trail) and rate it against a guide written first.
- What Should an Early-Career Assessment Measure, and Is Yours an Aptitude Test? Measure the job's actual work behaviors, not general aptitude. The vendor questions a repainted aptitude test can't answer, and the federal rule behind each one.
- Delegation, Description, Discernment, Diligence: What Each One Looks Like at Work Delegation, Description, Discernment and Diligence, shown inside a single quarterly attrition analysis. Only one of the four leaves a trace in the finished file.
- How Do You Grade a Take-Home When Every Submission Is Polished? Stop rating the artifact. Grade three decisions instead: what the candidate assumed, what they checked outside the prompt, and what they rejected. Ask for a short log.
- How Do You Hire Someone Who Can Tell When AI Is Wrong? Stop testing production. Hand over a real task with one wrong thing in the source, then grade four acts: which claim was checked, against what, when, and what changed.
- Six Categories of Hiring Assessment and What Each Can Claim Cognitive, personality, situational judgment, job knowledge, work sample, structured interview: what each licenses you to conclude, and how to choose backwards.
- The Validity Table Everyone Quotes Was Corrected in 2022 The 2022 re-analysis puts structured interviews at .42 and job knowledge tests at .40 close behind. The 1998 numbers on most vendor decks run .10 to .20 high.
- Can an Internal Candidate Really Move Into an AI-Heavy Role? Liking the tools is not evidence. Read three artifacts from a task in the destination role (the frame, the refusal, the check) before approving an internal move.
- Live Coding, Take-Home, or AI-Allowed Work Sample: Which Catches Real Skill? Watched coding measures composure, take-homes lose signal wherever the deliverable is a document, and an AI-allowed sample works only if someone authored the constraint.
- Is Your Assessment Broken After the Latest Model Release? The assessment isn't broken. A model release lowered what the task costs to finish, so a fixed cut score passes more people. Re-anchor the bar per occupation.
- One AI-Skills Assessment for Every Department, or Per-Role? One rubric across every department, one case per role: what has to be shared for results to compare, and what has to be local for the assessment to be job-related.
- Should You Run a Paid Trial Instead of an Assessment? A paid trial wins only when a week of the real work is reachable and the finalist can take the week. The full cost: your hours, theirs, and the employment you created.
- Paying for the Take-Home Buys a Shorter Take-Home You are generally not required to pay for a short exercise whose output you never use. Pay anyway: a flat rate, everyone at that stage, and the assignment gets shorter.
- Personality Explains a Few Percent, and Candidates Can Move It Square the validity vendors quote for conscientiousness: four to nine percent of the variation in performance. Use the profile for interview questions, never a cutoff.
- How Do You Pilot an Assessment Vendor Before Making It a Hiring Gate? Run the vendor's assessment beside your live process, act on none of it, and read the results against how your own hires performed. Fix the cohort and criterion first.
- Is a Prompt Engineering Certificate Worth Anything, or Should You Give a Work Sample? A prompt engineering certificate proves a course was finished. Here is the honest ranking of four credentials, and the 45-minute work sample that beats all of them.
- A Rank Order Is a Weaker Claim Than a Bar Write the criterion from the work, judge each candidate against it with the evidence attached, and compare only inside the group that already cleared it.
- What Should You Look for in a Candidate's AI Chat Log? Skip turn count and phrasing. Read where the first move aimed, which claim got a source demanded, what output was refused, and what was checked outside the chat.
- What Replaces an Online Assessment ChatGPT Solves in Thirty Seconds? Stop testing for the answer. Hand campus candidates occupational material with a defect in it, let them use AI, and grade what they framed, opened and refused.
- How Do You Score an Interview Answer the Candidate Produced With AI? Rate four items (framing, demanded evidence, output rejection, verification) against anchors written in your field's own evidence standard. One rubric, filled in.
- How Do You Screen for AI Judgment in Finance or Marketing? The six behaviors that show AI judgment in a diff are just as visible in a memo, a model or a brief. What each looks like in finance, marketing, consulting and product.
- Measure How Fast They Work With AI, or How Well They Judge It? Measure throughput only where a wrong output is cheap to catch and cheap to undo. Everywhere else, measure judgment and use the clock as a cap, not a score.
- Write the Answer Key Before You Send the Assignment A rubric with no reference answer anchors on whichever submission is read first. Do the assignment yourself, write the key from it, then grade one submission twice.
- Do Take-Home Assignments Still Tell You Anything if Candidates Use AI? A take-home an assistant can finish from the brief alone measures nothing. Put a real ambiguity in the material, then grade the questions asked and the checks run.
- Take-Home Assignment or Live Working Session: Which Shows AI Judgment? A live session shows AI judgment when the real work is fast and observable. When it runs across days, run a capped take-home plus a 20-minute walkthrough of the choices.
- What Does 'AI Fluency' Mean on a Job Description, and How Do You Test It? It means six different things across six functions. Write the requirement from the occupation's own tasks, then test one of those tasks with an assistant open.
- What's the Difference Between a Validated and a Bias-Audited Assessment? A bias audit is arithmetic on selection rates across sex and race groups; validation is evidence tied to one specific job. A clean audit proves nothing about your role.
- What Do You Need From an Assessment Vendor for a Bias Audit or EEOC Inquiry? The validity report, the job analysis, the impact figures, the auditor's letter. What each proves, who at the vendor writes it, and the reply that should stop a purchase.
- What Should an AI Skills Assessment Measure Before You Pay? Buy for job-anchored behavior, findings you can open, occupational data dated this year, and a report the candidate also receives. Tool familiarity predicts nothing.
- AI Fluency Is Four Competencies, Not a Tool List AI fluency is working with AI effectively, efficiently, ethically and safely, in four competencies. Nobody certifies it, so the employer writes the bar.
- What Should a Work Sample Test Now That AI Can Produce It? AI compresses output quality, so the artifact stops separating candidates. Keep the task, grade two acts: the framing before generating, and the check outside the chat.
- The Resume Screen Is Regulated AI. A Human-Scored Work Sample Usually Is Not. Decades of test litigation made assessments look like the risky stage. The 2023 and 2025 rules moved the exposure to the automated screen instead. What actually changes.
- Work Sample or AI-Collaboration Exercise: Which Predicts Performance? Neither wins outright. A classic work sample predicts while day-one work still resembles it; the AI-collaboration exercise predicts once an assistant drafts first.
- Work Sample, Interview, or Assessment: Which Catches AI Dependence? Work sample, interview or AI-skills test? One question decides: can a candidate who can't do the work pass with an assistant open? Only the recorded session holds.
- Write the Assignment Brief So Two Submissions Can Be Compared The prompt is the easy half. Name the time budget, the audience, what sits out of scope, and the tool rule, or you grade four assignments against one rubric.
This beat is written by Bill Nguyen & Olive.