Assessment design
A Situational Judgement Test Has a Key, Not One Right Answer
There's a scoring key, but not one right answer per scenario. Your result combines every item, so no single choice decides it. On the question that worries people most, the research answers a narrower point: situational judgement tests predict job performance about equally well whether the instruction asks what you should do or what you would do. Answer as the person who would actually hold the job, and keep AI out of the assessment itself.
The takeThe prep industry's advice to pick the most proactive option is a guess, not a key, and it is worst on the tests built around one employer's actual priorities. A scenario about escalating a problem has a different best answer at a hospital than at a startup. Reading the role and the organisation is the thing the test is asking about, and it is preparation that survives contact with a scoring key you never get to see.
Where Olive fits
Open a role and see what the work shows
If the assessment an employer sends you is Olive, using the AI assistant is the point: you work a real assignment openly with it, and a person writes or edits six findings describing how you framed the problem, what you delegated and what you verified. You're granted the identical report the employer reads, free.
Assess your shortlistIs There One Right Answer?
No, though there is a key. The test is scored, which means somebody decided in advance how good each response is, and your result combines every scenario rather than resting on one of them. What sits in that key is not published.
A situational judgement test estimates knowledge of what a good response looks like, which is a narrower thing than whether a person can do the job. Hiring teams reading what an early-career assessment should measure are told to ask a vendor, in writing, what evidence sits behind each rating. The research is modest in the same direction: in the 2022 re-analysis of the selection literature, these tests estimate at .26 against rated job performance, once performance ratings are corrected for their own unreliability 1.
A moderate predictor, in other words, of the kind that can run at the top of a large funnel.
Two hedges travel with that number. Virtually all the studies behind it were run on people already in the job, not on an applicant pool being sorted. The estimate also averages across instruments that measure quite different things 1.
That is part of why a scenario test is a weak place to end a hiring process. Modelling a screening stage built only from methods that run at volume, situational judgement tests among them, Berry and colleagues found the best combination topping out at .51, against .61 when a structured interview is in the mix 6. Those are modelled ceilings under stated assumptions, not results anyone observed, and no real screen is weighted the way the model weights one 6.
So a scenario test is best read as one leg of something longer, not the whole judgement.
Should You Answer Honestly or Strategically?
Answer honestly, and know how far the evidence goes: on predicting rated job performance, the two versions of the question perform the same. Sackett and colleagues report the studies by instruction type: 96 asked what a person should do, 22 asked what a candidate would do. Mean observed validity was .20 for both, rising to .26 after correcting for unreliable performance ratings 1.
That equality covers one property and no others: the scores line up with rated job performance about equally well 1.
How easy each wording is to fake is a separate question, and there the two do differ 1.
So the evidence does not settle the trade prep sites describe, honesty against a better score. What it does show is that a test asking how you would behave is no weaker than one asking what a person should do.
Research on interviews gives grounds to treat the what-would-you-do format as shadeable. An experiment on faking in structured interviews found past-behaviour questions, the "tell me about a time when" kind, more resistant to faking than situational questions asking what you would do 4. A meta-analysis comparing the two formats put past-behaviour questions with descriptively anchored rating scales at a higher mean validity, .63 against .47 for situational questions with the same scales 5.
Both findings are old, and neither describes an online scenario test taken alone. The 2002 estimates in particular rest on assumptions later shown to be too generous 5.
It is one reason a scenario test is a weak place to stop.
Read the Scenario Before You Rank the Options
Read each item for who carries the cost and what the role is accountable for, then answer as the person who would hold that job. A well-built scenario sets two reasonable duties against each other rather than putting one sensible option beside three foolish ones. The choice that fits a hospital may not fit a startup.
Preparing for that is not gaming it. In a predictive study of a situational panel interview, predictive validity was higher in a sample coached to convey relevant content accurately than in an uncoached one 7. That was 146 public safety incumbents in a human-scored panel, compared as two samples rather than randomly assigned, and not an online questionnaire 7.
The coaching studied was about content, not about tone or posture.
Take four things in with you when you open the test:
- What the posting says the role is responsible for, in the employer's own words
- Who in the scenario bears the cost if nothing happens
- Whether an option escalates, acts, or waits, and which of those the stated priorities favour
- Whether you would defend the same choice out loud in a later conversation
Answer the twentieth scenario the way you answered the second. Consistency is the part of this you actually control.
Two pages are worth reading before you sit one. Where your application actually goes after you hit submit walks the stages an application passes through after you send it. And whether to answer a personality test honestly covers the same choice on a personality questionnaire.
Don't Run It Through a Model
Don't, and the reason is not mysterious: at least one large employer rules it out in writing. The UK Civil Service publishes a candidate guide naming ChatGPT, Claude, Copilot and Gemini, and it splits their use into an acceptable list and an unacceptable one. Using AI to complete personality and ability assessments, situational judgement and numerical tests among them, sits on the unacceptable side 2.
The same guidance states the consequence. Candidates confirm before submitting that what they provide is true and accurate, and applications may be rejected where AI is used inappropriately at any stage 3. It draws one absolute line too: AI must not be used during a live assessment 3.
That is one employer's recruitment policy and it binds nobody else. The page names no detection method, publishes no rate, gives no examples and describes no appeal, so "may reject" is a stated power rather than a measured outcome 3. An employer that says nothing at all has not said yes.
The other reason has nothing to do with getting caught.
A model does not have the employer's key and does not know what that organisation rewards.
What it can produce is the average answer to a generic workplace scenario.
The acceptable half of the same guide points somewhere more useful: refining and clarifying your own ideas, researching public information about the organisation you are applying to, and checking the spelling, grammar and clarity of what you have written 2. Reading a company's published priorities the evening before you sit its scenario test is squarely on that list. So is thinking, out loud or on paper, about how you would handle the conflicts the job keeps producing.
Common questions
Is there a single correct answer to a 'what would you do' question?
No. The test is scored against a key that rates the options, and your result combines every scenario rather than a pass or fail on one. The key is not published, so nobody outside the employer can tell you what it prefers. In the 2022 re-analysis of the selection literature, situational judgement tests estimate at .26 against rated job performance, which places them among the moderate predictors rather than the decisive ones.
Should I answer how I would really behave, or what I think they want?
Answer as you would behave. The 2022 re-analysis found the same validity for both instruction types: 96 studies asked what a person should do, 22 asked what a candidate would do, and both came out at .20 observed and .26 after correction. That equality is about predicting job performance and nothing else, and the two wordings do differ on how easily they can be faked.
Can I use ChatGPT to answer a situational judgement test?
At least one large employer says no in writing. The UK Civil Service's candidate guide puts completing personality and ability assessments, including situational judgement and numerical tests, on its unacceptable list, and says applications may be rejected where AI is used inappropriately at any stage. That is one employer's policy, with no published rate and no examples, and it binds nobody else. Where an employer's own rules are silent, the guide's acceptable list is still a reasonable model: research and clarity help, completing the assessment for you does not.
Do these tests actually predict anything?
Moderately. In the 2022 re-analysis, situational judgement tests estimate at .26 against rated job performance, and virtually all the studies behind that figure were run on people already in the job rather than on applicants being sorted. A modelled screening stage made only of methods that scale, these tests among them, tops out at .51, against .61 when a structured interview is available. Those are ceilings under stated assumptions, not observed results.
Why add an interview to a scenario test?
One reason in the research: a hypothetical answer is easier to shade than an account of something you did. An experiment on faking in structured interviews found past-behaviour questions more resistant than situational ones, and a meta-analysis comparing the two formats put past-behaviour questions with anchored rating scales at a higher mean validity, .63 against .47. Both findings predate AI scoring, and the 2002 estimates rest on assumptions later shown to be too generous, so the comparison between formats holds up better than the numbers do.
Is preparing for one cheating?
No. A predictive study of a situational panel interview found higher validity among candidates coached to convey relevant content accurately than among uncoached ones, across 146 public safety incumbents, though the two groups were not randomly assigned. Learning what a role is accountable for, and thinking about how you would handle its common conflicts, is that kind of preparation. Hunting for a leaked answer key is not.
References
- 1. Revisiting Meta-Analytic Estimates of Validity in Personnel Selection: Addressing Systematic Overcorrection for Restriction of Range static1.squarespace.com The .26 operational validity estimate for situational judgment tests, and the finding that mean observed validity was .20 for both knowledge instructions (96 studies) and behavioural instructions (22 studies), which is the article's central claim that honest and strategic instruction types predict equally well.
- 2. A candidate's guide to artificial intelligence (AI) in recruitment civil-service-careers.gov.uk The named acceptable and unacceptable AI uses, including the placement of completing situational judgement and numerical tests on the unacceptable list, and the acceptable list the article's closing paragraph points to.
- 3. A candidate's guide to artificial intelligence (AI) in recruitment - Why authenticity is important civil-service-careers.gov.uk The stated consequence the article quotes as a power rather than a measured outcome: the truth-and-accuracy declaration at submission, the statement that applications may be rejected where AI is used inappropriately at any stage, and the absolute bar on AI during a live assessment.
- 4. Measuring faking in the employment interview: development and validation of an interview faking behavior scale. pubmed.ncbi.nlm.nih.gov The Study 6 finding that past-behaviour questions were more resistant to faking than situational questions, which the article uses to explain why a scenario test is usually paired with an interview.
- 5. Asking applicants what they would do versus what they did do: A meta-analytic comparison of situational and past behaviour employment interview questions doi.org The .63 against .47 mean validity comparison between past-behaviour and situational questions with descriptively anchored rating scales, and the staleness of the 2002 estimates that the article states alongside it.
- 6. Insights from an Updated Personnel Selection Meta-analytic Matrix: Revisiting General Mental Ability Tests' Role in the Validity-Diversity Tradeoff filiplievens.squarespace.com The modelled screening-stage ceiling of .51 for methods that run at volume, situational judgment tests included, against .61 when a structured interview is available, which supports the article's claim that a scenario test rarely ends a process.
- 7. How does coaching interviewees affect the validity of a structured interview? doi.org The finding that content-focused coaching raised predictive validity in a situational panel interview across 146 public safety incumbents, which supports the article's claim that honest preparation is not gaming the test.
7 sources, numbered by first appearance. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.