SOC 11-3021 · Computer and Information Systems Managers
Product management, tested on the work
Turn a vague request into a spec, where the vagueness is the test and the AI will happily resolve it for you.
The assignment
What the candidate is asked to do
A short product specification written from an under-specified request, with an AI assistant available.
Time: 40 to 60 minutes, hard-capped 15 to 20 minutes past whatever the case states and never above 75, with a short written debrief after. Paused time does not count against either.
SOC 11-3021
Computer and Information Systems Managers. The benchmark is derived from live postings mapped to that code and re-derived each quarter.
What the reviewer watches for
The product management failure modes
Each maps to a scored dimension, so the reviewer's finding arrives with the moment attached.
Is the decision the spec has to settle stated before any spec text exists?
Whether the opening move asks what is true, what is wrong, or what the options are — and whether a constraint or criterion is named in the same breath.
Are the invented requirements traced back to something, or accepted as given?
Whether sources are demanded for a specific claim rather than requested in general — and whether at least one is actually opened.
Which parts of the spec are written by hand, and which are generated?
Whether the split between the assistant's work and the candidate's own acts is deliberate rather than convenient, read from the record of both.
Why this bank
Why we built the product management bank
Problem framing is the dimension product roles fail most often, and the one an interview conversation is least able to observe.
This bank is live and has passed author review at launch quality.
https://olive.is/benchmarks · Edition 2026 Q3Named up front
Turn a vague request into a spec, where the vagueness is the test and the AI will happily resolve it for you.
The construct
Six dimensions, scored separately
- D1 · Problem framing Did the first move go after understanding, or straight for output?
- D2 · Evidence sourcing Did they demand evidence for the claim that mattered?
- D3 · Delegation boundary What did they keep, and what did they hand over?
- D4 · Working structure Did anything exist between the brief and the answer?
- D5 · Output rejection Was anything the assistant produced refused, and on what grounds?
- D6 · Verification Was anything tested against the world, and did the result change something?
SOC 11-3021
Run one product management role, free
Ten attempts a month against this bank, with the full six-dimension report on every one. No card, no salesperson.