Interviewing
When the Software Drafts the Code, Interview for the Note
Interview a clinic billing candidate on documentation judgment rather than code recall, because the coding software already drafts the code. Three situations cover it: a suggested code the note does not support, a denial that turns on medical necessity, and a payer rule applied to the wrong plan. A coding certification filters for baseline knowledge and answers none of the three. Ask every candidate the same three situations, and write down what a good answer sounds like before the first interview.
The takeThe part of this job that credentials measured is the part the software now does first. What a certification exam does not show is whether a person opens the chart before accepting a suggestion that looks right. Hire for that reflex. A biller who clears the queue at speed and never queries a thin note is not fast, and the cost does not disappear; it lands on somebody else, months later, in an audit letter. Interview for the pause.
Where Olive fits
Open a role and see what the work shows
An interview can get a candidate to describe how they would check a drafted answer against the record. Olive puts the checking in front of them as work: a role-grounded assignment in healthcare revenue cycle, an assistant that will do all of it if nobody stops it, and a human reviewer who reads what happened and writes or edits every finding with the timestamp behind it.
Assess your shortlistWhat Did the Encoder Actually Take Off the Desk?
The encoder took code lookup and left the judgment that depends on reading a chart. Medicare's own claims processing manual, in its section on evaluation and management visits, says what that reading has to settle. Medical necessity is the overarching criterion for payment, the volume of documentation should not be the primary influence on the level billed, and documentation should support the level of service reported 1.
An encoder drafts against the note. Whether the note holds is a reading.
A level of service is a claim about what the visit required, not about how much text the note contains.
Even an encoder that produced a defensible draft nine times in ten would be a real gain. The tenth draft is what the hiring is for.
The tenth does not announce itself.
In a Boston Consulting Group field experiment, consultants worked through realistic tasks, one of which was deliberately chosen to sit outside what the model could do. On that task, consultants using GPT-4 reached the correct answer 19 percentage points less often than the ones working without it 2.
Nothing in that setting resembles a billing queue: one task, one sample, and a 2023 model that is no longer current. What survives all of that is the part worth carrying into an interview: the people doing the work could not tell which side of the line the task sat on.
A billing queue has the same line running through it, and nothing on the screen marks where it falls.
Give Them a Note That Does Not Support the Code
Situation one is a suggested level the documentation will not carry. Hand the candidate a redacted note and the code the software drafted from it, then ask what happens next. A strong answer goes to the chart and stays there: what the note records, what is missing, and what would have to be documented before the higher level could stand.
Listen for the provider query. A candidate who says they would go back to the clinician and ask what was actually done has named a clean way to close the gap.
Listen harder for the compliance line, named unprompted, in whatever words the candidate uses. The Office of Inspector General's 2000 compliance guidance for individual and small group physician practices lists upcoding the level of service as one of the billing risk areas that had been among the most frequent subjects of that office's investigations and audits. The guidance defines it as billing for a more expensive service than the one actually performed 3.
The same list carries a subtler entry. Clustering, in that guidance, is coding one or two middle levels of service exclusively, on the theory that some will run high, some low, and the charges will average out over time 3.
A default setting and a team that approves defaults can produce that pattern without anyone choosing it.
Weak answers sound reasonable while they last. They talk about accuracy in general, or about how the tool has been right before, or they say they would flag the claim without saying to whom.
This case does not test prompting. Where the drafting sits inside a fixed application, the skill in question is override and escalation: knowing when to refuse the drafted answer, and where a refusal goes.
What Separates an Appeal From a Resubmission?
An appeal argues that the service was warranted. Against a medical-necessity denial, a resubmission changes a field and hopes. Situation two hands the candidate that kind of denial and asks what goes in the letter. The answer worth hiring names three things: the coverage policy the payer applied, the clinical facts already in the record that meet it, and the deadline.
In one federal review, denials were wrong often enough to make the letter worth writing. Reviewing a sample of Medicare Advantage payment denials, the Office of Inspector General found that 18% of the denied requests met Medicare coverage rules and the plan's own billing rules 4.
The causes in that report will sound familiar at any billing desk. Most of those payment denials came from human error during manual claims processing review, such as overlooking a document, and from system processing errors, such as a system that was not programmed or updated correctly 4.
Most prior authorization denials by Medicare Advantage insurers in 2024 went uncontested. Of the requests they denied, 11.5% were appealed, and more than eight in ten of the appealed denials were overturned in whole or in part 5.
Prior authorization sits earlier in the cycle than a claim denial, so take that as a direction rather than as the rate on your own remittances.
Ask the candidate to walk through an appeal they actually wrote, with no patient detail, before handing over an invented case. Ask for the instance first, the hypothetical only where there is none, because a real appeal carries the payer, the policy and the outcome with it.
A candidate who has never built one will often describe a resubmission instead, and that is the point to hand over the invented case.
The Rule That Belongs to Another Plan
Situation three is the quiet one. A rule the software applies correctly for one payer gets applied to a plan it does not govern, the claim goes out clean, and the denial arrives three weeks later. Ask how the candidate would catch it. The answer to listen for names the document the rule lives in and who publishes it.
A strong answer checks the plan before it checks the code. It treats the tool's rule set as one payer's reading of a policy, and it can say which policy.
A weak answer treats the tool's behavior as the rule. Asked where a rule comes from, it names a screen.
One compliance sentence belongs in the interview out loud, and the Office of Inspector General's guidance supplies it. Written standards and procedures should ensure that coding and billing are based on medical record documentation 3.
No software appears in that sentence.
The case against interviewing this way is throughput. A clinic clearing several hundred claims a day is buying speed, and a biller who opens the chart on every suggestion is slower than one who works the queue. These three cases do not tell you who is faster.
Write the Scale Before the Note Goes in the Packet
The scale gets written before the redacted note goes into the packet. Anchors are the sentences saying what each rating means. A billing manager who hires once or twice a year is in the exact condition under which a rating turns into a comparison between this candidate and whoever came last. Three cases, one scale, and a sentence for every point on it.
The three cases give the scale its wording, and its two ends are the place to start. A top answer on the note goes to the chart and names what is missing; a bottom answer accepts the drafted level and moves on. On the denial, the top names the coverage policy, the facts in the record that meet it and the deadline, while the bottom describes a resubmission. On the misapplied rule, the top names a plan and the bottom names a screen.
The gain from writing anchors down has been measured in one kind of interview, and it is modest. Levashina and colleagues' review reports Taylor and Small's meta-analysis of 19 past-behavior interview studies: interviews using anchored rating scales showed higher validity against the outcomes those studies measured, .35 against .26, and slightly higher agreement between raters, .77 against .73 6.
Those figures come from past-behavior interviews compared across studies rather than from one interview run twice, so anchors travel with whatever else careful teams do. Situational cases like these three were not what those studies measured.
Write them so a second reader could apply them. Every number on the scale needs an observable behavior, and "opened the chart" is observable in a way that "good judgment" never is.
Then ask the same three of everyone, in the same order, on the same scale, with agreement in advance about what an acceptable answer sounds like. The Office of Personnel Management's practical guide defines a structured interview by exactly those three properties 7.
Fixing the questions does not mean reading from a card. Fix the questions and the anchors; the follow-ups can differ, and on the note case the follow-up worth having is what the candidate would ask the clinician.
The queue looks the same on the new hire's first morning. What sits beside it is different: three cases, a written scale, and a record of how each candidate answered. The first time the encoder drafts a level the note will not carry, that record says whether you hired for it.
Common questions
Does a coding certification still count for anything?
Yes, as a floor. A certification is evidence that somebody has studied the code sets and passed an exam on them, which is worth having on a billing desk and is also the part the software now does first. What it does not tell you is whether the person goes back to the chart when a drafted code looks generous, whether they build an appeal on clinical grounds when a denial calls for one, or whether they check which plan a payer rule actually governs. Keep the credential as a screen. Do not let it stand in for the interview.
Should the exercise use a real note or an invented one?
A redacted real note, de-identified before it goes into an interview packet, and the same one for every candidate. An invented note tends to make the error obvious, because whoever wrote it knew where to put it. A real note carries the ordinary mess that makes the judgment hard: a history entered in one place and contradicted in another, a procedure described without the detail the level needs. Check with whoever handles privacy at the clinic first, and keep the redaction rules identical for everyone who sees it.
How do you interview a biller who has never used an encoder?
An encoder is not what the three interview situations test: a drafted code the note does not support, a denial to appeal, and a payer rule applied to the wrong plan. None of them requires having used a particular application: each one asks what the candidate does when a suggestion, a denial or a rule does not match the record in front of them. Run them unchanged. A biller who has worked from paper policies can answer the note situation well and may need to be shown how the drafting step works. Tool familiarity comes with onboarding. The reading habit does not.
What does a weak answer to the denial question sound like?
A weak answer to a medical-necessity denial sounds like a resubmission. The candidate changes a code, or adds a modifier, or says they would send it back through and see what happens, without naming the coverage rule the payer applied or the facts in the record that meet it. A second tell is silence about the deadline. Appeal windows differ by payer, and a candidate who has actually written appeals raises the deadline without being asked.
Should the interview test how fast someone works the queue?
Test it separately, and know which of the two is being measured. Speed and documentation judgment are different measurements, and one exercise scoring both will quietly reward the candidate who accepts the drafted code, because accepting is faster than checking. If throughput matters for the role, time a clean batch on its own and run the three situations untimed. Then decide in advance how much weight each carries, rather than discovering the trade in a debrief.
References
- 1. Medicare Claims Processing Manual, Chapter 12: Physicians/Nonphysician Practitioners, section 30.6.1 (Rev. 13316, issued 07-24-25) ✓ cms.gov Supports the article's statement, scoped to the manual's section on evaluation and management visits, of what a reading of the chart has to settle: medical necessity is the overarching criterion for payment in addition to the individual requirements of an E/M visit code, the volume of documentation should not be the primary influence upon which a level of service is billed, and documentation should support the level of service reported.
- 2. Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality (Working Paper 24-013) ✓ mitsloan.mit.edu Supports the claim that on one task deliberately chosen to sit outside the model's capability, consultants using GPT-4 reached the correct answer 19 percentage points less often than the control group. Cited for the inference that the people doing the work could not tell which side of the capability line the task sat on; one task, one sample, a 2023 model.
- 3. OIG Compliance Program for Individual and Small Group Physician Practices, 65 FR 59434 ✓ oig.hhs.gov Supports three claims in the article: that upcoding the level of service provided appears in the list the guidance introduces as the billing risk areas that have been among the most frequent subjects of investigations and audits by the OIG, defined in footnote 20 as billing for a more expensive service than the one actually performed; that clustering, in footnote 19, is the practice of coding or charging one or two middle levels of service codes exclusively under the philosophy that some will be higher, some lower, and the charges will average out over an extended period; and that written standards and procedures should ensure that coding and billing are based on medical record documentation. The guidance separately lists CPT and ICD-9-CM codes used for claims submission being supported by documentation and the medical record, but that sits in a list of examples of internal documentation guidelines a practice might use rather than in the should-ensure sentence, so it is not relied on here. The guidance's sentence making every physician who provides or supervises a service responsible for correct documentation sits in Appendix A and is scoped to the teaching physician setting, so it is not relied on here.
- 4. Some Medicare Advantage Organization Denials of Prior Authorization Requests Raise Concerns About Beneficiary Access to Medically Necessary Care (OEI-09-18-00260) ✓ oig.hhs.gov Supports the claim that among the payment requests Medicare Advantage organizations denied in OIG's sample, 18 percent met Medicare coverage rules and MAO billing rules, and that most of those payment denials were caused by human error during manual claims processing reviews and by system processing errors.
- 5. Medicare Advantage Insurers Made Nearly 53 Million Prior Authorization Determinations in 2024 ✓ kff.org Supports the claim that 11.5% of denied prior authorization requests were appealed to Medicare Advantage insurers in 2024 and that more than eight in ten appealed denials were overturned in whole or in part. Cited as a direction on appeals, and the article states in the next sentence that prior authorization is not a claim-denial rate.
- 6. The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature ✓ doi.org Supports the anchored rating scale figures used in the article: across 19 past-behavior interview studies, criterion-related validity of .35 with anchored scales against .26 without, and interrater reliability of .77 against .73.
- 7. Structured Interviews: A Practical Guide ✓ opm.gov Supports the definition of a structured interview by three properties: every candidate asked the same questions in the same order, a common rating scale, and interviewers agreeing in advance on what an acceptable answer looks like.
7 sources, numbered by first appearance. Every one was opened and checked against the claim it carries. How Olive sources claims
General guidance for hiring teams. What works at one company and one volume may not transfer to yours.
Olive assesses how a person works with AI. It does not detect AI-written documents, and it never produces a score, a ranking, or a match percentage for a person. Candidates read the same report the employer reads.