Why pre-screen behavioural economists before the research panel
This field has been through a public reckoning about replication, and how a candidate talks about that is genuinely diagnostic. Someone still citing famous effects as settled has not been paying attention; someone who treats every finding as suspect cannot advise anyone. What you want sits between: a researcher who runs adequately powered studies, reports effect sizes rather than only significance, and has watched an intervention fail at scale. A short screen gets to that quickly.
What actually matters when screening Behavioral Economist candidates
- 01
Technique and experimental design
Probe how they design experiments: randomisation units, power calculations, pre-registration on AsPredicted or OSF, choice of RCT versus discrete choice experiment or conjoint, and treatment arms.
- 02
Results that went somewhere
Ask which of their findings changed a product, price, form or policy: default enrolment changes, reminder wording, framing tests, and the measured lift in take-up or savings.
- 03
Troubleshooting and reproducibility
Test how they handle noisy or failed studies: attrition, non-compliance, Hawthorne effects, multiple comparisons corrections, replication attempts, and decay of nudge effects over time.
- 04
Documentation and collaboration
Look for analysis code in R or Stata under version control, documented pre-analysis plans, and evidence of translating results for product managers, policymakers or ethics review boards.
Pre-screening questions to ask Behavioral Economist candidates
12 questions grouped by what they test. Ask the same set in every screen and score answers on a consistent scale, or send them as an async video screen and compare answers side by side.
Design that supports causation
3 questions01Can you discuss your experience designing and conducting behavioural experiments?
Listen forRandomisation and control described concretely, with power calculated in advance rather than sample size decided by convenience.
Sample size determined by what was available, or before-and-after comparisons presented as experiments.
02What are your strategies for ensuring the robustness of your experimental data?
Listen forPre-registration or an analysis plan set in advance, with a stated position on how they avoid selecting the analysis after seeing results.
Analysis decided after the data arrives, or multiple outcome measures tested with no correction or pre-specification.
03How do you ensure diversity and representativeness in your experimental samples?
Listen forAwareness of who convenience samples exclude, with a case where a result did not hold in a different population.
Student or online panel samples generalised to the public, or no consideration of who is missing from the sample.
Interventions deployed
3 questions04Describe a time you applied behavioural principles to a real-world problem.
Listen forAn intervention that actually ran in a service or product, with the measured effect and their own role in getting it deployed.
Laboratory studies presented as applied work, or an intervention designed but never implemented.
05Can you share an example of using a behavioural intervention to change a decision?
Listen forA specific mechanism with the effect size stated, plus honesty about whether the effect persisted beyond the trial period.
Large percentage effects quoted with no baseline, or no follow-up on whether the change lasted.
06Can you explain a project where behavioural data informed a policy or product decision?
Listen forA decision that changed with their contribution separated from the wider team's, and what the evidence could not settle.
Research that ended as a report, or influence claimed with no decision named.
Honest about effect sizes
3 questions07How do you measure the impact of behavioural interventions?
Listen forEffect sizes reported alongside significance, with a control group and a stated view on practical rather than statistical importance.
Reports significance only, or a statistically significant effect too small to justify the implementation cost.
08Tell me about a time your findings contradicted what was expected.
Listen forA null or reversed result reported honestly, with what they checked before concluding and what they published or told the client.
Only positive results to report, or a null result reframed until it supported the original hypothesis.
09Discuss your experience using controlled testing in behavioural research.
Listen forDuration and sample decided in advance, with a refusal to stop early when an interim result looked favourable.
Tests stopped when the desired result appeared, or no fixed stopping rule set before the test began.
Explaining to decision-makers
3 questions10What challenges do you meet when explaining behavioural concepts to non-experts?
Listen forA usable answer given while the caveats survive, with uncertainty expressed in terms a decision-maker can act on.
Overstates confidence to make a recommendation land, or buries the audience in method until nobody can act.
11How do you approach ethical considerations in behavioural experiments?
Listen forA clear position on informed consent, deception and where influencing choice becomes manipulation, with a design they refused.
Ethics treated as approval obtained, or no line they would decline to cross when nudging behaviour.
12Can you discuss working with other disciplines on behavioural projects?
Listen forWork alongside operational teams who had to implement the intervention, with a design changed because it was unworkable in practice.
Hands a design to an operational team with no involvement, or no experience of an intervention failing at implementation.
How to score responses
Score every candidate on the same four criteria immediately after the screen. At this stage you are shortlisting for panel interviews, not making the final call.
Technique and experimental design
35%5Names sample size calculations, clustering decisions and pre-registered hypotheses for a specific trial, and explains why that design beat alternatives.
Results that went somewhere
25%5Cites shipped interventions with baseline and post numbers, plus honest mention of trials that produced null or negative effects.
Troubleshooting and reproducibility
25%5Describes diagnosing a suspicious result, re-running or bounding it, and adjusting inference rather than defending the original headline.
Documentation and collaboration
15%5Shares reproducible code and clear write-ups, and gives an example of persuading non-economists to accept an inconvenient finding.
How a candidate talks about replication is the fastest read on this field. A one-way video screen surfaces it before a research panel spends an hour on theory.
Try it on HirevireScreening FAQ
Process basics
How long should a pre-screening round for a behavioural economist take?
Fifteen minutes across eight to ten questions, answered async. Enough to test experimental design, hear one intervention that was actually deployed, and establish how they handle effect sizes and replication.
How much should academic publication count?
As evidence of method rather than of applicability. For applied roles, an intervention that ran in a real service and produced a measured change is worth more than a paper, because the difficulty is in deployment rather than in design.
Evaluating answers
What is the strongest signal when screening a behavioural economist?
How they discuss replication. Researchers with judgement can name an effect they no longer rely on and explain why. Candidates who cite well-known results as established, with no reference to the last decade of replication work, are working from a stale reading list.
How do I judge intervention claims without a research background?
Ask for the effect size and the sample, not the percentage improvement. A well-run trial with a modest effect is far more useful than a dramatic result from a small sample, and a researcher who leads with the modest number is telling you they understand the field.
Go deeper on this role
Sanat has been hiring since 2012 and watching the recruitment industry change up close ever since, and turned that screening process into Hirevire's video screening platform. LinkedIn
