Why pre-screen impact measurement analysts before the interview
The structural problem in this role is that the organisation paying for the evaluation wants a particular answer. Attribution is genuinely hard, comparison groups are often unavailable, and there is always a way to frame a flat result as encouraging. Analysts worth hiring have reported a programme that did not work. A short screen asks for that evaluation and what happened next.
What actually matters when screening Impact Measurement Analyst candidates
- 01
Outcomes that landed
Ask which programmes they measured end to end: theory of change built, indicators defined, baseline and endline data collected, and what the findings changed in programme design or funding.
- 02
Stakeholder facilitation
Probe how they got programme staff, beneficiaries and funders to agree on indicators, including handling pushback when data showed weak results or added frontline reporting burden.
- 03
Regulatory and policy command
Check working knowledge of IRIS+, SDG indicators, SROI methodology, the Impact Management Project five dimensions, and reporting regimes such as GRI, CSRD or B Corp verification.
- 04
Evidence and reporting
Test analytical craft: survey design, attribution versus contribution, counterfactuals, difference-in-differences or matched comparison, plus the tools used (Stata, R, Power BI, KoboToolbox, SurveyCTO).
Pre-screening questions to ask Impact Measurement Analyst candidates
12 questions grouped by what they test. Ask the same set in every screen and score answers on a consistent scale, or send them as an async video screen and compare answers side by side.
Evaluations they ran
3 questions01What is your experience with programme evaluation?
Listen forEvaluations run end to end, with design, data collection and reporting all part of their work.
Involvement limited to compiling reported figures, or evaluations designed entirely by others.
02Can you describe a time your measurement results changed a decision?
Listen forA specific decision that changed, including one where a programme was reduced or stopped.
Results used only for reporting, or no decision they can trace to their analysis.
03Can you discuss a difficult project where measuring impact was hard?
Listen forGenuine difficulty described, such as no baseline or no comparison, with how they proceeded.
Difficulty described as data access, or claims made despite the design not supporting them.
Attribution honest
4 questions04What is your experience with qualitative and quantitative methods?
Listen forBoth used with a reason, and qualitative work treated as evidence rather than illustration.
Qualitative work reduced to quotations, or quantitative claims made from tiny samples.
05Which metrics do you consider most important when measuring impact?
Listen forOutcome measures preferred over activity counts, with any proxies acknowledged openly as proxies.
Outputs reported as impact, or metrics chosen because they are easy to collect.
06Have you developed a custom metric, and how did you approach it?
Listen forA measure built from a clear definition, tested for whether it moves with real change.
Composite indices built without validation, or weightings chosen to produce a good number.
07What experience do you have with data analysis and statistical methods?
Listen forAnalysis done personally, with the limits of the design understood before conclusions are drawn.
Analysis outsourced entirely, or significance claimed from underpowered comparisons.
Weak data disclosed
3 questions08How do you handle incomplete or inconsistent data?
Listen forGaps investigated and disclosed in the reporting, with the effect on conclusions stated.
Gaps filled silently, or missing follow-up data assumed to look like the responses received.
09How do you ensure data accuracy and integrity in your analysis?
Listen forCollection checked at source, with validation and an audit trail from raw data to reported figure.
Reported figures accepted from delivery teams, or no trace back to underlying records.
10How would you improve an existing measurement framework?
Listen forFramework simplified towards what will actually be used, with weak indicators removed.
Improvement by adding indicators, or frameworks designed for reporting rather than learning.
Reports bad findings
2 questions11Can you describe presenting complex findings to a non-technical audience?
Listen forFindings delivered plainly, with uncertainty and negative results kept in rather than softened.
Presentations built around positive findings, or caveats moved to an appendix.
12How do you approach engagement with stakeholders during an evaluation?
Listen forStakeholders involved early to agree the questions, with independence maintained when results disappoint.
Findings negotiated with the programme team, or conclusions softened after review.
How to score responses
Score every candidate on the same four criteria immediately after the screen. At this stage you are shortlisting for panel interviews, not making the final call.
Outcomes that landed
30%5Names specific programmes, cites outcome numbers with sample sizes, and shows where evaluation findings altered delivery or budget decisions.
Stakeholder facilitation
25%5Describes co-designing indicator sets with delivery teams, running validation workshops, and presenting unflattering findings to funders without softening them.
Regulatory and policy command
25%5Explains framework choice for a given funder, cites specific indicator codes or SROI proxies used, and knows each method's limits.
Evidence and reporting
20%5Distinguishes contribution claims from causal ones, describes a specific counterfactual approach used, and produces dashboards and narratives non-analysts act on.
Somebody paid for the evaluation and wants a particular answer. A one-way video screen asks about a negative one.
Try it on HirevireScreening FAQ
Process basics
How long should a pre-screening round for this role take?
Fifteen minutes across eight to ten questions, answered async. Enough to establish evaluations they ran, test their method, and hear how they handle uncomfortable findings.
How technical does this role need to be?
Enough to design a comparison group and analyse the data themselves. An analyst who commissions all the quantitative work will end up accepting whatever framing the consultant provides.
Evaluating answers
What is the strongest signal when screening this role?
A negative finding they reported. Analysts with integrity have delivered one and can describe the reaction. Anyone whose evaluations all showed success has been writing reports, not evaluating.
How do I judge their method?
Ask how they handle attribution. Credible answers distinguish contribution from attribution, and describe what a comparison group would have required in practice. Anything simpler than that is overclaiming.
Go deeper on this role
Sanat has been hiring since 2012 and watching the recruitment industry change up close ever since, and turned that screening process into Hirevire's video screening platform. LinkedIn
