Interview scorecard template

AI Model Auditor interview scorecard

Evaluate AI Model Auditor candidates across 4 weighted areas: technical depth, real incidents and findings, risk judgement, and getting things fixed. Technical depth leads at 35%, so check fluency in fairness metrics (equalized odds, demographic parity), explainability tooling such as SHAP or LIME, and frameworks like NIST AI RMF or ISO 42001. Use the rubric to compare role-specific evidence consistently.

See AI scoring
security compliancealgorithmic biaseu ai actmodel auditmodel cards
TL;DR
For technical depth, look for evidence the candidate names specific metrics and thresholds used, explains where SHAP misleads, and maps controls to NIST AI RMF or ISO 42001 requirements. For real incidents and findings, look for evidence the candidate walks through named audits end to end, citing disparity numbers found, drift detected, and the concrete disposition each finding received. Apply the written 1–5 anchors to every answer, record the evidence behind each rating, and use the factor weights to reach a consistent overall assessment.
Complete evaluation framework

What to assess and how to score it

Review the evidence signals before interviewing. Then use the anchored descriptions—not instinct alone—to choose the score that best matches each answer.

01
Evaluation factor

Technical depth

35% weight

Check fluency in fairness metrics (equalized odds, demographic parity), explainability tooling such as SHAP or LIME, and frameworks like NIST AI RMF or ISO 42001 clauses.

Evidence to listen for

  • Command of the specific attack surface, tooling, and controls the role covers
  • Understands how the underlying system works, not just how the tool reports on it
  • Can explain an attack or control chain end to end
  • Distinguishes what they found themselves from what a scanner flagged

Five-point scoring guide

1
Poor

Tool operator only; no understanding of the systems underneath.

2
Needs Improvement

Runs tooling but cannot explain findings or how the attack works.

3
Satisfactory

Solid working knowledge; depth thins outside familiar tooling.

4
Very Good

Strong command of the domain; explains attack and control chains clearly.

5
Excellent

Names specific metrics and thresholds used, explains where SHAP misleads, and maps controls to NIST AI RMF or ISO 42001 requirements.

02
Evaluation factor

Real incidents and findings

30% weight

Probe actual audits performed: model type, data lineage reviewed, findings logged, and whether a system was blocked, retrained, or shipped with documented caveats.

Evidence to listen for

  • Brings specific incidents, findings, or audits they personally worked
  • States their own role rather than the team's
  • Describes what was actually at risk and what changed afterwards
  • Can talk about a finding that turned out to be wrong

Five-point scoring guide

1
Poor

No hands-on work; knowledge is entirely certification or coursework.

2
Needs Improvement

Limited exposure; cannot describe their contribution to an incident.

3
Satisfactory

Real casework with adequate detail; ownership sometimes vague.

4
Very Good

Specific incidents with clear personal scope and what changed after.

5
Excellent

Walks through named audits end to end, citing disparity numbers found, drift detected, and the concrete disposition each finding received.

03
Evaluation factor

Risk judgement

20% weight

Assess how they rank harms: proxy discrimination in credit or hiring models, hallucination in customer-facing LLMs, versus cosmetic documentation gaps under EU AI Act tiers.

Evidence to listen for

  • Prioritises by actual exploitability and business impact, not raw severity scores
  • Can argue for accepting a risk as well as fixing it
  • Knows the difference between a finding and a problem
  • Does not cry wolf or wave things through

Five-point scoring guide

1
Poor

Treats every finding as critical, or waves real risk through.

2
Needs Improvement

Follows severity scores mechanically; no business context.

3
Satisfactory

Reasonable prioritisation; less confident arguing for risk acceptance.

4
Very Good

Prioritises by exploitability and impact; can justify accepting a risk.

5
Excellent

Separates high-impact harms from paperwork issues, justifies severity with affected population size, exposure, and reversibility of the decision.

04
Evaluation factor

Getting things fixed

15% weight

Test their record moving data scientists and product owners to act: reopened model cards, added guardrails, retraining schedules, sign-off gates in the MLOps pipeline.

Evidence to listen for

  • Writes findings engineers can act on rather than a wall of output
  • Has persuaded a team to fix something they did not want to fix
  • Explains risk to executives in business terms
  • Works with the org rather than policing it

Five-point scoring guide

1
Poor

Adversarial with engineering; findings never get fixed.

2
Needs Improvement

Reports are unactionable; no influence beyond raising tickets.

3
Satisfactory

Adequate reporting; relies on mandate rather than persuasion.

4
Very Good

Actionable findings and a real record of getting fixes shipped.

5
Excellent

Describes findings that changed deployment practice, naming the owner engaged, the control added, and how closure was verified post-release.

Evidence-led prompts

Interview questions for a AI Model Auditor

Use these prompts to surface evidence for the weighted factors above and compare candidates against the same role-specific criteria.

  1. 01

    Can you discuss previous projects where you audited a model?

  2. 02

    Describe a time when you identified a significant issue during an audit.

  3. 03

    Can you give an example of a model that improved as a result of your audit?

  4. 04

    Can you describe your experience with model validation and verification?

  5. 05

    What techniques do you use for stress testing models?

See the complete AI Model Auditor question set
Put this rubric to work

Score every candidate against the same standard

Add these weighted factors to Hirevire and let AI evaluate recorded answers against your rubric.

Explore AI Scorecards