Interview scorecard template

Natural Language Processing (NLP) Engineer interview scorecard

Evaluate Natural Language Processing (NLP) Engineer candidates across 4 weighted areas: technical proficiency, systems and trade-offs, evidence and rigour, and collaboration and communication. Technical proficiency leads at 35%, so probe depth in transformer architectures, tokenization choices (BPE vs SentencePiece), fine-tuning methods like LoRA or SFT, and frameworks: PyTorch, Hugging Face, spaCy, vLLM. Use the rubric to compare role-specific evidence consistently.

See AI scoring
software datallm finetuningnlptokenizationtransformers
TL;DR
For technical proficiency, look for evidence the candidate explains attention internals, tokenizer trade-offs and adapter tuning from direct implementation, naming model families, context limits and quantization schemes used. For systems and trade-offs, look for evidence the candidate weighs recall, latency budgets and serving cost with numbers, and can defend rejecting a heavier model for a distilled one. Apply the written 1–5 anchors to every answer, record the evidence behind each rating, and use the factor weights to reach a consistent overall assessment.
Complete evaluation framework

What to assess and how to score it

Review the evidence signals before interviewing. Then use the anchored descriptions—not instinct alone—to choose the score that best matches each answer.

01
Evaluation factor

Technical proficiency

35% weight

Probe depth in transformer architectures, tokenization choices (BPE vs SentencePiece), fine-tuning methods like LoRA or SFT, and frameworks: PyTorch, Hugging Face, spaCy, vLLM.

Evidence to listen for

  • Command of the languages, frameworks, and data tools the role actually uses
  • Understands correctness, performance, and failure modes, not just syntax
  • Has opinions on testing and can justify them
  • Reads and reasons about code they did not write

Five-point scoring guide

1
Poor

Cannot work independently; fundamentals are missing.

2
Needs Improvement

Weak fundamentals; output needs heavy review.

3
Satisfactory

Competent for the role; needs guidance on complex or unfamiliar work.

4
Very Good

Strong practitioner; handles hard problems with little guidance.

5
Excellent

Explains attention internals, tokenizer trade-offs and adapter tuning from direct implementation, naming model families, context limits and quantization schemes used.

02
Evaluation factor

Systems and trade-offs

25% weight

Ask how they chose between prompting, RAG and fine-tuning, handled inference latency, GPU cost per million tokens, and embedding store selection for production traffic.

Evidence to listen for

  • Reasons about scale, latency, cost, and failure before writing code
  • Names the trade-off they chose and what they gave up
  • Understands the data lifecycle end to end
  • Anticipates what breaks at ten times the volume

Five-point scoring guide

1
Poor

No thinking beyond the immediate task; no awareness of scale or failure.

2
Needs Improvement

Limited architectural awareness; struggles with design decisions.

3
Satisfactory

Works within an existing design; makes sound local decisions.

4
Very Good

Designs for scale and maintainability; articulates trade-offs clearly.

5
Excellent

Weighs recall, latency budgets and serving cost with numbers, and can defend rejecting a heavier model for a distilled one.

03
Evaluation factor

Evidence and rigour

25% weight

Test evaluation practice: held-out sets, F1 versus BLEU or ROUGE, human annotation agreement, hallucination and bias checks, and regression suites before shipping a model.

Evidence to listen for

  • Validates results rather than trusting output
  • Knows how their work is measured and what a bad result looks like
  • Can describe a time their own analysis or model was wrong and how they caught it
  • Careful about data quality, leakage, and silent failure

Five-point scoring guide

1
Poor

Ships unvalidated work; no notion of how correctness is checked.

2
Needs Improvement

Validates superficially; misses obvious quality or leakage issues.

3
Satisfactory

Reasonable checks in place; rigour drops under time pressure.

4
Very Good

Validates thoroughly; can name a real error they caught in their own work.

5
Excellent

Cites concrete baselines and metric deltas, describes annotation guidelines and inter-annotator agreement, and admits where offline scores misled them.

04
Evaluation factor

Collaboration and communication

15% weight

Look for work with annotators, linguists, product owners and MLOps on data pipelines, model cards, and explaining why a model output cannot be fully controlled.

Evidence to listen for

  • Explains technical work to non-technical stakeholders
  • Gives and takes code or peer review constructively
  • Documents enough that the work survives their absence
  • Aligns with team process rather than working around it

Five-point scoring guide

1
Poor

Cannot work in a team; resistant to feedback.

2
Needs Improvement

Communication issues create rework; lone-wolf tendencies.

3
Satisfactory

Adequate team member; documentation and review participation are light.

4
Very Good

Communicates well; reliable reviewer and collaborator.

5
Excellent

Translates model limitations into plain product language, writes usable model documentation, and negotiates labelling scope with domain experts.

Evidence-led prompts

Interview questions for a Natural Language Processing (NLP) Engineer

Use these prompts to surface evidence for the weighted factors above and compare candidates against the same role-specific criteria.

  1. 01

    Can you describe specific projects that involved natural language processing?

  2. 02

    Have you worked on sentiment analysis, and can you share the details?

  3. 03

    Can you describe how you have implemented text summarisation?

  4. 04

    What is your approach to ensuring high quality results on these tasks?

  5. 05

    Are you familiar with text classification, and how do you evaluate it?

See the complete Natural Language Processing (NLP) Engineer question set
Put this rubric to work

Score every candidate against the same standard

Add these weighted factors to Hirevire and let AI evaluate recorded answers against your rubric.

Explore AI Scorecards