LLM Evaluator – Model Response Analyst (Remote)

About the role

We are looking for a detail‑oriented LLM Evaluator (Model Response Analyst) to assess and improve the output of large language models. Working remotely with a global team, you will help ensure AI‑generated content meets standards of factuality, safety, and alignment.

Key responsibilities

• Evaluate and rank model‑generated text using complex rubrics covering factuality, coherence, safety, instruction‑following, and creativity.
• Compare multiple model responses to the same prompt and justify the preferred output.
• Provide concise feedback to modeling and training teams about recurring failure patterns.
• Craft adversarial prompts to expose biased, harmful, or insecure model behavior.
• Collaborate with QA to refine evaluation guidelines for ambiguous or edge‑case scenarios.
• Participate in cross‑checking sessions to calibrate scoring standards and maintain inter‑rater reliability.
• Analyze underperforming outputs, hypothesize root causes, and flag novel model behaviors for research.

Required profile

• Minimum 2 years of professional experience in computational linguistics, data analysis, technical writing, NLP quality assurance, or cognitive science.
• Bachelor’s degree in Computer Science or a related field.
• Proven experience with Reinforcement Learning from Human Feedback (RLHF) data collection and evaluation.
• Experience sourcing, cleaning, and annotating datasets for LLM fine‑tuning or evaluation.
• Ability to analyze evaluation scores, conduct calibration sessions, and design A/B experiments.

Required skills

• Prompt engineering
• RLHF data collection
• Dataset annotation
• A/B testing for AI models
• Experiment design and analysis