QA/Test Engineer

$60–90/hr

65 people hired

RemoteGlobalTechnology3-5 YearsContractPosted Aug 11, 2026

About this role

QA/Test Engineer responsible for ensuring complex AI evaluation benchmark tasks are accurate, unambiguous, correctly graded, and resistant to shortcuts. The role involves deeply reviewing technical tasks and reference solutions, executing workflows, designing comprehensive test cases, investigating edge cases, debugging environments, and improving quality-review processes. You will collaborate with AI researchers and task authors to ensure benchmark results reliably measure frontier AI model performance.

Responsibilities

Design comprehensive test cases and edge-case validations.
Review technical tasks and reference solutions for accuracy and ambiguity.
Execute and validate complex multi-step benchmark tasks.
Debug task environments and automated checks using Python.
Identify grading gaps, shortcuts, and unexpected AI agent behaviors.
Develop repeatable QA checklists and review processes.
Document defects and provide actionable feedback to task authors.
Collaborate with researchers to maintain benchmark quality and reliability.

Qualifications

MSc or PhD in a STEM field or equivalent engineering/research experience.
1+ years of experience in QA, test engineering, research engineering, or software engineering with strong quality ownership.
Demonstrated expertise designing test cases and quality-review processes.
Strong end-to-end debugging and problem-solving skills.
Working proficiency in Python and Git.
Comfortable navigating unfamiliar codebases and technical environments.
Exceptional attention to detail and technical documentation skills.
AI training, model evaluation, or AI-generated work review experience preferred.
Ability to independently solve ambiguous and open-ended technical problems.
Must be US-based and available approximately 35 hours per week.