Humanities Evaluation Specialist

Skills

About this role

In this role, you'll apply your humanities expertise to help train next-generation AI systems by developing rigorous, high-difficulty evaluation material. You'll create original questions and well-supported answers that challenge AI models on interpretation, argument, context, and analytical reasoning, while researching and verifying information through primary texts and reputable sources. This opportunity is suited for humanities professionals and academics with strong analytical writing, close-reading, research, and question-formulation skills who can independently produce precise, nuanced, and defensible evaluation content.

Humanities Evaluation Specialist

Responsibilities

• Develop original, high-difficulty question-and-answer pairs that assess AI models on interpretation, argument, context, and analytical reasoning within the humanities.
• Research and verify answers using primary texts and reputable sources, providing authoritative citations and clear reasoning.
• Design prompts that require deep analytical and interpretive insight rather than surface-level factual recall.
• Test questions against AI models and iteratively increase difficulty while maintaining clarity and correctness.
• Write precise, unambiguous questions and well-supported answers with strong attention to detail.
• Incorporate reviewer feedback and maintain alignment with project guidelines and quality standards.

Qualifications

• Advanced academic background or professional experience in a humanities discipline such as history, literature, philosophy, linguistics, classics, art history, or journalism.
• Exceptional analytical writing, close-reading, and question-formulation skills.
• Demonstrated ability to document reasoning and sources clearly and meticulously.
• Experience creating challenging, nuanced questions that assess understanding beyond surface-level facts.
• Strong self-direction and reliability when delivering expert-level work independently in a remote environment.
• Familiarity with AI model evaluation or training is a plus but not required.