We are looking for AI Benchmark & Red-Teaming Specialist candidates for a project delivered through the hiring partner.
What you'll do
- Discover hard-to-surface facts
- Build precise research tasks
- Test research tasks directly against leading LLMs
- Diagnose the exact failure mechanisms in the model's search and reasoning paths
- Shape AI evaluation frameworks
- Ensure next-generation frontier models are tested against rigorous, non-trivial real-world research scenarios
- Deeply research niche topics to identify factual ground truths that cannot be surfaced via simple, direct search queries
- Rigorously verify facts against primary or highly credible sources
- Cross-check for contradictory evidence
- Document exact source URLs
- Draft clear, answerable benchmark tasks designed specifically to expose structural weaknesses in model research strategies
- Execute tasks against designated SOTA models to confirm they reliably trigger intended model failures
- Trace model run logs, search trajectories, and intermediate outputs to pinpoint exactly where and why the model failed (distinguishing genuine model limitations from ambiguous prompts)
- Apply validated "model-breaking" research patterns across new domains
- Review tasks authored by peer trainers
What you need
- 1+ years in investigative research, fact-checking, AI benchmark task authoring, red-teaming, or technical QA review.
- Proven experience authoring, testing, or reviewing tasks for AI benchmarks such as GAIA, BrowseComp, SimpleQA, SWE-bench, Terminal-Bench, or Humanity’s Last Exam (HLE).
- Intermediate-to-advanced web search and Open Source Intelligence (OSINT) skills; ability to navigate academic archives, local records, or obscure databases.
- Strong analytical capability to inspect model reasoning chains, search logs, and source selection to diagnose root causes of failure.
- Ability to author clear, unambiguous benchmark prompts with bulletproof, primary-sourced answer keys.
Nice to have
- Bachelor or higher degree from relevant engineering domain
- Prior experience specifically designing prompt-injection or reasoning-trap tasks that break SOTA models.
- Advanced OSINT capabilities or published investigative/academic research demonstrating deep source verification.
- Niche domain expertise in areas rich with hard-to-find records (e.g., local government filings, regional sports history, legal archives, historical patents).
- Experience serving as a lead reviewer, benchmark auditor, or quality owner on AI evaluation projects.
Expertise
Who you work with
Project and contracting process: the hiring partner. Applications continue on the provider's website.
Pay and hours
This role does not publish a rate, for 0 to 40 h/week. The work is remote.
Who can apply from where
The provider has not confirmed country eligibility. Check the original offer before applying. With a profile, Tier1 checks your country and languages against this role and every other before you apply.