We are looking for Senior Software Engineer - AI Evaluation / Coding Agents ($100-150/hr) candidates for a project delivered through the hiring partner.
What you'll do
- Evaluate and improve AI coding models by working with coding agents across real-world repositories and assessing the quality of their work.
- Review generated code and agent behavior, determine whether solutions are technically correct, identify failure modes, and create evaluation signals and feedback used to improve model performance.
- Design, refine, and iterate on rubrics and evaluation criteria for preference data.
- Review and label data with a high quality bar, catching subtle issues others might overlook.
- Build and maintain pipelines and infrastructure supporting data generation, collection, and evaluation workflows.
- Synthesize findings from data work into clear write-ups, updates, and recommendations for the team.
- Collaborate closely with researchers and engineers to translate qualitative judgment into scalable processes.
What you need
- 5+ years of hands-on software engineering experience.
- Strong proficiency in Python, TypeScript/JavaScript, Go, or another major production language.
- Experience working in substantial real-world codebases.
- Strong code-review skills and technical judgment.
- Ability to clearly explain why an implementation is correct, incorrect, or could be improved.
- Strong written communication.
- Experience using modern LLMs or AI coding tools.
Nice to have
- Experience with LLM evaluation, coding agents, RLHF, preference data, rubric design, or post-training.
Expertise
Who you work with
Project and contracting process: the hiring partner. Applications continue on the provider's website.
Pay and hours
This role pays $100–$150/hr. The work is remote. That is about 56% above the $80 an hour median for software roles open now.
Who can apply from where
The provider has not confirmed country eligibility. Check the original offer before applying. With a profile, Tier1 checks your country and languages against this role and every other before you apply.