AI Quality Analyst (Personalization) - Indonesian

• Designing and executing multi-turn conversational prompts (typically 1-5 turns) that require the AI to utilize your personal information and experiences.
• Evaluating model responses based on your intent from the starting prompt, checking if the personalization was appropriately applied.
• Analyzing responses for Grounding issues, ensuring claims about you are supported by evidence and not flawed inferences or hallucinations.
• Assessing Integration quality to ensure personal data is woven naturally into the response without robotic "overnarrating".
• Rigorously evaluating and stack-ranking two model responses side-by-side (SxS) to determine which is overall more helpful, easy to use, and enjoyable.
• Writing clear, defensible rationales for your comparisons, explicitly referencing where issues or positive aspects occurred in the conversation.
• Extracting and verifying "Debug Info" from the model to confirm that chat summaries and data sources were properly utilized.
• Maintaining strict data hygiene by deleting evaluation conversations to prevent them from polluting your future chat history.

• Ability to read and write in Indonesian with a high degree of proficiency, as Indonesian is the focus language for this project.
• Willingness to use your primary personal Google account (not a testing account) and enable personal data sources for a genuine assessment.
• Full-time availability in your local time zone is required. We are staffing a global, 24-hour operations team.
• Demonstrated ability to evaluate nuanced and ambiguous AI responses, specifically assessing personalization quality.
• Experience in designing creative, multi-turn starting prompts based on personal context to thoroughly test the model's capabilities.
• Understanding of personalization concepts, including the ability to identify incorrect personalization, poor inferences, and forced connections.
• Ability to review Side-by-Side (SxS) model responses and spot subtle differences in naturalness and overnarrating.
• Superior ability to write clear, concise, and structured rationales for model rankings, explicitly referencing specific turn numbers.
• Ability to provide constructive feedback and detailed annotations.
• Excellent communication and collaboration skills.
• Self-motivated and able to work independently in a remote setting.
• Desktop/Laptop setup with a good internet connection.

• Experience in data annotation, AI quality evaluation, content moderation, or a related role.

RequiredAlso useful

Also listed: analytical thinking, attention to detail, written communication.

Project and contracting process: Turing. Applications continue on the provider's website.