Research Engineer, Gemini AutoRater

Snapshot

Advance research and engineering in large language models, to improve AI feedback across Gemini (AutoRaters for evaluation, Generative Reward models, Self-critic capability of the core Gemini model).

At Google DeepMind, we value diversity of experience, knowledge, backgrounds and perspectives and harness these qualities to create extraordinary impact. We are committed to equal employment opportunity regardless of sex, race, religion or belief, ethnic or national origin, disability, age, citizenship, marital, domestic or civil partnership status, sexual orientation, gender identity, pregnancy, or related condition (including breastfeeding) or any other basis as protected by applicable law. If you have a disability or additional need that requires accommodation, please do not hesitate to let us know.

About us

Artificial Intelligence could be one of humanity’s most useful inventions. At Google DeepMind, we’re a team of scientists, engineers, machine learning experts and more, working together to advance the state of the art in artificial intelligence. We use our technologies for widespread public benefit and scientific discovery, and collaborate with others on critical challenges, ensuring safety and ethics are the highest priority.

The role

The Gemini AutoRater team aims to push the research frontier of LLM's critiquing ability across various capabilities (Quality, Coding, Factuality, Instruction Following, etc). We work on AI feedback across Gemini evals and post-training, specifically:

• AutoRaters for evaluation 

• Generative reward models 

• “Self-Critic” and self-verification capability of Gemini models 

These models are used across Gemini (e.g. OneRecipe).

You will be working alongside a world-class team of researchers and engineers to develop and advance the next generation of frontier AI models. Come work with us if you would like to pioneer work in this direction!

This role requires experience with LLM training and evaluation.

Key responsibilities

As a Research Engineer on the Gemini AutoRater team, you will be at the forefront of evals and post-training research in Gemini. Typical responsibilities include:

• Training AutoRaters and reward models (SFT and RL*F), evaluating them against human raters and deploying them in production (e.g. OneRecipe leaderboard).

• Analyzing model error patterns and continuously improving them: collecting data, new research ideas, etc

As part of your role, you will collaborate with numerous research and engineering teams working on Gemini models and their applications. You will deeply understand the relationships between the AutoRater workstream and other modeling capabilities. You will document and regularly present your work internally within the team, and to senior stakeholders.

About you

In order to set you up for success as a Research Engineer at Google DeepMind, we look for the following skills and experience:

• Degree in machine learning, statistics or related fields. 

• Strong hands-on experience with LLMs and foundation models

• Strong end-to-end system building and prototyping skills.

In addition, the following would be an advantage: 

• Self-directed researcher who can drive new research ideas from conception, experimentation, to productionization in a rapidly shifting landscape.

• Experience with internal ML frameworks such as Gemax Scale/Prod and the Evergreen ecosystem.

• A track record on landing research impact within multi-team collaborative environments under senior stakeholders.