James Assiene

Research Engineer @Google DeepMind

London, GB

Applying Google DeepMind\'s research to solve real-world problems. Main projects- VQA for Lookout: Lookout is an app from Google whose goal is to help blind and low vision users to better understand their visual environment. We added a Visual Question Answering feature (VQA) based on DeepMind\'s Flamingo to enable them to find the information they want in an image. I wasresponsible for the Visual Question Answering model itself (data mixture, post-training, and evaluation)using tools like Python and JAX. I was also responsible for some of the safety requirements, implementing them with Python, Scikit-learn and Tensorflow. This feature is available on the Lookout app in the US, UK, Canada and India notably. More details in Astra for video games: The goal is to build a real-time multimodal agent based on Project Astra that will be a gaming companion for mobile games. My focus is on synthetic data for game knowledge and search tool use, as well as visual understanding improvement for games. Announcement in https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/#ai-game-agents- Flamingo for YouTube Shorts: Shorts videos often lack proper textual descriptions, making them difficult to search for users. We used DeepMind\'s Flamingo to automatically generate internal descriptions for these videos so as to enhance users\' experience from a search point of view on YouTube Shorts. I participated in the modeling effort (post-training and evaluation) using tools like Python and JAX. As of late 2024, more than 2 billions shorts videos have been processed using our work. More details in