Today we’re joined by Akshita Bhagia, a senior research engineer at the Allen Institute for AI. Akshita joins us to discuss OLMo, a new open source language model with 7 billion and 1 billion variants, but with a key difference compared to similar models offered by Meta, Mistral, and others. Namely, the fact that AI2 has also published the dataset and key tools used to train the model. In our chat with Akshita, we dig into the OLMo models and the various projects falling under the OLMo umbrella, including Dolma, an open three-trillion-token corpus for language model pretraining, and Paloma, a benchmark and tooling for evaluating language model performance across a variety of domains.

🔔 Subscribe to our channel for more great content just like this: https://youtube.com/twimlai?sub_confi...


🗣️ CONNECT WITH US!
===============================
Subscribe to the TWIML AI Podcast: https://twimlai.com/podcast/twimlai/
Join our Slack Community: https://twimlai.com/community/
Subscribe to our newsletter: https://twimlai.com/newsletter/
Want to get in touch? Send us a message: https://twimlai.com/contact/


📖 CHAPTERS
===============================
00:00 - Introduction
02:07 - What is OLMo?
06:34 - The Dolma dataset
11:15 - Training a general LLM
12:11 - OLMo in Industry
13:36 - Datasheets vs Dolma tools
15:28 - Lessons from training OLMo
23:15 - Evaluations
31:37 - Open vs Closed source
34:39 - Instruction tuning
35:04 - The future of OLMo
36:34 - Conclusion


🔗 LINKS & RESOURCES
===============================
Paloma: A Benchmark for Evaluating Language Model Fit - https://arxiv.org/abs/2312.10523
OLMo: Accelerating the Science of Language Models - https://arxiv.org/abs/2402.00838


📸 Camera: https://amzn.to/3TQ3zsg
🎙️Microphone: https://amzn.to/3t5zXeV
🚦Lights: https://amzn.to/3TQlX49
🎛️ Audio Interface: https://amzn.to/3TVFAIq
🎚️ Stream Deck: https://amzn.to/3zzm7F5