Scale AI

Contrary to prior work, new research from Scale finds that LLMs continue to learn new knowledge during post-training following a power law similar to well known pre-training scaling laws. Let’s dive in 👇 The Superficial Alignment Hypothesis suggests that most of a language model's knowledge and skills come from its initial training. Post-training a model is about giving it the right style and format. However, our research team found that when evaluated appropriately on reasoning benchmarks  LLMs continue to learn and apply new information to better tackle complex questions. Specifically, they found that just like pre-training scaling laws, post-training performance scales as a power law against the number of fine-tuning examples. What this implies is the Superficial Alignment Hypothesis is an oversimplification of how models learn. Relying on just human preference votes alone can be misleading, especially for complex reasoning tasks. Evaluating models using both human preference and objective reasoning benchmarks provides a more holistic picture of a model's true capabilities. Read the full paper here from authors Mohit Raghavendra, Vaskar Nath, and Sean Hendryx: https://lnkd.in/gXNzCgvD

We are proud to announce Defense Llama, the LLM built on Meta's Llama 3 to support American national security missions. Defense Llama empowers our service members and national security professionals to apply generative AI to defense-related questions and scenarios, such as planning operations and understanding adversary vulnerabilities. We collaborated with Meta and defense experts to use fine-tuned data to configure the parameters of Defense Llama. Defense Llama is available now, exclusively in controlled U.S. government environments within Scale Donovan. Learn more: https://lnkd.in/gwuxeYrj

Long before most enterprises saw AI’s potential, Alexandr Wang had a vision. As a kid growing up in Los Alamos, New Mexico, Alex was surrounded by science and technology. After enrolling at MIT, he began experimenting with AI by tackling small, everyday problems. What started as a personal side project — trying to track when his fridge needed restocking — sparked a realization: data was the key to unlocking AI’s future. In 2016, he founded Scale AI to help companies harness the power of high-quality data. Today, Scale provides the data infrastructure that powers AI models for some of the world’s largest AI enterprises, helping them build customized AI agents using their proprietary data. Index Ventures partner Mike Volpi caught up with Alex at Scale’s new San Francisco office. They discussed the evolution of AI, the challenges of moving projects from prototype to production, the geopolitical dynamics shaping the future of AI, and more. https://lnkd.in/gPw4-YU6

We are excited to announce Xiaote Z. as the first General Manager of Outlier, which is part of the Scale family of products and services dedicated to advancing GenAI through specialized human expertise. Xiaote will drive the next phase of Outlier growth–actively addressing feedback and keeping an eye on the future. She will focus on three key pillars: 1️⃣ Best-In-Class Platform: leading with contributor experience 2️⃣ Reliability and Transparency: improving pay transparency and contributor support 3️⃣ More Opportunity and Flexibility: increasing contributor choice and testing Expert Match, a feature that enables customers to select their expert team. As we take this next step, we're committed to our mission to advance AI through expert-driven, high-quality data, ensuring that both AI and its development process benefit humanity. Hear from Xiaote on this exciting next step for Outlier: https://lnkd.in/g8vp5Ad2

Hosted by Scale CEO Alexandr Wang and entrepreneur and investor, Nat Friedman, the AI Leadership Summit brought together the world’s foremost AI leaders and industry executives to explore the blueprint to develop and implement AI. Attendees discussed frontier AI's role in enterprise, advances in LLM capabilities and evaluation, U.S.-China strategic dynamics, and what’s on the horizon for the industry. Special thanks to partners Amazon Web Services (AWS), Coatue, and NFDG.

🔥We just heard from keynote speaker Michael Kratsios, Managing Director at Scale AI and former Chief Technology Officer of the United States, in a fireside chat moderated by Keegan McBride (Oxford Internet Institute, University of Oxford). Michael gave an enlightening talk on the geopolitical dimensions of AI, how to balance regulation with innovation, trends in AI policy, and the importance of sector-specific regulation.

Our CEO Alexandr Wang was named to Business Insider’s AI Power List 2024. The list recognizes top players working on a host of challenges and opportunities across AI. Join us on our mission to accelerate the development of AI applications: https://scale.com/careers Read more: https://lnkd.in/g9ib9Zkj

Today, the The White House released the National Security Memorandum on AI which acknowledges the importance of American AI leadership in supporting national security objectives.  We appreciate the Administration’s commitment to partnering with industry to promote and secure the foundational capabilities that power AI development. Our team has been dedicated to these initiatives since our founding and is proud to bring advanced AI capabilities to the USG. Learn more about the national security work taking place in our St. Louis AI Center here: https://lnkd.in/g4ux9z7h See the White House’s full announcement here: https://lnkd.in/efSbJjMr

Our researchers at Scale have developed a novel method to evaluate LLM output during generation instead of waiting until it’s complete. Here’s why that’s a big deal 👇 Traditional methods for training language models, like RLHF, only evaluate the final product of their outputs. As a result, mistakes aren’t caught until it’s too late. The new approach developed by the Scale team is inspired by goal-conditioned reinforcement learning. It allows a reward model to provide early feedback throughout the generation process, guiding generations during training or inference onto the right track. This is like a GPS recalculating when you go off route, instead of waiting until you've reached the wrong destination. This method offers better control, accuracy, and potential for broader societal benefits through more controllable language models. Learn more about the methodology and approach here: https://lnkd.in/g5byZTvD

This week we hosted a red-teaming event in the UK Parliament with MPs and Peers in London! 🇬🇧 Our red-teaming experts led MPs, Peers, and policy heads in an exercise to run adversarial attacks against an AI system and experience first-hand how AI can be exploited to elicit unwanted behaviors. They gained insight into the capabilities and limitations of AI and learned how models can be defended, evaluated and improved. Thanks to Andrew Pakes for sponsoring the event, and to all MPs, Peers and policy heads who attended. Following the opening of our London office, we look forward to deepening our roots in the UK tech ecosystem and continuing the conversation with the UK government on how to safely and effectively leverage AI for public good.