Our mission is to advance humanity's understanding of AI by examining the inner workings of advanced AI models (or “AI Interpretability”). As a research-driven product organization, we bridge the gap between theoretical science and practical applications of interpretability. We're building critical infrastructure that empowers developers to understand, edit, and debug AI models at scale, ensuring the creation of safer and more reliable systems. Goodfire is a public benefit corporation headquartered in San Francisco.
External link for Goodfire
San Francisco, CA, US
Goodfire reposted this
Prompting and activation steering are just two sides of the same coin! New research from Goodfire researchers. Pretty insane how well their Bayesian belief model predicts empirical results. Read more here: https://lnkd.in/eSGMjrwP
Goodfire reposted this
LLMs memorize a lot of their training data. But where do "memories" live inside models? How are they stored? How much are they involved in different tasks? 📌 Jack Merullo and Srihita Vatsavaya's new paper investigates all of these questions! They found a signature of memorization in model weights, and use it to edit models, generally removing the ability to recite text verbatim. This reveals a spectrum of different model capabilities - some which rely heavily on memorization (like factual recall), and others which do perfectly fine without it (pure reasoning). In addition to answering fundamental scientific questions, this points to new applications - like being able to turn memorization up or down for different tasks, or making lightweight agents that excel at reasoning rather than encyclopedic knowledge.
Goodfire reposted this
44M real users’ data secured by our AI interpretability platform Ember. ⤵️ We partnered with Rakuten AI to translate frontier interpretability research into real production value - keeping personal user data safe without slowing down Rakuten’s AI agents. PII detection is a common concern in enterprise AI systems. In production, it requires methods that are: - Lightweight enough to run efficiently at scale - High-recall, so no sensitive data slips through - Trained only on synthetic data, since customer data can’t be used Using Ember, we built interpretability-based classifiers to catch PII with techniques that outperform black-box guardrails on recall, latency, and cost. Our methods were 15–500× cheaper than state of the art LLM-as-a-judge approaches. Huge thanks to Nam Nguyen, Dhruvil Gala, Myra Deng, Michael Byun and Daniel Balsam for leading the charge on this project at Goodfire, and to our collaborators at Rakuten - Yusuke Kaji, Kenta Naruse, Felix Giovanni Virgo, Mio Takei, and others who were early believers in Goodfire and our vision of interpretable AI. We’re excited about helping enterprises build safe, intentionally designed AI systems. If you’re interested in exploring what a partnership could look like, I’d love to chat.
Goodfire reposted this
Are you a high-agency, early- to mid-career researcher or engineer who wants to work on AI interpretability? We're looking for several Research Fellows and Research Engineering Fellows to start this fall. Fellows will work on areas like interp for scientific discovery, causal analysis, representational structure of memorization/generalization, dynamics of representations, and more. We're looking for a range of skillsets - e.g. RL, Bayesian inference, distributed systems, signal processing, and API infrastructure. Fellows will collaborate with senior members of our technical staff, contribute to core projects, and work full time in person in our San Francisco office. Full post and links to apply: https://lnkd.in/eum9VZhq
Goodfire reposted this
Last year alone, AI copilots wrote over 250 billion lines of code. Impressive as that sounds, we still don’t completely understand how they do it, or why they occasionally get things wrong. Two researchers from Lightspeed-backed companies, Anthropic and Goodfire, are helping shape the new discipline of mechanistic interpretability, aka the science of explaining how neural networks make predictions. Anthropic Researcher Jack Lindsey and Goodfire Co-Founder and Chief Scientist Tom McGrath recently joined Lightspeed Partner Nnamdi Iregbulem at our #GenSF meetup to discuss the urgent need for interpretability, the challenges and benefits of scale, and creating a lie detector for AI models. Watch the full meetup on this week’s episode of the Generative Now Podcast: https://lnkd.in/gDRuBm2G
Goodfire reposted this
Our researchers at Goodfire see major boosts from using AI agents, and not just for coding. E.g., when told only to “find interesting SAE features” in Evo 2 (a DNA model), our agent rediscovered a result from the Evo 2 paper! After several months refining agentic workflows, we've learned that a) research agents are crucially different from software agents, and b) we see a huge productivity boost when researchers figure out how to use them effectively. Why? Because experimental research: - benefits from rapid iteration - is often parallelizable - has outputs (experiment plots) that humans can validate quickly Agents excel here! And the #1 lesson for making them effective at research: give them Jupyter notebook access. We're open-sourcing Scribe, an MCP-based system that lets agents actually run notebook cells and receive Jupyter outputs - text, errors, images, etc. Our latest blog post has more details!
Goodfire reposted this
"Suppose every [model] is just a big Mixture of Experts that's been smushed together. Can we separate out the experts? ...it's like model unmerging." Lee Sharkey explains the intuition behind his team's bet on a new paradigm for explaining and editing models:
Goodfire is excited to announce a collaboration with Mayo Clinic! We're working to improve personalized patient outcomes by extracting richer, more reliable signals from genomic & digital pathology models. That could mean novel biomarkers, personalized diagnostics, and more. Read the full announcement of the collaboration: https://lnkd.in/gzAcGBj4
Goodfire reposted this
Arc Institute's DNA foundation model, Evo 2, was trained on genetic data from all domains of life. What has it learned about the natural world? Our latest interpretability research found something really fascinating: the model represents the evolutionary "tree of life" as a manifold within its neural activations. Evo 2 implicitly understands how closely related different species are, without relying on naive sequence similarity. Instead, it has developed an internal map of species relationships that mirrors billions of years of evolutionary history - and we can pull that map out of the model’s internals! This case study tells us a couple things: - Scientific models learn rich representations of the natural world - Emerging interpretability techniques can help us uncover these structures of latent knowledge Those are the first steps toward something I’m very bullish on: extracting new-to-science insights from these models. If we can understand how models represent complex concepts from biology or other fields, we unlock opportunities for tons of applications in medicine, biotech, science, and more. Shoutout to Michael Pearce for leading this work, and you can read the full research here: https://lnkd.in/erRZcfgN