Manuel Tanzi

Fine-tuning for making expert, domain-specific models? Not so fast! I often get asked whether companies should fine-tune LLMs to internalize the knowledge required for their particular use case or domain. The answer I give is probably not…. There is research suggesting that large language models struggle to acquire new factual knowledge through fine-tuning. Novel knowledge is learned more slowly than knowledge consistent with what the model already knows. This same research also showed that when knowledge is eventually learned from novel examples, there is a linear increase in the model's tendency to hallucinate. Ouch! So what can you do? What should you do? RAG is one approach, but that comes with complexity and its own challenges: RAG pipelines are more complex, with larger storage costs, higher memory and compute requirements (due to longer contexts demanded by the additional context) and higher latency, due to the need to query an external index. In the long term, storing knowledge natively in the model's parameters may also provide generalization advantages, as the model can relate different pieces of knowledge in its parameters. This is particularly apparent for complex or indirect queries, where simple retrieval augmentation may fall short. A very exciting recent paper from Meta introduced a new approach called Active Reading. This approach leverages synthetic data to have LLMs generate a range of diverse training data based on a closed body of knowledge. By having the LLMs read and restructure the data in many and varied ways and training on that enlarged, restructured corpus, you can significantly improve the model's retention of the contained facts. Active Reading applies the same principles observed in human studying, allowing the model itself to propose multiple study strategies — e.g., paraphrasing, knowledge linking, active recall, etc. — and instantiates these different strategies on a document-by-document basis. This process results in a highly diverse and contextually grounded signal which can then be trained on. The authors demonstrate huge gains vs. vanilla fine-tuning: +313% and +160% (relative improvement over vanilla fine-tuning) on SimpleQA and FinanceBench respectively. They also trained a SOTA 8B model for factual QA, demonstrating the utility of the technique at pre-training scale (1T tokens). It should be noted that the Active Reading paper focuses on knowledge acquisition; that traditional fine tuning can still be useful for instilling style, format, reasoning patterns, or other behaviors. Learning Facts at Scale with Active Reading https://lnkd.in/e7FCAq-3 Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? https://lnkd.in/e_REAVZB

267

12 Comments

The story of hybrid architectures is honestly fascinating! I've been diving deep into why Transformers became the default choice, and looking at new model architectures. It's not because "attention is all you need" (though catchy!) It's because they exploited GPU parallelism so beautifully. But attention scales quadratically with context. With only attention and nothing else, it gets inefficient as we max out our context windows with the latest context engineering best practices - and try to run LLMs everywhere, on phones, laptops, desktops. Anyways, there's really cool work taking the same principle of fitting the model architecture to the hardware for efficiency + scale. You want a private real-time audio LLM on your phone? Yep. You want an LLM with 1000x higher rate limit on your laptop, so you can run an obscene number of agents in parallel? Yep. OK, so that brings me to the latest and possibly the greatest in this space! I recently learned more about Liquid AI's work. Their new foundation model LFM2 just dropped and they got this highly-coveted-in-AI top left triangle diagram comparing other LMs that run efficiently on CPU (see diagram). They used an evolutionary algo based on a research approach called STAR to discover new hybrid attention+X model architectures for CPUs to use in laptops and desktops, including AMD Ryzen. So instead of all attention blocks, the algo composed a gated convolution block to mix in. They found that 10 of those convs + 6 attention blocks were best for one of their CPU hardware targets. And the conv block looks like this: ```py def lfm2_conv(x): B, C, x = linear(x)      # input projection x = B*x                        # gating (gate depends on input) x = conv(x)                 # short conv x = C*x                        # gating x = linear(x) return x ``` I find it super cool that an algorithm composed this. Basically, let the algorithm discover what works, rather than forcing our preconceptions. Kind of a core principle of deep learning: just backprop to your dreams. The deeper insight: This is doubling down on Transformer's big win and maybeee moving from "one architecture to rule them all" to "let the hardware guide the design." Kinda like having specialized brain regions that each speak their native computational language. The challenge: There have been an absurdly huuuge number of optimizations (over)fit for attention and autoregressive models, from the super optimized kernels up to the GUIs. Makes sense, but this means that feeling these improvements as an end user can be difficult. And don't get me wrong, we still need attention, it's just not all we need :)

377

24 Comments

Pretty exciting new work from Kuaishou in the Generative Recommender domain. They replaced the entire video recommendation funnel (retrieval - ranking - value model) with a single encoder/decoder architecture trained on user action sequences with next-item prediction task, similar to how LLMs are pre-trained to predict the next token. The key to make this all work is their tokenizer, which is trained to map video content and metadata to semantic item IDs — so that similar videos share similar IDs, making generation feasible even at billion-scale vocabularies. On top of that, they apply reinforcement learning with user feedback to break through the ceiling of the traditional recommender stack. Based on their A/B tests this resulted in +0.5% App Stay time in the Kuaishou app. This finding shows just how much headroom there is left on top of traditional recommender systems! More about this in my blog: https://lnkd.in/gCr6k-MD

148

2 Comments

𝗛-𝗡𝗲𝘁: 𝗔 𝗡𝗲𝘄 𝗪𝗮𝘆 𝘁𝗼 𝗕𝘂𝗶𝗹𝗱 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀 𝗪𝗶𝘁𝗵𝗼𝘂𝘁 𝗧𝗼𝗸𝗲𝗻𝗶𝘇𝗲𝗿𝘀 We’ve come a long way with LLMs, but one thing still feels stuck in the past: tokenizers. They're fragile, hand-tuned, and struggle with things like spelling errors, multilingual data, and code. That’s where H-Net from Carnegie Mellon steps in — a new model that learns directly from raw bytes, skipping tokenizers altogether. _____ What Makes H-Net Different? → Adapts to Any Data: Handles English, Chinese, DNA, and more without a fixed vocabulary. → More Efficient: Matches bigger models with fewer parameters and similar computing power. → Handles Messy Data: Processes typos or noisy inputs without extra training. Inside H-Net: How It Processes Data Efficiently → Uses Raw Bytes: Starts with basic data units (like single characters). → Groups Data: Learns to combine bytes into meaningful chunks during training. → Builds in Stages: Processes chunks step by step for better understanding. → Saves Power: Uses Transformer tech with efficient models to work faster. → Stays Stable: Applies a smoothing trick (EMA) to handle tricky data. Performance Insights: A 760M H-Net matches a 1.3B traditional model with the same computing power. It shines in tough areas like DNA and Chinese text (We don’t yet know if it can scale to giant models). _____ 𝗧𝗿𝘆 𝗶𝘁 𝘆𝗼𝘂𝗿𝘀𝗲𝗹𝗳:   • H-Net Research Paper: https://lnkd.in/gEMV3nm6  • H-Net GitHub: https://lnkd.in/gfcis8JR Share this with your network if you found it helpful!♻️ Follow for more AI tips. #HNet #TokenFreeAI #AIResearch #CMU #LanguageModels #GenomicsAI #TransformerModels #AIInnovation

80