Yaobang Deng

Comet - LLM model monitoring, trace every step along with the evaluation & ranking to see which models provide more value for our problem is really an advantage to stay ahead in the solution offering process.

1

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

180

2 Comments

"Are large language models superhuman chemists?" So what these researchers did was make a test -- a benchmark. They made a test of 7,059 chemistry questions, spanning the gamut of chemistry: computational chemistry, physical chemistry, materials science, macromolecular chemistry, electrochemistry, organic chemistry, general chemistry, analytical chemistry, chemical safety, and toxicology. They recruited 41 chemistry experts to carefully validate their test. They devised the test such that it could be evaluated in a completely automated manner. This meant relying on multiple-choice questions rather than open-ended questions more than they wanted to. The test has 6,202 multiple-choice questions and 857 open-ended questions (88% multiple-choice). The open-ended questions had to have parsers written to find numerical answers in the output in order to test them in an automated manner. In addition, they ask the models to say how confident they are in their answers. Before I tell you the ranking, the researchers write: "On the one hand, our findings underline the impressive capabilities of LLMs in the chemical sciences: Leading models outperform domain experts in specific chemistry questions on many topics. On the other hand, there are still striking limitations. For very relevant topics the answers models provide are wrong. On top of that, many models are not able to reliably estimate their own limitations. Yet, the success of the models in our evaluations perhaps also reveals more about the limitations of the exams we use to evaluate models -- and chemistry -- than about the models themselves. For instance, while models perform well on many textbook questions, they struggle with questions that require some more reasoning. Given that the models outperformed the average human in our study, we need to rethink how we teach and examine chemistry. Critical reasoning is increasingly essential, and rote solving of problems or memorization of facts is a domain in which LLMs will continue to outperform humans." "Our findings also highlight the nuanced trade-off between breadth and depth of evaluation frameworks. The analysis of model performance on different topics shows that models' performance varies widely across the subfields they are tested on. However, even within a topic, the performance of models can vary widely depending on the type of question and the reasoning required to answer it." And with that, I'll tell you the rankings. You can log in to their website at ChemBench.org and see the leaderboard any time for the latest rankings. At this moment I am seeing: gpt-4: 0.48 claude2: 0.29 GPT-3.5-Turbo: 0.26 gemini-pro: 0.25 mistral_8x7b: 0.24 text-davinci-003: 0.18 Perplexity 7B Chat: 0.18 galactica_120b: 0.15 Perplexity 7B online: 0.1 fb-llama-70b-chat: 0.05

3

For the past few weeks, I’ve been exploring JAX and Flax, and I’m really appreciating their design philosophy. A key decision they’ve made is to avoid “reinventing the wheel.” Rather than duplicating functions like dataset loading or checkpointing, JAX/Flax taps into the strengths of PyTorch and TensorFlow, which already offer robust solutions. With a few adjustments, you can seamlessly use these libraries within the JAX/Flax ecosystem for data handling, saving both time and resources. For checkpointing, Flax initially had its own system with `flax.training.checkpoints`, but this has since been replaced by Orbax, which enhances model persistence. However, recent changes in JAX's random number generation (RNG) format have introduced some challenges. The new random key format `dtype=key<fry>` is fully compatible with Flax itself, but Orbax has yet to adopt it, which means that saving checkpoints with dropout layers now requires additional manual adjustments to the key type, adding a layer of complexity. This experience highlights a broader challenge for Flax: its reliance on external libraries that may not always keep pace with its own updates. While the Flax community is actively innovating, dependency on other libraries means they’re sometimes at the mercy of external update cycles. It’s a reminder that in fast-evolving ecosystems, dependencies can sometimes introduce bottlenecks that are outside the control of the primary library's community. Despite these challenges, it’s exciting to see how JAX and Flax continue to push the boundaries, even if it means navigating some temporary friction along the way. #MachineLearning #DeepLearning #JAX #Flax #DataScience #TensorFlow #PyTorch #DeepLearning

12

Hi LinkedIn Community,   Short version:  Grad student building low-code tool for ML data pipelines. Goal: Make ML dev work enjoyable again, not just endless debugging. Ask: Shadow sessions (see Calendly link below) Longer version: As ML devs, we often leave the office thinking, “Do I actually love doing my work?” because we had spent most of the day on inefficient whack-a-mole debugging and maintenance work. I want to get us back to where we can love coding for those reasons we originally chose to pick up coding in the first place.   A lot of AI tools are restrictive and end up mangling codebases, and I believe the current state of AI coupled with strong design principles can enable much better code native developer experiences. I’ve gone through prototypes/testing and have some competing hypotheses. I’d love to observe coders working on ML data pipelines (happy to sign NDAs if necessary, can also be personal projects instead). If you're open to sharing your expertise, please schedule a shadow session: https://lnkd.in/eBW6p6qE Let's make ML data development more efficient and actually enjoyable! #MachineLearning #DataPipelines #LowCode #DevTools #AI #SoftwareEngineering #GradStudent #Research #MLOps #DataScience #ArtificialIntelligence #ProductivityTools #DeveloperExperience #CodeQuality #TechInnovation #ComputerScience #DataEngineering #CloudComputing

84

8 Comments

Introducing AGRaME for Any-Granularity Ranking with Multi-Vector Embeddings to rank at varying levels of granularity while maintaining encoding at a single (coarser) level. Link to paper: https://lnkd.in/gC2TtAWk We propose a multi-granular contrastive loss for training multi-vector approaches and validate its utility with both sentences and propositions as ranking units. We demonstrate the application of proposition-level ranking to post-hoc citation addition in retrieval-augmented generation, surpassing the performance of prompt-driven citation generation. Work done in collaboration with Apple Knowledge Platform Team Omar Attia Yunyao Li (hiring) Saloni Potdar

64

🚀 Excited to share my latest in-depth blog post exploring a cutting-edge AI framework that combines Hyperdimensional Memory Matching (HDMM), Neuro-Symbolic Context Mapping (NSCM), Evolutionary Reinforcement Scoring (ERS) powered by Genetic Algorithms (GAs) and Linear Regression for advanced feature prioritization and decision-making. This framework isn’t just another iteration of AI — it’s a paradigm shift. By leveraging hyperdimensional computing for high-capacity memory representation, symbolic reasoning for human-like interpretation and evolutionary algorithms for dynamic optimization it creates a system that’s scalable, interpretable, and robust. Key highlights: 📈 HDMM enables unmatched scalability for high-dimensional data encoding and matching. 🧠 NSCM bridges symbolic reasoning with data-driven models for enhanced decision-making. ⚙️ ERS with GAs optimizes features in a way that’s both adaptive and efficient. 🧮 Linear Regression integrates seamlessly for actionable insights and predictions. This isn’t just about better AI — it’s about creating systems that mimic cognitive processes, adapt dynamically, and deliver transparent and interpretable results. Let’s push the boundaries of what’s possible in AI! 💡 Read the full article here #ArtificialIntelligence #MachineLearning #HyperdimensionalComputing #NeuroSymbolicAI #GeneticAlgorithms #Innovation #FutureOfAI

1

I wanted to share some insights from a recent paper from Amazon Science in which authors tackled an important challenge in language modeling: minimizing sequence truncation. In language models, truncating input sequences might lead to a loss of crucial information, especially when the models are dealing with long-context tasks such as document summarization or story generation. The authors introduce a clever solution by leveraging a heuristic from the Best Fit Bin Packing problem, an NP-hard problem, to reorder input documents and reduce truncation. The proposed approach helps retain more information compared to other strategies by optimizing how sequences are packed into the model's context window. Authors show that the bin packing solution had biggest performance gain reading comprehension (+4.7%), natural-language inference (+9.3%), context following (+16.8%), and program synthesis. They also found that best-fit packing helped prevent closed-domain hallucination. Authors claim that while the experiments primarily focused on LLM pretraining, best-fit packing is broadly applicable to fine tuning as well. https://lnkd.in/gWquw3ud

28

3 Comments

AirLLM - running inference on a 4GB GPU card

2

There’s been a lot of discourse around why LLMs can or can’t solve math problems lately and I’ve been wondering why that’s so important to us (because LLMs suck at math.) TL;DR two reasons: 1. Using computers for math is something we’ve been doing since we invented computers and even before and 2. Doing math is not the goal, reasoning is. https://lnkd.in/eVkJQuXy

113

3 Comments

Day 5 of learning in public 🚀 Traditional fine-tuning methods for large language models often face challenges like high computational requirements, catastrophic forgetting, and single-task specialization. Enter LoRA (Low-Rank Adaptation), a game-changing technique that addresses these limitations, enabling efficient and practical fine-tuning of large language models. LoRA introduces a novel approach to fine-tuning by adding a small number of task-specific weights to the pre-trained model instead of updating all the weights. Here's how it works: 1. Low-Rank Matrices: For each layer in the pre-trained model, LoRA introduces a pair of low-rank matrices: one for the weights and one for the biases. 2. Adaptation: During fine-tuning, LoRA updates these low-rank matrices instead of modifying the pre-trained weights directly. 3. Composition: At inference time, the low-rank matrices are composed with the pre-trained weights to produce the final, adapted weights for the fine-tuned model. The low-rank nature of these matrices is the key to LoRA's efficiency, dramatically reducing the memory and computational requirements for fine-tuning. This makes it possible to adapt large language models even on consumer-grade hardware like CPUs or edge devices. But LoRA's advantages don't stop there. It can match or even outperform the performance of full fine-tuning while using far fewer updatable parameters. Additionally, the LoRA updates are reversible, allowing the model to revert to its original state, and composable, enabling a single pre-trained model to be fine-tuned for multiple tasks simultaneously. To illustrate the power of LoRA, let's dive into a practical example. Here, I'll demonstrate how to implement LoRA from scratch and fine-tune the DistilBERT model using LoRA on the IMDB dataset for sentiment analysis. https://lnkd.in/g-Tz2NU7 As you can see, LoRA is a remarkable technique that addresses the limitations of traditional fine-tuning methods, making it more efficient, practical, and versatile. By introducing low-rank, task-specific updates, LoRA enables efficient adaptation while preserving the model's original knowledge and allowing for multi-task specialization. As the demand for deploying large language models on resource-constrained devices continues to grow, techniques like LoRA will make these powerful models more accessible and practical.

4

Is GPT-4o good at table recognition? Not quite! OCR looks fine, but tripped by spanning cell table structure! 🤩 However, our UniTable, with only 124M param trained by self-supervision, shows great understanding! 👉 UniTable paper: arxiv.org/abs/2403.04822 👉 UniTable code: https://lnkd.in/e4a7YMxJ #MachineLearning #GPT-4o #Table #Document

85

7 Comments

🚀 Calling all developers, data scientists, and tech enthusiasts! I’ve been working on CoreRec, a cutting-edge graph-based recommendation engine, alongside other exciting projects like ML Annotation at NVIDIA. My GitHub is where you’ll find experiments in advanced recommendation systems, deep learning, and algorithmic modeling—ranging from transformers with augmented attention to graph neural networks. If you’re passionate about machine learning, recommendation systems, or open-source innovation, let’s connect, collaborate, and push boundaries together. Explore my projects and let’s discuss the future of tech: https://lnkd.in/e3HGzdwj #GitHub #CoreRec #GraphAI #RecommendationSystems #DeepLearning #OpenSource #TechInnovation

8

Have you wondered how the decision boundary of in-context learning in LLMs compares to traditional models like Decision Trees and KNN? 🤔 Excited to share our new research, which uncovers unexpected irregularities and non-smoothness in LLMs' in-context decision boundaries! 🔍 📄 Paper: https://lnkd.in/gZAKEJH6 🐦 Tweet summary thread: https://lnkd.in/gY9xkk6d 💻 code: https://lnkd.in/gCsgQk8r Here are some key findings 📌 : - Increasing the number of in-context examples can enhance LLM accuracy on both linear and non-linear classification tasks. But how reliable are these in-context classifiers? We probe their decision boundaries to find out. - By visualizing the decision boundaries, we show that SOTA LLMs, ranging from 1B to large closed-source models such as GPT-3.5-turbo and GPT-4o, all exhibit different non-smooth, irregular decision boundaries, even on simple linearly separable tasks. 🔍 - How do these irregularities arise? 🤔 We study various factors that impact decision boundary smoothness in LLMs, including in-context example count, quantization levels, label semantics & examples order. Then, we identify methods to improve the smoothness of the boundaries. 🛠 - First, increasing in-context examples does not guarantee smoother decision boundaries. While classification accuracy improves with more in-context examples, the decision boundary remains fragmented. - Quantization Impact: Reducing precision from 8-bit to 4-bit can significantly affect areas near the decision boundary, especially in regions with high uncertainties, potentially flipping decisions in these regions. - Label Sensitivity: Decision boundaries are sensitive to label names and example order. Shuffling in-context examples and labels can alter the model’s decision boundaries, indicating a dependency on the semantic prior knowledge of the labels. - 🛠️ Can we improve decision boundary smoothness in LLMs through training? We show that fine-tuning on simple linearly separable tasks can improve the smoothness of decision boundaries and generalize to more complex non-linear, multi-class tasks, enhancing robustness. - Further, fine-tuning the token embedding and attention layers leads to smoother decision boundaries, while fine-tuning the linear prediction head alone does not. - We also explore uncertainty-aware active learning: By adding labels for the most uncertain points to the in-context dataset, we can smoothen the decision boundary more efficiently. - 📚 Effect of Language Pretraining: Transformers trained from scratch on synthetic classification tasks can learn smooth in-context decision boundaries for unseen problems. Thanks for reading about our work! 💡 For a more detailed summary, please check out our tweet thread: https://lnkd.in/gY9xkk6d! 🌟 Work done with amazing collaborators 🙌 Tung Nguyen and our advisor Aditya Grover at UCLA!

65

I just finished watching Stanford’s lecture on "𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗟𝗮𝗿𝗴𝗲 𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗠𝗼𝗱𝗲𝗹𝘀" by Yann Dubois and it’s one of the most insightful resources I’ve come across. It was packed with information—I had to pause and replay a lot to really get it, but it was so worth it! Here's the link: https://lnkd.in/eW5ZpE3B Here’s a quick snapshot of what stood out to me: 1️⃣ Data quality over quantity: Pretraining success relies more on clean, high-quality datasets than simply amassing huge volumes of data. 2️⃣ Alignment with real-world needs: Fine-tuning with human feedback bridges the gap between technical accuracy and practical utility. 3️⃣ Smarter scaling: Techniques like mixed-precision training and efficient GPU usage make it possible to train massive models without excessive costs. As someone who’s just been in the field not long ago, this lecture was a wake-up call to how much there is to learn—but also how exciting the journey is. If you’re like me, trying to figure out how to navigate building LLM applications, this is a must-watch. #AI, #MachineLearning, #NaturalLanguageProcessing, #LanguageModels

15

2 Comments

OpenAI has announced the release of o1 model which contains chain-of-thoughts (CoT) Critics says that openAI were already using it before. LLMs, reinforcement learning (RL) and CoT is coming together in o1. o1 models introducing inference compute over pretraining & fine tuning paradigm. Instead of solely depending on the trained data, o1 can "think" before generating output using RL and CoT. CoT prompting simulates human-like reasoning by breaking down complex tasks into a sequence of logical steps, allowing models to articulate their thought processes. #openai #openaio1 #reinforcementlearning #chainofthoughts

17

I'm excited to say that my two latest publications are now live on TD Commons! These papers introduce advancements in Large Language Model (LLM) fine-tuning and deployment methodologies. The first focuses on a new QLoRA-Blend methodology, which enables small LLMs to outperform larger state-of-the-art models in target domains with minimal financial investment. By integrating multiple domain-specific QLoRA adaptations using Spherical Linear Interpolation (SLERP), this technique significantly enhances accuracy and efficiency in Retrieval-Augmented Generation (RAG) systems. This work can be used to extend other full SFT, DPO, and RLHF training methods. QLoRA-Blend lowers the barrier to fine-tuning LLMs effectively by making high-performance AI more accessible and cost-effective—especially for those who are GPU-poor. The second publication proposes LLM deployment approaches that facilitate the use of LLMs on mobile, edge, and low-resource hardware. By leveraging techniques such as selective pruning, QLoRA-Blend fine-tuning, and our new approach called SynthRAG, which increases both the speed and accuracy of retrieved sources compared to traditional RAG pipelines, these methodologies achieve impressive accuracy and speed in real-time applications on consumer-grade hardware for local CPU-only inference. This work is important as it democratizes access to cutting-edge AI technology, allowing for high-level performance on widely available devices. We are excited to use these advancements in upcoming projects that will push the boundaries of what's possible with AI/ML technology. Stay tuned for the official archived version of our paper to be released soon, showcasing how we used these breakthroughs to develop an LLM that can work in enterprise networking hardware to help diagnose, analyze, and remediate problems within networks. Check out the full publications here: https://lnkd.in/g2qANSdc https://lnkd.in/g-HvKk_m

25

2 Comments

Excited to share my latest research paper preprint on arXiv. 🎮🖥️ Adrenaline: Adaptive Rendering Optimization System for Scalable Cloud Gaming (https://lnkd.in/e8RwSPKD) In this work, we tackle the scalability challenge of cloud gaming services at the edge cloud, introducing Adrenaline - a new system that optimizes game rendering quality by considering both user-side service quality and server-side cost. Key highlights: • Cloud gaming at the edge cloud provides low-latency connections, but fixed compute capacity poses challenges for multi-user serving. • High-quality game frames generated on the server may be ineffective due to lossy compression during transmission by a user's network condition. • Adrenaline adaptively optimizes content generation costs on the server while considering user-side quality, reducing resource waste. • Our network-condition-aware optimization achieves up to 24% higher service quality and serves 2x more users with the same resource footprint. • Easy integration with modern game engines and existing cloud gaming service architectures Check out more details with the full paper! #CloudGaming #EdgeComputing #ResearchPaper

35

2 Comments