Mahmoud Abo Khamis

Introducing RouteLLM: a sophisticated routing framework developed with Berkeley-LMSys in collaboration with Anyscale. RouteLLM optimizes query handling by dynamically selecting between high-performance proprietary LLMs and cost-effective open-source models, cutting costs by over 2x without sacrificing quality. Using human preference data and LLM-as-a-judge for data augmentation, our routers evaluate query complexity to choose the appropriate model. Rigorous testing on benchmarks like MMLU and GSM8K confirms our cost-efficient, high-quality performance. Explore our open-source code, models, and preference data on GitHub: https://lnkd.in/gJyijZRx, and try our online demo: https://lnkd.in/gME-GC2a Explore More about RouteLLM and learn how it can revolutionize your LLM applications: Read our Blog here: https://lnkd.in/gfPD-u-y LLMsys Blog here: https://lnkd.in/ga_MgERE Full research paper here: https://lnkd.in/gqRy7Pjy

81

1 Comment

Less than 20 lines of code is all it takes to get started with Llama-3 on Intel Xeon Processors. Get started with the tutorial below. #IntelXeon #LLM #Developer

6

The paper "Scalable MatMul-free Language Modeling" by Rui-Jie Zhu et al. introduces a method to eliminate matrix multiplication (MatMul) operations in large language models (LLMs), significantly reducing memory usage and computational cost. Their MatMul-free models perform comparably to state-of-the-art transformers, scaling effectively up to 2.7 billion parameters. They achieved memory reductions of up to 61% during training and over 10x during inference by using a custom FPGA hardware solution, which highlights the efficiency of lightweight operations for future language model accelerators. For further details, you can read the full paper https://lnkd.in/eZidqJuk

2

The parameters of compiler optimizations interact in complex and non-trivial ways, and finding optimal combinations remains an open research challenge. There are several approaches to discovering effective combinations of these optimization parameters. Recently, Michael Canesche (Cadence Design Systems) and Gaurav Verma (Stony Brook University) have had promising results using auto-tuning with coordinate descent. To visualize this process, imagine that each optimization represents a 'dimension' in a search space. The combination of all optimizations and their respective parameters forms a vector in this space. Now, consider that each point in this space is mapped to a real number representing the performance of a kernel (assuming a fixed input). Coordinate descent is a search technique that assumes the performance curve is convex. Based on this assumption, it aims to find the optimal point by varying one optimization parameter at a time. Explore as a Storm, Exploit as a Raindrop: On the Benefit of Fine-Tuning Kernel Schedulers with Coordinate Descent: https://lnkd.in/dgxxfdYR The Droplet Search Algorithm for Kernel Scheduling: https://lnkd.in/dBfQid9z

77

NVIDIA AI Releases the TensorRT Model Optimizer: A Library to Quantize and Compress Deep Learning Models for Optimized Inference on GPUs Quick read: https://lnkd.in/gfGP7uWg GitHub: https://lnkd.in/gfj62278 NVIDIA NVIDIA AI Huizi Mao #ai

18

Mixture-of-Agents as an Event-Driven System 🤖☎️ This paper by Wang et al. shows you how to ensemble smaller LLMs to create a system that can outperform state-of-the-art larger models. We’ve implemented this paper in a fully async, event-driven workflow system thanks to Ravi Theja Desetty - treat each “small LLM” as an event-driven step that can process incoming events and respond to it, independently and in parallel. ✅ Take full advantage of processing an entire batch of requests  ✅ Cleaner, readable code LlamaPack: https://lnkd.in/gRuqQGk2 Learn more about workflows: https://lnkd.in/gqWwz7Xs

311

9 Comments

LLMs have become more capable with better training and data. But they haven’t figured out how to “think” through problems at test-time. The latest research from Scale finds that simply scaling inference compute–meaning, giving models more time or attempts to solve a problem–is not effective because the attempts are not diverse enough from each other. 👉 Enter PlanSearch, a novel method for code generation that searches over high-level "plans" in natural language to encourage response diversity. PlanSearch enables the model to “think” through various strategies before generating code, making it more likely to solve the problem correctly. The Scale team tested PlanSearch on major coding benchmarks (HumanEval+, MBPP+, and LiveCodeBench) and found it consistently outperforms baselines, particularly in extended search scenarios. Overall performance improves by over 16% on LiveCodeBench from 60.6% to 77%. Here’s how it works: ✅ PlanSearch first generates high-level strategies, or "plans," in natural language before proceeding to code generation. ✅ These plans are then further broken down into structured observations and solution sketches, allowing for a wider exploration of possible solutions. This increases diversity, reducing the chance of the model recycling similar ideas. ✅ These plans are then combined before settling on the final idea and implementing the solution in code. Enabling LLMs to reason more deeply at inference time via search is one of the most exciting directions in AI right now. When PlanSearch is paired with filtering techniques—such as submitting only solutions that pass initial tests—we can get better results overall and achieve the top score of 77% with only 10 submission attempts. Big thanks to all collaborators on this paper including: Evan Wang, Hugh Zhang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen H., Sean Hendryx, Summer Yue 👉 Read the full paper here: arxiv.org/abs/2409.03733

179

2 Comments

Yes! Our #ICSE25 is accepted! Now that I got your attention, this was actually a month ago. But we do propose a new architecture smell on modularity: files that are in the same module but should be separated or the other way round. Interested? Link to the pre-print below. Kudos to Chenxing Zhong, the real hero behind this work!

64

2 Comments

Torch Compile is here to optimize inference in large language models (LLMs) with minimal overhead. In Benjamin Marie's latest article, explore how adding just one line of code can improve decoding performance. #Llama LLM

14

In-Context Learning at Extreme Scales: My Thoughts As large language models (LLMs) continue to expand their context capacities, the domain of in-context learning (ICL) at extreme scales has piqued my interest. A recent paper by researchers at Carnegie Mellon University and Tel Aviv University provides a detailed exploration into this area, unveiling both impressive capabilities and notable challenges. 🔍 Key Insights: Performance Improvement: The study demonstrates that ICL can improve significantly by integrating hundreds to thousands of examples into the context window—far beyond traditional few-shot settings. This is particularly impactful in complex tasks like 77-way banking intent classification, where long-context ICL matches or surpasses traditional finetuning. Input Sensitivity: Long-context ICL shows reduced sensitivity to input shuffling compared to shorter contexts. However, organizing examples by label dramatically reduces performance, emphasizing the need for diverse context configurations. Source of Gains: Performance gains primarily arise from accessing more relevant examples, rather than aggregating knowledge across many examples during encoding. 📉 Challenges & Drawbacks: Sensitivity to Example Order: Despite reduced sensitivity, the performance still varies with input order, indicating potential instability in practical scenarios. Performance Saturation: The models often do not fully leverage their extended context capacities, showing performance saturation before reaching the maximum context length. Computational Cost: The increased computational cost associated with using large context windows could limit practical deployment, particularly in resource-constrained environments. Efficiency Issues: There is a noted inefficiency in how models utilize the encoded information within large datasets, suggesting that current architectures may need refinement to better use available data. 🚀 Future Directions: The paper suggests several exciting avenues for future research, including optimizing the trade-offs between computational costs and performance, exploring model adaptation strategies, and refining ICL approaches for specific applications or datasets. As we push the boundaries of what LLMs can achieve, effectively leveraging their full context could unlock new capabilities. This area is ripe for further exploration and could lead to major advancements in how we deploy AI. For those at the cutting edge of language model development, I highly recommend this paper for its thorough analysis and forward-looking perspective. Check out the full work at https://lnkd.in/d7AmGh4y. What are your thoughts on the future of in-context learning at such scales and the challenges discussed? #AI #MachineLearning #DeepLearning #LanguageModels #InContextLearning

17

Learn how you can convert a PyTorch model, trained on the web, to a mobile-optimized format. Madhur Zanwar describes the process using a few lines of code. He also discusses his journey and how one approach didn't yield satisfactory results while trying to run it in a React Native app. #machinelearning #pytorch #computervision #edgeml #reactnative #objectdetection #yolov5

14

We are happy to announce that our paper "SAUC: Sparsity-Aware Uncertainty Calibration for Spatiotemporal Prediction with Graph Neural Networks" has been accepted by ACM SIGSPATIAL 2024! 📜 Paper link: https://lnkd.in/efgH22K5 📑ArXiv: https://lnkd.in/evvjYd2A This paper addresses the challenge of uncertainty quantification in spatiotemporal deep learning, which is often overlooked in favor of deterministic predictions. Sparse, highly-granular spatiotemporal datasets pose additional difficulties. To tackle this, the authors propose the Sparsity-aware Uncertainty Calibration (SAUC) framework, which adjusts uncertainty for both zero and non-zero values. They first transform deterministic spatiotemporal Graph Neural Networks (ST-GNNs) into probabilistic models and then calibrate them using quantile methods. Experiments show that SAUC improves variance fitting and reduces calibration errors by 20% in sparse datasets like traffic accident and crime predictions, demonstrating its theoretical and practical value. We extend our heartfelt thanks to our esteemed collaborators: Dingyi Zhuang, Bu Yuheng, Guang Wang, Jinhua Zhao! #UncertaintyQuantification #SpatiotemporalData #GraphNeuralNetworks #DeepLearning #ACMSIGSPATIAL #AIResearch #DataScience #SparseData

14

PyTorch 2.5.1 has been released with fixes for RPM based distributions, arm64 distributions, torch.compile crashes, MPS crashes, as well as several issues related to attention regressions observed in 2.5.0. For more information, please refer to the release notes at https://hubs.la/Q02Wc2JX0

338

2 Comments

My paper on Interpretable Tensor Fusion (https://lnkd.in/gjTxusR5) which can capture both linear combinations and multiplicative interactions of multimodal data is on Arxiv. Will also be presenting the work at IJCAI - 24 in Korea.

136

4 Comments

Delve into the recently published paper "Layer-Sensitive Neural Processing Architecture for Error-Tolerant Applications". 📝 Authored by: Zeju Li; Qinfan Wang; Zihan Zou; Qiao Shen; Na Xie; Hao Cai; Hao Zhang; Bo Liu Volume: 32, Issue: 5, May 2024 Neural network (NN) operation has high requirements for storage resources and parallel computing, which bring huge challenges to the deployment of NNs in Internet-of-Things (IoT) devices. Consequently, this work proposed a low-power NN architecture, comprising an energy-efficient NN processor and a Cortex-M3 host processor to achieve state-of-the-art (SOTA) end-to-end inference at the edge. 🔗 Read more on IEEE Xplore here: https://loom.ly/MwvAlBw #PaperHighlight #TVLSI #VeryLargeScaleIntegrationSystems #VeryLargeScaleIntegration

13

#highlycitedpaper Title: A Review of Research on #SignalModulation Recognition Based on #DeepLearning Views 4864, Citations 18 Authors: Wenshi Xiao, Zhongqiang Luo, and Qian Hu Read more here: https://lnkd.in/eDhjaKdx   #mdpielectronics #openaccess #electronics

1

Microsoft Research introduced OmniParser – a breakthrough in multimodal large model for GUI agents, now available on Hugging Face! OmniParser is a compact screen parsing module that leverages the power of large vision-language models like GPT-4V to transform UI screenshots into structured elements. This enables AI agents to interact with user interfaces (UI) effectively using purely visual inputs. With OmniParser, researchers are able to tackle some of the toughest challenges in GUI-based agent technology: Reliable identification of interactive icons within the UI. Deep understanding of UI semantics to accurately map actions to specific regions on the screen. Powered by models like GPT-4V, OmniParser takes intelligent agents to the next level, enabling them to perform precise actions across diverse operating systems and applications. In fact, it recently achieved top performance in the WindowsAgentArena benchmark! 🎉 OmniParser is now available open-source on GitHub!  https://lnkd.in/gZge9yta It's an interesting read for vision language model researchers and enthusiasts. #MicrosoftResearch #AI #MachineLearning #VisionLanguageModels #OmniParser #GPT4V #HuggingFace #ArtificialIntelligence #UserInterface #ScreenParsing

16

1 Comment