Ting Liu

Reading Building LLMs for Production by Louis-François Bouchard and Louie Peters. At the moment I am reading chapter VIII: Advanced RAG. Looking forward to read the chapter on Deployment this weekend. In general, I like the the style of the book. It is easy to read and understand. I think this book is a good starting point for people who are going to put LLMs in production for the first time. A very practical guide. #LLMs #AI #books

27

2 Comments

Video of our recent Genetic and Evolutionary Computation Conference (GECCO) talk on "Survival-LCS: A Rule-based #MachineLearning Approach to Survival Analysis". Awarded best paper in the Evolutionary Machine Learning Track. https://lnkd.in/gtfuqeut

26

This weekend I read “Zero-shot Tokenizer Transfer” by Minixhofer et al, 2024. In this paper the authors tackle the limitation of the dependence of language models (LM) to their tokenizer. In their approach, the authors propose to “mitigate” this limitation by changing the original tokenizer to an arbitrary one such that the performance of the LM does not degrade. To do this, the authors train a network that takes the tokenizer as input and predicts the embeddings for a base LM. Once trained, the authors claim that the network generalizes to new tokenizers for different LLMs. To me this paper is interesting because studies the implications of how the units of the vocabulary at hand (tokens) have to the language model. Being able to map across representations seems to be an important computing operation in artificial and biological networks. Paper: https://lnkd.in/gU-6Ja7h

4

Interesting paper about the limitations of the generative AI. Actually Yann LeCun talks about this limitation as inherited problem in the existing models for long time. Knowing the limitation helps to align expectations. https://lnkd.in/dCid9B_J #generativeai #llm #artificailintelligence #ai

7

3 Comments

Amidst a range of commercial black box LLMs from large enterprises, we are also seeing Open-Source Machine Learning Frameworks for Advancing Large Language Models in Healthcare. Check out Hippocrates, an open-source LLM framework specifically developed for the medical domain https://lnkd.in/gw5xHdUD

1

FOCUSING ML HEALTHCARE RESEARCH ON VERIFIABLE, EQUITABLE PATIENT BENEFITS ⬇️ This paper focuses on shifting the application of ML in healthcare from mere technical novelty to achieving equitable impact. It critically examines how the current incentive structures in healthcare ML research prioritize technological innovation over real-world clinical outcomes, and proposes a reorientation towards impactful applications. 1️⃣ Emphasizes the need to distinguish between ML applied to healthcare data for technical challenges and ML designed with the primary goal of solving clinical problems. 2️⃣ Highlights the common disconnect between healthcare ML research goals and tangible clinical impacts, suggesting that it should aim for direct improvements in patient care. 3️⃣ Critiques the healthcare ML community for often neglecting meaningful deployment and integration of technologies within healthcare settings, leading to limited practical use despite technical advancements. 4️⃣ Proposes a series of "Impact Challenges" designed to refocus efforts on real-world health equity and effective clinical applications of ML technologies. 5️⃣ Discusses structural barriers within the field, such as funding biases towards novelty, and offers recommendations for promoting research that addresses these barriers to enhance the real-world applicability and impact of healthcare ML. ✍🏻 Aparna Balagopalan, Ioana Baldini, Leo Anthony Celi, Judy Gichoya Liam McCoy, Tristan Naumann, Uri Shalit, Mihaela van der Schaar, Kiri L. Wagstaff. PLOS Digital Health. April 15, 2024. DOI: 10.1371/journal.pdig.0000474 💡 Focusing on equitable impact is not just ethically imperative but also essential for the sustainable integration of AI technologies in healthcare, where they can genuinely contribute to health outcomes and not exacerbate existing disparities. 💥 Join our community of 30.000 curious minds! ✅ Subscribe to our newsletter for your front-row seat to the latest groundbreaking studies. Get started here: https://lnkd.in/eR7qichj.

144

7 Comments

🔗 Simulate PyTorch Models with Simulink! 🚀 Engineers, unlock new possibilities by simulating PyTorch and other Python-based models directly in Simulink using co-execution blocks. This integration streamlines your workflow, enabling seamless model testing and validation. 🔍 What's Inside: >> Co-execution of Python models in Simulink << >> Enhanced simulation and validation processes << 👉 Dive into the details: https://spr.ly/6041o1cOX #DeepLearning #Simulink #Python #ModelSimulation #Engineering #MATLAB #MAthWorks #PyTorch

9

Kolmogorov–Arnold Networks seem like a paradigm shift in ML architectures. Curious to see how they evolve. "While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. " https://lnkd.in/gdPk_qCQ

28

I recently read “Towards Monosemanticity: Decomposing Language Models with Dictionary Learning” by Bricken et al 2023. Published at transformer-circuits.pub In this paper the authors discuss the use of autoencoders to extract interpretable features from a one-layer transformer model. The approach aims to improve the understanding and transparency of artificial neural networks. To do this they use the concept of dictionary learning which is a method used to decompose the activations of the transformer model into a set of basis features that are sparse and interpretable. This involves training sparse autoencoders to identify a dictionary of monosemantic features (or features with a single meaning). These monosemantic features in single units (artificial neurons) reminds me of the “grandmother neuron” theory in neuroscience which proposes that specific neurons respond to specific highly complex stimuli. However, in real brains, it is widely believed that complex representations are encoded by the collective activity of groups of neurons (and not monosemantic). But going back to the paper, I really like the visualization tools and illustration to make such complex networks more transparent (as opposed to black boxes). Paper: https://lnkd.in/gR87pJcs

3

Foundational models, or large language models (LLMs), train at the scale of the entire internet to acquire the vocabulary, syntax, and narrative structure of human language. Fine-tuning involves using a smaller, domain-specific dataset to grant additional specificity to a pre-existing foundational model that has already acquired patterns and features from massive underlying training datasets. By utilizing the existing knowledge embedded in a pre-trained language model, we unlock high-performance language understanding, but fine-tuning helps to get the right answer - not just an answer that sounds right - with a minimum of hallucinations or incorrect info. In healthcare, out-of-the-box, foundational models are often too generic to sufficiently answer the precision of a healthcare query. Foundational language models are trained on a massive scale of data, which includes jokes, incorrect information, misunderstandings, and misinformation, along with the sum total of human knowledge. The problem is, if you ask a large language model to give you factually correct information when it has been trained on BOTH accurate and inaccurate information - you’re playing a bit of a dangerous game. Enter fine-tuning language models. In the old days, we used to build a single model for a single task - for example, classifying procedures into evidence-based or not as laid out by clinical practice guidelines, for patients with particular disease markers. Each disease would have its own model, which took a long time to build and didn’t scale particularly well, and we struggled with the sparsity and inherent high-dimensionality of healthcare data. By fine-tuning LLMs, however, we're able to specialize those foundational models to achieve greater domain specificity and specialization, leverage the unique predictive and planning capabilities of LLMs, and cut down on the incorrect information that such models inevitably produce, while still preserving the underlying language prowess of the LLM. If improved accuracy and specificity wasn't enough of a selling point, there's another notable upside to fine-tuning foundational models for use in a clinical context. In terms of computational efficiency, using LLMs out of the box on specific tasks can require massive compute (and spend) that amounts to computational overkill for the problem. Some LLMs charge per query, some by compute load or time - either way, as a business running LLMs in a production environment, minimizing the number of LLM calls means optimizing your compute expenditures, too. When LLMs fail at scale in healthcare, it's usually along the fault line of failure to specify adequately. Fine-tuning large language models along domain-specific lines is one way to effectively productize genAI for healthcare use cases. #genAI #LLM #ai #healthcareAI #healthtech #largelanguagemodels #generativeAI #chatGPT

61

2 Comments