Zhou Fang

A nice optimization that leverages redundance between the key and value representations, and also between heads. It probably conveys information through it more effectively, correlating keys (and hence queries) with values, which explains why it works better than multi-head attention.

8

1 Comment

Our paper "FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learning" has been accepted at the #ACCV2024. We introduce a FedRepOpt optimizer for Federated Learning that boosts model performance on low-end devices by modifying optimizer gradients based on model-specific hyperparameters, achieving faster convergence times and robustness across different data distributions. Highlights: -The first systematic study on training RepOpt-based models in FL settings sheds light on their importance and sets a baseline for FL implementation. -We propose FedRepOpt and validate its effectiveness under different FL configurations. Our experiments demonstrate that models using FedRepOpt significantly boost performance by 16.7% and 11.4% over RepGhost-style and RepVGG-style networks, with accelerated convergence times of 11.7% and 57.4%. -We show that FedReOpt excels in cross-silo scenarios and cross-device setups, even in non-iid cases. 📄 Preprint: https://lnkd.in/gjkvZY9i Our implementation is based on the 🌼 Flower framework (Flower Labs). Feel free to check out our code using the following link. 🖥️ Code: https://lnkd.in/ghi2wQp4 Work done with Yasar Abbas Ur Rehman, Pedro Porto Buarque de Gusmão, Lai-Man Po, Ruby Ma, Yuyang Xie #TCLAILAB #AI #FederatedLearning #FL #ACCV2024 #ComputerVision #Reparameterization #CNN #FLOWER

17

I'm happy to annouce that our work "Toleo: Scaling Freshness to Tera-scale Memory using CXL and PIM" accepted to ASPLOS'24 (delayed to ASPLOS'25 for presentation) is now available on arxiv. 🌟Scale trusted memory size from hundreds of MB to tens of TB by expanding the span of trusted from a single trusted processor to an entire platform including intelligent memories. 🌟Design a new scheme of freshness protection that reduces the space requirement by 50x. 🌟Reduce deployment cost by spacing sharing one intelligent memory device among multiple CPUs. Confidential computing is key to assuring data privacy in the cloud. It protects the privacy and integrity of code and data while they're processed on untrusted third-party systems. However their security guanrantees are weekened to opt for scalability, ease of deployment and lower cost etc. In particular, Intel abondoned protection against memory replay attack in their attempt to bring SGX to Xeon Scalable processors. Our work Toleo brings back the freshness protection to trusted memory with the scalibility to protect TBs of data. We leverage trusted smart memory connected through a secure CXL IDE network as the root of trust, and eliminates the need for an unscalable Merkle tree to protect memory freshness. Futhermore, our space optimizations make it feasible for one 168 GB Toleo smart memory device to provide freshness to a 28 TB CXL-expanded main memory pool in a rack server for a negligible performance overhead.

65

LLMs achieve adult human performance on higher-order theory of mind tasks https://lnkd.in/g-A_JFr7 The paper examines the ability of large language models (LLMs) to develop higher-order theory of mind (ToM) - the human capacity to recursively reason about multiple mental and emotional states (e.g. "I think that you believe that she knows"). The authors introduce a new handwritten test suite called "Multi-Order Theory of Mind Q&A" to evaluate five LLMs against a newly gathered adult human benchmark on ToM tasks. The results show that GPT-4 and Flan-PaLM reach adult-level and near adult-level overall performance on ToM tasks, respectively. Notably, GPT-4 even exceeds adult human performance on 6th order ToM inferences. The findings suggest an interplay between model size and finetuning in the realization of ToM abilities in LLMs. The best performing models appear to have developed a generalized capacity for theory of mind reasoning. Given the critical role higher-order ToM plays in human cooperative and competitive behaviors, these results have significant implications for user-facing applications of large language models that need to understand recursive mental state reasoning. In summary, state-of-the-art LLMs like GPT-4 demonstrate human-level or better ability to make higher-order theory of mind inferences, an important milestone with real-world implications.

21

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

Can Differential Transformer solve the attention noise issue in long-context LLMs? Just several weeks after NVIDIA published a paper on how LLMs struggle at the limits of their context windows (which I discussed in my last post: https://lnkd.in/ghW3ZdK4), researchers from Microsoft Research and Tsinghua University introduced a potential solution: the 𝗗𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝘁𝗶𝗮𝗹 𝗧𝗿𝗮𝗻𝘀𝗳𝗼𝗿𝗺𝗲𝗿. Traditional Transformers can handle large sequences, but they often suffer from 𝗮𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗱𝗶𝘀𝗽𝗲𝗿𝘀𝗶𝗼𝗻 as the context size increases. The standard attention mechanism tends to spread focus too thin, over-allocating attention to irrelevant tokens, which makes it difficult for the model to pinpoint and prioritize the most relevant information. This results in a drop in accuracy, especially when key details are buried within massive contexts. The Differential Transformer aims to fix this by working like an 𝗮𝘁𝘁𝗲𝗻𝘁𝗶𝗼𝗻 𝗮𝗺𝗽𝗹𝗶𝗳𝗶𝗲𝗿. It calculates attention as the difference between two separate softmax maps, producing a sparser and more precise attention pattern. This is similar to how differential amplifiers in electrical engineering cancel out common-mode noise by using the difference between two signals. This approach can 𝗯𝗼𝗼𝘀𝘁 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 𝘂𝗽 𝘁𝗼 𝟳𝟲% when working with contexts up to 64K tokens. What’s even more remarkable is that it matches FLOPs and parameter counts with the original Transformer, achieving these improvements 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗲𝘅𝘁𝗿𝗮 𝗰𝗼𝗺𝗽𝘂𝘁𝗮𝘁𝗶𝗼𝗻𝗮𝗹 𝗰𝗼𝘀𝘁. Will this technique bring language models to the next level of scalability, like how 𝗿𝗲𝘀𝗶𝗱𝘂𝗮𝗹 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀 enabled deeper networks or how 𝗱𝗿𝗼𝗽𝗼𝘂𝘁 helped reduce overfitting? Only time will tell, but it’s certainly an exciting development to watch. Full details in the paper linked in the comments. #MachineLearning #ArtificialIntelligence #AI #LLM #GenAI #NeuralNetworks

83

7 Comments

simple AI paper review : When using existing machine learning models without explicit causal information, there is a risk of misinterpreting correlations as causal relationships, particularly in complex scenarios with sequential effects. This paper introduces an advanced methodology for analyzing pixel circuit design, integrating Explainable AI (XAI) techniques, specifically SHAP values, to address these challenges. By constructing a Directed Acyclic Graph (DAG) in collaboration with domain experts, the complexity of interconnections was reduced, improving causal analysis accuracy. The methodology enhances understanding of causal mechanisms in pixel circuitry, demonstrating the effectiveness of combining machine learning with XAI and gaining acceptance among experts. REF : https://lnkd.in/g6QesN8r

11

I'm excited to announce that our talk on the Agent-based AI Framework for ROS 2 has been accepted for ROSCon 2024 🐢! In recent years, there have been significant advancements in Visual-Language-Action (VLA) models. However, these advancements have not yet enabled these models to independently perform real-life robotic tasks. Rather than solely pursuing cutting-edge AI models aimed at complete transformations, we prioritize practical applications that deliver clear benefits in real-world scenarios. Our approach emphasizes seamless integrations and the creation of robust tools to effectively support everyday robotic tasks. If you're interested or have any questions, please feel free to reach out! #ros #ros2 #roscon #robotics #vla

8

Sharing my new article on Intel GPU parallelism! 🌟 It features a detailed breakdown of matrix addition using SYCL and PyTorch, highlighting how efficient scheduling and workgroups maximize performance on Intel hardware. 💻✨ #Intel #GPU #PyTorch #ParallelComputing

19

I'm excited to share our latest research, published in Engineering Applications of Artificial Intelligence, where we present Episodic-Guided Prioritized Experience Replay (EPER)—a novel training method to tackle sample inefficiency and improve policy learning in deep reinforcement learning for autonomous navigation. EPER leverages an episodic memory that stores successful episodes, combined with Prioritized Experience Replay (PER) to enable the agent to prioritize learning from high-reward states. With the addition of Q-diversity regularization, EPER also promotes strategic exploration, contributing to the exploration-exploitation trade-off. Tested across complex driving scenarios (e.g., highway driving, merging, roundabouts, intersections), EPER outperformed both traditional PER and state-of-the-art baselines. Read more here: https://lnkd.in/etVRaFUg Hossein Hassani, Soodeh Nikan, Abdallah Shami #reinforcementlearning #deeplearning #autonomousdriving

43

2 Comments

🏃 Faster PyTorch inference is here. Thanks to the integration of Arm Kleidi on Arm Neoverse, there's up to 2x performance uplift in PyTorch's Eager and Graph modes. 💪 That's not all. Work is underway to supercharge generative AI in PyTorch with Llama and Gemma by up to 18%. Learn more about these performance gains in my colleague @Ashok Bhat's blog. https://lnkd.in/eK7bVYf5

21

Hey Folks, Another day, another breakthrough in large language models: "iLLM-TSC: Integrating Reinforcement Learning and Large Language Models for Traffic Signal Control Policy Improvement." This innovative approach combines reinforcement learning (RL) with large language models (LLMs) to enhance traffic signal control. Traditionally, RL has been used to make decisions based on observed data. However, real-life events are often not considered. By integrating LLMs, real-time events can now be validated, leading to more effective traffic control. Visit this paper for more information : https://lnkd.in/deE8KsDY #AI #ArtificialIntelligence #ReinforcementLearning #GenAI #LLM

3

🏆 Honored to announce that our paper was recognized with the #BestPaperAward at the IEEE-EMBS Conference on Biomedical Engineering & Sciences! Paper title: Sleep Stage and Apnea Classification from Single-Lead ECG Using Artificial and Spiking Neural Networks Our research introduces a comprehensive comparison between traditional and brain-inspired AI approaches for sleep analysis. 📈 Working with an exceptional team we’ve achieved significant results: - State-of-the-art performance in Sleep-Wake detection (94.51% accuracy) - Advanced Wake-NREM-REM classification (81.61% accuracy) - Detailed five-stage sleep classification (68.43% accuracy) - Our Spiking Neural Network (SNN) achieved 89.55% accuracy in detecting sleep apnea events from single-lead ECG Our approach, using only a single-lead ECG signal, demonstrates how sleep monitoring can become more accessible and comfortable while maintaining clinical-grade accuracy. Immense gratitude to my co-authors: Alexandru Vasilache, Tao Hu, Maximilian Alexander Nicolas Themistocli, Sven Nitzsche, Jens Juhl, Christina Erler, Silvester Fuhrhop (Dr.) and Professors Wilhelm Stork and Juergen Becker. Looking forward to seeing how this technology can transform sleep diagnostics and improve patient care! #AI #Healthcare #HealthAI #SleepScience #MachineLearning #Research #IEEE #BiomedicalEngineering

72

5 Comments

Introduction to Implementing Numerical Deep Learning Models in MATLAB and Models for Signal Anomaly Detection and Audio Tasks https://lnkd.in/gKewcat4

3

Happy about our latest paper titled "Deduplicator: When Computation Reuse Meets Load Balancing at the Network Edge", which was recently accepted for publication by IFIP Networking 2024. All the props go to my senior Ph.D. student, Md. Washik Al Azad. This is the 9th (!) publication during his Ph.D., which is a substantial accomplishment for a systems student. The rest of his papers have been published at IEEE PerCom (best paper runner up award), the IEEE Internet of Things Journal (impact factor > 10), IEEE CLOUD, and the IEEE Wireless Communications Magazine (impact factor > 10), among other venues. In this paper, we explore whether we can achieve the contradictory in nature functions of load balancing and computation reuse at the network edge. To this end, we propose an edge middlebox, called the Deduplicator. Our experimental results demonstrate that we can facilitate computation reuse and distribute the load among edge servers at the same time. You can find a preprint of our paper on arXiv: https://lnkd.in/dkQqq9XM Our code is also available on GitHub: https://lnkd.in/dq3HXJSZ #research #cloudcomputing #edgecomputing #loadbalancing #distributedsystems #systems #computernetworks #middleboxes

45

6 Comments