Apache Spark

Bay Area folks: one week from today! 🙌 Join the 𝗔𝗽𝗮𝗰𝗵𝗲 𝗦𝗽𝗮𝗿𝗸 𝗛𝗮𝗽𝗽𝘆 𝗛𝗼𝘂𝗿 on Thursday, November 13 at 5:00–6:30 PM at the Databricks Mountain View office, right after the Open Lakehouse + AI Mini Summit. Grab swag, appetizers, and drinks while connecting with contributors, committers, and maintainers across the Apache Spark community. See you there! 👋 🎟️ Register on Luma: https://luma.com/7or3ih6m #apachespark #opensource #oss

📣 Join us for 𝗢𝗽𝗲𝗻 𝗟𝗮𝗸𝗲𝗵𝗼𝘂𝘀𝗲 + 𝗔𝗜 𝗠𝗶𝗻𝗶 𝗦𝘂𝗺𝗺𝗶𝘁 in Mountain View, CA on November 13! Sandy Ryza (Databricks) will present "𝗗𝗲𝗰𝗹𝗮𝗿𝗮𝘁𝗶𝘃𝗲 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲𝘀: 𝗪𝗵𝗮𝘁'𝘀 𝗡𝗲𝘅𝘁 𝗳𝗼𝗿 𝗔𝗽𝗮𝗰𝗵𝗲 𝗦𝗽𝗮𝗿𝗸™", covering how Spark Declarative Pipelines (SDP) makes it dramatically easier to build robust Spark pipelines by abstracting away orchestration and complexity. SDP extends beyond individual queries to enable a mix of batch and streaming pipelines, keeping multiple datasets fresh, and this session shares a broader vision for openness, standardization, and community momentum. What to expect: 🔹 Core concepts behind SDP 🔹 Where the architecture is headed 🔹 What this shift means for users and Spark engineers building procedural code 👉 Register here: https://luma.com/OLMS-1113 #opensource #apachespark #spark #oss

📣 Join us for the next Open Lakehouse + AI webinar: 𝗚𝗣𝗨-𝗔𝗰𝗰𝗲𝗹𝗲𝗿𝗮𝘁𝗲𝗱 𝗔𝗽𝗮𝗰𝗵𝗲 𝗦𝗽𝗮𝗿𝗸™ 𝗗𝗮𝘁𝗮 𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗮𝗻𝗱 𝗠𝗲𝘁𝗮𝗱𝗮𝘁𝗮 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 𝗳𝗼𝗿 𝗚𝗲𝗻𝗔𝗜 𝗪𝗼𝗿𝗸𝗹𝗼𝗮𝗱𝘀. Discover how NVIDIA’s GPU-accelerated Data Science Platform streamlines data processing and metadata capture for GenAI applications. Get a firsthand look at the platform’s architecture, APIs for ingestion, processing, and retrieval, and how it leverages 𝗥𝗔𝗣𝗜𝗗𝗦 𝗔𝗰𝗰𝗲𝗹𝗲𝗿𝗮𝘁𝗼𝗿 𝗳𝗼𝗿 𝗔𝗽𝗮𝗰𝗵𝗲 𝗦𝗽𝗮𝗿𝗸™ to accelerate the pipelines with open source technologies like Apache Spark™, Rapids, Delta Lake, and Kubeflow. Don’t miss this live session on December 9 at 9 AM PT! 🔗 REGISTER: https://lnkd.in/e9EMCk8C 🎤 Niranjan Nataraja (NVIDIA), hosted by Jules Damji (Databricks) #opensource #apachespark #genai #oss #deltalake

This is exciting news for early Spark developers who want to fiddle with the latest upcoming features! Have a go!

We just finished hosting Apache Spark™ Connect webinar/meetup with NVIDIA's German "Gera" Shegalov and Erik Ordentlich. An excellent technical presentation on the merits of #SparkConnect with #MLlib boosted by NVIDIA's Accelerator. 🔹 Speaks volumes to the benefits of Spark Connect and its extensibility for such connectors by the Spark developer community 🔹 Demonstrates the use of CPU and GPU performance benefits for Spark MLlib algorithms 🔹 Shares reference architecture that shows best practices 🎥 If you missed it, watch here: https://lnkd.in/etUYzsGA #OpenSource #NVIDIA #ApacheSpark

Join us on October 29 at 9:30 AM PT for a deep dive into Apache Spark™, Spark Connect, and Spark ML at NVIDIA! We’ll explore how Spark Connect, starting in Spark 3.4 and extended to MLlib in Spark 4.0+, enables new client connections and stability benefits. ✅ The session will also highlight how NVIDIA GPU-accelerated plugins for Spark SQL and ML deliver end-to-end acceleration with no code changes—with performance up to 9x faster at 80% lower cost. 📅 October 29 ⏰ 9:30–10:30 AM P 📍 Live online (LinkedIn, X & YouTube) RSVP here 👉https://luma.com/0zffq605  🎤 Speakers: Erik Ordentlich & German "Gera" Shegalov, hosted by Jules Damji #opensource #sparkconnect #nvidia #apachespark #oss

🚀 Massive data, mission‑critical pipelines, zero excuses. “Scaling Apache Spark at OpenAI” with Chao Sun reveals how to make Apache Spark™ reliable at scale. You’ll see: 🔹 Production patterns for cluster management, job submission, and access control. 🔹 Dynamic scaling strategies to handle spiky, multi-tenant workloads. 🔹 How Apache Spark™ + Delta Lake underpin large-scale pipelines—and the lessons learned keeping them fast and dependable. Two tracks. One epic afternoon. Open Lakehouse + AI Mini Summit. 🙌 📍 Mountain View 🗓️ Nov 13 🔗 Register → https://lu.ma/OLMS-1113 #OpenLakehouse #ApacheSpark #DeltaLake #DataInfra #BigData #DataOps

Take a read on Python Data source

In the Bay Area? Join us for the Apache Spark Happy Hour on November 13 from 5:00 to 6:30 PM at the Databricks Mountain View Office! 🎉 This event brings together contributors, committers, and maintainers from the Apache Spark community for an evening of networking, conversation, and fun. Enjoy free swag, light bites, and drinks while connecting with new and familiar faces. This informal happy hour is a great way to keep the energy going, share ideas, and celebrate what makes the Apache Spark community special. 🌟 🔗 Register: https://luma.com/7or3ih6m #opensource #apachespark #oss #spark

Happening next Wednesday! 🚀 Join us on October 29 at 9:30 AM PT for a technical session on Apache Spark™, Spark Connect, and Spark ML with NVIDIA. We’ll walk through how Spark Connect—introduced in Spark 3.4 and extended to MLlib in Spark 4.0+—enables new connection models, and how NVIDIA’s open source GPU-accelerated plugins bring up to 9x performance gains with 80% cost savings for Spark SQL and ML workloads. This session includes: 🔹 A working pattern for Spark Connect and accelerated ETL and ML for lakehouses 🔹 The benefits of NVIDIA’s  GPU acceleration for Spark SQL and MLlib 🔹 Live Q&A 📅 October 29 ⏰ 9:30–10:30 AM PT 📍 Live online (LinkedIn, X & YouTube) Don’t miss it—RSVP here 👉 https://luma.com/0zffq605    #opensource #sparkconnect #nvidia #apachespark #oss Jules Damji German "Gera" Shegalov Erik Ordentlich