James Briggs

"I love complicated, costly, & slow pipelines..." words you will NEVER say. Tools like Fivetran, Rivery, Prophecy, and dbt Labs help, no doubt. But what if you want something unified, end-to-end, and built right inside of Databricks for the most common pipeline scenarios? Until now, most of what we had were scattered components with a few connectors here, Delta Live Tables there, and Unity Catalog tying all the data together. This is changing with Databricks' LakeFlow. This interview with Bilal Aslam (product leader over LakeFlow) covers a lot of details around LakeFlow, including: -What can LakeFlow Connect do? -What is happening to Delta Live Tables? Hint: Only positive things you can take advantage of. -Where does LakeFlow fit in the crowded ingestion, transformation, and orchestration space? -Much more! I hope you enjoy this video and that it helps you be better prepared to take advantage of LakeFlow. https://lnkd.in/dGguHMET

62

6 Comments

📝 "This AI Agent Should Have Been a SQL Query" Inspired by the title of a conference talk from a few years back, I ran a thought experiment: can we build agentic systems in the form of (streaming) SQL queries, running on top of #ApacheFlink? 1️⃣ Agents Need to Interact With LLMs 2️⃣ Agents Should Be Event-Driven 3️⃣ Agents Need Context 4️⃣ Agents Require Memory Which ones of these building blocks do exist, which ones are missing? And most importantly, if we can build agents that way, is it a good idea? Find the answer in my latest blog post! 👉 https://lnkd.in/gHEi-R6j

144

11 Comments

What I love about #ApacheIceberg is how many different workloads folks are running on the same copy of data. Graph queries are a really cool example. Instead of copying data into specialized graph databases, PuppyGraph lets you run graph traversals directly on your existing Iceberg tables. Pretty cool stuff. But here's the thing - to get truly great performance, you need to ensure your Iceberg tables are well maintained and optimized. That's where Ryft comes in. We make sure your Iceberg tables are automatically maintained and optimized, no matter what engine you are using. It was terrific to work with Zhenni Wu and Weimo Liu, to create this blog post on how you can use Ryft with PuppyGraph to run graph queries on a fully optimized Iceberg lake! https://lnkd.in/giwnwMuc #ApacheIceberg #GraphQueries

99

2 Comments

It's been a minute since I've written on ZeroCopy, our blog on the future of data infrastructure, but it's been worth the wait. In my latest post: Why Kafka + Iceberg Will Define the Next Decade of Data Infrastructure We look at how the 2 defacto standards of data management can combine to become greater than the sum of their parts and revisit an old data architecture friend that helps explain the game changer we've been working on at Streambased. Check it out: https://lnkd.in/ecnQ69Yz

63

2 Comments

I started exploring AI agents because I want to run them purely on Snowflake. I began investigating the available components within Snowflake for tooling. Before diving into LangChain and workflows, this first blog post explains how to securely connect external APIs to Snowflake and tests the connection using the Perplexity API. 🔍 What’s covered? ✅ Secure API key storage with Snowflake Secrets ✅ Outbound API access control via Network Rules ✅ Handling API calls with Stored Procedures ✅ Logging best practices for debugging & monitoring 📖 Read my full breakdown at Recordly's website👉 https://lnkd.in/dnhSg7EB #Snowflake #AI #DataEngineering #Security #PerplexityAI

45

We tested Snowflake's Gen2 warehouses on 2 billion query profiles across 14K warehouses here at Capital One. Here is where we saw improvement: 🍎 Approximately 30% of queries fell into categories that could benefit from Gen2—specifically those involving DELETE, UPDATE or MERGE. 🍎 Long-running DML operations: A significant subset of queries running longer than 30 minutes were performing complex DML tasks. These are prime candidates for Gen2 migration. 🍎 Large scans: Numerous queries scanned over 1 TB of data, where Gen 2’s improved scan performance can provide material runtime reduction. 🍎 Combined heavy patterns: A smaller but critical set of queries both scanned over 1 TB and ran for 30 minutes or more, often involving DML operations—these represent the highest potential gain from Gen2. These findings support a targeted migration strategy: focus on warehouses with a significant concentration of eligible workloads rather than migrating all warehouses indiscriminately. Check out the full blow written by our resident expert and all around awesome guy - Ganesh Bharathan https://lnkd.in/g8XufZaV

292

14 Comments