Alex DeBrie

Now is the time to migrate from #ApacheSolr to #OpenSearch For years, Apache Solr has been a reliable search workhorse - but the search landscape is evolving fast. With Solr’s development cadence slowing and community momentum thinning, teams aiming to modernize their search stack are increasingly looking to OpenSearch Project instead. Here’s why forward-thinking engineering leaders are making the switch: - Modern Search Capabilities Built-In: OpenSearch delivers vector search, hybrid semantic + lexical search, and native machine learning integrations - enabling richer, AI-powered search experiences that users expect today. - Cloud-Native & Fully Managed Options: With Amazon OpenSearch Service and other managed offerings, your team can stop worrying about cluster ops, upgrades, and maintenance and focus on delivering more value. - Rich Analytics & Observability: OpenSearch Dashboards offers integrated visibility into search performance and system health - without stitching together third-party tools. - Better Total Cost of Ownership: Even if self-managed Solr looks cheaper up front, the operational overhead - infrastructure, tuning, bug fixes - adds up. OpenSearch often wins the ROI game when you factor in people and time. Yes - migration isn’t a one-to-one lift-and-shift: you’ll map schemas to OpenSearch mappings, translate query logic to its Query DSL, and adapt configurations. But it's also not that big of a deal. We just completed a few migrations like that. With careful planning, parallel environments, and a strong proof-of-concept, you can mitigate risk and unlock future growth. Migration guide is in the comments :) Let’s modernize search — and get more value for every buck invested. 💡

59

7 Comments

We're excited to announce that all the ADBC drivers Columnar distributes can now be used in Java. The latest ADBC release adds a JNI layer that lets Java applications load drivers, run queries, and efficiently fetch results in Apache Arrow format. David Li and I wrote a blog post with quickstart examples and a performance comparison to JDBC. Link in comments 👇

163

15 Comments

Three years ago, I had a client who knew their fraud patterns were shifting in real time. Their detection model was running on overnight batch data. By the time the model saw the signal, the window had closed. The honest answer then was: real-time architecture at enterprise scale is expensive, complex, and operationally heavy. Kafka clusters, custom connectors, dedicated engineering capacity just to keep the pipes running, before a single line of business logic gets written. Databricks' Zerobus Ingest changes that calculation meaningfully. Sub-five-second latency, fully managed, directly into the lakehouse. The infrastructure overhead that made real-time a boardroom trade-off is compressing fast. Worth watching, not because the technology is impressive, but because it removes an excuse that a lot of data strategies have been hiding behind. The question for data and AI leaders now isn't "can we afford to stream." It's "what decisions are we still making on yesterday's data, and what's that actually costing us?"

2

Most teams adopt Apache Airflow to schedule pipelines. Then reality sets in... Pipelines don't just run on a clock. They react to external events. They depend on other systems. They need custom logic that doesn't fit neatly into built-in operators. Suddenly, your DAGs aren't just schedules — they're infrastructure. Data Pipelines with Apache Airflow, Second Edition by Julian de Ruiter, Ismael Cabral, Kris Geusebroek, Daniel van der Ende, and Bas Harenslak shows what it takes to make that shift. If you're already running Airflow in production or about to, this is the difference between, "it works," and, "it scales." Get your copy: • Manning: https://hubs.la/Q042MBjY0 • Amazon: https://hubs.la/Q042MLzx0

35

SQL Down Under show 94 with guest Ben Weissman discussing vectors, REST, and AI in SQL Server 2025 is now published ! https://lnkd.in/gR4ZrgV3

26

Real-time data streaming has always required complicated architectures and weeks of development work to get right. Databricks Zerobus Ingest is designed to change that. CRN's Rick Whiting spoke with Senior Director of Product Management Bilal Aslam about the general availability of the fully managed, serverless ingest service that streams data directly into the lakehouse with sub-five-second latency. On what this means for teams building real-time applications: "It's minutes to hours of development, not weeks and months." Read the full story: https://lnkd.in/gfF8sf7b

303

3 Comments

Building RAG on your lakehouse data? Join us on Jan 22 for the Apache Iceberg Meetup. 🎟️ RSVP: https://lnkd.in/gNwJbcU9 Matt Guolei Yi (Apache Doris PMC) will discuss how Apache Doris 4.0 now runs analytical queries, full-text search, AND vector search in a single SQL engine, no ETL to a separate vector database. Why this matters for RAG: - Most teams extract data from their lakehouse → load into vector dbs like Pinecone/Weaviate → struggle to keep embeddings in sync with source data. - With Apache Doris, your vectors live alongside existing tabular data. Users can essentially query Iceberg tables, filter with SQL, rank with embeddings, all in one place. Also on the agenda for the meetup: - Periyasamy Sivakumar from OCBC Bank on building ML/AI data foundations with Iceberg. - Shelvia Hotama from Confluent on streaming data challenges. We're honored to co-host the event with Starburst, and Confluent. 📅 Date: Thursday, Jan 22, 2026, 6:30–9:00 PM (GMT+8) 📍 Location: Crédit Agricole CIB #Iceberg, #RAG, #Lakehouse

11

2 Comments

During a recent build, I initially stored raw files directly in PostgreSQL. PostgreSQL supported it through TOAST, but the design quickly exposed its limits. Large values were being compressed, chunked into multiple TOAST rows, and reconstructed on every read. That added overhead on I/O, increased WAL generation, and pushed VACUUM into overtime. None of this was related to core business logic; it was purely a consequence of using the database for something it isn’t optimized for. I restructured the system by moving all file storage to S3 and limiting PostgreSQL to metadata only. This separated hot transactional data from cold binary objects. S3 provided efficient large-object handling with streamable reads and writes, while PostgreSQL returned to predictable relational performance with significantly reduced bloat and background maintenance. The result was a simpler, more efficient architecture: – Lower latency for metadata queries – No TOAST fragmentation or reassembly overhead – Reduced WAL volume and faster replication – More linear scaling behavior – Cleaner backup and restore processes The change simplified the architecture, query paths got shorter, and latency improved across the board.

26