Confluent, Inc., a pioneer in data streaming, and Databricks, a leader in Data and AI, have significantly expanded their partnership to integrate Confluent’s comprehensive Data Streaming Platform with Databricks’ Data Intelligence Platform. This collaboration aims to equip enterprises with real-time data capabilities, facilitating AI-driven decision-making. A key aspect of this expansion is the integration of Confluent’s Tableflow with Databricks Unity Catalog, ensuring seamless data governance across operational and analytical systems for more efficient AI application development.
Tackling the AI Data Challenge
As enterprises accelerate the adoption of AI applications, the need for reliable, real-time data becomes more pressing. However, only 22% of enterprises are confident that their current IT infrastructure can support these AI initiatives. One of the most significant challenges they face is bridging the gap between operational systems, where data is generated, and analytical systems, where insights are derived. The existence of these systems in separate silos leads to fragmented tools, teams, and processes, preventing businesses from leveraging real-time data effectively. This disconnect stifles AI innovation, particularly for advanced use cases.
“For companies to maximize returns on their AI investments, they need their data, AI, analytics and governance all in one place,” said Ali Ghodsi, co-founder and CEO, Databricks. “As we help more organizations build data intelligence, trusted enterprise data sits at the center. We are excited that Confluent has embraced Unity Catalog and Delta Lake as its open governance and storage solutions of choice, and we look forward to working together to deliver long-term value for our customers.”
“Real-time data is the fuel for AI,” said Jay Kreps, co-founder and CEO, Confluent. “But too often, enterprises are held back by disconnected systems that fail to deliver the data they need, in the format they need, at the moment they need it. Together with Databricks, we’re ensuring businesses can harness the power of real-time data to build sophisticated AI-driven applications for their most critical use cases.”
Also Read: Databricks Acquires BladeBridge to Boost Data Migration
Enabling AI with Real-Time Data
To close this gap, Confluent and Databricks are introducing new integrations that enable real-time data interoperability, fostering better collaboration across business teams. The bidirectional integration between Confluent’s Tableflow with Delta Lake and Databricks’ Unity Catalog, a unified and open governance solution for data and AI, ensures that real-time data across operational and analytical systems remains discoverable, secure, and trustworthy.
Delta Lake, an open-format storage layer developed by Databricks, was originally built for streaming use cases requiring rapid data ingestion. Now, it is the most widely adopted lakehouse format, handling over 10 exabytes of data daily. With Tableflow now integrating with Delta Lake, operational data becomes instantly available within Delta Lake’s extensive ecosystem. This allows Confluent and Databricks customers to leverage various AI and data processing engines, including Apache Spark, Trino, Polars, DuckDB, and Daft, within Unity Catalog.
Furthermore, custom integrations between Tableflow and Databricks’ Unity Catalog ensure that metadata—a crucial component for AI applications—is automatically applied to data exchanged between platforms. This enhancement makes operational data more accessible and actionable for data scientists and analysts working in Databricks while simultaneously improving analytical data accessibility for application developers and streaming engineers in Confluent. Additionally, Confluent’s Stream Governance suite will provide upstream governance and metadata, enabling fine-grained governance, comprehensive stream lineage, and automated data quality monitoring in Unity Catalog.
“Leveraging proximity to generation sources is a key factor not just in the energy sector, but also in the field of data,” said Dr. Dora Simroth, Head of Data and AI Engineering, E.ON Digital Technology. “Confluent and Databricks are already essential technologies in our Data and AI stack. These integrations will allow our practitioners to work on a single source of well-defined and timely data for both our operational and analytical plane. By partnering, Confluent and Databricks open up a faster path for us to build data and model-centric digital solutions.”
With these new capabilities, Confluent’s operational data becomes a first-class citizen within Databricks, while Databricks data is easily accessible across enterprise processors. AI applications consuming streaming data topics and data analysts working with tables will now have a unified, real-time view of data, enabling faster and more intelligent AI-driven decision-making. This seamless integration between enterprise applications, analytics, and governance is pivotal for scaling AI innovation across organizations.