DataWorkers Posted on May 31 • Originally published at dataworkers.io Why We Bet on MCP (And What We're Still Figuring Out) # dataengineering # ai # opensource # mcp When we started building Data Workers, we had to make a foundational decision: how do our AI agents connect to the dozens of tools in a modern data stack? We could build custom integrations for each tool. We could use existing orchestration frameworks. Or we could bet on the Model Context Protocol (MCP). We bet on MCP. Here is why, and what we are still figuring out. What MCP Actually Is MCP is an open protocol, originally developed by Anthropic, that standardizes how AI models interact with external tools and data sources. Think of it as a USB-C port for AI — a universal connector that lets an AI agent talk to any tool that implements the protocol. The ecosystem has exploded. There are now 12,230+ MCP servers available, covering everything from databases to CI/CD tools to cloud platforms. A year ago, this number was in the hundreds. Why We Chose MCP Over Custom Integrations The math is simple. Data Workers needs to connect to warehouses (Snowflake, Databricks, BigQuery, Redshift), orchestrators (Airflow, Dagster, Prefect), transformation tools (dbt, Spark), catalogs (Unity Catalog, Datahub, Hive Metastore), BI tools (Tableau, Looker, Power BI), and more. Building and maintaining custom integrations for each of these is a full-time job for a team our size. With MCP, we get a standard interface. If a tool has an MCP server, our agents can connect to it. We are building custom MCP servers for each agent in our swarm. What Is Working Rapid prototyping. Our Incident Debugging Agent prototype connected to Snowflake query logs, dbt manifests, and Airflow DAGs through MCP in days, not weeks. Composability. Because each agent has its own MCP server, agents can share context through the protocol. When the Incident Debugging Agent identifies a data quality issue, it can invoke tools from the Quality Moni
LIVE