6 Capability Layers · 20 MCP Tools · 232 Tests · Portfolio Project

Your AI Data Engineer
lives in your editor

Describe your analytics needs in plain English. DataMCP connects to your Postgres database, maps the schema, generates a production-quality dbt project, wires up dashboards, schedules orchestration, and streams real-time data — all without leaving your coding assistant.

20
MCP tools
across 6 layers
<15m
From DB connection
to runnable dbt project
0
SQL written
by hand
∞
Knowledge base
that compounds
🐍 Python · FastAPI · uv 🔧 dbt · dlt · Dagster 📊 Metabase · Postgres ⚡ Kafka · Debezium 🧠 SQLite Knowledge Base 🔐 OS Keychain Auth
✉ Request Repository Access
The Problem

Analytics is a wiring problem
disguised as a staffing problem

The tools all exist. dbt, Dagster, Metabase, Kafka — the ecosystem is mature. The problem is the assembly cost: a skilled data engineer needs one to three weeks just to wire everything together for the first time. Then another week whenever the schema changes.

01
Schema chaos, no documentation
Every team has a production database nobody has fully mapped. Foreign keys undocumented. Column names ambiguous. Business logic buried in application code. A data engineer starts from zero every time.
02
Weeks of scaffolding toil
Staging models, sources, marts, tests, KPI definitions, scheduler configs, dashboard connections — all written by hand, each requiring deep expertise in tools the rest of the team doesn't know.
03
Maintenance is a full-time job
Schema drifts silently. Dashboards break. Pipeline tests fail with cryptic messages. Without a dedicated data engineer monitoring things, the analytics layer rots faster than the product evolves.
How It Works

Describe it once.
DataMCP builds the rest.

DataMCP runs as an MCP server inside Claude Code, Cursor, or Codex. The developer stays in their editor. Every tool call is logged to a local knowledge base that makes every subsequent interaction smarter.

Claude Code — DataMCP Session
user › Connect to our production Postgres and set up analytics ⟶ connect_database(host="db.prod.internal", db="appdb", schema="public") Connected. Found 24 tables, 187 columns, 9 detected foreign key relationships. ⟶ introspect_schema() Schema graph built. Tables mapped with types, nullability, PKs, FKs, row counts. ⟶ infer_domain() Domain: SaaS / Subscription platform. Confidence: 94% Signals: users, subscriptions, plans, invoices, events tables detected. ⟶ infer_model_candidates() Recommended: 24 staging models · 6 intermediate joins · 4 fact tables · 3 dimensions ⟶ generate_staging_models() Generated 24 staging models with column renaming, casting, not_null + unique tests. ⟶ generate_mart_models() Generated fct_subscriptions, fct_invoices, dim_users, dim_plans — schema.yml included. ⟶ generate_kpis() Defined 12 KPIs: MRR, Churn Rate, ARPU, Trial-to-Paid, DAU/MAU, NRR… ⟶ run_dbt(command="run") ✓ 37 models completed. 0 errors. 0 warnings. Total time: 11 minutes 42 seconds. Zero SQL written by hand.
The Pipeline

Five stages. Fully automated.

Every DataMCP session follows the same reproducible flow — from raw Postgres to analytics-ready marts, with tests, documentation, and monitoring at each stage.

01
Connect & Introspect
Postgres connection established. Full schema graph built — tables, columns, types, PKs, FKs, row counts, indexes.
02
Infer Domain
Schema patterns analysed. Business domain classified. Model candidates ranked. Developer confirms before generation.
03
Generate dbt Project
Staging models, marts, tests, sources, KPIs — all generated from Jinja templates. Committed to the repo.
04
Execute & Validate
dbt run and dbt test executed. Failures surfaced with plain-English explanations and fix suggestions.
05
Monitor & Alert
Freshness SLA tracking per table. Consumer lag for streaming. Dagster pipeline health. All in the knowledge base.
Six Capability Layers

One server.
The entire data stack.

Each stone adds a complete layer of functionality — independently useful, but designed to compose. A team can adopt Stone 1 today and unlock the full stack as they grow.

🗄️
Stone 1 — Core Analytics
Postgres → dbt · Production Complete
Connect to any Postgres instance, map the full schema, infer the business domain, generate a complete dbt project with staging models, mart layer, KPI definitions, and data quality tests — all in under 15 minutes.
Live dbt Postgres Jinja
📊
Stone 2 — BI Dashboards
dbt → Metabase · Production Complete
Automatically connect the mart layer to Metabase — provision the database connection, sync table metadata, scaffold dashboards from KPI definitions, and propagate schema changes without manual intervention.
Live Metabase REST API Auto-sync
⏱️
Stone 3 — Orchestration
Dagster Runtime · Production Complete
Scaffold and manage a live Dagster project — start and stop the runtime, poll pipeline state via GraphQL, enforce freshness SLA sensors, and surface run history and failures through the knowledge base.
Live Dagster GraphQL SLA Sensors
🔌
Stone 4 — Ingestion
dlt Pipelines · Production Complete
Generate and execute dlt pipelines to ingest data from external sources. Detect source-to-staging drift automatically. Surface last run status, row counts, and alignment issues directly from the MCP.
Live dlt Drift Detection Multi-source
⚡
Stone 5 — Streaming
Kafka · Production Complete
Provision Kafka topics, generate Postgres sink connector config (Debezium / JDBC), monitor consumer lag per partition, and generate streaming-aware incremental dbt models — with real-time freshness tracking.
Live Kafka Debezium Incremental dbt
💬
Stone 6 — Chat GUI
Non-developer Interface · In Progress
A Streamlit-based chat interface backed by the full knowledge base — schema graph, KPIs, freshness history, pipeline logs, Metabase chart metadata. RAG-powered answers grounded in the team's specific data model.
Partial Streamlit RAG SQLite KB
The Differentiator

A knowledge base that
compounds every week

Persistent Context

The longer you use it,
the smarter it gets

Every DataMCP action is stored in a local SQLite knowledge base — schema snapshots, model generations, KPI definitions, pipeline run history, query logs, Metabase chart metadata, Kafka connection configs. This is the context that makes every subsequent interaction faster and more specific.

  • Schema evolution tracked across every introspection run
  • Model generation history with before/after diffs
  • KPI definitions versioned alongside mart models
  • Pipeline run log with row counts and failure context
  • Natural language query history for RAG retrieval
  • Freshness SLA violations logged with timestamps
Knowledge Base Schema

SQLite — 12 Tables

Persisted locally in the project directory. Grows with the project. Never leaves the team's environment.

connections schemas tables generated_models kpi_definitions pipeline_runs query_history freshness_checks metabase_connections metabase_charts kafka_connections dlt_runs
Generated Code Quality

Code a senior data engineer
would be proud of

Every file DataMCP generates lives in the team's repository, under their version control. The generated dbt models follow naming conventions, include column-level documentation, apply correct materialisation strategies, and ship with data quality tests. There is no proprietary format. The output is standard dbt.

  • Staging models with column renaming and type casting
  • Mart layer with fact and dimension separation
  • not_null + unique tests on every primary key
  • relationships tests on all detected foreign keys
  • Column-level docs in schema.yml for every model
  • Streaming-aware incremental models for Kafka sources
Generated dbt Project

Full project structure from one prompt

All files committed to the repo. Readable, editable, owned by the team.

dbt_project.yml profiles.yml sources.yml stg_*.sql int_*.sql fct_*.sql dim_*.sql schema.yml metrics.yml macros/
Build Quality

Built to production standard
from day one

Every stone ships with full test coverage, typed interfaces, and CI-ready structure. The MCP protocol contract is stable across all six stones.

232
Tests passing
CI-enforced threshold
20
MCP tools
registered & dispatched
6
Capability stones
delivered
100%
Schema coverage
staging per table
MCP Tool Reference

20 tools. Every layer covered.

Each tool is a typed Python function registered with the MCP server. Tools are organised by capability layer and can be composed by the AI in any sequence to complete complex data engineering tasks.

Schema · Stone 1
connect_database
Establish a Postgres connection. Stores credentials in OS keychain. Returns connection summary.
Schema · Stone 1
introspect_schema
Map all tables, columns, types, PKs, FKs, indexes, and row counts into the schema knowledge graph.
Schema · Stone 1
infer_domain
Classify the business domain (SaaS, e-commerce, fintech…) from schema patterns. Confidence-scored.
Schema · Stone 1
infer_model_candidates
Rank and recommend staging models, intermediate joins, facts, and dimensions based on the schema graph.
Schema · Stone 1
generate_er_diagram
Produce a Mermaid ER diagram from the schema graph for documentation and code review.
Generation · Stone 1
generate_staging_models
Generate one staging model per source table with column renaming, casting, not_null + unique tests.
Generation · Stone 1
generate_mart_models
Generate fact and dimension mart models with schema.yml documentation and relationship tests.
Generation · Stone 1
generate_kpis
Define domain-specific KPIs as dbt metrics — name, formula, owning model, grain, cadence.
Generation · Stone 1
generate_dbt_macros
Generate reusable dbt macro library for the project — type-safe column casting, surrogate keys.
Execution · Stone 1
generate_dbt_project
Scaffold the full dbt project skeleton — dbt_project.yml, profiles.yml, directory structure.
Execution · Stone 1
run_dbt
Execute dbt run, test, or docs generate. Capture failures with plain-English explanations.
Execution · Stone 1
execute_sql
Run arbitrary SQL against the connected Postgres database. Results returned as structured data.
Monitoring · Stone 1
check_freshness
Compare table update timestamps against expected cadence. Surface staleness warnings.
Knowledge · Stone 1
query_knowledge_base
Natural language search over the accumulated knowledge base — schemas, models, KPIs, run history.
BI · Stone 2
connect_metabase
Provision a Metabase database connection pointing to the Postgres instance. Stores config in KB.
BI · Stone 2
sync_metabase
Trigger Metabase schema sync and wait for completion. Surfaces new tables and field changes.
BI · Stone 2
create_metabase_dashboard
Scaffold a Metabase dashboard from the KPI definitions — one question per KPI, auto-visualised.
Orchestration · Stone 3
scaffold_dagster
Generate a complete Dagster project with dbt asset definitions, freshness sensors, and schedules.
Orchestration · Stone 3
get_pipeline_status
Query live Dagster pipeline state via GraphQL. Returns run status, asset materialisation history.
Ingestion · Stone 4
scaffold_dlt
Generate a dlt pipeline for a specified source (Stripe, HubSpot, Shopify…) with destination config.
Ingestion · Stone 4
run_dlt
Execute a dlt pipeline subprocess. Parse row counts. Persist run record to knowledge base.
Ingestion · Stone 4
explain_model
Generate plain-English documentation for a dbt model — column meanings, business context, lineage.
Ingestion · Stone 4
query_data
Natural language to SQL — translate a question into a query, execute it, and return results.
Streaming · Stone 5
provision_kafka_topic
Create a Kafka topic with configurable partitions, replication, retention. Stores config in KB.
Streaming · Stone 5
configure_postgres_sink
Generate Debezium or JDBC sink connector JSON config for streaming data into Postgres.
Streaming · Stone 5
check_consumer_lag
Query consumer group lag per partition. Flag high-lag partitions. Returns structured lag report.
Architecture

One server. Six layers.
One knowledge base.

DataMCP is a single MCP server process with a typed tool registry. All tools share a common knowledge base client and a database connection pool. The MCP protocol contract is stable — adding new tools never changes the interface for existing ones.

AI Coding Assistants
Claude Code · Cursor · Codex
MCP client layer — calls DataMCP tools via the Model Context Protocol. Developer stays in their editor throughout.
↓
MCP Server — datamcp
Tool Registry · Dispatcher · Connection Pool
20 typed tools registered via FastMCP. Dispatcher routes calls. Shared Postgres connection pool and KB client injected into every tool handler. OS keychain for credential storage.
↓
Core — Stones 1–2
Schema · dbt · Metabase
Introspection, domain inference, dbt project generation, Jinja2 templates, Metabase API client, ER diagram.
Runtime — Stones 3–4
Dagster · dlt · Query
Dagster process manager (GraphQL polling), dlt subprocess runner, NL-to-SQL, drift detection, model explanation.
Streaming — Stone 5
Kafka · Debezium · Lag
kafka-python admin client (optional extra), consumer lag polling, connector config generation, streaming dbt templates.
↓
Knowledge Base
SQLite — 12 Tables — Persistent Context
Schema graph snapshots, generated models, KPI definitions, pipeline run history, Metabase chart metadata, Kafka configs, freshness logs, query history. Idempotent migrations on startup.
Generated Artefacts
dbt Project · Dagster Project · dlt Pipelines
All generated files written to the project repository. Standard formats — dbt YAML, Python — no proprietary lock-in. Version-controlled and editable by any engineer.
Technology Stack

Standard tools.
No proprietary lock-in.

DataMCP is built on the established open-source data engineering ecosystem. Every tool it generates uses standards that any data engineer already knows.

Core Server
🐍 Python 3.11+ ⚡ FastMCP 🔧 uv (dependency management) 🔐 keyring (OS keychain) 📦 pyproject.toml
Data Transformation
🔧 dbt-core 🐘 dbt-postgres 📝 Jinja2 templates 📊 dbt metrics 🧪 dbt test suite
Ingestion & Orchestration
🔌 dlt (data load tool) ⏱️ Dagster 🔗 Dagster GraphQL API 🐘 psycopg2 📡 SQLAlchemy
Streaming
⚡ Apache Kafka 🐍 kafka-python (optional) 🔄 Debezium 🔌 JDBC Sink
BI & Visualisation
📊 Metabase 🌐 Metabase REST API 📈 Mermaid ER diagrams
Knowledge Base & Testing
🗄️ SQLite (local KB) 🧪 pytest 🎭 unittest.mock 📏 pytest-cov 🔍 mypy (type checking)
Delivery Roadmap

Six stones delivered.
The ecosystem expands.

Each stone was scoped to be independently useful — a team gets real value from Stone 1 alone on day one. The full stack unlocks progressively as stones are added.

Stone 1
Postgres + dbt Foundation
Schema introspection, domain inference, full dbt project generation, KPI definitions, data quality tests, freshness monitoring, SQLite knowledge base.
✓ Complete
Stone 2
Metabase BI Integration
Metabase connection provisioning, schema sync, KPI-driven dashboard scaffolding, chart metadata stored in knowledge base.
✓ Complete
Stone 3
Dagster Orchestration
Dagster project scaffolding, runtime start/stop, GraphQL pipeline polling, freshness SLA sensor, streaming table skip logic.
✓ Complete
Stone 4
dlt Ingestion Pipelines
dlt scaffold and execution, row count parsing, KB persistence, source-to-staging drift detection, pipeline status surfacing.
✓ Complete
Stone 5
Kafka Streaming Layer
Topic provisioning, Postgres sink config, consumer lag monitoring, streaming-aware dbt incremental templates, hybrid batch/stream freshness tracking.
✓ Complete
Stone 6
Chat GUI Interface
Streamlit chat frontend with RAG-powered answers grounded in the full knowledge base. Non-developer access to the same infrastructure engineers manage in their editor.
✓ Partial
Stone 7
Multi-Database Support
Database adapter interface enabling MySQL, Snowflake, BigQuery, and DuckDB. Write once, run on any supported backend. Connector marketplace architecture.
Planned
Stone 8
Alternative Stacks
SQLMesh and Trino as transformation alternatives. Grafana and Superset as BI alternatives. Stack configuration profiles — user selects tools, DataMCP configures itself.
Planned
Stone 9
Enterprise & Compliance
Data lineage tracking, column-level access controls, audit log for all MCP tool calls, PII detection and masking, SOC 2 compatible run history.
Planned
Competitive Moat

Depth of context,
not breadth of features

01
Context compounds every week
After six months of use, DataMCP knows a team's schema evolution history, KPI decisions, pipeline failure patterns, and query habits better than any new hire. That knowledge is not portable — it compounds continuously in the local knowledge base.
02
Zero adoption friction
DataMCP plugs into Claude Code, Cursor, and Codex — tools developers are already using. There is no new platform to learn, no new dashboard to open, no migration required. Adoption is one MCP server entry in a config file.
03
Output is owned by the team
Every file DataMCP generates lives in the team's repository, under their version control. Standard dbt. Standard YAML. Standard Python. The lock-in comes from context depth, not from format capture. Teams can always eject — but they won't want to.

Ready to see DataMCP in action?

The full source — MCP server, 20 tools, 6 stones, Jinja2 templates, SQLite knowledge base, Dagster integration, Kafka connectors, and 232 passing tests — is available on request. Built as a portfolio project demonstrating production-grade AI data engineering.

✉ Request Repository Access
Nihildev Nandakumar · nihilsdesk@gmail.com · Available for data engineering and AI systems roles