By Stijn Dolphen
Choosing the right analytics tool shapes how quickly and reliably companies turn raw data into insights. For years, dbt Core has been the backbone of that transformation, a trusted standard in the data ecosystem. But now, a new challenger is emerging: SQLMesh, a stateful, flexible framework offering a fresh take on managing data workflows. Is SQLMesh ready to challenge dbt Core's position?
Spoiler: It’s a very promising contender, but like most things in data, the reality is layered. Here’s what stood out when we put SQLMesh and dbt Core head-to-head.
The Playground
To run a fair test, we built an end-to-end pipeline using the Medallion Architecture (Bronze, Silver, Gold) on Databricks and loaded it with the Airline On-Time Performance dataset from Harvard Dataverse. That’s over 120 million rows, nearly 12GB uncompressed, and packed with real-world complexity.
To track SQLMesh model state with minimal overhead, we used DuckDB (more on that later). We recreated the same data flows in both SQLMesh and dbt Core to compare how each handles complex scenarios like state changes, backfills, and scheduling.
Our goal was to see how each handled the challenges that often happen in operational environments.
How They Compare, Feature by Feature
Virtual Environments
Before launching changes, you need a safe environment to test without risking production.
Setting up a prod and dev environment in SQLMesh was really easy to do. You spin up an isolated virtual environment without needing to fork your entire project or replicate complex configs. This makes experimentation cheap and fast, which is especially handy for small teams or fast-moving projects.
dbt Core requires more setup: separate profiles, targets, and often multiple schemas. For projects working with large volumes of data, this means running the same models multiple times across environments. That duplication directly impacts both compute and storage usage and those costs add up quickly.
Incremental Models
In large datasets, you don’t want to reload everything, just update what’s changed.
dbt Core is stateless, it doesn’t really remember what happened before unless you explicitly build that logic in with is_incremental() along with timestamp comparisons. This works fine... until it doesn’t. Backfilling two random days? That’s on you to handle. Missed a load? You need to know exactly where and why.
SQLMesh, on the other hand, tracks state. It knows what ran, what didn’t, and what needs fixing. That’s powerful. It’s a strong feature to make managing targeted backfills or spotting missed data feel much more controlled.
But, do you always need state? State introduces complexity too, you need a persistent SQL database behind the scenes. You also wonder what happens during version upgrades or corrupted states. dbt doesn’t worry about any of that, and that’s part of its charm.
Data Observability
Before making changes, it’s crucial to see what exactly will be impacted, and catch mistakes early. Here’s how both tools approach observability and quality control.
SQLMesh introduces a “plan” step - much more than just a build. It shows what models will run, what changed, and what downstream models are affected. It’s like version control, dependency graph, and QA assistant rolled into one.
You can inspect the plan, validate it, even roll it back. dbt Core doesn’t offer this natively - you’d need CI pipelines and artifact inspection to replicate the view.
SQLMesh’s focus on lineage and stateful tracking means it can highlight breaking changes downstream, before you deploy. It tracks when things were backfilled, shows how models connect, and stores snapshots that support both time-travel and rollbacks.
Better observability leads to safer deployments, faster issue detection, and greater confidence across teams.
Scheduling
Data pipelines often need to run on schedules - nightly updates, hourly refreshes, etc.
Scheduling is a bit of an apples-to-oranges comparison here. dbt Core doesn’t come with a scheduler. Most teams connect it to Airflow, Prefect and Dagster to handle orchestration. It’s flexible, but it does mean extra setup and tooling, especially for small teams or simpler workflows.
SQLMesh, in contrast, comes with basic scheduling capabilities built in. You can define schedules per model directly in your project, making it possible to manage orchestration without needing an external orchestrator - at least for simpler use cases.
So... Is SQLMesh Better?
Not quite. But it’s close. And in some ways, it’s ahead.
dbt Core has earned its status as the backbone of analytics workflows today. It’s supported by a massive community, documentation, and an ever-growing library of packages that help teams move faster. dbt’s capabilities work across a wide range of databases (eg. MS SQL Server, which isn't fully supported yet in SQLMesh) and integrates easily with orchestration tools, BI platforms, and enterprise stacks. That level of maturity gives teams the confidence to scale with dbt and rely on it in production.
SQLMesh, meanwhile, is the one to watch. It brings thoughtful defaults, stateful execution, integrated planning, and a smooth development experience, all built for teams that want more control without having to piece everything together.
This comparison focused on the open-source versions. Features from dbt Cloud and the upcoming Semantic Data Framework (SDF) integration weren’t part of this analysis, but they’re worth watching. Both ecosystems are evolving fast, and a deeper comparison could easily go further.
Why This Matters
Choosing between dbt Core and SQLMesh isn’t just about comparing features, it’s about shaping how your organization builds, scales, and trusts its data infrastructure.
At Dataroots, we help teams make these choices with confidence, blending deep technical expertise with a strong focus on business impact.
Both dbt Core and SQLMesh offer powerful paths forward. The best choice depends on your platform maturity, operational needs, and innovation goals.
👉 Curious how SQLMesh or dbt Core could fit your next move? Reach out to the authors, they’d be happy to explore it with you!
