By Sophie De Waele
At Mediahuis, we were tasked with migrating all data pipelines from our legacy platform to Databricks with a 6-month deadline. Migrating close to 300 data pipelines onto Databricks in six months is already a stretch. Now imagine doing this while handing full ownership to product teams who have never touched Git, CI/CD, or Databricks before. On paper, both projects individually would be challenging to deliver in six months, combined it felt near impossible. What changed the outcome at Mediahuis was a web agent we built on Claude. Here's what we built, what we delivered, and the biggest lesson we learned along the way.
When the timeline stopped being realistic
Our goal was ambitious: move almost 300 pipelines off a legacy platform onto Databricks, in six months while all teams were handling their regular day-to-day responsibilities. The platform team, DAP (Data & Analytics Platform), has historically handled several responsibilities across two dozen teams ranging from managing every CI/CD pipeline to creating custom tools for users.
With the migration to Databricks, we also bundled it with a deliberate shift-left initiative where teams, not the platform team, would now own their data products end to end. DAP are now enablers rather than executors.
This meant that teams suddenly had to learn, all at once, under deadline: Git and GitOps, CI/CD, the Databricks platform itself, config-based management, and data access management. With the pressure to migrate all of their pipelines, learning everything on top of this was unrealistic.
Closing the gap
We didn't have time to train every team on several new systems before they could start migrating pipelines. So, we turned to a solution that was new to our team of Platform & Data Engineers: Can we build an AI agent that lets team describe what they need, in plain language, and handle the how?
That became DapBot — a web-based agent, built on Claude, that teams could talk to directly:
- "Help me migrate my Python notebook."
- "I'd like access to the following data sets."
- "Set up an ingestion for this data source."
Under the hood, DapBot narrowed the problem to four scoped skills, each an end-to-end workflow:
- Platform configuration — managing team identities, resources, and Unity Catalog access.
- GitLab operations — turning a request into a merge request, with the right config changes attached.
- Ingestion setup and validation — setting up and checking new data ingestions against DAP's framework.
- Job migration — converting legacy pipeline definitions into their Databricks equivalents.
What it actually delivered
DapBot handled over 500 merge requests and became the primary migration tool for 95% of teams. Two things made that adoption stick:
- Setup work took minutes instead of days.
Configuration work that used to need a DAP engineer became a conversation, with consistent, templated output every time.
- Non-technical teams could do multi-system work unsupervised.
Simple prompts like "provide data access for my team" produced a complete merge request touching every required config file, which is work that previously required knowing how several systems fit together, now done by people who'd never touched Git.
One team put this to the test early. They had close to 50 pipelines to move; one team member ran all but one migration through DapBot, at one session per notebook, and finished in three weeks. The rest of their time could now be used to validate data and get familiar with Databricks itself. By giving teams the means to act without waiting on DAP or a training cycle, we turned what looked impossible into a realistic goal.
The lesson: an AI agent still needs scope
DapBot got good enough at hiding the underlying processes that we ended up missing our original goal with it. We wanted teams to become independent, but DapBot became their dependency. What should've been a tool that taught them as they went ended up being a bit too good at hiding the complexity instead. Months in, questions like "How do I set up data access for my team?" were still coming up on the days DapBot was unavailable, a sign that DapBot had been managing that resource for teams without ever explaining it.
It's a reminder that the job was never just "get the task done." The shift-left only succeeds if teams end up understanding the systems they now own, not just successfully invoking a tool that owns them on their behalf.
The tool itself was a great success and fast-tracked teams onto our new platform, but there were scoping decisions we should've made more explicit from the beginning.
As Data Engineers, we learned that AI products still require a clear understanding of scope and target audience, just like any other product. For most agents, hiding the “how” is the preferred outcome, but in our case, we should have put more focus on making sure it enabled users to understand their processes. Even with that gap, DapBot has proven to be popular amongst the teams and has shown how strong AI enablement can be. With some tweaks on our end, teams can continue to have a Platform or Data Engineer in their pocket.
The takeaway
Six months to migrate 300+ pipelines, with most of the teams doing that migration starting from zero platform experience, was not a realistic ask on its own. An AI agent, built on Claude and scoped to a handful of well-defined tasks, closed enough of that gap so that teams could face the daunting learning curve with more confidence.
AI agents are genuinely powerful for exactly this kind of work, provided they're built and scoped with the same discipline as any other product. Give teams a helper, not a black box, and acceleration and ownership can both survive the six-month deadline.