Why Data Migrations Are Always Messier Than You Think

Data migration doesn't break your data. It shows you how fragile it already was – and has been for years. However, what can break everything else – the timeline, the budget, the team – is underestimating what you're actually doing. Data migration shouldn’t just be a “line item in the project plan”. It's the continuos and iterative work of getting your data right so your business can operate right.

Data migration shows up in every program whether it is customer onboarding, system replacement, a modernization initiative, or an M&A integration – and it is always messier than anyone expects.

Data migration is consistently the highest-risk, most time-consuming activity in any systems change. And the reasons it goes sideways are remarkably predictable – even if teams keep getting surprised by them.

After years of working with financial institutions, consulting firms, and software companies on this exact problem, I've seen the same four patterns show up again and again. Understanding them is half the battle. The other half is knowing what it takes to get ahead of each one  –  the right approach, the right tooling, and the right mindset  –  before they compound into something program-threatening.

Every Production System Carries Operational Debt

People talk about technical debt in code. But production systems carry something broader: business operational debt. Years of workarounds, bolt-ons, manual overrides, and undocumented exceptions that kept the business running. When you migrate, that debt doesn’t stay behind. It shows up as data – messy, inconsistent, and full of edge cases nobody remembers creating.

This is why upfront and ongoing data profiling is critical at the start and throughout any migration. When you can see the completeness, distribution, and quality of your data within minutes rather than weeks, you’re working from reality instead of assumptions. A project manager who knows upfront that a critical date field is missing in 500 records can plan around it. One who discovers this for the first time three months in is managing a crisis.

The Problem Lives in the Handoffs

Here’s something I see on every program: the person who knows the business rule is not the person who writes the data rule. Between them, there’s a chain of handoffs – analysts, engineers, sometimes third-party consultants – and every stop is a lossy connection. Context gets dropped. Intent gets reinterpreted. By the time a transformation rule gets coded, it may reflect what someone thought the requirement was, not what it actually was.

The compounding effect is brutal. One misunderstood business rule becomes a transformation error, which becomes a reconciliation break, which becomes a go-live delay. If the person who knows the answer could act on it directly – without the chain of handoffs – most of these breaks never happen.

Most Programs Start from the Wrong End

It's worth separating two things that often get conflated: lift-and-shift and data migration. Lift-and-shift is moving or replicating data without logical change to data. A true data migration is something different. It's an opportunity to land in a target state – often with a data model change – that supports how the business operates going forward, not how it operated before.

That distinction changes where you should start. The typical instinct is to start with what you have: pull out the source data, understand it, and then figure out where it goes. That feels logical. But starting from the source means you can invest significant effort in mapping and transformation before you fully understand what the target actually requires. Gaps appear slowly – or worse, after significant work has already been done.

A target-centric approach flips this. Start with what the new system requires, then work backward to understand how your current data fits – or doesn’t. AI-powered mapping can predict field matches between source and target schemas in seconds, giving teams a starting point that would otherwise take days, weeks or months of manual side-by-side comparison. That head start changes the trajectory of the entire program.

In Financial Services, Complexity Is Structural

Not all data migrations are created equal. When you’re migrating investment or financial applications, the complexity isn’t just about volume – it’s structural. Financial data doesn’t live in one place. Positions, counterparties, reference data, and transactions are scattered across systems, each with their own rules, formats, and interdependencies.

At this level of referential complexity, you need more than a mapping spreadsheet. You need metadata that actively connects every migration step – so when one field changes, everyone downstream knows about it. And if you’re dealing with legacy mainframe systems, the challenge compounds further: the business logic that governs how data was calculated, stored, and routed is buried in COBOL modules that may not have been documented in decades.

How Zengines Helps You Get Ahead to Avoid the Mess

Data migration isn’t a side activity that happens at the end of a program. It’s the connective tissue of every systems change – whether you’re modernizing legacy systems, managing mainframes, or meeting new regulatory compliance requirements. We built Zengines to treat it that way.

Every problem I described above has a direct answer in our platform.

• Operational debt hiding in your data? Zengines profiles your source data automatically – surfacing completeness gaps, format inconsistencies, and quality issues in minutes instead of weeks, so your team plans from reality, not assumptions.
• Challenging handoffs between business and technical teams? Our platform keeps analysis, mapping, transformation, and reconciliation in one place, so the person who knows the business rule can act on it directly – no chain of handoffs, no lost context.
• Starting from the wrong end? Zengines is target-centric by design: AI predicts field mappings between your source and target schemas in seconds, giving teams a validated starting point that would otherwise take days of manual comparison. AI also generates transformation rules to ensure the data gets the right business logic treatment.
• And the structural complexity of financial data? Our platform maintains active metadata that connects every migration step, so changes upstream are visible downstream – across every table, every relationship, and every transformation rule.

When legacy mainframes are part of the equation, Zengines goes further. Our contextual data lineage capability parses COBOL, RPG, and PL/1 code to extract the embedded business logic, calculation rules, and data flows that have been locked inside these systems for decades – giving your team the transparency to reverse-engineer requirements in minutes, not months.

The result: business analysts are 6x more productive, migrations move 80% faster, and transformation rules are generated from plain English prompts – so the people closest to the business drive the process without waiting on engineering resources.

The programs that go smoothly aren’t the ones with the simplest data. They’re the ones that saw the potential messiness early, connected the right people to the right decisions, and had the tooling to act on what they found.

If your organization is planning a migration or modernization initiative, schedule a demo with our team to see how Zengines turns the messiest part of your program into the most predictable one.

If you're searching for contextual data lineage, you've probably already discovered something frustrating: most lineage tools tell you surface-level relationships between data points–where data came from and where it went–but not much else.

You're left staring at a diagram that shows Table A feeds into Table B, which outputs to Table C. Technically accurate. But when a risk analyst asks why a capital reserve figure changed overnight, or a regulator wants to know exactly which source system contributed to a reported metric and under what transformation logic, the map answers none of it.

Where data came from and where it went is the starting point. What analysts, risk teams, and compliance officers actually need is the context: what logic touched it, what conditions applied, what changed, and what business rule was in effect at the time. That's the difference between a lineage map and lineage you can actually use.

The Problem with Traditional Data Lineage

Traditional data lineage tools were designed to answer a narrow question: where did this data come from, and where did it go?

That was a reasonable starting point decades ago. But for organizations managing complex legacy estates today – particularly mainframes or midranges running COBOL, RPG, etc. – surface-level mapping barely scratches the surface of what you need.

Consider what happens when a regulator asks you to explain how a specific calculation is derived. You can show them a data flow diagram. They'll nod politely. Then they'll ask: "But why is it calculated this way? What business rule drives this? When did this logic change, and why?"

The traditional lineage tool has no answer.

Or consider a modernization project where your legacy system produces one result and your new platform produces another. Is that difference significant? Is it a bug? Is it an intentional business rule that was never documented?

Without context, you're back to the same approach that's been failing for decades: finding someone who remembers, hoping the documentation exists, or spending weeks tracing through cryptic code.

What Contextual Data Lineage Actually Means

Contextual data lineage goes beyond mapping data flows. It captures the intent and reasoning behind how systems were built – the business logic, decision contexts, and institutional knowledge embedded in decades of code evolution.

A Gartner analyst recently described this capability as "knowledge and logic extraction" – and noted that it represents an emerging category distinct from traditional lineage tools.

The distinction matters because context transforms raw lineage data from overwhelming output into actionable intelligence:

• ‍Without context: You know that Field X flows through Program Y and ends up in Report Z. You have no idea why the program applies a specific multiplier, under what conditions it branches, or what business requirement drove that logic forty years ago.‍
• With context: You understand that the multiplier exists because regulatory requirements changed in 1987, that the branching logic handles different asset types, and that the specific calculation matches the methodology documented in your compliance framework – or doesn't, which is exactly what you needed to identify.

This is the difference between data and understanding.

Why Raw Lineage Data Isn't Enough

Here's what some vendors don't tell you: lineage data can be extraordinarily rich and detailed, yet still fail to be useful.

We learned this directly from customers. They told us that comprehensive lineage output – no matter how accurate – was overwhelming. Compliance teams would receive massive data dumps and have no idea where to start. Business analysts would get technically correct diagrams that didn't answer the questions they were actually asking.

The problem isn't the data. The problem is that data without context forces you to become an archaeologist, piecing together meaning from fragments.

What teams actually need is the ability to ask a question and get an answer – in plain language, with business context, in a timeframe that makes the answer useful.

What This Looks Like in Practice

When context is embedded in your lineage approach, the scenarios that typically take weeks or months become manageable in hours or minutes.  See the examples below:

Legacy system modernization

Your organization is migrating off the mainframe to a modern cloud-based platform. The project is stuck in the analysis phase–and has been for months, because no one can confidently explain how the legacy system actually works.

Here's the scenario that plays out constantly: you run a transaction through the old system and get one result. You run the same transaction through the new platform and get a different result. The old system says the interest accrual is $5.00. The new system says $15.62.

Which one is right? More importantly, why are they different?

With the new system, you can trace the logic – the code is documented, the team that built it is still around. But the legacy system? That calculation was written forty years ago, modified dozens of times since, and the people who understood it have long since retired. You're left reverse-engineering requirements from cryptic COBOL modules, hoping you find the answer before the project timeline slips again.

This is where contextual lineage changes everything. Instead of weeks of system archaeology, analysts can trace the calculation back through its entire history – seeing not just what the logic does, but why it was written that way, when it changed, and what business requirement drove each modification. They can determine whether the $5.00 reflects an intentional business rule that needs to be replicated in the new system, or an outdated approach that can be safely left behind.

Without this context, modernization projects stall. Teams can't confidently port or decommission legacy systems because they can't prove the new platform handles every scenario correctly. With contextual lineage, what used to take months of investigation becomes a matter of minutes – and teams can finally move from analysis to action.

Regulatory response and audit readiness

A regulator demands lineage-based evidence. An auditor spot-checks in real time. Failure to respond accurately and quickly exposes the company to fines, consent orders, or worse. Without contextual lineage, compliance teams spend months manually assembling fragmented documentation, chasing down tribal knowledge, and hoping nothing was missed. With it, they generate audit-ready responses immediately and handle live questions on the spot – transforming regulatory exposure into regulatory confidence.

Data feed or vendor replacement

Your business wants to swap an outdated data feed or vendor for a more modern alternative. Sounds straightforward, but decades of modifications have buried the answer to a simple question: which feed is actually being used today? Teams spend weeks hunting through systems, hoping they've found the right source. Get it wrong and you've got data corruption or system failures. With contextual lineage, analysts trace back to the exact source in minutes with complete confidence – eliminating weeks of effort and the risk of replacing the wrong feed.

Onboarding new team members

Your mainframe experts are retiring, and their institutional knowledge is walking out the door with them. New team members face a wall of undocumented legacy code with no way to get up to speed. Contextual lineage translates that complexity into plain language, allowing new analysts to orient themselves to unfamiliar systems in hours instead of months – preserving critical knowledge before it's lost.

The Shift from Data Extraction to Understanding

Traditional tools extract data. The next generation extracts understanding – and packages it so people can actually use it.

This isn't a feature difference. It's a category difference.

Legacy platforms like Collibra were built for metadata management and governance workflows. They're valuable for those purposes. But when it comes to unlocking the institutional knowledge trapped in legacy systems, they weren't designed for the depth of analysis that complex modernization and current compliance initiatives require.

What's needed is a fundamentally different approach: one that translates complex legacy code into plain language with business context, allows self-service access without requiring technical expertise in legacy languages, and curates rich lineage output into formats that compliance teams, business analysts, and project managers can actually address.

Finding Contextual Data Lineage

If you're evaluating lineage tools, the questions to ask are:

1. Does it just map data flows, or does it expose business logic?
2. Can it explain legacy code into language business users understand?
3. Does it provide context around why calculations exist, not just that they exist?
4. Can compliance teams use it directly, or does every question require a COBOL or RPG specialist?
5. Is the output actionable, or is it just overwhelming?

The answers will quickly reveal whether you're looking at surface-level lineage or something that can actually solve the problems you're facing.

Zengines provides contextual data lineage for legacy systems, helping enterprises understand, manage, and modernize their most critical legacy assets. Our platform translates complex COBOL, RPG, and other legacy code into plain English with business context – enabling teams to answer questions in minutes instead of weeks.

Something structural is shifting in consulting - and the firms paying attention are rethinking how they staff, price, and deliver client work as a result.

Clients are pushing back on people-heavy, time-and-materials engagements. They're asking harder questions about what they're actually paying for, and in some cases they're building internal capabilities rather than renewing multimillion-dollar consulting contracts. The era of charging by the hour for work that AI can now accelerate dramatically is under visible pressure - and the consulting industry is feeling it.

Nowhere is this tension more acute than in financial services technology delivery, where data migration sits at the center of nearly every major transformation program. It's the workstream that consumes the most analyst hours, carries the most project risk, and is most likely to determine whether a client engagement ends with confidence or – in the worst case – with a lawsuit.

The firms finding a path forward are the ones investing in AI-powered delivery capabilities - not as a marketing claim, but as a genuine operational shift that changes what they can promise and reliably deliver.

The Billing Model Under Pressure

The numbers behind the shift are striking. Business Insider reported in November 2025 that McKinsey disclosed roughly a quarter of its global fees now come from outcomes-based arrangements - a notable departure for an industry where traditional time-based billing has dominated for decades. EY's leadership has openly acknowledged the same pressure, with executives suggesting that AI could push consulting toward a "service-as-software" model where clients pay for results rather than labor. PwC, meanwhile, reduced its global headcount by more than 5,600 in 2025 - a signal that the people-heavy delivery model is already under structural strain.

The underlying tension is straightforward: AI makes consultants dramatically more productive, but most revenue models still depend on billable hours. A task that once required 60 hours can now be completed in 6. If firms deploy AI aggressively, they either earn less revenue for the same work or they have to fundamentally rethink how engagements are scoped and priced.

Buyer expectations are shifting - clients increasingly want to pay for results. The pressure is real and it's intensifying. Consulting firms that once relied on junior teams to churn through data-heavy work are now discovering that clients can replicate that output with an off-the-shelf AI tool and a couple of their own analysts - and they're asking why they should keep paying consulting rates for it.

When Delivery Fails, the Stakes Are High

The pressure on consulting firms isn't only coming from pricing conversations. It's coming from clients who are done absorbing the cost of programs that don't deliver.

In September 2025, Zimmer Biomet filed a $172 million lawsuit against Deloitte Consulting over a botched SAP S/4HANA implementation. The complaint alleged that Deloitte misrepresented its capabilities, assigned undertrained and constantly rotating offshore teams, and concealed system defects before a July 2024 go-live that left the company barely able to ship products, issue invoices, or generate basic sales reporting. The total damages sought included $94 million in fees paid to Deloitte, $15 million in additional remediation invoiced by Deloitte itself, and $72 million in Zimmer Biomet's own post-go-live costs.

The case is still working through the courts. But regardless of outcome, it illustrates a broader dynamic: clients are no longer absorbing failed technology programs quietly. They are quantifying the damage and pursuing accountability. And for the consulting firms delivering these programs, the risk profile of a poorly managed implementation has grown considerably.

In financial services -- where a data error doesn't just cause operational disruption but can trigger regulatory scrutiny, client relationship damage, and audit findings - the consequences of delivery failure are even more pronounced. A migration that goes wrong at a bank or asset manager isn't just a project problem. It's a systemic risk event.

Where Financial Services Delivery Is Uniquely Hard

Financial services technology programs put consulting teams in a particular bind. The work is genuinely complex, the data is dense, and the tolerance for error is narrow - yet the pressure to compress timelines and control costs is as high here as anywhere.

Consider what a typical data migration engagement looks like in this space. A bank modernizing its legacy infrastructure, an asset manager consolidating data after an acquisition, or an insurance carrier migrating off a legacy policy administration system - each arrives with decades of client data stored in formats that weren't designed for portability. Position histories across multiple asset classes. NAV records from prior administrators. Interest calculations embedded in COBOL modules that haven't been touched since the 1990s. Counterparty hierarchies full of historical exceptions and overrides.

The consulting team's job is to move all of that accurately, quickly, and in a way that satisfies both the client's operational requirements and the regulatory frameworks that govern their data. BCBS-239 for global systemically important banks. ORSA and Solvency II for insurers. The compliance dimension means that reconciliation isn't just a technical milestone - it's an evidence-gathering exercise that regulators will review.

And yet, this is precisely the work that has traditionally been done manually: analysts comparing schemas side by side, writing transformation rules by hand, iterating with target systems through slow feedback loops. It's time-intensive, expertise-dependent, and difficult to scale.

The Legacy System Problem No Standard Playbook Solves

A significant share of financial services programs involve migrating data off legacy systems - mainframes running COBOL, AS/400 environments running RPG, or custom platforms whose original developers retired years ago. For consulting teams, this creates a structural challenge that sits upstream of everything else: the source system is a black box.

The business logic governing how data is calculated, transformed, and stored in these systems was often never externally documented. It lives in the code - in tens of thousands of COBOL modules, in conditional branching logic written to solve a specific business problem and never touched again. When a consulting team needs to understand why a risk calculation produces a particular result, or how two legacy fields need to be combined before they can map to a target schema, they often have no reliable starting point.

The traditional answer has been to engage the institution's mainframe specialists - a small, typically overburdened group who are simultaneously managing live operations and fielding questions from the migration project. Analysis that should take days can take weeks. And when those specialists retire, the institutional knowledge goes with them.

Contextual data lineage changes this calculus entirely. AI-powered platforms can parse thousands of COBOL or RPG modules and surface the calculation logic, data flows, field relationships, and branching conditions embedded in legacy code - in minutes rather than months. For consulting teams, this means arriving at the analysis phase with a structured, searchable map of what the legacy system actually does, before a single record is moved.

That foundation changes everything that follows. Learn more about what contextual data lineage reveals in legacy financial systems.

How AI Changes the Delivery Economics and Predictability

For consulting firms navigating the shift toward outcomes-based pricing, AI-powered data migration tooling offers a concrete path to better margins and better delivery - simultaneously.

The efficiency gains are measurable and meaningful. Business analysts on AI-assisted migration projects work up to 6x faster. Migrations complete up to 80% faster overall. and the work that once required senior technical resources increasingly flows through analysts with the right platform behind them.

In a financial services context, these gains show up in specific, high-stakes ways:

• Early data profiling gives consulting teams a clear picture of source data quality - completeness rates by field, currency distributions, unique values, anomalies -- before execution is underway. For a senior project manager, this visibility is the difference between a defensible timeline and one that keeps slipping.
• Predictive field mapping eliminates the blank-spreadsheet start. AI-generated mapping predictions -- ranked by confidence, ready for review -- compress what is typically the most expertise-heavy step into a validation exercise. Business analysts can own the work end-to-end, without requiring a senior SME to perform what is fundamentally an analyst-level task.
• AI-assisted transformation handles the precision that financial data demands: standardizing identifiers, reformatting currency codes, reconciling accounting bases, applying calculation logic consistently across millions of records. Work that previously required a systems engineer can be completed by a business analyst -- which directly affects how engagements are staffed and priced.
• A single, governed platform replaces the sprawl of spreadsheets, email threads, and siloed tools that typically fragment a migration engagement. Every mapping decision, transformation rule, and data file lives in one place -- visible to every teammate, maintained in a consistent state, and managed through a structured workflow rather than tribal knowledge. For consulting firms, this is what makes delivery governable at scale: the engagement doesn't live in an analyst's head, it lives in the platform.

For consulting firms, the deeper advantage is structural. Zengines is the single source of migration truth -- where every decision is made, every rule is stored, and every teammate works from the same live picture. Profiling feeds mapping. Mapping feeds transformation. Transformation feeds testing. The engagement lives in the platform, not in any one person -- which means it's scalable, transferable, and consistently deliverable regardless of who is staffed on the next one.

From Effort to Outcomes: Making the Transition Real

The shift to outcomes-based delivery isn't just a pricing conversation - it's an operational one. Firms can't credibly commit to delivery outcomes on fixed-fee or risk/reward structures if their underlying methods are still dependent on manual, labor-intensive processes that are inherently unpredictable.

This is the core reason why AI tooling matters so much for consulting firms right now. It's not about replacing consultants - it's about giving delivery teams the infrastructure to make commitments that they can keep. When field mapping is AI-assisted, reconciliation is automated, and data quality is profiled upfront, project timelines become far more predictable. And predictability is the prerequisite for outcomes-based pricing.

Firms building these capabilities are finding that they compete differently. They can take on fixed-fee engagements with genuine confidence rather than aggressive contingencies. They can staff programs leaner without sacrificing quality or pace. They can have more credible conversations with financial services clients who have been burned before and are scrutinizing methodology more carefully than they used to.

The Big 4 and major systems integrators are all investing in AI platforms - EY's AI Agentic Platform, Deloitte's Zora AI, KPMG's and PwC's respective investments - but rolling out new tooling across thousands of staff, multiple service lines, and global operations takes time.

The firms moving fastest are the ones being strategic about where AI solves the most acute delivery problems first. In financial services technology programs, that means data migration and legacy system analysis.

What This Means for How Firms Compete

Financial services clients have long memories when it comes to failed implementations. Many have lived through at least one program where data issues surfaced late, caused delays, and required expensive remediation. They ask harder questions in proposal stages now, and they're paying close attention to how prospective partners describe their methodology - not just their credentials.

Consulting firms that can demonstrate AI-powered migration capabilities as a concrete, operational practice - not just a line on a capability slide - are differentiating themselves in a market where the work is increasingly scrutinized and the pricing conversation is shifting. That differentiation translates directly into faster delivery, lower cost, reduced probability of late-stage surprises, and more defensible outcomes for clients whose data environments are regulated and complex.

The firms that navigate this moment well won't be the ones that simply talk about AI. They'll be the ones that have embedded it where delivery risk is highest - and in financial services technology programs, that starts with data.

For more on the specific challenges that make legacy financial system migrations difficult to de-risk without the right tooling, see Why It's So Hard to Leave the Mainframe.

Delivering a financial services technology program that involves data migration or legacy system analysis?

Zengines partners with consulting firms and systems integrators to accelerate data migration delivery, unlock legacy system business logic, and produce the audit-ready documentation that financial services clients and regulators require. Schedule a demo to see how it works, or explore our resources library for more on AI-powered data conversion and contextual data lineage.