Why the Layer Diagram Isn't the Hard Part
The weakness of a layered architecture is not the layers. It is that a diagram describes structure and says nothing about discipline.
Consider the questions a reporting platform has to answer at three in the afternoon, under pressure. Which notebook produced this table. Where this column originally came from. Why the figure moved overnight. Whether the person asking is allowed to see it.
None of those are architectural questions. All of them are answered by convention, or not at all.
When every pipeline is built its own way, each answer costs someone an afternoon. Multiply that across a growing platform and the cost stops being an inconvenience. It becomes the reason people quietly build their own copy of the data instead.
This is the objection any honest conversation about data platforms has to face. Most of us have seen a well-architected platform that nobody could maintain.
Those platforms were not badly designed. They were inconsistently built.
A Better Approach
At Qubiqon, we recently worked with a manufacturing group facing exactly this pattern: a large SAP estate, a lakehouse that kept growing, and a platform that became harder to reason about with every table added.
Our approach was not to redesign the architecture. The layers were sound. We changed how they were built.
lakehouse layers · 2 ingestion paths, 3 layers
We started with the repository, not the pipelines. Every table has exactly one notebook that produces it, and the folder structure mirrors the catalog exactly. Finding the code behind a table became navigation rather than search.
We made the names carry the metadata. Numbered layers sort correctly in any catalog explorer, a catalog per layer means access is granted once at the top, and technical source names stop at the raw layer. Above raw, an analyst never sees an abbreviation.
We separated logic from orchestration. Each notebook holds its transformation in SQL and declares its target and sources in a single configuration cell. The run order is derived from those declarations rather than maintained by hand, so the dependency graph cannot drift from the code.
We made the view layer the contract. Reporting reads views over the star schema and never the tables underneath, which means the physical model can be restructured without breaking a single report.
The architecture did not change. It simply stopped being the only thing holding the platform together.
Conventions Aren't Bureaucracy
The common misreading of this is that conventions slow a team down.
They do the opposite. Each one removes a decision that would otherwise be made differently every time, by a different person, under time pressure. One notebook per table removes the question of where a table is produced. A repository shaped like the catalog removes the search. Names that carry their layer and environment remove the guesswork about what an object is before you open it.
The point is not elegance. It is that a question about any table has one place to go and one answer waiting there.
There is a second effect that is easy to miss. Keeping transformations in SQL meant the people who defined the business logic could read and review it themselves. On an SAP platform that matters more
than usual, because the bottleneck is rarely the engineering. It is knowing which of tens of thousands of tables holds the answer, and that knowledge sits with the module specialists, not the engineers.
The Business Impact
The engagement changed how the platform is worked with day to day:
• Every table traceable to the single notebook that produced it
• A dependency graph derived from code rather than maintained beside it, so it cannot drift
• Reporting decoupled from the physical model by a view layer, allowing tables to be restructured without breaking reports
• One governed definition per metric, rather than one per report author
• Access granted at catalog level, so raw is not reachable by accident
• New engineers productive without a guided tour of the codebase
What Comes Next
A prediction, stated with the confidence each part deserves.
Near-certain for 2026: catalog-level governance and column lineage become the default expectation rather than a maturity milestone. Teams will stop reconstructing join chains by reading notebooks.
Likely: declarative, configuration-driven orchestration displaces hand-maintained job graphs on most new platforms. Once the dependency order is derived from the code, maintaining it separately looks like the anomaly it is.
Contested: whether transformation logic belongs in SQL or in a framework. The engineering arguments favour the framework. The reviewability argument, which decides whether a business specialist can check your logic, favours SQL. On ERP platforms I would bet on SQL for longer than most people expect.
Speculative: agents that write and maintain pipelines. The capability will arrive before the conventions that make their output reviewable, and teams without those conventions will inherit code nobody can trace.
The platforms that age well will not be the ones with the most sophisticated architecture. They will be the ones where every table can still explain where it came from.
