Methods · Topic T6

How the life cycle of a data project at scale should be structured

Reviewed October 18, 2026

It prints in full from the browser, on A4 or Letter.

Block 1

The problem

Designing a data platform and the life cycle of what runs on it so that it grows through integrations without multiplying copies, without mixing clients and without taking untested changes to production.

Summary in three lines

The starting point is the data’s path and its consumers, not the components: where it comes from, what transformation it still needs to be trustworthy and who consumes it, with which engine —people, applications or agents—. That is how Vesting’s data platform was designed from scratch on Microsoft Fabric.
What is common is kept apart from what is configurable: an event contract and early normalization so everything speaks the same language, layers that move from intake to consumption, a single copy of the data in an open format and isolation as a property of the architecture —one workspace per client—, not as a filter.
The life cycle does not end at publishing: development, test and production kept apart, changes versioned and promoted between environments, a reconciliation that stops publishing when a figure does not add up, and retiring whatever no longer answers a need.

Block 2

The map

From the source to whoever decides, and from the change to production

Swipe to see the whole map.

Legend of shapes and colors

Actor — who does the work
Data — what is stored and read
Control — a rule that gets checked
Person — a person decides
External — what comes from outside

Reading the map, node by node

Integrations deliver events that must meet a contract: required fields, context identifiers and common states. Intake keeps the raw event, preparation validates and normalizes it, and the consumption tables serve it as a single copy, in an open format. The governed semantic model is the definition read by whoever decides and also by applications and agents, which query the measure instead of reinterpreting the raw table. Each client has its own workspace on top of the same structures. Whatever does not add up when the data is used reaches observation and lineage, which ask for a change: the change is born in development, passes a reconciliation test and a deployment pipeline promotes it to production.

Node 1 Sources and integrations (External) — Whatever produces the events: at Vesting, the agents’ integrations and the flows that orchestrated them.
Node 2 Event contract (Control) — Required fields, context identifiers and common states; whatever fails it shows up at intake.
Node 3 Intake: the raw event (Data) — The lakehouse keeps every session’s detail so what happened can be rebuilt.
Node 4 Preparation and quality (Control) — Validations inside the pipeline: missing values, unexpected types, duplicates and relationships that do not close.
Node 5 Consumption tables (Data) — Relational structures and aggregates for predictable questions, as a single copy and in an open format.
Node 6 Governed semantics (Data) — Relationships, measures and row-level security: the definition everyone consumes.
Node 7 Whoever decides (Person) — Product, operations and process leaders: they read dashboards built on the same meaning.
Node 8 Applications and agents (Actor) — New consumers: they read the governed measure instead of reinterpreting the raw table.
Node 9 One workspace per client (Control) — The access boundary matches the client’s boundary, on top of the same common structures.
Node 10 Change in development (Data) — Every change is born in its development environment, versioned.
Node 11 Test and reconciliation (Control) — The key figure in the new model must match the current model’s and the source’s.
Node 12 Deployment pipeline (Control) — Promotes the change from one environment to the next, with a way back to the previous version.
Node 13 Observation and lineage (Control) — From the dashboard to the model, the table and the pipeline: who answers for each asset, what depends on it and what no longer answers a need and gets retired.

Connections

from Sources and integrations to Event contract (delivers)
from Event contract to Intake: the raw event (complies)
from Intake: the raw event to Preparation and quality
from Preparation and quality to Consumption tables (normalizes)
from Consumption tables to Governed semantics
from Governed semantics to Whoever decides (decides)
from Governed semantics to Applications and agents (queries)
from One workspace per client to Intake: the raw event (isolates)
from Whoever decides to Observation and lineage (what does not add up)
from Observation and lineage to Change in development (asks for a change)
from Change in development to Test and reconciliation
from Test and reconciliation to Deployment pipeline (adds up)
from Deployment pipeline to Preparation and quality (promotes)

Block 3

The phases

1
Phase 1
Map the data’s path and its consumers
Inputs
- The process, the questions and the decisions the data must support.
- The sources and their integrations.
Activities
- Identify who consumes each piece of data and with which engine: SQL, Spark, the semantic model or an agent.
- Decide what level of detail is kept and how often it is needed.
- Name, for each piece of data, the copy that will be the reference.
Artifacts
- The map of the path, from the source to each consumer.
Who decides
Whoever answers for the data platform.
Control that closes the phase
If the answer to “which copy is the reference” is “several”, the architecture is not finished yet.
Evidence
Before creating an artifact in Fabric, the author’s practice is to answer three questions: who is going to consume this data and with which engine, what transformation it still needs to be trustworthy, and which of the possible copies will be the reference.
2
Phase 2
Set the event contract and normalize at intake
Inputs
- The events each integration brings.
- The common entities: session, request, response, state.
Activities
- Define required fields and context identifiers that travel from the very first event.
- Validate the structure at intake and make the exceptions visible.
- Separate what is common, which makes comparison possible, from what is configurable, which keeps things moving.
Artifacts
- The event contract.
- A common representation of the events.
Who decides
Whoever designs the platform, together with product.
Control that closes the phase
A change in the event structure is validated at intake before it can touch a semantic model.
Evidence
At Vesting, a new agent or a new client could not require a new table for something that already had a name on the platform; with 27 agents and 12 clients, early normalization was the key decision of the architecture.
3
Phase 3
Design the layers and serve a single copy
Inputs
- The normalized events.
- The workloads: Spark, SQL and the semantic model.
Activities
- Keep the detail at intake and move through increasingly trustworthy layers, with quality rules.
- Assign each workload to the right engine: the lakehouse for engineering and Spark, the warehouse for relational work and T-SQL.
- Isolate by client with separate workspaces on top of the same structures.
Artifacts
- The lakehouse with the detail and the consumption tables.
- One workspace per client.
Who decides
Whoever designs the platform.
Control that closes the phase
Every transformation has a place and a reference: neither the data nor the logic is duplicated without a reason.
Evidence
At Vesting, separating intake, preparation and consumption let a new integration come in without touching the previous ones: intake absorbed each agent’s variation and, from preparation onward, everything spoke the same language.
4
Phase 4
Model the meaning for people, applications and agents
Inputs
- The consumption tables.
- The definitions agreed with the business.
Activities
- Build the star schema, the measures and row-level security, tested by impersonating roles before publishing.
- Choose the storage mode by the freshness, volume and cost each use needs.
- Decide what an agent reads: the governed measure, or the table with its interpretation written down.
Artifacts
- The governed semantic model.
Who decides
The business agrees on the definition; the platform turns it into a rule.
Control that closes the phase
A critical measure is not freely recalculated: whoever consumes it reads it from the model.
Evidence
When two people arrive with different figures for the same indicator, the problem is solved in the semantics and in the agreed definition, not in the visual; at Banco Pichincha an indicator with two definitions was a governance problem before it was a DAX problem.
5
Phase 5
The life cycle in Microsoft Fabric: development, test and production kept apart
Inputs
- The change that is meant to be published.
- The current asset and who depends on it.
Activities
- Version the semantic model as a project, in a Git repository.
- Promote the change between workspaces separated by environment with deployment pipelines.
- Reconcile the key figure: the new model, the current one and the source have to match.
- Know which people, processes or products depend on the asset before changing it.
Artifacts
- The versioned change and its reason.
- The recorded reconciliation.
Who decides
Whoever answers for the asset.
Control that closes the phase
A difference that cannot be explained stops publishing.
State of the field
Microsoft builds Fabric CI/CD on Git integration and on deployment pipelines across development, test and production stages; since each of them links one workspace to one environment, it advises one workspace per environment.^[6]^[7]
6
Phase 6
Observe, optimize and retire
Inputs
- Real use and the capacity consumed.
- The published assets and their lineage.
Activities
- Walk the lineage from the dashboard to the model, to the table and to the pipeline that loaded it.
- Step in first at the layer where the waste starts, not at the last one.
- Retire whatever stopped answering a need, telling its consumers.
Artifacts
- The lineage and the retirement decision.
Who decides
Whoever answers for the platform.
Control that closes the phase
No asset stays current in appearance only: either it has consumers and an owner, or it is retired.
State of the field
Fabric spreads each operation’s consumption over time —interactive ones over a minimum of five minutes, background ones over 24 hours— and, if the capacity gets overloaded, it first delays new interactive operations by 20 seconds and then rejects them.^[9]

Block 4

The design decisions

What is common and what is configurable in Vesting’s data platform
When yes
Common: the fundamental events, the identifiers, the states and the metrics, with shared definitions that make comparison possible.
When no
Configurable: each client’s or agent’s own attributes, inside a structure that already knows how to take them and without redesigning the platform.
Why
Evidence
Vesting’s data platform was designed with the minimum level of standardization that kept the startup’s agility without giving up traceability or quality.
Judgment
What is common is the condition for comparing; what is configurable, the condition for not slowing down.
Progressive layers or a single step
When yes
Layers, when there are several sources, quality rules to apply and different consumers that must read the same thing.
When no
A direct step, when the source already arrives clean, a single question consumes it and another layer would only add copies and latency.
Why
State of the field
The Azure Databricks documentation presents the medallion architecture —bronze, silver and gold— as a recommended best practice, but not a requirement; Microsoft calls it the recommended design approach for Fabric.^[4]^[3]
Judgment
Layers are not dogma: each one has to add value without losing traceability, and the one that adds nothing is surplus.
Lakehouse, warehouse or both
When yes
Both, when there is a functional reason: the lakehouse to keep and transform highly detailed events; the warehouse for relationships, metrics and predictable queries.
When no
Both just in case, without a rule on where each transformation happens or which copy is the reference.
Why
Evidence
At Vesting the lakehouse kept the agents’ events —session, request, response, time, state, cost— and the warehouse organized what the product and operations dashboards needed.
State of the field
In OneLake the Fabric engines —T-SQL, Spark, Analysis Services— store tables in Delta Parquet, so the same data is read by another engine without copying it.^[1]
Isolating by client: a workspace of its own or a filter in the model
When yes
One workspace per client, when the access boundary must match the client’s boundary and a change for one cannot touch another.
When no
Only a row-level security filter, when all the data has a single owner and a badly written filter is an acceptable risk.
Why
Evidence
At Vesting isolation between the 12 clients was solved with separate workspaces on top of the common event structures: the boundary did not depend on a well-written filter.
Evidence
Row-level security stayed in the model, defined as DAX filters on the dimensions and tested by impersonating roles before publishing.
Copying the data or reading it where it lives
When yes
Reading it where it lives, with a shortcut or mirroring, when the source is already in an open format or in another workspace and freshness matters.
When no
Copying it, when the source needs a query-optimized layout or cannot meet what is required of it without a copy.
Why
State of the field
Microsoft recommends shortcuts to include data without copying it and proposes mirroring a source once so the other workspaces consume it through shortcuts, without creating another copy.^[2]
Judgment
Every copy needs a written reason: the copy without one is the one that will show a different figure tomorrow.
Direct Lake, import or DirectQuery
When yes
Direct Lake for what must be seen right away on Delta tables; import for the aggregated history; DirectQuery for the detail that does not fit in memory and is rarely queried.
When no
A single mode for the whole model, when there are questions with different freshness, volume and cost.
Why
Evidence
At Vesting a single model combined the three modes: Direct Lake for operational monitoring, import for the history and DirectQuery to drill down to a session’s events.
State of the field
Microsoft separates Direct Lake on OneLake, which does not fall back to DirectQuery and is the recommended option for new models, from Direct Lake on the SQL analytics endpoint, which can fall back to DirectQuery and become slower.^[8]

Block 5

What is new in 2026

State of the field
A single copy in OneLake, in Delta or in Iceberg
What changes from common practice
OneLake virtualizes metadata across open formats: an Iceberg table is read as Delta in Fabric and a Delta table is read by Iceberg readers, with no manual conversion; when a conversion fails or drops a column, it is written to a log.^[1]^[13]
Evidence
At Vesting the tables lived in Delta on OneLake and the model read them from there; Apache Iceberg left the author’s list of tools for lack of use.
State of the field
Shortcuts and mirroring: reading data where it lives
What changes from common practice
Shortcuts make data from another workspace, another cloud or another lake show up in OneLake without copying it; mirroring replicates an operational database as Delta tables or synchronizes only an external catalog, and Microsoft proposes mirroring once and consuming everywhere.^[1]^[2]
Judgment
Before copying, it is worth asking whether a shortcut is enough: fewer copies are fewer figures that can contradict each other.
State of the field
Declarative layers with materialized lake views
What changes from common practice
Materialized lake views, defined in Spark SQL or —in preview, and with full refresh only— in PySpark, express the bronze-to-silver-to-gold transformations as declarative statements: Fabric tracks their dependencies, refreshes them in order and enforces quality rules at every stage; Microsoft warns they do not fit non-SQL logic or streams that need sub-second updates.^[5]
Evidence
The quality rules live inside the pipelines: missing values, unexpected types, duplicates and relationships that do not close show up before they reach the model.
State of the field
Fabric CI/CD: Git, deployment pipelines and per-stage variables
What changes from common practice
Fabric CI/CD brings together Git integration, deployment pipelines across development, test and production, a variable library with values per stage and a deployment plan that sets the order in which items deploy.^[6]
Evidence
The semantic model is versioned as a project —with Tabular Editor or in PBIP format, in a Git repository— to compare a change and go back to the previous version without rebuilding anything by hand.
State of the field
The Fabric data agent on top of the governed model
What changes from common practice
A Fabric data agent answers in natural language over lakehouses, warehouses, semantic models and KQL databases: it only generates read queries, with the credentials and permissions of whoever asks, and its answers are capped at 25 rows and 25 columns.^[10]
Evidence
An agent that reads a governed measure inherits its definition; one that reads a raw table inherits the obligation to interpret it, and that obligation has to be written down somewhere.
State of the field
Fabric and Databricks: a bridge between catalogs
What changes from common practice
Many organizations register their data in Unity Catalog, within Azure Databricks; Fabric can mirror that catalog without moving or replicating the data —it only copies the structure and reads the data through shortcuts—, so it reads the same tables without copying them, with changes that can take from seconds to minutes to show up.^[11]
Evidence
On the author’s profile, Databricks and Snowflake are listed as prototype; Microsoft Fabric is the platform used end to end, at Vesting.
State of the field
The DP-600 as assessed from October 19, 2026
What changes from common practice
The DP-600 guide asks for configuring version control for a workspace, creating deployment pipelines, analyzing the impact on whatever depends on a lakehouse or a model, and choosing between Direct Lake on OneLake and on the SQL analytics endpoint.^[12]
Evidence
The DP-600 was earned in December 2024, within the author’s role at Vesting, while the platform the syllabus describes was being built.

Block 6

Failure modes and how to anticipate them

What breaks
Every integration brings its own table
How it shows up
A new client or agent forces rewriting pipelines and metrics that already existed.
Control that catches it
An event contract and early normalization: whatever already has a name is not modeled again.
What breaks
Two figures for the same indicator
How it shows up
Two people arrive at the meeting with different results for the same metric.
Control that catches it
A definition agreed with its owner, turned into a measure of the governed model.
What breaks
A copy without an owner becomes someone’s truth
How it shows up
Several copies of the same data and nobody knows which one is the reference.
Control that catches it
One reference per piece of data, a written reason for every copy and shortcuts before copies.
What breaks
One client sees another client’s data
How it shows up
A badly written filter or a model shared between audiences that should not cross.
Control that catches it
One workspace per client and row-level security tested by impersonating roles before publishing.
What breaks
A published change moves a figure silently
How it shows up
A deployment that finishes fine and still produces a different total.
Control that catches it
Reconciling the key figure against the current model and the source; an unexplained difference stops publishing.
What breaks
Capacity runs out and the dashboard gets slow
How it shows up
Interactive operations delayed or rejected; consumption rising without more users.
Control that catches it
Stepping in at the layer where the waste starts before buying more capacity, and watching the shared capacity.
What breaks
An agent reinterprets a raw table
How it shows up
Answers with figures that do not match the official measures.
Control that catches it
The agent queries the governed measure; the raw table only with its written interpretation.
What breaks
A retired asset is still in use
How it shows up
Reports and models nobody maintains but someone keeps opening.
Control that catches it
Lineage to know its consumers, notice of the change and an explicit retirement.

Block 7

Metrics and controls

What is measured
Events that meet the contract at intake
Threshold
All of them; one that fails shows up as an exception and does not move on silently.
Who approves the step to production
Whoever designs the platform.
What is measured
Key figures reconciled before publishing
Threshold
New model, current model and source match; an unexplained difference stops publishing.
Who approves the step to production
Whoever answers for the asset.
What is measured
Critical measures with one definition and one owner
Threshold
All of those used by more than one report or by an agent.
Who approves the step to production
The business owner.
What is measured
Copies of the same data
Threshold
One reference per piece of data; every extra copy with its written reason.
Who approves the step to production
Whoever designs the platform.
What is measured
Assets with known consumers and owner
Threshold
None stays current without an owner or consumers.
Who approves the step to production
Whoever answers for the platform.
What is measured
Use of the shared capacity
Threshold
Below the limit on average, with a margin before throttling starts.
Who approves the step to production
Whoever pays for and manages the capacity.
What is measured
Access across clients
Threshold
None: each client in its own workspace and row-level security tested before publishing.
Who approves the step to production
Whoever answers for the data.

The thresholds are examples of the method, not figures from a specific project.

Block 8

Evidence from projects

EvidenceLevel: Real use
Vesting: the design of its data platform
Between August 2023 and January 2025 the data platform of an agent startup was designed and implemented at Vesting: 12 clients integrated, 27 agents in the inventory, 1,000 events per day, 120 tables and 20 GB at the close, with real-time monitoring of up to 23 agents at once.
See where
EvidenceLevel: Real use
Early normalization as an architecture decision
A common representation of the events from intake onward made metrics comparable across the 27 agents: the lakehouse kept the raw event and the following layers turned it into session, request, response, agent and client.
EvidenceLevel: Real use
Lakehouse and warehouse on OneLake, with combined modes
The lakehouse kept the events and allowed transforming them with Spark, the warehouse served in T-SQL what the dashboards needed, and the models combined Direct Lake, import and DirectQuery according to freshness, volume and cost.
EvidenceLevel: Real use
Governance by design: one workspace per client and row-level security
Each client with its own workspace, permissions and models on top of the common event structures; row-level security defined with DAX filters and tested by impersonating roles before publishing.
EvidenceLevel: Real use
Reconciling and versioning semantic models
In the author’s Power BI models, the total of a key measure in the new model must match the current model’s and the source’s —three figures, one single truth—, and the model is versioned as a project in Git to go back without rebuilding anything.
EvidenceLevel: Real use
Keeping development and production apart at Fundación CTIC
At Fundación CTIC, with more than 25 analytics products in use, keeping a solution’s states apart is what lets a model be corrected without some 75 users seeing a half-finished figure.
EvidenceLevel: Real use
Banco Pichincha: preparing in separate steps
In 2023, reorganizing the ETL processes into separate steps for connecting, preparing and building the model’s tables cut analysis times by 35%.

Block 9

Limits and scope

Evidence
The author knows Eventhouse and KQL at the query level, like SQL —the DP-600 requires working with SQL, KQL and DAX—, not from a project.
Evidence
On the author’s profile, Databricks and Snowflake are listed as prototype; Airflow and PySpark, in study; on September 27, 2026 dbt and Apache Iceberg left the list of tools for lack of use.
Evidence
This publication does not describe experience with AWS or Redshift: the end-to-end data platform in the evidence is Microsoft Fabric, at Vesting.
Evidence
The public Fabric piece with Colombian open data is still in exploration: it is not presented as built.
Judgment
What should be done differently today: start new models with Direct Lake on OneLake and leave DirectQuery only for the detail that does not fit in memory.
Judgment
Between Fabric and Databricks the choice should not follow the engine but where the organization’s governance already lives and who runs it; with the catalog mirrored, the question stops being one or the other.

Block 10

Sources and review date

Reviewed October 18, 2026

[1]
What is OneLake?
Microsoft Learn · March 9, 2026
https://learn.microsoft.com/en-us/fabric/onelake/onelake-overview
[2]
Unify data with OneLake shortcuts and mirroring
Microsoft Learn · July 15, 2026
https://learn.microsoft.com/en-us/fabric/onelake/unify-data
[3]
Understand medallion architecture for Fabric with OneLake
Microsoft Learn · February 12, 2026
https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture
[4]
What is the medallion lakehouse architecture? (Azure Databricks)
Microsoft Learn · September 11, 2026
https://learn.microsoft.com/en-us/azure/databricks/lakehouse/medallion
[5]
What are materialized lake views in Microsoft Fabric?
Microsoft Learn · September 30, 2026
https://learn.microsoft.com/en-us/fabric/data-engineering/materialized-lake-views/overview-materialized-lake-view
[6]
What is CI/CD in Microsoft Fabric?
Microsoft Learn · August 23, 2026
https://learn.microsoft.com/en-us/fabric/cicd/cicd-overview
[7]
CI/CD for pipelines in Data Factory in Microsoft Fabric
Microsoft Learn · June 10, 2025
https://learn.microsoft.com/en-us/fabric/data-factory/cicd-pipelines
[8]
How Direct Lake works
Microsoft Learn · September 22, 2025
https://learn.microsoft.com/en-us/fabric/fundamentals/direct-lake-how-it-works
[9]
Understand the Fabric capacity throttling policy
Microsoft Learn · August 14, 2026
https://learn.microsoft.com/en-us/fabric/enterprise/throttling
[10]
Fabric data agent concepts
Microsoft Learn · May 11, 2026
https://learn.microsoft.com/en-us/fabric/data-science/concept-data-agent
[11]
Mirroring Azure Databricks Unity Catalog
Microsoft Learn · October 24, 2025
https://learn.microsoft.com/en-us/fabric/mirroring/azure-databricks
[12]
Study guide for Exam DP-600: Implementing Analytics Solutions Using Microsoft Fabric
Microsoft Learn · August 16, 2026
https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/dp-600
[13]
Use Iceberg tables with OneLake
Microsoft Learn · July 30, 2026
https://learn.microsoft.com/en-us/fabric/onelake/onelake-iceberg-tables

Summary in three lines

From the source to whoever decides, and from the change to production

Legend of shapes and colors

Reading the map, node by node

Connections

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

What is common and what is configurable in Vesting’s data platform

Why

Progressive layers or a single step

Why

Lakehouse, warehouse or both

Why

Isolating by client: a workspace of its own or a filter in the model

Why

Copying the data or reading it where it lives

Why

Direct Lake, import or DirectQuery

Why

A single copy in OneLake, in Delta or in Iceberg

Shortcuts and mirroring: reading data where it lives

Declarative layers with materialized lake views

Fabric CI/CD: Git, deployment pipelines and per-stage variables

The Fabric data agent on top of the governed model

Fabric and Databricks: a bridge between catalogs

The DP-600 as assessed from October 19, 2026

Vesting: the design of its data platform

Early normalization as an architecture decision

Lakehouse and warehouse on OneLake, with combined modes

Governance by design: one workspace per client and row-level security

Reconciling and versioning semantic models

Keeping development and production apart at Fundación CTIC

Banco Pichincha: preparing in separate steps