Methods · Topic T8

How decisions should be made with models

Reviewed October 21, 2026

It prints in full from the browser, on A4 or Letter.

Block 1

The problem

Making decisions with models without fooling oneself: every model has to beat a baseline, be validated the way operations will face it, reach the decision maker with its limits and be monitored after delivery, and optimization has to be formulated so that its result can be verified.

Summary in three lines

The problem starts from the process: what is anticipated, on which unit, how far ahead and which decision changes with the result; and before the algorithm, the baseline the model has to beat.
The model is validated the way operations will face it —temporal order, a leak-free test, error by segment— and reaches the decision maker with its card and its limits, as a signal and not a decision.
When the question is what to choose and not what will happen, the problem is formulated as optimization, twice, and simulated under variability; and every published model is monitored, because drift gives no warning.

Block 2

The map

From the process problem to a monitored decision

Swipe to see the whole map.

Legend of shapes and colors

Actor — who does the work
Data — what is stored and read
Control — a rule that gets checked
Person — a person decides
External — what comes from outside

Reading the map, node by node

The decision owner defines the problem and its horizon; the baseline is set first, and then the variables are built from what exists in time. The candidate model is validated in temporal order and measured on a leak-free test; its errors are read by segment and written into the model card, and the prediction is delivered where the decision is made. The decision maker receives a signal; when the question is what to choose, optimization comes in, and simulation adds scenarios to what is delivered. In operation, drift monitoring alerts the decision owner.

Node 1 Decision owner (Person) — Whoever answers for the decision the model has to improve and sets which error costs more.
Node 2 Problem and horizon (Control) — What is anticipated, on which unit, how far ahead and which decision can change with the result.
Node 3 Baseline (Control) — A simple rule, a historical average or the current way of deciding: what the model has to beat.
Node 4 Data known in time (Data) — Only what exists at prediction time: nothing from the future leaked into training.
Node 5 Candidate model (Actor) — Models compete with the same preprocessing and are chosen by their error, not by their novelty.
Node 6 Temporal order (Control) — Training uses the past and measurement uses later periods, the way operations will face it.
Node 7 Leak-free test (Control) — The metrics come from data training never saw, with preprocessing fitted on the training data only.
Node 8 Errors per segment (Data) — The confusion matrix, the error by route, class or group and what each kind of error costs.
Node 9 The model card (Data) — Data, split, method, metrics, verdict against the baseline and limits, written down.
Node 10 Delivery at the decision (Actor) — The prediction arrives with its context in the dashboard or report already in use, before the decision closes.
Node 11 Decision maker (Person) — Receives a signal, not a decision: prioritizes, reviews and answers for the action.
Node 12 Monitoring and drift (Control) — Inputs, distribution, performance, use and impact, monitored after publishing and not only before.
Node 13 Optimization (Actor) — The same problem as integer programming and as constraint programming: the optima must match.
Node 14 Simulation (Actor) — Compares policies and assignments under variability before intervening in operations.

Connections

from Decision owner to Problem and horizon (defines)
from Problem and horizon to Baseline (first)
from Baseline to Data known in time (then)
from Data known in time to Candidate model (trains)
from Candidate model to Temporal order (is validated)
from Temporal order to Leak-free test (is measured)
from Leak-free test to Errors per segment (shows)
from Errors per segment to The model card (is written)
from The model card to Delivery at the decision (is delivered)
from Delivery at the decision to Decision maker (informs)
from Decision maker to Monitoring and drift (in operation)
from Monitoring and drift to Decision owner (alerts)
from Decision maker to Optimization (to choose)
from Optimization to Simulation (under variability)
from Simulation to Delivery at the decision (adds scenarios)

Block 3

The phases

1
Phase 1
Formulate the problem from the decision
Inputs
- The decision to be improved and whoever answers for it.
- The process, its events and the current way of deciding.
Activities
- Define the phenomenon, the unit of analysis and the prediction horizon against the decision horizon.
- Set the baseline and the cost of each kind of error before training.
- Decide whether the question is to anticipate or to choose: prediction or optimization.
Artifacts
- The problem definition: phenomenon, unit, horizon and decision.
- The written baseline.
Who decides
The decision owner.
Control that closes the phase
No model is trained without a baseline and without a decision its result can change.
Evidence
At TransMilenio the unit was the route in a time band of a type of day, with a monthly horizon: the next month's schedule is decided before that month starts, and a prediction arriving later was of no use.
2
Phase 2
Build the variables from what exists in time
Inputs
- The events, states and sources of the process.
- The exact moment each prediction will be produced.
Activities
- Build every variable with a meaning, a source, a time window and its relation to the unit.
- Discard whatever only appears after the event to be anticipated.
- Use the same transformation at inference as in training.
Artifacts
- The set of variables with their source and window.
- The versioned transformation rules.
Who decides
Whoever builds the model, with whoever knows the process.
Control that closes the phase
No variable gets in if it cannot be computed before the target event.
Evidence
At Banco Pichincha every variable had to be computable with the information available before the event: a delinquency model that uses information from after the default is perfect in the lab and useless in operation.
3
Phase 3
Validate the way operations will face it
Inputs
- The candidate models and the baseline.
- Periods later than those the model saw.
Activities
- Measure on later periods, never with a random split in a temporal problem.
- Read the confusion matrix and the error by segment, not only the overall metric.
- Check calibration when the output is used as a probability.
Artifacts
- The verdict against the baseline.
- The map of errors by segment.
Who decides
The decision owner, with whoever builds the model.
Control that closes the phase
A model that does not beat the baseline does not pass, and the verdict says so.
Evidence
At TransMilenio a good average could hide errors in the critical segments: getting most low-demand routes right and missing the peak-hour trunk lines is a good average and a bad prediction.
4
Phase 4
Deliver where the decision is made, with the model card
Inputs
- The approved model, its card and its limits.
- The dashboard, report or flow the decision maker already uses.
Activities
- Present the prediction with the context that allows it to be interpreted.
- Keep the date, version, data and population of every prediction.
- Define what happens when the input data do not meet expectations.
Artifacts
- The prediction in the product already in use.
- The record that allows every prediction to be reconstructed.
Who decides
The decision maker, with the owner of the analytics product.
Control that closes the phase
No prediction arrives without context or after the decision closes.
Evidence
At Banco Pichincha the predictions arrived through Power BI, together with the customer's historical and contextual information, in the same dashboards that more than 50 business users had adopted: the delivery channel was part of the model's design.
5
Phase 5
Monitor the model and decide what to do with what shows up
Inputs
- The inputs, the predictions and the actual outcome when it arrives.
- The use the people in charge make of the prediction.
Activities
- Monitor five levels: input quality, distribution, observed performance, use and impact.
- Compare the prediction with what actually happened, segment by segment.
- Decide for each deviation: investigate the source, recalibrate, retrain, restrict or roll back to an earlier version.
Artifacts
- The model monitoring dashboard.
- The record of every decision taken about the model.
Who decides
The model owner, with the decision owner.
Control that closes the phase
No alert is left without a decided action.
Evidence
At TransMilenio the actual outcome arrived with the month's fare collection: the prediction by route and time band was checked against what actually happened.

Block 4

The design decisions

The baseline before the algorithm
When yes
Always: before training, what the model has to beat is written down —a simple rule, a historical average or the current way of deciding— and the verdict can say «does not beat it».
When no
When nobody can say which decision would change with the result: then the problem is not yet a predictive one.
Why
Evidence
At Banco Pichincha the baseline was the prediction that already existed, and the model improved on it by up to 35%; in Probeta DS the baseline is in plain sight by design —majority class and logistic regression—.
Judgment
The sophistication of the algorithm is worth nothing on its own: what counts is how much it improves on the way decisions are already made.
Validate in temporal order, never with a random split
When yes
When the phenomenon has a temporal dimension —demand, churn, delinquency—: training uses the past and measurement uses later periods.
When no
When the observations depend neither on time nor on the order in which they arrive: a stratified random split is enough.
Why
Evidence
At TransMilenio the demand model was trained on the previous months and measured with RMSE on the following ones, never with a random split that let the future show up inside training.
Judgment
An accuracy that operations will not repeat is worse than a modest one: it promises what nobody will get.
Evaluate by the error that costs, not by accuracy
When yes
When the classes are imbalanced or a false positive and a false negative do not cost the same: the confusion matrix and the threshold are read against that cost.
When no
When the two errors cost the same and the classes are balanced: the overall metric works as a first reading.
Why
Evidence
At Banco Pichincha calling a customer who was not going to leave cost a call, and not calling the one who was leaving cost the customer: that is why the analysis looked at where the errors piled up and not only how many there were.
How an optimization problem is formulated: twice, and the optima have to match
When yes
When the decision is choosing a combination of resources under constraints —assigning, scheduling, balancing—: the problem is formulated as integer programming and as constraint programming, and a difference between their optima exposes a faulty translation.
When no
When the question is what will happen and not what is worth choosing: a predictive model with its baseline is enough there.
Why
Evidence
In the author's Computational Paper every problem is formulated twice —mixed-integer programming with Pyomo and HiGHS, and constraint programming with OR-Tools— and their optima must match or the model was translated wrong; giving up one of the two formulations requires a written justification.
State of the field
Systems that use an LLM to translate a natural-language statement into an optimization model have mostly been evaluated on toy or synthetic problems; with 223 real MIPLIB 2017 problems reconstructed, the ones that do well on those tests degrade substantially.^[10]
Simulate before intervening when variability decides
When yes
When the result depends on queues, competing resources and times that vary: discrete-event simulation compares scenarios before the operation is touched.
When no
When the average is enough for the decision and changing it does not alter the dynamics of the system.
Why
Evidence
At Cafam, in 2021, the FlexSim model of medicine dispatch with the logic of the new WMS showed where a reasonable rule in one activity produced waiting in another: the queue was not where the average said it would be.
Judgment
A simulation does not predict what will happen: it compares scenarios under written assumptions, and it is worth what those assumptions are worth.
A signal for the decision maker, not an automatic decision
When yes
When the action requires additional knowledge, judgment or accountability: the model prioritizes or ranks, and the authorized person decides.
When no
When the action is verifiable, reversible and low-cost, and reviewing each case would turn the person into the bottleneck.
Why
Evidence
In Banco Pichincha's churn and delinquency models the model produced evidence to support a decision, not an automatic decision replacing the judgment of the people in charge.
State of the field
In Microsoft Fabric the predictions are written to OneLake and Power BI reports read them in Direct Lake mode, directly and without copying them into the semantic model.^[1]
Monitor after publishing: drift gives no warning
When yes
Whenever a model goes into operation: inputs, distribution and performance are monitored against those of training, with thresholds and a defined response for each alert.
When no
When the model answers a single question that is not repeated: writing down its limits and not using it again is enough.
Why
State of the field
Changes in data distribution, training-serving skew, data quality problems, shifts in the environment and changes in consumer behavior can all make a model stale, and its performance can degrade to the point that it fails to add business value.^[5]
Evidence
At Vesting, between August 2023 and January 2025, real-time monitoring of up to 23 agents collected inputs, outputs, times, statuses and cost per session, and took them to Microsoft Fabric and Power BI, where a degradation showed up as a trend and not as a complaint.

Block 5

What is new in 2026

State of the field
The experiment and the model are recorded automatically
What changes from common practice
In Microsoft Fabric data science, autologging with MLflow captures each run's parameters, metrics and items during training, and with MLflow 3 every logged model stays linked to the run that produced it, with its parameters, its metrics and the datasets it was trained on.^[2]^[3]
Judgment
Autologging keeps what happened; the baseline and the acceptance criterion are still written before the first run.
State of the field
The prediction reaches the dashboard without copies
What changes from common practice
Predictions written to OneLake are read from Power BI in Direct Lake mode, without copying them into the semantic model: the prediction stays up to date without redundant data movement.^[1]
State of the field
Drift as a signal with a threshold
What changes from common practice
Azure Machine Learning offers monitoring signals for tabular data —data drift, prediction drift, data quality, feature attribution drift and model performance—, and Microsoft's study guide for the exam on operationalizing machine learning and generative AI solutions includes detecting and analyzing data drift and configuring retraining or alert triggers when thresholds are exceeded.^[5]^[6]
Judgment
A drift signal opens a review; retraining because a distribution changed can erase exactly what needed investigating.
State of the field
Foundation models for time series
What changes from common practice
GIFT-Eval compares statistical, deep learning and foundation models across 23 datasets with more than 144,000 time series, and a 2026 paper argues that a standard patch Transformer, trained with a straightforward protocol, reaches state-of-the-art zero-shot forecasting, without training on the series it forecasts, with an analysis of what drives it: scale, data composition and training techniques.^[7]^[8]
Judgment
A foundation model comes in as one more candidate: it is validated in temporal order and against the same baseline as any other.
State of the field
The LLM as a formulation assistant
What changes from common practice
OptiMUS-0.3 uses an LLM to turn the natural-language description of a linear or integer programming problem into a mathematical model and solver code, and then evaluates and corrects what it generated; its authors find that the system's architecture drives the result more than the model's capability.^[9]
Judgment
A formulation proposed by an LLM is verified like any other: with a second, independent formulation and optima that match.
State of the field
MLOps in levels, not as a leap
What changes from common practice
Microsoft's MLOps maturity model describes five levels, from work without MLOps to automated operations in which drift or regression signals trigger retraining, and notes that an organization often shows characteristics of more than one level at the same time.^[4]

Block 6

Failure modes and how to anticipate them

What breaks
Information from the future in training
How it shows up
A near-perfect metric in the lab that operations do not repeat.
Control that catches it
Variables only from what is available at prediction time, preprocessing fitted on training only and a too-good metric treated as suspicious.
What breaks
A random split in a temporal problem
How it shows up
The error in operation is much larger than in validation.
Control that catches it
Validation on periods later than those the model saw.
What breaks
High accuracy on an imbalanced class
How it shows up
The model gets the majority right and fails exactly on the cases that justified building it.
Control that catches it
Confusion matrix, per-class metrics and a threshold set by the cost of each error.
What breaks
A model with no baseline
How it shows up
Nobody can say whether the model improves on the current way of deciding.
Control that catches it
The baseline is set before training and the verdict can say «does not beat it».
What breaks
The average that hides the segment
How it shows up
An acceptable error in aggregate and a systematic one on the routes, classes or customers that matter.
Control that catches it
Error by segment wherever the prediction will be used.
What breaks
The prediction that arrives late
How it shows up
The estimate shows up after the decision has closed.
Control that catches it
The prediction horizon is aligned with the decision horizon.
What breaks
Unmonitored drift
How it shows up
The model degrades and nobody notices until someone complains.
Control that catches it
Inputs, distribution, performance, use and impact monitored in operation, with thresholds and a defined response.
What breaks
A badly translated formulation
How it shows up
The solver returns an optimum that does not respect a real constraint.
Control that catches it
Two independent formulations whose optima must match.

Block 7

Metrics and controls

What is measured
Improvement over the baseline
Threshold
Beats it on the test; if it does not, the verdict says so and the model does not pass.
Who approves the step to production
The decision owner.
What is measured
Error in the segments that will be used
Threshold
Within the agreed margin in every segment, not only in aggregate.
Who approves the step to production
The decision owner, with whoever builds the model.
What is measured
Calibration of the probabilities
Threshold
The announced probability matches the observed frequency in the groups that get prioritized.
Who approves the step to production
Whoever allocates the capacity to intervene.
What is measured
Timeliness of the prediction
Threshold
Always before the decision closes.
Who approves the step to production
The decision owner.
What is measured
Variables available at prediction time
Threshold
All of them, checked before training.
Who approves the step to production
Whoever builds the model.
What is measured
Drift alerts handled
Threshold
All of them, each with a decided action.
Who approves the step to production
The model owner.
What is measured
Agreement of the two formulations
Threshold
Equal optima on every instance; if they differ, the model was translated wrong.
Who approves the step to production
Whoever formulates the problem.

The thresholds are examples of the method, not figures from a specific project.

Block 8

Evidence from projects

EvidenceLevel: Real use
TransMilenio: demand by route and time band, still in use after the departure
At C&M Consultores, between July 2021 and May 2022, a scikit-learn model predicted the system's demand by route, type of day and time band with a monthly horizon; it was validated with RMSE respecting temporal order, went into the demand report the units scheduled with, stayed in use after the author left and contributed to a reported 20% improvement in system performance.
See where
EvidenceLevel: Real use
Banco Pichincha: churn and delinquency, with the person deciding
At Banco Pichincha, in 2023, scikit-learn models for customer churn and delinquency went into production with reported results above 90% accuracy and up to a 35% improvement over the existing predictions; the evaluation looked at where the errors piled up, and the predictions reached the decision makers through Power BI.
See where
EvidenceLevel: Prototype
Probeta DS: what it guarantees by construction
Probeta DS, one of the author's applications, runs Python, Pandas and scikit-learn in the browser with Pyodide: two models compete against two baselines in plain sight —the majority class and a logistic regression— and the verdict says «does not beat it» when it does not; the five metrics are computed on the test set, with the confusion matrix beside them; leakage is impossible because preprocessing is fitted on the training half only, and a downloadable model card keeps data, split, method, metrics, verdict and limits.
See where
EvidenceLevel: Real use
Optimization formulated twice
In the driver-and-bus assignment paper, produced with the Computational Paper, the two independent formulations —integer programming and constraint programming— gave 14,251 identical optima in 14,251 comparisons, on the real route topology from the system's open data and with a declared synthetic generator.
See where
EvidenceLevel: Real use
FlexSim at Inglopres and Cafam: simulate before intervening
At Inglopres, in 2016, discrete-event simulation with FlexSim compared alternatives for the machinery rental operation before changing it; at Cafam, in 2021, the model of medicine dispatch with the logic of the new WMS was validated against the operation the testing team knew firsthand: if the model queued where the warehouse did not, the error was in the model.
See where
EvidenceLevel: Real use
Frozen criteria and published negative results
In the author's research the thresholds that decide whether a thesis holds are frozen before a single result exists, and whatever comes out is published: in the pilot paper, two of the three principles evaluated were refuted by their own metrics, and that is how they were written up.
See where
EvidenceLevel: Prototype
Bus bunching: the policy, against the best classic control
In the research on bus bunching, the proposed policy is tested in a discrete-event simulation with ground truth known by construction and compared against the best classic control, never against doing nothing; the experiences measured come from a simulated corridor, and its technical sheet claims no figures for the real system.
See where

Block 9

Limits and scope

Evidence
There is no case of observed drift of the author's own that can be told with its figures: what exists is the infrastructure to see it, built at Vesting.
Evidence
The prescriptive layer was not built in a job: at Vesting the third layer, scenario simulation on top of the event base, was missing, and the driver–vehicle–route assignment was spotted in TransMilenio's data, but its formulation and its answer came later, in a research line.
Evidence
The magnitudes of the assignment and bunching work come from synthetic instances and a simulated corridor: the mechanism is demonstrated; real figures require an operator's data.
Evidence
PyTorch and TensorFlow are exploration: neural networks and deep learning, with no case in production; the models in production are scikit-learn ones.
Evidence
Of MLOps, the half of taking models to production and sustaining them has been done, not the half of packaging and orchestrating with containers; the AI-300 path, in progress, replaced the retired DP-100 and is not presented as earned.
Judgment
MLflow, time-series foundation models and LLM formulation assistants come in here as state of the field: the experience this piece claims is that of block 8, at the level the corpus gives it.
Judgment
What is worth doing differently today: write down the baseline and the acceptance threshold before opening the first notebook, not once the model already looks promising.

Block 10

Sources and review date

Reviewed October 21, 2026

[1]
What is Data Science in Microsoft Fabric?
Microsoft Learn · December 3, 2025
https://learn.microsoft.com/en-us/fabric/data-science/data-science-overview
[2]
Autologging in Data Science
Microsoft Learn · May 15, 2026
https://learn.microsoft.com/en-us/fabric/data-science/mlflow-autologging
[3]
MLflow 3 in Fabric Data Science
Microsoft Learn · May 15, 2026
https://learn.microsoft.com/en-us/fabric/data-science/mlflow-3-overview
[4]
MLOps maturity model
Microsoft Learn (Azure Architecture Center) · June 3, 2026
https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model
[5]
Azure Machine Learning model monitoring
Microsoft Learn · January 27, 2026
https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring
[6]
Study guide for Exam AI-300: Operationalizing Machine Learning and Generative AI Solutions
Microsoft Learn · February 18, 2026
https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/ai-300
[7]
GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation
arXiv · November 11, 2024
https://arxiv.org/abs/2410.10393
[8]
Revisiting the Generic Transformer: Deconstructing a Strong Baseline for Time Series Foundation Models
arXiv · February 6, 2026
https://arxiv.org/abs/2602.06909
[9]
OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale
arXiv · June 26, 2026
https://arxiv.org/abs/2407.19633
[10]
Constructing Industrial-Scale Optimization Modeling Benchmark
arXiv · May 26, 2026
https://arxiv.org/abs/2602.10450

Summary in three lines

From the process problem to a monitored decision

Legend of shapes and colors

Reading the map, node by node

Connections

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

Inputs

Activities

Artifacts

The baseline before the algorithm

Why

Validate in temporal order, never with a random split

Why

Evaluate by the error that costs, not by accuracy

Why

How an optimization problem is formulated: twice, and the optima have to match

Why

Simulate before intervening when variability decides

Why

A signal for the decision maker, not an automatic decision

Why

Monitor after publishing: drift gives no warning

Why

The experiment and the model are recorded automatically

The prediction reaches the dashboard without copies

Drift as a signal with a threshold

Foundation models for time series

The LLM as a formulation assistant

MLOps in levels, not as a leap

TransMilenio: demand by route and time band, still in use after the departure

Banco Pichincha: churn and delinquency, with the person deciding

Probeta DS: what it guarantees by construction

Optimization formulated twice

FlexSim at Inglopres and Cafam: simulate before intervening

Frozen criteria and published negative results

Bus bunching: the policy, against the best classic control