Simulation Is Not Evidence: Rehearsing Consequential Decisions

A simulation can make assumptions executable. It cannot make them true.

That distinction matters most when the decision is consequential: changing an approval policy, automating a handoff, reallocating limited review capacity, or releasing an agent into a workflow where errors propagate. A convincing animation is easy to mistake for knowledge. A pile of generated personas is easy to mistake for research. A precise metric is easy to mistake for a measured outcome.

I use decision rehearsal for a narrower practice: encode a proposed world, run explicit policies inside it, preserve the trace, and compare what those assumptions produce. The result is a synthetic hypothesis with an audit trail. It is useful because it exposes the model. It remains synthetic because the world being executed is still one we specified.

The minimum simulation contract#

A dashboard is not automatically a simulation. Neither is a forecast, an agent conversation, or a Monte Carlo chart. A decision rehearsal needs at least these parts:

• Agents with roles, goals, local information, and bounded abilities.
• Legal actions that say what each agent may do in each state.
• Explicit world state that owns queues, resources, permissions, evidence, and process status.
• Time that advances and changes what actions cost or become possible.
• Constraints such as capacity, deadlines, authorization, money, inventory, or attention.
• Branching that holds a shared history constant before applying a different policy.
• Traces that record state-changing events in order, with the seed and scenario version needed for replay.
• Metrics derived from completed traces rather than written into the story after the fact.

If the actors can say anything, take any action, and teleport between states, the system is closer to improvisation than simulation. If the chart changes when a label changes, the metric is presentation, not evidence. If a counterfactual silently receives a different starting state, it is not a controlled comparison.

The useful discipline is to make every important transition answerable: Who acted? Was the action legal? What state did it read? What state did it change? How much simulated time passed? Which policy authorized the transition? Can the prefix be reproduced exactly?

Five claims that should not collapse into one#

The words around modeling often imply more certainty than the artifact has earned. I separate five evidence states.

Synthetic hypothesis#

A synthetic hypothesis executes invented agents, inputs, rules, or distributions. It can expose contradictions, queue effects, unsafe permissions, and hidden assumptions. It does not establish how customers, operators, markets, or institutions actually behave.

Calibrated model#

A calibrated model has been compared with relevant observed data, and its parameters or outputs have been adjusted within a documented validation regime. Calibration is always scoped: to a population, time window, measurement process, and set of conditions. A calibrated model can still fail under distribution shift or outside its validation envelope.

Forecast#

A forecast predicts an observed quantity at a future time and can be scored after that time arrives. It needs a forecast horizon, an uncertainty statement, and a record of prior predictions. A decision rehearsal can generate possible futures without becoming a forecast.

Digital twin#

A digital twin implies an ongoing relationship with a specific physical or operational counterpart. It needs current observations, identity and synchronization rules, model maintenance, and a defined correspondence between the twin and the thing it represents. A generalized workflow model is not a digital twin merely because its interface looks realistic.

Operational evidence#

Operational evidence comes from the real workflow: observed behavior, accepted transactions, measured incidents, provider responses, production traces, or other authoritative records. A simulation may help design what to observe. It does not replace the observation.

These classifications can change over time, but only when new evidence changes them. Publishing a simulator changes availability. It does not create calibration or external validation.

Validation is a program, not a badge#

Validation starts before a run.

The scenario needs provenance for every important input: source, date, unit, transformation, rights, and known limitations. Synthetic values must be labeled synthetic. Licensed or confidential inputs need an explicit right to be used for the intended purpose. A plausible number with unknown custody is not a stronger input than an obviously invented number; it is merely harder to audit.

The implementation then needs conformance checks:

• Identical seeds reproduce identical traces.
• Declared random variables are the only source of run-to-run variation.
• Illegal actions fail without mutating state.
• A branch reproduces the original prefix exactly.
• Metrics are recomputed from the event trace.
• Scenario versions and manifest hashes travel with exported results.

Sensitivity testing asks a different question: does a claimed result survive reasonable changes to uncertain inputs? If changing one weakly supported parameter reverses the policy ranking, the output should foreground that dependence. The run did not identify a winner. It identified an assumption that controls the decision.

Calibration and retrospective testing come later, when appropriate observed data and usage rights exist. A good validation report says which behaviors matched, which did not, which periods and populations were used, and what remains outside the model. It should also preserve failed calibrations. Otherwise the final model looks inevitable when it was selected from many discarded alternatives.

Human authority stays outside the model when it must#

A simulation can compare an auto-advance threshold with a review gate. It cannot authorize the real system to auto-advance. It can estimate queue pressure under specified arrivals. It cannot decide which people should bear the operational consequences.

This is especially important when agents participate. An agent in a rehearsal receives only the actions assigned to its role. A human approval remains a modeled event, not a permission for a live tool call. Refusal, escalation, retry, and rollback need first-class states because happy-path completion is not the only behavior worth measuring.

The output should therefore be a decision aid with named assumptions, not a recommendation disguised as inevitability. The accountable human or organization still owns the real intervention.

Rehearsal's current evidence state#

Rehearsal is currently one synthetic private research prototype.

Its existing onboarding scenario includes stateful agents, constrained actions, simulated time, queues, deterministic seeds, append-only traces, branchable histories, and trace-derived metrics. Those are implementation facts supported by private source and local tests. The scenario uses no live customer data. It has no public runtime, no external pilot, no calibration against operational behavior, and no independent validation.

The intended public release will make a bounded demonstration inspectable: two policy runs, their manifests, traces, metrics, and hashes. If that release occurs, the evidence level will move from private implementation evidence to a live public artifact. The output will still be a simulated hypothesis.

That boundary is not modesty language added after the interesting work. It is part of the product contract.

A research backlog, not a product catalog#

The longer research program may explore other domains: evidence-review queues, studio capacity, pricing migrations, tool-permission evaluations, cooperative inventory, physical bench processes, sports decisions, and civic systems. These are candidate scenarios, not announced products.

The next planned scenario is a completely synthetic workflow-authority model. It will test what happens when evidence arrives complete, partial, or conflicting; when review capacity is constrained; and when a structural check is or is not allowed to imply readiness. The value of that scenario is not a prediction about a named organization. It is a reusable test of authority boundaries, queue policy, rework, and unsafe transitions.

One shared simulation kernel and two materially different scenarios would be stronger evidence than a dozen names. A buyer discovery conversation would be stronger evidence of demand than another landing-page card. An authorized pilot would be stronger evidence of utility than a polished synthetic chart.

What a rehearsal may legitimately conclude#

A well-governed simulation can say:

• Under these explicit rules and inputs, policy A produced fewer unsafe transitions than policy B.
• This result depends heavily on reviewer capacity and evidence-arrival assumptions.
• These failure paths were absent from the original happy-path design.
• These events and measurements should be instrumented in a bounded real-world pilot.

It cannot, without further evidence, say:

• Customers will behave this way.
• The intervention will reduce real churn, cost, or risk.
• The model represents the current operational system.
• A simulated approval makes a real action safe or authorized.

The purpose of rehearsal is to make a decision more inspectable before reality makes it expensive. Its credibility comes from preserving the line between the world we encoded and the world we have observed.