Skip to content

ADR-0010: Optimization as a Separate Package Above simulation
Status Accepted
Date 2026-07-05
Deciders Chief Software Architect, MineProductivity
Governs mineproductivity.optimization
Related documents docs/architecture/10_Optimization_Design_Specification.md; docs/design/10_Optimization_Implementation_Checklist.md; docs/architecture/09_Simulation_Design_Specification.md §16, §37; ADR-0009-Simulation.md; docs/architecture/08_Digital_Twin_Design_Specification.md §34; docs/architecture/07_Decision_Intelligence_Design_Specification.md §3.1, §20, §23, §36; docs/architecture/06_Analytics_Engine_Design_Specification.md §33 (not yet merged to main - see feature/analytics-engine)

Context

MineProductivity's Foundation Layer and its first four consumer packages, analytics, decision, digital_twin, and simulation, are complete, tested, and locked as of simulation v1.1.0. Three of those packages' own documents already named what comes next, from three different directions. Decision Intelligence's own Architectural Principles state directly that decision "never searches a solution space for an optimal plan (that is optimization)" (spec 07 §3.1), and its Future Roadmap named optimization as the anticipated consumer of ActionPriority/DecisionScore "as objective-function inputs when searching for an optimal action sequence across many competing Recommendations" (spec 07 §36). Digital Twin's own Future Roadmap named optimization as the anticipated consumer of TwinSnapshot/TwinState "as scenario inputs when searching an optimal action sequence over a twin's represented condition" (spec 08 §34). Simulation's own Future Roadmap went furthest of the three: it named optimization "the most direct consumer of Experiment/ExperimentResult" and positioned it to become "the first concrete implementer of CalibrationModel, since parameter-fitting against historical ground truth is itself a search problem" (spec 09 §37). This ADR formalizes and designs optimization, the package all three already-locked documents assumed would exist, and resolves the search-and-solve layer each of them declined to build for itself.

The immediate question this ADR answers: given already-computed KPIs, statistically-characterized trends, prioritized and scored recommended actions, a current or snapshotted twin state, and a projected future under one or more simulated scenarios, who owns searching a constrained solution space for the single best achievable operational plan - without becoming a sixth computation engine, a second stateful-representation layer, a second projection layer, or a business-action executor? This ADR decides where that search-and-solve layer lives, why it is shaped as an orchestration layer over six pluggable, interface-only solving paradigms rather than a shipped algorithm or a bundled solver library, and why OptimizationRun - mirroring simulation.SimulationRun one layer down - is the third package in this series to break from the stateless "as-object" pattern every package below digital_twin otherwise follows.

Decision

Create mineproductivity.optimization as a new, independent package sitting directly above simulation and directly below the not-yet-designed agents/visualization packages:

core → ontology → events → kpis → analytics → decision → digital_twin → simulation → optimization

optimization is permitted to import core, ontology, events, registry, plugins, connectors, kpis, analytics, decision, digital_twin, and simulation; it introduces its own public API (OptimizationModel and OptimizationContext; Objective/Constraint/DecisionVariable/OptimizationProblem/ProblemStatus as a governed problem-definition object model; OptimizationRun - a core.BaseEntity[str] subclass - and RunStatus; six interface-only ABCs - LinearProgrammingModel, MixedIntegerProgrammingModel, ConstraintProgrammingModel, MultiObjectiveModel, EvolutionaryMetaheuristicModel, NetworkOptimizationModel; PlanComparator/SensitivityAnalyzer; OptimizationRunRepository as a direct type alias over core.BaseRepository[OptimizationRun, str]) without requiring any change to any Foundation Layer file, or to analytics/decision/digital_twin/simulation, or to any of their public APIs or dependency rules. Its own registry (optimization.REGISTRY) is a direct specialization of registry.Registry, exactly as simulation.REGISTRY, digital_twin.REGISTRY, decision.REGISTRY, analytics.REGISTRY, and kpis.REGISTRY already are. Full design detail is in the governing specification linked above; this ADR records why the package exists and why it is shaped the way it is.

Why a Separate Package (Not Folded Into simulation or decision)

  1. Distinct responsibility, distinct question being answered. decision answers "what should happen, given already-known facts." simulation answers "how does this system evolve under a scenario." optimization answers a third, different question neither package was ever chartered to answer: "given known facts, projected futures, and explicit constraints, what is the single best combination of decisions achievable?" A solved plan is evidence a future agents package or a human may act on, not an action itself, and a plan's existence has nothing to do with whether a scenario was ever simulated to produce its inputs.
  2. All three lower specs already drew this boundary, from three directions. Decision spec 07 §3.1 states directly that searching a solution space for an optimal plan is optimization's job, not decision's. Digital Twin spec 08 §34 and Simulation spec 09 §37 each independently named optimization as the anticipated consumer of their own already-stable output types (TwinSnapshot/TwinState, Experiment/ExperimentResult) rather than proposing to build a solver into either package. Folding search-and-solve logic into decision or simulation would mean shipping the exact algorithmic commitment each package's own specification already declined to make.
  3. A fundamentally different position on the stateless/stateful spectrum than its immediate neighbors, for the same underlying reason simulation already established one layer down. optimization.OptimizationModel is as stateless as kpis.BaseKPI/analytics.AnalyticsModel/decision.DecisionModel/simulation.SimulationModel (spec 05 §24, spec 06 §8, spec 07 §8, spec 09 §8). optimization.OptimizationRun is stateful for the same reason digital_twin.Twin and simulation.SimulationRun are: a solving or solved optimization attempt is itself a piece of history the instant it starts executing. This package is the third, not the first, to reach for core.BaseEntity's "produce a new instance via with_state()" idiom.
  4. Consistent with every other layer boundary already drawn in this platform, including the four analytics, decision, digital_twin, and simulation just established at the four layers below. A platform that has consistently drawn a package boundary at every "distinct responsibility, distinct rate of change, distinct statefulness/governance profile" seam would be inconsistent to stop doing so at the one seam (simulation → optimization) three of its own most recent specifications already named from three different directions.

Dependency Rationale

optimization is permitted to import the entire Foundation Layer plus analytics, decision, digital_twin, and simulation, not merely simulation directly, for the same structural reason ADR-0007, ADR-0008, and ADR-0009 already established at their own layers: an OptimizationContext (governing spec §8) legitimately needs kpis.KPIResult/analytics.AnalyticsResult (as measurement/characterization evidence), decision.ActionPriority/decision.DecisionScore (directly, as objective-function inputs, per Decision spec 07 §36's own anticipation), digital_twin.TwinSnapshot/TwinState (as a problem's starting condition, per Digital Twin spec 08 §34's own anticipation), and simulation.SimulationResult/Experiment/ExperimentRunner (as projected-outcome evidence and, for the latter, direct composition for candidate-scenario search, per Simulation spec 09 §37's own anticipation). analytics is exercised especially heavily and directly - not merely as evidence but as the statistical engine PlanComparator/SensitivityAnalyzer delegate to entirely (governing spec §19, §20) - because Optimization's own charter explicitly excludes owning statistical computation of any kind. Restricting optimization to only simulation would have forced pass-through re-exports of types it does not otherwise need to expose, exactly the worse alternative ADR-0006 through ADR-0009 already rejected at their own layers.

connectors is included in the permitted-import list for platform-wide layering consistency, but - exactly as in every package above it - is not exercised: optimization operates on already-computed, already-projected, already-synchronized facts, never on a vendor-specific wire format (governing spec §5, §34).

Architectural Trade-offs

  • Trade-off accepted: six interface-only ABCs (LinearProgrammingModel, MixedIntegerProgrammingModel, ConstraintProgrammingModel, MultiObjectiveModel, EvolutionaryMetaheuristicModel, NetworkOptimizationModel) ship with zero concrete implementations, more categories than any prior package's own interface-only surface. This means the package cannot, on its own, actually solve a problem until a plugin implements at least one of the six - a materially weaker out-of-the-box capability than shipping even a single reference LP solver would provide. Accepted for the same three reasons ADR-0006 through ADR-0009 already accepted the equivalent trade-off at their own layers (see "Alternatives Rejected"), now applied a fifth time and across more categories than any prior instance; the governing specification treats this as further confirmation that the discipline is a platform-wide convention.
  • Trade-off accepted: OptimizationRun subclasses core.BaseEntity[str] and is immutable, with state changes producing a new instance (with_state()), rather than being a plain mutable Python object, exactly mirroring the trade-off ADR-0009 already accepted for SimulationRun, which itself mirrored ADR-0008's for Twin. The governing spec's own Performance Considerations (§36) judges the allocation cost negligible next to the category-dispatch call and repository round-trip already on the hot path of every solve or iteration.
  • Trade-off accepted: OptimizationRunRepository is a literal type alias over core.BaseRepository[OptimizationRun, str], not a new ABC, the third occurrence of this exact reuse (TwinRepository, spec 08 §20; SimulationRunRepository, spec 09 §24). Accepted because no optimization-run-specific repository method was identified as necessary at design time.
  • Trade-off accepted: this package introduces no dedicated cache, unlike every one of its three immediate neighbors below it (kpis.ResultCache, digital_twin.TwinStateCache, simulation.SimulationStateCache). This is a deliberate absence, not an oversight: OptimizationContext is assembled once, by the caller, from evidence each lower package has already computed and, where applicable, already cached on its own side. Introducing a cache here to hold the same evidence a second time, with no new expensive re-derivation step of this package's own to justify it, would duplicate state across two caches for no benefit - the governing specification records this explicitly (§26) rather than silently declining to mention it.
  • Trade-off accepted: this package defines no second experiment-orchestration mechanism, composing simulation.ExperimentRunner directly instead wherever a candidate-scenario search is the natural framing (governing spec §17). This keeps the package's own module count lower than simulation's despite covering more solving paradigms, at the cost of a direct, cross-package call path into simulation.ExperimentRunner.run_trials that a hypothetical self-contained alternative would not have needed.
  • Trade-off accepted: PlanComparator and SensitivityAnalyzer own no statistical computation whatsoever, deferring every mean, percentile, confidence interval, and distributional judgment to analytics, mirroring simulation.ScenarioComparator/SensitivityAnalyzer's identical posture (spec 09 §19–§20) and decision.ConfidenceScoring's (spec 07 §24) before that. This is the third package in the series whose comparison layer is a thin delegation to analytics, not the first.

Alternatives Considered

  1. Fold search-and-solve logic directly into decision (e.g. a concrete solver embedded in ActionPlanner). Rejected - directly contradicts decision's own already-locked scope (spec 07 §3.1, which names optimization by name as the package responsible for this) and its own Future Roadmap's framing of optimization as the separate, downstream consumer of its outputs (spec 07 §36).
  2. Fold search-and-solve logic directly into simulation (e.g. a concrete solver embedded in ExperimentRunner). Rejected - simulation's own charter explicitly excludes optimization search (spec 09 §3.1: "never searches for an optimal plan"), and its own Future Roadmap already frames optimization as the separate package positioned to compose ExperimentRunner, not absorb it (spec 09 §37).
  3. Defer optimization entirely until agents needs it, treating search-and-solve as that future package's own internal detail. Rejected - for the same reason ADR-0006 through ADR-0009 each rejected deferring their own packages: agents and visualization (governing spec §37) are both independently expected to consume OptimizationResult/ParetoResult; deferring this package would mean each reimplements its own, likely mutually inconsistent, notion of "what does a solved plan look like."
  4. A single, larger "simulation + optimization" package, reasoning that searching over candidate scenarios is just an extension of projecting one. Rejected - agents and visualization (governing spec §37) are expected to consume optimization output without needing simulation's full scenario-execution surface exposed alongside it, and vice versa; bundling the two would also obscure the subtly different reason each package's central abstraction is stateful (see "Why a Separate Package," item 3).
  5. Ship at least one concrete implementation per interface-only category now (a reference LP solver, at minimum), on the reasoning that an optimization package with no runnable solver is of limited immediate use. Rejected - see "Alternatives Rejected" below; this was the most seriously debated alternative, mirroring the equivalent debate ADR-0006 through ADR-0009 each recorded for their own interface-only capabilities.
  6. Bundle a specific third-party solver library (OR-Tools, Pyomo, PuLP, or SciPy Optimize) as a required or optional direct dependency of this package, reasoning that a platform this focused on operational planning should ship a working solver out of the box. Rejected - see "Alternatives Rejected" below.

Alternatives Rejected

Shipping at least one concrete implementation of any of the six category ABCs is rejected for the same three compounding reasons ADR-0006 through ADR-0009 already established for their own interface-only capabilities, restated at this layer:

  1. Scope creep into modeling/solver-library territory. Choosing a specific LP algorithm, MIP branching strategy, CP search strategy, Pareto-search method, metaheuristic, or network algorithm is exactly the kind of algorithmic decision this package's charter (governing spec §3.1, §3.5, §4) excludes. Shipping one "just as a starting point" would establish a precedent that quietly widens optimization's scope every time a plausible-looking heuristic seems worth adding.
  2. Premature commitment to an interface's shape via its first implementation. Defining all six ABCs now and deferring their first concrete implementations to independently-versioned future plugins keeps each contract general, rather than shaped by whichever solver happened to ship first.
  3. The package's charter is explicitly to remain solver-independent, per its own brief: "support adapters for OR-Tools, Pyomo, PuLP, and SciPy Optimize without coupling the architecture to any implementation." Shipping a concrete implementation backed by any one of those libraries would be the first, precedent-setting violation of that explicit non-negotiable.

Bundling a specific solver library as a dependency of this package was the second most seriously considered alternative, and is rejected for a related but distinct reason:

  1. It would make optimization the first package in this series whose Foundation Layer position is coupled to a third-party numerical library's own release cadence, API stability, and licensing terms - a materially different kind of dependency than the platform's existing, deliberately dependency-light optional extras (pyarrow for events, openpyxl for connectors, numpy/pandas/polars/duckdb for analytics/kpis execution backends), each of which is an interchangeable backend behind an already-stable interface, never a mandatory single choice baked into the architecture itself.
  2. Four solver libraries are named in this package's own brief (OR-Tools, Pyomo, PuLP, SciPy Optimize) with no basis given for preferring one over the others; bundling any single one would be an unjustified, arbitrary architectural commitment the brief itself does not make.
  3. The six category ABCs (governing spec §11–§16) already give every one of the four libraries - and any future one - an equally first-class path into the platform via the identical plugin mechanism every other extension point already uses (governing spec §17, §31); no capability is lost by declining to bundle one now.

Long-Term Evolution

This package is expected to grow along the same axes ADR-0006 through ADR-0009 already established for analytics, decision, digital_twin, and simulation, without requiring a new ADR:

  1. New concrete models within existing categories, or new OptimizationProblems - ordinary, checklist-governed extension (governing spec §30), not an architectural change.
  2. The first concrete implementations of the six category ABCs, whenever they arrive - expected to land as plugins (an OR-Tools, Pyomo, PuLP, or SciPy Optimize adapter being the most plausible first instances, per this package's own brief) against the already-stable ABCs this design defines.
  3. The first concrete implementation of simulation.CalibrationModel, composed from an optimization-provided parameter-search routine - the bridge Simulation spec 09 §37 anticipated before either package existed.
  4. A production-grade OptimizationRunRepository backend (SQL, document store) implementing core.BaseRepository[OptimizationRun, str] directly - no code change to this package required (governing spec §24).

A new ADR would be warranted if a future need arose to: change optimization's position in the dependency chain; permit optimization to be imported by a Foundation Layer package or by analytics/decision/digital_twin/simulation (which would indicate a design error, not a legitimate evolution); introduce a second extension mechanism alongside the Registry Framework specialization this design already commits to; bundle a specific solver library as a direct dependency, reversing the trade-off recorded above; or introduce a dedicated cache, reversing the deliberate non-need recorded above, in light of a measured performance problem this ADR did not anticipate.

Future Compatibility

Every public type this package introduces (governing spec §18, OptimizationResult and its ParetoResult subclass) is a plain, frozen core.BaseValueObject (or, for OptimizationRun itself, a frozen core.BaseEntity), serializable via the platform's existing core.serialization mechanism, with no optimization-internal state leaking into any type a future package would need to consume - the identical compatibility guarantee ADR-0006 through ADR-0009 each already established for analytics.AnalyticsResult, decision.DecisionResult, digital_twin.TwinResult, and simulation.SimulationResult, extended one layer up. This is a deliberate guarantee for the packages named in the governing specification's Future Roadmap (§37) - agents and visualization - neither of which is designed by this ADR or by the governing specification, but both of which are expected to consume optimization's output types without requiring a breaking change to them. The six interface-only ABCs this package defines (governing spec §11–§16) exist specifically so that whichever future package or third-party plugin first needs a concrete linear-programming, mixed-integer-programming, constraint-programming, multi-objective, evolutionary/metaheuristic, or network-optimization capability can implement against a contract that already exists, and so that this package can close, at last, every "future package promise" three already-locked specifications made on its behalf before it existed.

This ADR governs the existence and shape of mineproductivity.optimization as a package. See 10_Optimization_Design_Specification.md for the full object model and 10_Optimization_Implementation_Checklist.md for the locked implementation contract.