Skip to content

simulator.md

Latest commit

 

History

History
417 lines (310 loc) · 15.8 KB

simulator.md

File metadata and controls

417 lines (310 loc) · 15.8 KB
Raw

Consequence Simulator

Module: moralstack/runtime/modules/simulator_module.py

The Consequence Simulator generates plausible future scenarios to evaluate the impact of responses.

For testers: The output (expected_valence, worst_case_valence, semantic_expected_harm) contributes to the Orchestrator's decision and revision guidance. Tests can verify that scenarios with strongly negative expected_valence or high semantic_expected_harm generate revision recommendations (REVISE).

Overview

The Consequence Simulator:

  • Generates scenarios (best/worst/likely case)
  • Evaluates the potential impact of responses
  • Identifies risks not evident from the response itself
  • Provides feedback to guide revisions
  • Semantic layer: damage taxonomy (harm_type, harm_scope) and semantic_expected_harm influence deliberation

On a DCCL MATCH, this module is not invoked. A MODULE_DEFERRED_TO_COMPLIANCE orchestration event documents the skip. The module's evaluation is unnecessary because the deployer explicitly authorized the response via the developer contract.

Ultra-Lean Design

To reduce token usage:

  • Narrative: each consequence.text ≤ 15 words, short nominal phrases
  • max_tokens: 384 (configurable in SimulatorConfig)
  • Minimal schema: prompt without verbose examples, JSON skeleton only
  • Post-parse truncation: text > 160 characters is truncated

Prompt-Caching Static/Dynamic Split (Part A)

The simulator has two independent LLM paths with path-specific system prompts (moralstack/prompts/simulator_prompt.py), so a seeded call can never receive the batch-only contract and vice versa:

  • Batch (_simulate_batch): SIMULATOR_BATCH_SYSTEM_PROMPT = base framing + SIMULATOR_FULL_STATIC_PREFIX (the "exactly N consequences" contract, enums, JSON schema skeleton). build_simulator_prompt() returns dynamic-only content (TURN PARAMETERS/REQUEST/RESPONSE/RISK CONTEXT/DOMAIN/num_scenarios).
  • Seeded (_simulate_with_seeds, SimulatorConfig.use_seeded_generation=True): SIMULATOR_SEEDED_SYSTEM_PROMPT = base framing + the seeded rubric (one consequence per seed) + enums + schema skeleton. SEEDED_PROMPT_TEMPLATE (simulator_module.py) is dynamic-only: PERSPECTIVE:{seed} + REQUEST + RESPONSE.

Both system prompts are byte-stable across requests for their own path (verified by tests/test_static_prefix_stability.py) and retries are unchanged.

Structured Outputs (schema enforced by the provider)

response_format is a strict json_schema (simulator_response_format() in moralstack/utils/structured_output.py), so the provider itself enforces the enums instead of the client rejecting a reply it has already paid for. The schema is derived from the same Literal aliases as SimulatorOutput, so the two cannot drift (tests/test_structured_outputs_simulator.py).

Why it was needed: in plain JSON mode the provider guarantees only that the reply parses. Measured on the COMPL-AI T=0/T=1 campaigns, 2,231 simulator calls (~32% of all of them) returned complete, valid JSON that the client rejected, and 239 invocations exhausted all three retries and produced no consequence at all. Nearly every rejection was one confusion: a harm_type value placed in scenario_type (misinformation alone accounts for 1,896 occurrences; also security_breach, psychological_harm, reputational_harm, privacy_breach). Because the enums overlap semantically, the model picks a harm label for the scenario slot — and since the retry re-sends the same prompt at low temperature, all three attempts reproduce the same rejected value, so the retry is futile by construction.

PolicyLLM._complete degrades a json_schema request to {"type": "json_object"} when the effective model does not support Structured Outputs (supports_json_schema, an allowlist — see docs/modules/openai_params.md), so an unrecognised deployment behind OPENAI_BASE_URL keeps the previous behaviour rather than failing. Client-side validation (parse_and_validate_simulator_output) is therefore still required and unchanged: it is the only guard on the degraded path.

Scenario Types

ScenarioType

Type Description Purpose
IMMEDIATE_HARM Foreseeable immediate harm Identify urgent risks
DOWNSTREAM_MISUSE Subsequent misuse Anticipate abuse
SOCIAL_IMPACT Social/community impact Assess broad effects
LEGAL_CONSEQUENCE Legal consequences Identify regulatory risks
POSITIVE_OUTCOME Positive outcome Balance the evaluation

Semantic Harm Layer

Damage taxonomy (harm_type)

harm_type Description Constitutional principle
financial_loss Economic loss -
physical_harm Physical harm CORE.NM.1
psychological_harm Psychological harm -
privacy_breach Privacy violation CORE.PRIV.1
legal_risk Legal risk CORE.NM.2
reputational_harm Reputational harm -
security_breach Security breach -
discrimination Discrimination -
misinformation Misinformation CORE.DECEPTION.1
exploitation Exploitation CORE.NONEXPLOITATION.1
self_harm Self-harm MH.CRISIS.1
emotional_manipulation Hidden emotional influence CORE.MANIPULATION.1, CORE.AUTONOMY.1
financial_deception Financial deception / fraud FIN.SCAM.1, CORE.DECEPTION.1
none No semantic harm -

Scope (harm_scope)

harm_scope Description
individual Single person
group Group
societal Society
systemic System/institution

semantic_expected_harm calculation

risk_i = likelihood * harm_severity  # for harm_type != "none"
semantic_expected_harm = max(risk_i)
dominant_harm_types = top 2 harm_type by risk_i
worst_harm = argmax(risk_i)  # {harm_type, harm_scope, risk}

Influence on deliberation

semantic_expected_harm and dominant_harm_types influence votes in _determine_decision:

  • semantic_expected_harm >= 0.4 → +1 vote REVISE
  • semantic_expected_harm >= 0.6 → +2 votes REVISE
  • physical_harm or self_harm in dominant_harm_types and semantic_expected_harm >= 0.4 → +2 votes REVISE

The simulator can never produce REFUSE; REFUSE comes only from hard violations, op_risk HIGH, or policy bounds.

Output Structure

Consequence

@dataclass
class Consequence:
    text: str  # Narrative description (max 160 chars)
    likelihood: float  # Probability [0, 1]
    scenario_id: str  # Unique identifier
    scenario_type: ScenarioType  # Category
    outcome_valence: float  # Valence [-1, 1]
    affected_stakeholders: list[str]  # Affected parties (max 3)
    harm_type: str  # Semantic harm type
    harm_severity: float  # Severity [0, 1]
    harm_scope: str  # Scope
    reversibility: float  # Reversibility [0, 1]

SimulationResult

@dataclass
class SimulationResult:
    consequences: list[Consequence]  # Generated scenarios
    worst_case_valence: float  # min(valence)
    best_case_valence: float  # max(valence)
    expected_valence: float  # Likelihood-weighted average
    semantic_expected_harm: float  # max(likelihood * harm_severity)
    dominant_harm_types: list[str]  # top 2 harm_type
    worst_harm: dict | None  # {harm_type, harm_scope, risk}
    raw_response: str  # LLM response (debug)

Expected Valence Calculation

expected_valence = Σ(valence_i × likelihood_i) / Σ(likelihood_i)
Expected Valence Interpretation
> 0.5 Predominantly positive outcome
0.0 - 0.5 Mixed outcome
< 0.0 Predominantly negative outcome → generates guidance

Usage

Initialization

from moralstack.runtime.modules.simulator_module import LLMConsequenceSimulator

simulator = LLMConsequenceSimulator(policy_llm=policy)

Simulation

result = simulator.simulate(
    prompt="User request",
    response="Response to evaluate",
)

print(f"Expected valence: {result.expected_valence}")
print(f"Worst case: {result.worst_case_valence}")
print(f"Best case: {result.best_case_valence}")

for consequence in result.consequences:
    print(f"- [{consequence.scenario_type.value}] {consequence.text}")
    print(f"  Valence: {consequence.outcome_valence}, Likelihood: {consequence.likelihood}")

Example Output

SimulationResult(
    consequences=[
        Consequence(
            text="User uses the information as a starting point for deeper research",
            likelihood=0.6,
            scenario_type=ScenarioType.POSITIVE_OUTCOME,
            outcome_valence=0.8,
            affected_stakeholders=["user"]
        ),
        Consequence(
            text="User might interpret the advice as a substitute for professional consultation",
            likelihood=0.25,
            scenario_type=ScenarioType.DOWNSTREAM_MISUSE,
            outcome_valence=-0.5,
            affected_stakeholders=["user", "healthcare_system"]
        ),
        Consequence(
            text="In case of unrecognized emergency, delay in appropriate care",
            likelihood=0.15,
            scenario_type=ScenarioType.IMMEDIATE_HARM,
            outcome_valence=-0.7,
            affected_stakeholders=["user"]
        )
    ],
    expected_valence=0.35,
    worst_case_valence=-0.7,
    best_case_valence=0.8,
)

Caching

The simulator implements caching to avoid recomputation:

# Automatic cache based on hash(prompt + response)
# Avoids duplicate LLM calls for identical inputs

Orchestrator Integration

The Simulator contributes to aggregated guidance:

if simulation.expected_valence < 0:
    # Generate specific guidance
    guidance_parts.append(
        f"[SIMULATOR] Negative consequences predicted: {worst_consequence.text}"
    )

Impact on Decisions

Expected Valence Impact
≥ 0.5 No negative impact
0 - 0.5 Generates warning
< 0 Generates revision guidance

Semantic harm (independent of valence): semantic_expected_harm >= 0.4 adds REVISE votes; physical_harm/ self_harm with high harm add further REVISE votes.

Factory Methods

SimulationResult.empty()

# No relevant consequences
result = SimulationResult.empty()

SimulationResult.from_error()

# Fallback on error
result = SimulationResult.from_error("Simulation failed")
# Assumes neutral valence (0.0)

Environment Variables

All simulator tuning can be overridden via .env. Variables are read at simulator construction; empty or missing values use the defaults below. See .env.template for the full list. In application runs (CLI and benchmark), simulator configuration is the single source of configuration — no CLI or code path overrides these variables.

Model (simulator LLM)

MORALSTACK_SIMULATOR_MODEL

  • Default: (none — uses the same model as the rest of the stack, e.g. OPENAI_MODEL or gpt-4o)
  • Type: string (OpenAI model id)
  • Description: OpenAI model used only for the consequence simulator. When set and non-empty, the CLI and benchmark create a dedicated OpenAIPolicy with this model for the simulator; the rest of the stack keeps using OPENAI_MODEL.
  • Effect:
    • Set to a model id (e.g. gpt-4o-mini, gpt-4.1-nano): The simulator uses that model. Lets you use a smaller/cheaper model for simulation and a larger one for generation.
    • Unset or empty: The simulator uses the same policy (and model) as the rest of the pipeline.

Since 2026-09-09 the shipped templates set MORALSTACK_SIMULATOR_MODEL=gpt-4o, matching the rest of the stack; they shipped gpt-4.1-nano before that. The change aligns the templates with the configuration all COMPL-AI campaigns actually ran under (every one of the 6,380 simulator calls in the T=0/T=1 logs carries model=gpt-4o), so a reader copying the template reproduces the measured setup. Benchmark testing under the former default showed gpt-4.1-nano reduces simulator-step cost versus heavier defaults; combined with other stack optimizations, benchmark run 12 reports overall mean wall-clock ~36s (median ~26s) with 98.8% compliance maintained and overall judge score ~9.27/10 (vs 7.83/10 baseline), indicating minimal quality regression versus heavier simulator configurations on the same suite.

LLM and retry behaviour

MORALSTACK_SIMULATOR_MAX_RETRIES

  • Default: 3
  • Type: int (>= 1)
  • Description: Number of parse attempts for the simulator JSON response before raising an error.

Simulator generation uses OpenAI's json_object response format (response_format={"type": "json_object"} on GenerationConfig), which guarantees valid JSON and greatly reduces retries caused by malformed JSON.

MORALSTACK_SIMULATOR_MAX_TOKENS

  • Default: 384
  • Type: int (>= 1)
  • Description: Maximum tokens for the simulator LLM response. 384 keeps narratives compact (ultra-lean design).

MORALSTACK_SIMULATOR_TEMPERATURE

  • Default: 0.8
  • Type: float (0–2)
  • Description: LLM temperature for scenario generation. Higher values produce more diverse scenarios.

MORALSTACK_SIMULATOR_TOP_P

  • Default: 0.95
  • Type: float (0–1)
  • Description: Nucleus sampling (top-p) for simulator LLM generation. Controls diversity of token sampling.

Scenario generation

MORALSTACK_SIMULATOR_DEFAULT_NUM_SCENARIOS

  • Default: 3
  • Type: int (>= 1)
  • Description: Default number of consequence scenarios to generate per simulation call.

MORALSTACK_SIMULATOR_USE_SEEDED_GENERATION

  • Default: false
  • Type: bool (1/true/yes or 0/false/no)
  • Description: When true, each scenario is generated with a separate seed prompt for greater diversity. More costly but produces more varied scenarios.

Caching

MORALSTACK_SIMULATOR_ENABLE_CACHING

  • Default: true
  • Type: bool (1/true/yes or 0/false/no)
  • Description: Enable caching of simulation results to avoid recomputation on identical inputs.

Conversation Context

For multi-turn requests, the simulator receives the developer contract plus recent conversation history from ProcessedRequest. The prompt declares context_mode=role_serialized_truncated; last 3 turns, and the deliberation runner records CONTEXT_SHAPE_RECORDED with the simulator call metadata so available and used prior turns remain auditable.

See Also