Chaos Testing

Audience

Outcome

After this page you should know what this surface is for, which source files own the behavior, which public route or adjacent page to use next, and which validation command to run before changing the claim.

Source Truth

• Public route: security/chaos-testing
• Source document: helm-ai-enterprise/docs/public/security-and-trust/chaos-testing.md
• Public manifest: helm-ai-enterprise/docs/public-docs.manifest.json
• Source inventory: helm-ai-enterprise/docs/source-inventory.manifest.json
• Validation: corepack pnpm run docs:coverage, corepack pnpm run docs:truth, and npm run coverage:inventory from docs-platform

Do not expand this page with unsupported product, SDK, deployment, compliance, or integration claims unless the inventory manifest points to code, schemas, tests, examples, or an owner doc that proves the claim.

Troubleshooting

Overview

HELM chaos testing validates that the kernel degrades gracefully under adversarial conditions, maintaining security invariants (fail-closed, receipt generation, budget enforcement).

Scenarios

1. Error Budget Exhaustion

Goal: Verify that builder/promotion endpoints return 429 when budget drops below 20%.

# Inject chaos (burns 30% budget per call)
curl -X POST http://localhost:8080/api/ops/chaos/inject \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Repeat until budget < 20%
curl -X POST http://localhost:8080/api/ops/chaos/inject \
  -H "Authorization: Bearer $ADMIN_TOKEN"
curl -X POST http://localhost:8080/api/ops/chaos/inject \
  -H "Authorization: Bearer $ADMIN_TOKEN"

# Verify generated-spec creation is gated
curl -X POST http://localhost:8080/api/v1/workspaces/$WORKSPACE_ID/generated-specs \
  -H "Content-Type: application/json" \
  -d '{"goal":"test governed generated-spec flow"}'
# Expected: 429 Too Many Requests with X-Gate-Enforced: BudgetExhausted

2. Signer Unavailable

Goal: Verify Guardian returns DENY verdict when signer is unavailable.

// Test with nil signer
g := guardian.NewGuardian(nil, prg, registry)
decision, err := g.EvaluateDecision(ctx, req)
// Expected: err != nil (fail-closed)

3. Clock Drift

Goal: Verify that expired intents are rejected even with clock manipulation.

// Inject a clock that returns past time
exec := NewSafeExecutor(dispatchers, guardian, proofgraph)
exec = exec.WithClock(func() time.Time {
    return time.Now().Add(-2 * time.Hour) // 2 hours in the past
})

// Execute with a valid intent — should detect expiration
result, err := exec.Execute(ctx, intent)
// Expected: err contains "expired"

4. Concurrent Storm

Goal: Verify kernel handles 1000 concurrent tool calls without race conditions.

# Run race detector with concurrent access
cd core && go test ./pkg/guardian/... -race -count=1 -timeout 120s
cd core && go test ./pkg/executor/... -race -count=1 -timeout 120s
cd core && go test ./pkg/firewall/... -race -count=1 -timeout 120s

5. Policy Graph Mutation

Goal: Verify that mutating the PRG after Guardian creation doesn't affect in-flight decisions.

g := guardian.NewGuardian(signer, prg, registry)
// Modify PRG after creation
prg.AddRule("dangerous-tool", RequirementSet{...})
// Guardian should still use original snapshot

6. Malformed Input

Goal: Verify kernel rejects malformed JSON, oversized payloads, and invalid schemas.

# Malformed JSON
curl -X POST http://localhost:9090/v1/tools/execute \
  -H "Content-Type: application/json" \
  -d '{invalid json'
# Expected: 400 Bad Request

# Oversized payload (>1MB)
python3 -c "print('{\"data\":\"' + 'A'*2000000 + '\"}')" | \
  curl -X POST http://localhost:9090/v1/tools/execute \
  -H "Content-Type: application/json" \
  -d @-
# Expected: 400 or 413

Automation

All chaos scenarios are automated via the race detector (go test -race) and the TestPolicyFirewall_ConcurrentAccess test in the firewall package.

Diagram

flowchart TD
    subgraph Ingestion["1. Ingestion & Context Plane"]
        inject["Inject fault"]
        observe["Observe signal"]
        report["Update drill report"]
    end

    subgraph Evaluation["2. Evaluation & Policy Plane"]
        invariant["Check invariant"]
    end

    subgraph Execution["3. Execution & Verdict Plane"]
        block["Block release"]
    end

    subgraph Ledger["4. Tamper-Evident Ledger Plane"]
        receipt["Capture evidence"]
    end

    %% Operational Flow Edges
    inject --> observe
    observe --> invariant
    invariant --> receipt
    receipt --> report
    invariant -->|fails| block

    %% Premium Styling Rules
    style invariant fill:#2d3748,stroke:#4a5568,stroke-width:2px,color:#fff
    style receipt fill:#2f855a,stroke:#276749,stroke-width:2px,color:#fff
    style block fill:#e53e3e,stroke:#9b2c2c,stroke-width:2px,color:#fff

flowchart TD
    subgraph Ingestion["1. Ingestion & Context Plane"]
        inject["Inject fault"]
        observe["Observe signal"]
        report["Update drill report"]
    end

    subgraph Evaluation["2. Evaluation & Policy Plane"]
        invariant["Check invariant"]
    end

    subgraph Execution["3. Execution & Verdict Plane"]
        block["Block release"]
    end

    subgraph Ledger["4. Tamper-Evident Ledger Plane"]
        receipt["Capture evidence"]
    end

    %% Operational Flow Edges
    inject --> observe
    observe --> invariant
    invariant --> receipt
    receipt --> report
    invariant -->|fails| block

    %% Premium Styling Rules
    style invariant fill:#2d3748,stroke:#4a5568,stroke-width:2px,color:#fff
    style receipt fill:#2f855a,stroke:#276749,stroke-width:2px,color:#fff
    style block fill:#e53e3e,stroke:#9b2c2c,stroke-width:2px,color:#fff

Operational Readiness

Use this page as the public operating layer for Chaos Testing. The source of truth is helm-ai-enterprise/docs/public/security-and-trust/chaos-testing.md; if this page and the implementation disagree, update the source-backed doc and rerun the validation command before publishing.

Before relying on this surface, confirm three things: the source path above still exists, the referenced commands or contracts are still present in the owning repo, and the docs-platform export surfaces still show this page in search, Markdown, llms-full.txt, and MCP without exposing protected routes.

Validation command: corepack pnpm run docs:coverage && corepack pnpm run docs:truth. For website parity, also run npm run exports:boundary and npm run thin-pages:check from docs-platform.

Expected Output

A reader should leave with a concrete next action, the source file or contract to inspect, the command that proves the claim, and a clear boundary for what is public versus protected. For reference pages, the expected output is a correctly scoped request, schema, command, or diagnostic path. For operations pages, the expected output is a reproducible readiness or failure signal that can be attached to an evaluation or support thread.

Failure Modes

If the validation command fails, do not patch this page in isolation. First identify whether the drift is in code, generated contracts, source-owner docs, or the docs manifest. If the public page needs a protected deep link, describe the protected document by name instead of exposing its route. Commercial operator details, tenant data, key ceremonies, and deployment-sensitive internals stay in protected customer or staff docs; this public page only exposes the safe developer contract.