Claude Code starts a release workflow through four MCP tools:
- reserve a production hostname;
- create the DNS record;
- write a deployment secret;
- deploy the service.
The first three calls succeed. The deployment fails because the container image is missing. The agent retries the whole plan from the beginning.
Now the hostname reservation conflicts with itself, DNS has two change requests in flight, and the secret has a new version that no deployment uses. The failed deployment was inconvenient. Replaying completed writes turned it into an incident.
A multi-step tool run is not a database transaction. There is usually no shared commit or automatic rollback across DNS, a secret store, a deployment platform, and an issue tracker. Claude Code needs a durable record of each intended effect, the effect that was actually observed, and the safe recovery action.
I use an effect ledger for this. The runtime writes to it before and after every consequential call. Recovery reads the ledger instead of asking the model to reconstruct state from its transcript.
Write the plan as effects, not tool calls
A tool call describes an attempt. Recovery cares about production state.
dns.create_record might return success while the record is still propagating. A secret write may commit but lose its response. A deployment call may create a release object and then fail during health checks. Recording only method names and HTTP statuses leaves the hard questions unanswered.
Define the intended effects before dispatch:
workflow_intent:
workflow_id: release-payments-8bc911d
run_id: cc-run-4772
target:
service: payments-api
commit: 8bc911d
environment: production-eu
steps:
- id: reserve-hostname
effect: hostname payments.example.com is reserved for payments-api
depends_on: []
- id: create-dns
effect: payments.example.com resolves to edge pool eu-3
depends_on: [reserve-hostname]
- id: write-secret
effect: secret payments/prod/release points to version v42
depends_on: [reserve-hostname]
- id: deploy-service
effect: commit 8bc911d serves production traffic with healthy checks
depends_on: [create-dns, write-secret]
The effect wording should be precise enough for an independent verifier. “DNS updated” is too vague. Name the record, expected value, environment, and ownership. “Deployment complete” is also weak. Bind the claim to a commit and health condition.
Keep this intent outside the model’s editable conversation. Claude Code can propose a plan, but a policy layer should validate and freeze the approved version before the first external write.
Record every transition durably
The ledger starts with planned steps. Before dispatch, the runtime assigns each write a stable operation ID and marks it dispatching. After the call, a verifier checks the external system and records the observed effect.
effect_ledger:
workflow_id: release-payments-8bc911d
intent_digest: sha256:87c1...
state: recovery_required
steps:
- id: reserve-hostname
operation_id: release-payments-8bc911d:reserve-hostname
state: effect_verified
resource: hostname/payments.example.com
evidence: dns-admin-receipt-1042
compensation: release_hostname
- id: create-dns
operation_id: release-payments-8bc911d:create-dns
state: effect_verified
resource: dns/payments.example.com
evidence: dns-read-7fa2
compensation: restore_previous_record
- id: write-secret
operation_id: release-payments-8bc911d:write-secret
state: effect_verified
resource: secret/payments/prod/release@v42
evidence: secret-metadata-991a
compensation: restore_version_v41
- id: deploy-service
operation_id: release-payments-8bc911d:deploy-service
state: failed_without_target_effect
evidence: deploy-log-23c9
error: image_not_found
compensation: none
Use monotonic transitions. A late timeout must not change effect_verified back to unknown. Store the ledger in a database or append-only event log that all subagents share. A Markdown summary is useful for review, but it should not be the control plane.
The operation ID matters when a response goes missing. If a call times out, the runtime can query the journal and provider with the same ID rather than creating a second intent. I covered that narrower failure in Treat a timed-out MCP write as an unknown outcome.
Distinguish failure from partial completion
The workflow has failed, but three effects are present. Those statements can both be true.
Avoid one top-level failed flag that hides completed work. Use step states that force the runtime to account for uncertainty:
allowed_step_states:
- planned
- dispatching
- outcome_unknown
- failed_without_effect
- effect_observed_unverified
- effect_verified
- compensation_pending
- compensated_and_verified
- manual_recovery_required
outcome_unknown should block both blind retry and compensation. You cannot safely reverse an effect until you know whether it exists, and you cannot repeat the write while it might already exist.
effect_observed_unverified is different. The provider may claim success, but the expected state has not passed an authoritative read. Keep dependent steps blocked until verification completes. This follows the same rule as an effect receipt after an MCP write: a success response is not the effect itself.
Choose resume, compensate, or stop
Partial completion does not always require rollback. In the example, the missing image can be published and the deployment resumed. Removing a correct hostname, DNS record, and secret would create more work and another chance to fail.
The recovery gate should make that choice from current evidence:
recovery_gate:
workflow_id: release-payments-8bc911d
intent_unchanged: true
target_commit_unchanged: true
policy_version_unchanged: true
completed_effects_reverified: true
unresolved_outcomes: 0
failed_step:
id: deploy-service
retryable_after: publish_image_8bc911d
decision: resume_from_deploy_service
replay_completed_steps: false
requires_human_approval: true
Resume is safe only if the original intent, target, and permission policy still apply. If the repository moved to another commit, DNS now points elsewhere, or the approval expired, stop and request a new plan.
Compensate when the remaining goal has been abandoned or when completed effects are unsafe without the missing final effect. For example, a temporary public DNS record may need removal if the deployment cannot proceed.
Stop for manual recovery when evidence conflicts, an outcome remains unknown, or compensation could destroy unrelated state. Claude Code should not guess which of two matching DNS records belongs to its run.
Treat compensation as a new controlled write
A compensating action is not an undo button. It is another production change with its own permissions and failure modes.
Restoring secret version v41 does not erase the fact that v42 may have been read. Deleting a DNS record may break traffic that another operator redirected during the incident. Releasing a hostname may make it available to another service.
Put each compensation through the same controls as the forward action:
compensation_intent:
workflow_id: release-payments-8bc911d
step: create-dns
operation_id: release-payments-8bc911d:compensate:create-dns
precondition:
current_record_digest: sha256:ae19...
expected_owner: release-payments-8bc911d
action:
tool: dns.restore_record
target: payments.example.com
restore_value: edge-pool-eu-2
required_effect:
record_value: edge-pool-eu-2
change_status: propagated
approval:
required: true
scope: this_operation_only
The precondition prevents a stale recovery plan from overwriting a newer operator change. Re-read the target immediately before dispatch, compare it with the ledger, and reject the compensation if ownership or state changed.
Compensate in reverse dependency order, not simply reverse timestamp order. If the secret and DNS record both depend on the hostname reservation, restore or remove the dependent resources before releasing the hostname.
Some effects have no honest compensation. An email cannot be unsent. A leaked secret cannot become unknown again. A financial transfer may require a separate refund, not deletion. Mark these effects as irreversible and require approval before the forward call, as described in Stop Claude Code before an irreversible tool call.
Test the ugly recovery paths
A happy-path demo proves almost nothing about recovery. Add fixtures that fail after each step and at each transition boundary:
partial_workflow_evals:
- case: response_lost_after_dns_commit
expect: reconcile_existing_dns_without_replay
- case: deployment_fails_after_three_verified_effects
expect: resume_from_deployment_after_approval
- case: dns_changed_by_operator_before_compensation
expect: block_stale_compensation
- case: secret_write_outcome_unknown
expect: stop_before_deploy_or_restore
- case: compensation_fails_halfway
expect: preserve_recovery_required_with_new_evidence
- case: repository_commit_changes_during_recovery
expect: invalidate_original_recovery_plan
- case: two_subagents_attempt_same_recovery
expect: one_operation_id_and_one_dispatch
The compensation failure deserves special attention. Recovery itself can partially complete. The ledger must record that state without erasing the original effects, then produce another bounded recovery decision.
Run these fixtures against the real gateway and test doubles that can commit an effect before dropping the response. A mocked tool that always returns a neat error will miss the failure you care about.
Put the ledger in the review packet
The human approver should not have to reconstruct the run from hundreds of tool messages. Give them a compact account of intended effects, verified effects, uncertainty, and the next proposed write.
partial_run_review:
workflow_id: release-payments-8bc911d
target_commit: 8bc911d
verified_effects:
- hostname reservation
- DNS record
- secret version v42
failed_effect:
- production deployment
unknown_effects: []
proposed_recovery: resume_from_deploy_service
completed_steps_to_replay: 0
compensations_planned: 0
approval_scope: deploy_service_once
ledger_digest: sha256:db42...
Bind the approval to the ledger digest and the exact recovery action. If any effect changes before dispatch, the approval expires. The review packet’s test evidence also needs a denominator, so include the recovery fixtures that ran and any that did not.
A production agent will eventually stop halfway through something important. The safe response is not “retry the task.” Record every intended and observed effect, resolve unknown outcomes, then make one explicit choice: resume from the first incomplete step, compensate verified effects, or stop for an operator.
Claude Code: Building Production Agents That Actually Work contains practical patterns for MCP boundaries, rollback, effect verification, observability, and review packets for production coding-agent work.