incident review

Evidence for a Trading Incident Review

A practical guide to preserving the evidence needed to reconstruct systematic trading incidents, assess controls, and improve operations.

IMRYN Research · · 1282 words

Editorial scope: IMRYN explains infrastructure, execution and risk concepts for educational purposes, without presenting performance promises or investment advice.

Start with a reconstructable timeline

An incident review should preserve enough evidence for a later reader to answer a simple question: what did the system know, decide, send, receive, and do next at each meaningful moment? Build the record around a single timeline, using consistent timestamps, time-zone information, and an identified clock source. Include the start of abnormal behavior, detection, any automated safeguards, human intervention, venue acknowledgements, recovery steps, and the point at which normal operating conditions were confirmed.

Keep raw events separate from the reviewer’s interpretation. A timeline may contain a market-data interruption, an order submission, a rejected request, a risk-limit event, and an operator action. Later analysis can connect those events, but preserving the original sequence makes it possible to revisit assumptions as new evidence emerges.

Where systems communicate asynchronously, retain correlation identifiers that connect a decision to its input, an order to its venue response, and an alert to the process or person that handled it. Without those links, logs can be plentiful but still fail to explain causal relationships.

  • Record timestamps with timezone and clock-source context.
  • Preserve request, order, execution, alert, and intervention identifiers.
  • Distinguish raw event evidence from later annotations.

Preserve the decision context, not only orders

Order and fill records are essential, but they rarely explain why an action occurred. Preserve the decision context available to the system at the time: relevant market-data state, instrument and venue configuration, strategy or workflow version, parameters in effect, account or routing constraints, and the reason codes produced by the decision logic where available.

For an incident involving unexpected exposure or repeated execution attempts, the review should be able to determine whether the behavior followed the configured rules, resulted from stale or incomplete inputs, or occurred after a configuration change. That requires immutable copies or version references for the exact configuration and code or deployment artifact involved.

This is also where explicit risk limits should be documented. Retain the limits configured at the time, their scope, utilization immediately before and during the incident, any overrides, and the action each breached threshold was expected to trigger. A risk control cannot be evaluated only by whether an alert appeared; the review needs to show whether the control was active, observable, and effective within its intended boundary.

Capture observable execution across venues

Multi-venue execution creates multiple records of the same operational story. Preserve outbound order messages, venue acknowledgements, rejects, cancels, amendments, execution reports, and reconciliation results. Record the intended route as well as the route actually used, including any fallback behavior or routing decision that changed during the incident.

The evidence should support comparisons between internal state and external state. If an internal system marked an order cancelled while a venue later reported a fill, the review needs the messages and timestamps that reveal when those states diverged. Similarly, preserve connectivity and session information when it may explain delayed acknowledgements, duplicate submissions, or incomplete status updates.

Avoid reducing execution evidence to a final position snapshot. Final outcomes can conceal important intermediate risk, such as temporarily open exposure, repeated retries, incomplete cancel handling, or delayed monitoring. Observable execution means keeping enough detail to inspect the path from instruction to confirmed external outcome.

  • Keep both internal order state and venue-reported state.
  • Retain routing decisions, fallback paths, and venue response messages.
  • Include reconciliation outputs and unresolved mismatches.

Document human oversight and control actions

A useful incident record makes human oversight visible. Preserve who received an alert, when it was acknowledged, what information was available, what action was taken, and what authority or procedure supported that action. This is not about assigning blame; it is about understanding how operational controls functioned under pressure.

Record automated safeguards alongside human actions. Examples include a risk-limit block, a trading pause, a throttling event, a circuit-breaker response, or an escalation notification. For each, note the configured expectation and the observed outcome. If an operator overrode, paused, resumed, or changed a control, preserve the reason, approval path where applicable, and the exact configuration change.

Communications can be relevant evidence when they affect execution or recovery. Preserve concise incident-channel notes, handoff records, and decision summaries, while applying appropriate access controls and retention practices. A review does not need every conversation; it needs the decisions that changed the system’s operational state.

  • Log alert delivery, acknowledgement, and escalation.
  • Record pauses, overrides, resumes, and configuration changes.
  • Capture the rationale and authority for material interventions.

Keep the review reproducible

Later analysis is strongest when another reviewer can reproduce the investigative path without rebuilding it from memory. Preserve the incident data set with checksums or other integrity references, document known gaps, and identify the tools, queries, dashboards, and transformations used to create the review. If data was filtered, aggregated, normalized, or manually corrected, record that fact and the rule applied.

Reproducible evaluation does not require a perfect replay of every production condition. It requires a clear record of what evidence was used, what was unavailable, and how conclusions were derived. Where a replay or simulation is used to examine behavior, label it as an analytical exercise and retain the inputs, assumptions, and versioned environment needed to repeat it.

Separate confirmed findings from hypotheses and remediation proposals. A practical review can state that a message sequence is confirmed, that a causal explanation remains under investigation, and that a control change is proposed. This discipline prevents later readers from treating an early theory as established fact.

Example: a compact evidence decision aid

Example only: suppose monitoring identifies a short period of unexpected order retries after one execution venue stops acknowledging requests. The incident review should not begin by deciding that the venue, strategy logic, or operator caused the problem. It should first preserve the evidence needed to distinguish among those possibilities.

Use the following decision aid to decide whether a record is complete enough for later analysis. If any answer is no, record the gap explicitly and identify the owner and retention risk before continuing the investigation.

IMRYN presents systematic trading infrastructure and multi-venue execution, with guardrailed autonomy, risk controls, and continuous monitoring as public product concepts. In that context, this example is educational guidance for documenting infrastructure and operational-risk events. It does not describe a specific IMRYN deployment, claim any outcome, or provide investment advice.

  • Can each retry be linked to its triggering input, internal decision, and venue response or absence of response?
  • Were the active retry settings, risk limits, routing rules, and deployment version preserved?
  • Can the review show when monitoring alerted, whether a safeguard activated, and what a human operator did?
  • Can internal order state be reconciled with the venue’s later messages and final records?
  • Can another reviewer rerun the documented queries and reach the same factual timeline?

Frequently asked questions

What is the minimum evidence to preserve after a trading incident?

Preserve timestamped raw events, relevant market and execution messages, active configurations and risk limits, system and venue state, alerts, human interventions, and reconciliation results. Include identifiers that connect these records into one timeline and document any missing data.

Why are configuration versions important in an incident review?

Configuration versions show the exact rules, limits, routing settings, and deployment context active during the event. They help reviewers distinguish expected behavior under the configured system from behavior caused by stale inputs, changes, or control failures.

How should simulations be used in an incident review?

Use simulations as clearly labelled analytical tools, preserving their inputs, assumptions, code or environment version, and limitations. They can help explore possible behavior, but simulated or past results do not determine future outcomes.

Sources and further reading

These resources provide the wider reference frame. Product statements on this page are limited to the public information provided by IMRYN.

Who, how and why

Editorial responsibility: IMRYN Research

An automated assistant prepared a first draft. It then passed the published structure, similarity and unsupported-claim checks. Please report any useful correction through the main site.

Method, checks and corrections