Methodology
Evidence needs a label, a denominator and a limit.
A performance number without its data status, observation window, costs, benchmark and drawdown context is incomplete. This is the publication standard IMRYN uses for technical and performance explanations.
- Publisher
- IMRYN
- Last updated
- Method
- Product scope + primary guidance
Short answer
Short answer
IMRYN does not treat backtests, simulations, paper execution and live observations as interchangeable. Any published result should state what was measured, over which period, against what reference, under which assumptions and with which material limitations.
Label the evidence before interpreting it
Historical replay asks what a defined rule would have done on recorded data under specified assumptions. Simulation adds an execution model. Paper execution observes current conditions without committing the same capital and venue interaction as live trading. Live observation records actual operating outcomes. Each can answer useful questions, but none is a substitute for another.
The first line of a result should therefore identify its status. Mixing statuses inside one uninterrupted chart can create a false impression of continuity. If a methodology changes, the change date and affected period should be visible rather than silently rewriting the comparison.
- Historical or backtested: derived from past data and assumptions.
- Simulated or paper: observed without equivalent live execution and capital effects.
- Live: based on actual execution records for the named scope and period.
Define the observation window and denominator
Start and end dates, timezone, asset universe, venue coverage and inclusion rules determine what a metric means. A win rate, for example, can refer to orders, positions, days or completed strategies; those denominators are not comparable. A return can be gross or net, time-weighted or money-weighted, and expressed against different capital bases.
A useful publication defines the unit of analysis next to the metric, not in an inaccessible footnote. Missing observations, cancelled orders and partial outcomes should have an explicit treatment so that exclusions do not quietly favour the result.
- Name the period, timezone, instruments and included event types.
- Define the capital base and whether values are gross or net.
- State how missing, rejected, cancelled and partial observations are handled.
Costs and benchmark belong in the same frame
Fees, spreads, slippage, financing, subscriptions and market impact can materially change an outcome. In historical work these are estimates, not observed facts, and should be presented as assumptions. In live reporting they still require a defined reconciliation method.
A benchmark is useful only when its window, currency and exposure are comparable enough to answer the stated question. Choosing a reference after observing the result introduces selection bias. IMRYN’s publication rule is to explain why a benchmark was selected and where the comparison breaks down.
- Separate observed costs from estimated costs.
- Show the benchmark over the same dates and identify material exposure differences.
- Avoid implying that relative outperformance removes absolute loss or drawdown risk.
Return without a path hides risk
A final value does not show how the path behaved. Drawdown, volatility, concentration, time out of market, turnover and tail events provide different views of the risk taken. No single ratio captures every relevant failure mode.
Strategy selection and repeated testing can also overfit historical data. The more alternatives are tried, the more important it becomes to separate development data from evaluation data and to document what changed after seeing results.
- Pair outcome measures with drawdown and exposure context.
- Separate development, validation and later observation where practical.
- Record model, rule and parameter versions used for each measured interval.
Publication checklist and correction policy
A result is publication-ready only when a reader can identify its evidence status, scope, calculation method, costs, comparison basis and limitations. Charts should label axes and units, and narrative conclusions should stay within the questions the data can answer.
If an error is found, the correction should preserve the fact that a change occurred: what was corrected, when, why and whether the conclusion changed. This is more trustworthy than silently replacing a number. IMRYN does not publish placeholder returns or fabricate missing observations.
- Required: status, dates, denominator, costs, benchmark, risk context and limitations.
- Version material methodology changes and retain a correction note.
- Do not convert absence of data into zero, success or an estimated live result.
Evidence
Primary sources and further reading
These sources inform the general control and risk concepts on this page. They do not certify or endorse IMRYN.
- Mutual Funds, Past Performance
Investor.gov, U.S. Securities and Exchange Commission — Investor education on why past performance does not necessarily predict future results.
- Guidance on Effective Supervision and Control Practices for Algorithmic Trading Strategies
FINRA, Regulatory Notice 15-09 — Guidance covering testing, validation, monitoring, change control and records.
- AI Risk Management Framework Core
National Institute of Standards and Technology — Govern, map, measure and manage functions for continuous AI risk management.
FAQ
Questions answered directly
Does a backtest predict future performance?
No. It describes how defined rules behaved on selected historical data under assumptions. Future data, costs, liquidity and behaviour can differ.
Are simulated and live results equivalent?
No. Simulations and paper execution do not reproduce every live fill, queue, delay, market-impact, operational and capital constraint.
Why can two valid performance figures differ?
They may use different periods, capital bases, costs, inclusion rules, currencies or denominators. The methodology must define those choices.