Prediction Market Bot Backtesting: From Replay to Paper Trading

The test ladder

Model fills before optimizing signals

A buy crosses the ask and a sell crosses the bid. If the requested size exceeds the first level, walk the ladder and compute VWAP; if the ladder cannot support the order, record the unfilled fraction. A midpoint fill can remain as a best-case ceiling, but it should never be confused with executable performance.

Fees, latency, queue position and market impact require separate assumptions. Keep each assumption visible so a result can be reproduced rather than hiding all execution error inside one unexplained slippage constant.

Use one rule across Polymarket and Kalshi

DepthFeed normalizes recorded Polymarket and Kalshi books into one research shape while preserving venue identifiers and source metadata. The Backtest Lab can run presets or a sandboxed custom rule over resolved markets with midpoint, fixed-slippage or book-depth fills.

A surviving rule can move to Paper Trading. DepthFeed evaluates supported rules server-side or accepts signals from a per-strategy webhook, prices virtual fills from the live book and records an equity curve, open positions and trades. Sports backtests are supported; forward sports paper execution is not available yet.

Bot validation checklist

• Freeze a market universe and time range before tuning.
• Keep outcome and settlement fields unavailable to the entry function.
• Compare midpoint, fixed-slippage and depth-aware fills.
• Report unfilled size, missing books and rejected trades.
• Run sensitivity tests around every threshold and holding rule.
• Forward-test the final rule before connecting real execution.

Prevent look-ahead and survivorship bias

At each simulated decision, expose only fields whose source or observation time is no later than that decision. Settlement, final outcome, later status edits and post-event metadata must be unavailable to the entry function. An as-of join should select the latest preceding reference value, never the nearest value on either side.

Freeze the market universe using information available at the start of each test period. Testing only contracts that later became liquid, settled cleanly or remained in a current catalog creates survivorship bias. Keep inactive, failed and missing markets visible in the sampling report even when they are excluded for a stated reason.

Use train, validation and forward periods

Develop thresholds on a training window, make model and parameter choices on a separate validation window, and report the final historical result on untouched data. For short-lived prediction markets, split by event time or market family rather than randomly mixing observations from the same event across folds.

A final forward paper period is still necessary because all historical choices were influenced by the archive. Lock the code, parameters, market filters, fee assumptions and data version before the forward run begins. Changes during that period create a new experiment and should reset the clock.

Metrics a trading-bot backtest should report

Sensitivity and falsification tests

Move every important entry threshold, exit threshold, holding period, fee and latency assumption across a reasonable neighborhood. A genuine effect should degrade gradually; a result that exists at one exact value is more likely overfit. Re-run by venue, asset, market duration, liquidity bucket and calendar period to expose concentration.

Add deliberately hostile tests. Delay the signal, worsen execution by one or more ticks, cap available size, remove the best-performing day, and randomize entries while preserving trade frequency. The purpose is not to make every strategy fail; it is to learn which assumption is carrying the result.

Position sizing and portfolio constraints

A per-trade edge is not a portfolio. Model cash locked until settlement, simultaneous positions, correlated markets, maximum venue exposure and limits on one event or outcome family. Reject orders that exceed the stated capital rule instead of letting simulated leverage appear from nowhere.

Use filled quantity for exposure and PnL, not requested quantity. Mark open positions with a documented bid, ask, midpoint or liquidation rule, and show how that choice changes drawdown. Binary payouts cap upside per share but do not remove concentration, liquidity or timing risk.

From backtest to paper trading to live approval

Paper trading should consume the same live identifiers, book schema, signal format and position rules intended for production. Record rejected signals, unavailable size, actual observation age and the difference between expected and paper fills. Compare the forward distribution with the historical one before considering live execution.

If live routing is later connected, begin with human confirmation, strict notional limits, kill switches and independent reconciliation. A profitable backtest is evidence about one historical simulation. It is not evidence that credentials, order management, venue rules or operational recovery are safe.

Minimum experiment record

• Strategy code hash, parameter set and runtime version.
• Data query, venue, market universe, time range and schema version.
• Decision clock, source fields and explicit anti-leakage allowlist.
• Fill model, order size, fees, latency, queue and market-impact assumptions.
• Missing-data, duplicate, maintenance-window and rejection policies.
• Train, validation, test and forward-paper boundaries.
• Trade ledger, equity curve, aggregate metrics and excluded-market report.
• Author, run time and a permanent identifier for reproducing the experiment.

Example end-to-end bot experiment

Suppose a rule buys a binary outcome when a model probability exceeds the executable ask by five percentage points. Define the model version and feature clock, choose a frozen market universe, and calculate the signal only from preceding observations. The order enters at a fixed share size and walks the recorded ask ladder. If less than the required minimum fraction is available, reject it rather than shrinking only the losing trades after the fact.

Apply fees, hold or exit under a written rule, and settle from the official outcome after the trading period ends. Report midpoint and depth-aware results side by side, plus fill rate, slippage, rejected orders, exposure, drawdown and performance by liquidity bucket. Tune on the training set, choose once on validation, and evaluate untouched test markets.

Finally, lock the version and run it in paper trading. Compare signal frequency, observation freshness, fill rate and net outcome with the historical distribution. If the live environment behaves differently, investigate that mismatch before changing parameters. Any change begins a new paper experiment.

Preserve venue-specific mechanics

A shared normalized book does not make Polymarket and Kalshi interchangeable. Preserve native market identifiers, price units, side conventions, close times, settlement rules, maintenance behavior and capture method. Strategy logic may be portable while loaders, eligibility filters and reconciliation remain venue-aware.

Run venue-level results before combining them. Differences in market duration, liquidity, polling cadence, fees and settlement can make a pooled strategy look stable when it is carried by one venue. Cross-venue features also need strict time alignment; a quote observed later cannot be used to trade an earlier book on the other venue.

If a strategy trades related contracts across venues, size each leg against its own displayed ladder and report incomplete hedges. A simulated simultaneous fill is an assumption, not an observed fact. Add explicit delay and failure scenarios before describing the result as an arbitrage test.