Prediction Market Data Downloads: CSV, Parquet and API Exports

Match the format to the workload

Fields not to lose

• Venue, event, market and outcome identifiers in their native and normalized forms.
• Exchange/source time and receive/observation time as separate fields.
• Bid prices, bid sizes, ask prices and ask sizes at matching array positions.
• Capture method, interval or event sequence needed to interpret cadence.
• Settlement outcome and reference values stored without leaking into earlier rows.
• A stable null or missing convention that never silently becomes zero.

Partition for the questions you actually ask

Large depth archives should be partitioned by high-selectivity fields such as venue and date, then by a stable market or asset key where it matches the workload. A million tiny files can be as costly as one unpartitioned export, so measure the query engine rather than copying a generic lake layout.

Keep a manifest beside bulk exports. It should record schema version, generation time, requested filters, row or observation counts and any incomplete partitions. That manifest turns an opaque download into a reproducible research input.

DepthFeed delivery choices

DepthFeed's regular research path is REST for filtered historical windows and WebSocket for current normalized frames. Desk workflows can use the full archive and columnar delivery patterns for large studies. The browser tools use the same underlying schema, so an idea can move from a sample to a reproducible backtest without redefining the book.

Start with the smallest representative slice: one venue, one market window and enough observations to test parsing, ordering and a sized fill. Scale only after those invariants pass.

Two valid order-book file models

A nested snapshot model stores one observation per row with arrays for bid prices, bid sizes, ask prices and ask sizes. It is compact, preserves book boundaries and maps naturally to Parquet list columns or JSON. The consumer must validate that each price array matches its size array and that the arrays are correctly ordered.

A normalized levels model stores one row per observation, side and price level. It is convenient for SQL aggregation but multiplies row counts and requires a stable observation identifier to reconstruct a complete book. Neither model is universally better; the manifest should state which one is used and how deletions, empty books and partial observations are represented.

CSV schema for a small extract

Parquet layout for analytical workloads

For large historical-depth studies, Parquet usually reduces storage and scan cost because timestamps, identifiers and numeric arrays compress efficiently and query engines can skip unrelated partitions. Partition first by fields that consistently narrow reads, commonly venue and date, then test whether asset or market family improves the real workload.

Avoid encoding every market into its own tiny file. Small-file overhead can dominate query planning and object-store requests. Compact files to a measured target, keep row groups large enough for efficient scans, and publish checksums so a research run can prove exactly which objects it consumed.

Download quality-control checklist

• Compare manifest row counts with the sum of file or partition counts.
• Verify checksums before parsing and preserve the original manifest with the research output.
• Check timestamp monotonicity within each market without assuming all venues share one cadence.
• Reject mismatched price-size arrays, out-of-range prices and unexplained crossed books.
• Quantify duplicate observations, empty books, gaps and markets with no settlement mapping.
• Run a sample depth-walk and reconcile its VWAP against an independent implementation.
• Pin the schema version and record every transformation applied after download.

When to use a download, REST or WebSocket

Use a bulk download when the study repeatedly scans a large fixed historical range or must run close to the compute engine. Use REST when filters can substantially reduce the payload, when the workflow is exploratory, or when pagination and reproducible requests are more useful than local storage. Use WebSocket for current monitoring and forward collection, not as a substitute for an already retained archive.

Many serious workflows use all three: a versioned historical base in columnar files, REST for targeted refreshes and metadata, and WebSocket for the live tail. The critical design choice is a common identifier and ladder schema so the historical base and live continuation can be joined without rewriting the strategy.