PCAP market data is a raw capture of the binary packets the exchanges send to colocated subscribers of their direct market feeds. I need at least a sample of it to replay during simulations.

The PCAP file format originated with tcpdump/libpcap. It contains a header with software version info, followed by a series of packet records. Each packet record has a header with a timestamp and packet length, followed by the raw packet data. Packet sniffers like Wireshark use the PCAP file format.

Chicago Mercantile Exchange

The CME sells historical PCAP data through their DataMine platform. It is grouped into channels, where each channel contains some number of futures contracts. They deliver the data through automatic S3 transfers. The CME uses a simple binary encoding called MDP 3.0.

After a fair bit of hassle and communicating with sales, I finally got three days’ worth of channel 310 data (E-mini), totaling about one gigabyte compressed. They also sent me secdef files for the same three days.

NASDAQ TotalView

ITCH v5.0 is the binary encoding used by the NASDAQ for their TotalView data. They kindly make it very easy to get sample data from a public web page.

NYSE OpenBook Ultra

The NYSE group of exchanges also makes it easy to download sample data. Their data format is described in the XDP Integrated Feed Client Specification and the OpenBook Ultra Client Specification.