PCAP market data is a raw capture of the binary packets the exchanges send to colocated subscribers of their direct market feeds. It is useful for research because it can be replayed offsite in order to run simulations.
The PCAP file format originated with tcpdump/libpcap.[ref]https://www.tcpdump.org/[/ref] It contains a header with software version info, followed by a series of packet records. Each packet record has a header with a timestamp and packet length, followed by the raw packet data.[ref]https://datatracker.ietf.org/doc/id/draft-gharris-opsawg-pcap-00.html[/ref] Packet sniffers like Wireshark use the pcap file format.[ref]https://www.wireshark.org/[/ref]
Chicago Mercantile Exchange
The CME sells historical PCAP data through their DataMine platform.[ref]https://datamine.cmegroup.com/#/datasets/cme.pcap[/ref] It is grouped into channels, where each channel contains some number of futures contracts. They deliver the data through automatic S3 transfers.[ref]https://www.cmegroup.com/market-data/automatic-s3-transfer.html[/ref] The CME uses a simple binary encoding called MDP 3.0.
The CME gives out sample data upon request.[ref]https://www.cmegroup.com/confluence/display/EPICSANDBOX/Packet+Capture+Dataset#PacketCaptureDataset-SampleFiles[/ref] You have to first configure your S3 bucket policy to allow the transfer. And then you email sales at CMEDataSales@cmegrop.com. I actually had a lot of trouble finding the location of the S3 bucket policy generation tool. I was only able to follow the instructions and find the S3 Transfer Settings page after adding some data to the cart. After that, everything went smoothly.
UPDATE: After six days, I finally got the sample data. I first had to speak with a sales person and explain my use case. Then they sent me three days’ worth of channel 310 data (E-mini), totaling about one gigabyte compressed. They also sent me secdef files for the same three days.
NASDAQ TotalView
ITCH v5.0 is the binary encoding used by the NASDAQ for their TotalView data. They kindly make it very easy to get sample data from a public web page.[ref]https://emi.nasdaq.com/ITCH/Nasdaq%20ITCH/[/ref]
NYSE OpenBook Ultra
The NYSE group of exchanges also makes it easy to download sample data.[ref]https://ftp.nyse.com/Real%20Time%20Data%20Samples/[/ref] Their data format is described in the XDP Integrated Feed Client Specification[ref]https://www.nyse.com/publicdocs/nyse/data/XDP_Integrated_Feed_Client_Specification_v2.3a.pdf[/ref] and the OpenBook Ultra Client Specification.[ref]https://www.nyse.com/publicdocs/nyse/data/OpenBook_Ultra_Client_Specification.pdf[/ref]