Open a section from the left

Block tape and C ABI

study/block_tape.hpp stores recorded ticks in an append-only, columnar file. The writer batches rows and publishes a version:


pitlane::BlockTapeWriter w;
w.open("ticks.plt", 1);          // symbol 1; creates the file
w.append(ticks, n);              // copies rows into a staging block
w.commit();                      // flushes the partial block, appends a commit, fsyncs

append writes a block whenever 1024 rows are staged. commit is the only call that fsyncs and the only one that publishes a version; a writer that never commits leaves no visible version. rows() and versions() report progress and the destructor closes the file.


pitlane::BlockTapeReader r;
if (!r.open("ticks.plt")) return 1;           // bad magic, bad header, or truncated tail
pitlane::TapeMeta m{};
r.meta(m);                                     // versions, blocks, symbol, rows, bytes, first/last ts
pitlane::TapeTick buf[256];
uint32_t n = 0;
r.rows(0, INT64_MIN, INT64_MAX, 0, 256, pitlane::kTapeColLast, buf, 256, n);

open validates the magic, the format version, the column and block-row counts, and the whole commit chain; a missing file, a different format, or a truncated or trailing-garbage tail fails. meta fills TapeMeta. describe(version, TapeStats&) fills version, rows, and per-column min, max, last, and mean (a double). load_quotes(version, Quote*, cap, n) reads a whole version for the backtester. blocks_read / reset_blocks_read report how many blocks the queries touched.

On-disk layout

Part Size Contents
File header 64 B magic PLTAPE01, format version 1, 8 columns, 1024 block rows, symbol
Block 64 B + 64 KiB header (PLBLK001, row count, block sequence, first and last exch_us), then 8 columns × 1024 rows × 8 B in column-major order; unused tail rows are zero
Commit 64 B PLCMT001, version, cumulative rows, end offset, first and last exch_us, block count

Blocks are immutable. A commit publishes version N over exactly the rows written so far; the next commit publishes N+1 over the same prefix. Version 0 means the latest, and a version in 1..versions never changes. Reads take no lock and never map the file: pread fills one column run of one block at a time, so a file truncated after open cannot raise SIGBUS.

rows() semantics

rows(version, since_us, until_us, offset, limit, cols, out, cap, n):

Timestamps must be non-decreasing, because the block header and the in-block binary search assume it.

tape_source adapters

study/tape_source.hpp defines TapeLoadFn and TapeSourceApi{name, load}, so research code can take a function pointer instead of a concrete file. tape_load_block loads a whole version into Quote rows and reports the version read; tape_tail_block reads only the last cap ticks for live-style replay. Both return false when the path is missing, is not a block tape, or does not fit.

C ABI

include/pitlane/capi/pitlane_c.h is an optional C surface built into libpitlane_c.so; it is never linked into the hot path. pl_tick, pl_meta, pl_stats, pl_query, pl_bt_cfg, and pl_bt_metrics are plain C structs; pl_tape is opaque.

Function Returns Purpose
pl_tape_open(path) pl_tape* Open and validate a tape; NULL on a missing file, bad magic, or truncated tail
pl_tape_close(t) void Close and free the handle
pl_tape_meta(t, out) 1/0 Header summary
pl_tape_rows(t, q, out, cap, n) 1/0 Rows of q->version with time bounds, offset, limit, and projection
pl_tape_describe(t, version, out) 1/0 Per-column statistics
pl_tape_write(path, symbol, ticks, n) 1/0 Write n ticks as one committed version
pl_backtest(path, cfg, out) 1/0 Read the whole tape and run Backtester
pl_version() const char* Library version, for example "2.23.0"

A ctypes caller maps the structs directly:


import ctypes

lib = ctypes.CDLL("build/libpitlane_c.so")
lib.pl_version.restype = ctypes.c_char_p
print(lib.pl_version().decode())

class Tick(ctypes.Structure):
    _fields_ = [(name, ctypes.c_int64) for name in
                ("exch_us", "local_us", "bid_px", "ask_px",
                 "last_px", "bid_qty", "ask_qty", "last_qty")]

lib.pl_tape_open.argtypes = [ctypes.c_char_p]
lib.pl_tape_open.restype = ctypes.c_void_p
tape = lib.pl_tape_open(b"run/tape.plt")

python/pitlane does this and adds version, Tick, Tape, write_tape, and backtest; it is a cold-path research helper, so the core stays header-only.