Block tape and C ABI
study/block_tape.hpp stores recorded ticks in an append-only, columnar file. The writer batches rows and publishes a version:
pitlane::BlockTapeWriter w;
w.open("ticks.plt", 1); // symbol 1; creates the file
w.append(ticks, n); // copies rows into a staging block
w.commit(); // flushes the partial block, appends a commit, fsyncs
append writes a block whenever 1024 rows are staged. commit is the only call that fsyncs and the only one that publishes a version; a writer that never commits leaves no visible version. rows() and versions() report progress and the destructor closes the file.
pitlane::BlockTapeReader r;
if (!r.open("ticks.plt")) return 1; // bad magic, bad header, or truncated tail
pitlane::TapeMeta m{};
r.meta(m); // versions, blocks, symbol, rows, bytes, first/last ts
pitlane::TapeTick buf[256];
uint32_t n = 0;
r.rows(0, INT64_MIN, INT64_MAX, 0, 256, pitlane::kTapeColLast, buf, 256, n);
open validates the magic, the format version, the column and block-row counts, and the whole commit chain; a missing file, a different format, or a truncated or trailing-garbage tail fails. meta fills TapeMeta. describe(version, TapeStats&) fills version, rows, and per-column min, max, last, and mean (a double). load_quotes(version, Quote*, cap, n) reads a whole version for the backtester. blocks_read / reset_blocks_read report how many blocks the queries touched.
On-disk layout
| Part | Size | Contents |
|---|---|---|
| File header | 64 B | magic PLTAPE01, format version 1, 8 columns, 1024 block rows, symbol |
| Block | 64 B + 64 KiB | header (PLBLK001, row count, block sequence, first and last exch_us), then 8 columns × 1024 rows × 8 B in column-major order; unused tail rows are zero |
| Commit | 64 B | PLCMT001, version, cumulative rows, end offset, first and last exch_us, block count |
Blocks are immutable. A commit publishes version N over exactly the rows written so far; the next commit publishes N+1 over the same prefix. Version 0 means the latest, and a version in 1..versions never changes. Reads take no lock and never map the file: pread fills one column run of one block at a time, so a file truncated after open cannot raise SIGBUS.
rows() semantics
rows(version, since_us, until_us, offset, limit, cols, out, cap, n):
version0 is the latest; a value below 1 or aboveversionsfails.since_us/until_usare inclusive;INT64_MIN/INT64_MAXmean no bound. Blocks outside the range are skipped by their header timestamps and the edges are found by binary search inside a block.offsetandlimitapply after the time filter.limit0 copies up tocap, which is also the hard ceiling of one call.colsis a bitmask in on-disk order:kTapeColExch(bit 0) throughkTapeColLastQty(bit 7), withkTapeColAll(0xff); 0 selects all. Columns not selected keep whatever the output buffer held, so clear it when only part of a row is wanted.- The call returns false on a bad version or an I/O error;
nis the number of rows copied.
Timestamps must be non-decreasing, because the block header and the in-block binary search assume it.
tape_source adapters
study/tape_source.hpp defines TapeLoadFn and TapeSourceApi{name, load}, so research code can take a function pointer instead of a concrete file. tape_load_block loads a whole version into Quote rows and reports the version read; tape_tail_block reads only the last cap ticks for live-style replay. Both return false when the path is missing, is not a block tape, or does not fit.
C ABI
include/pitlane/capi/pitlane_c.h is an optional C surface built into libpitlane_c.so; it is never linked into the hot path. pl_tick, pl_meta, pl_stats, pl_query, pl_bt_cfg, and pl_bt_metrics are plain C structs; pl_tape is opaque.
| Function | Returns | Purpose |
|---|---|---|
pl_tape_open(path) |
pl_tape* |
Open and validate a tape; NULL on a missing file, bad magic, or truncated tail |
pl_tape_close(t) |
void |
Close and free the handle |
pl_tape_meta(t, out) |
1/0 | Header summary |
pl_tape_rows(t, q, out, cap, n) |
1/0 | Rows of q->version with time bounds, offset, limit, and projection |
pl_tape_describe(t, version, out) |
1/0 | Per-column statistics |
pl_tape_write(path, symbol, ticks, n) |
1/0 | Write n ticks as one committed version |
pl_backtest(path, cfg, out) |
1/0 | Read the whole tape and run Backtester |
pl_version() |
const char* |
Library version, for example "2.23.0" |
A ctypes caller maps the structs directly:
import ctypes
lib = ctypes.CDLL("build/libpitlane_c.so")
lib.pl_version.restype = ctypes.c_char_p
print(lib.pl_version().decode())
class Tick(ctypes.Structure):
_fields_ = [(name, ctypes.c_int64) for name in
("exch_us", "local_us", "bid_px", "ask_px",
"last_px", "bid_qty", "ask_qty", "last_qty")]
lib.pl_tape_open.argtypes = [ctypes.c_char_p]
lib.pl_tape_open.restype = ctypes.c_void_p
tape = lib.pl_tape_open(b"run/tape.plt")
python/pitlane does this and adds version, Tick, Tape, write_tape, and backtest; it is a cold-path research helper, so the core stays header-only.