StormByte-Buffer 4.0.0
C++26 buffer module of the StormByte suite
Loading...
Searching...
No Matches
StormByte-Buffer

Platform C++26 CMake License: LGPL v3 or commercial CI Sponsor

This repository is StormByte Buffer: byte streams, buffered I/O and typed item queues for the StormByte C++ suite. Those are three different things. A byte buffer does not carry a C++ object, and a Hopper does not carry octets.

It uses StormByte Base 3.0.0, StormByte-System 3.0.0 and StormByte-Logger 3.0.0. Base is downloaded on its own. Logger does not bring it. Base provides the owned UTF-8 and wide text types, the octet payload, and the wait primitives. Logger is used for pipeline pipes (Scope). Public headers live under StormByte/buffer/.

The suite is split on purpose. Base, Config, Crypto, Database, Logger, Multimedia, Network and System are other repositories. This one does not implement them.

Designed to interconnect

Pieces plug into each other only inside their own group.

  • Byte tips share ReadOnly / WriteOnly. Producer yields a Consumer over the same Ring. Pipeline transforms those stream buffers only. Bridge moves octets from a ReadOnly or an IO reader into a WriteOnly or an IO writer. Pumper owns a Bridge and runs Passthrough until EoF or failure.
  • IO leaves are also byte tips. BufferedFileReader is a BufferedReader. BufferedFileWriter is a BufferedWriter. A Bridge can steal a leaf by move. A Pipeline cannot take a leaf.
  • Hopper and Sink take C++ items. They do not read or write Safe::Binary, and a byte buffer is not a Hopper.

Typical wires:

  • Producer → Consumer (same ring).
  • Producer → Pipeline::Process → Consumer.
  • That Consumer → Bridge::Passthrough → file (or wrap the Bridge in a Pumper).
  • BufferedFileReader / BufferedFileWriter for seekable files with a page map.
  • Sink::To(key) >> other for a typed item, never for a byte chunk.

See Bytes, IO and Items.

What this module does

The rest of the suite

Module Role API
Base Exceptions, Expected, serialization, UUID, Safe /StormByte
Buffer This repository /StormByte-Buffer
Config Human-readable text and versioned binary documents /StormByte-Config
Crypto Hash, compress, encrypt, sign — Crypto++ stays private /StormByte-Crypto
Database One API over SQLite, PostgreSQL and MariaDB /StormByte-Database
Logger Stream logger with levels, headers, components and Scope /StormByte-Logger
Multimedia Decode, encode and containers without raw FFmpeg types /StormByte-Multimedia
Network Framed packets, Client/Server, IPv4/IPv6 TCP /StormByte-Network
System Processes, pipes, Device, host and environment /StormByte-System

Table of Contents

Documentation

DLL Boundary Contract (2.0.0)

DLL safety is conditional, not a promise of compatibility between different compilers or STL ABIs. MaybeSafe registrations apply to exact types only: the Generic / ReadOnly / WriteOnly / ReadWrite interfaces, Producer, Consumer, Bridge, Pumper, Pipe, Pipeline, buffered IO bases and file leaves, telemetry types and operation samples, IO Result and knobs, and Pumper::Parameters. Hopper<T> and Sink<T> are partial specializations of IsMaybeSafe. The exact-type macro does not accept a class template. Derived types do not inherit certification; each concrete provider and its owned state need their own audit and declaration.

Keep Buffer, Base and every participating concrete, template, element and callback provider loaded until their objects, owners and callbacks are released. Logger must also remain loaded while pipeline logger handles are used. Inline template callbacks are not automatically provider-local. Stop and join all users before destroying queues or their borrowed state; allocation failure in nonthrowing operations terminates the process.

Ring constructs, moves and destroys its private deque and synchronization state in Buffer; SharedFIFO constructs and destroys its mutex / condition variable out of line. Range constructors convert to Base-owned Safe::Binary in the caller and delegate to the provider constructor. This keeps private storage lifecycle in its provider, without transferring STL container ownership or certifying derived buffers. Pipe callable state is allocated with Safe::Heap::Allocate and released by the creator callback. It is not a std::unique_ptr.

Protected FIFO::HexDumpHeader() and Ring::HexDumpHeader() return StormByte::Safe::String, not std::ostringstream. Custom overrides must use the new return type; stream formatting stays local to the implementation.

Public headers do not export std::mutex, std::condition_variable, std::atomic or std::thread. Hopper, Sink, SharedFIFO, Ring and the IO engines wait on Safe::Mutex, Safe::ConditionVariable and Safe::Atomic. Notify borrows the caller's Safe::ConditionVariable and, in the stored-event overload, a Safe::Atomic<std::size_t>. A worker created by this module is a Safe::Thread, created and joined here. A thread you start in your own .cxx is yours. Do not pass a std wait object into Notify.

Installation

-DENABLE_TEST=ON keeps normal tests available locally and during PGO training. Package discovery fixtures are CI-only and require -DENABLE_CI_FIXTURES=ON, which is disabled by default and enabled explicitly by CI.

Needs a C++26 compiler, CMake 3.28 or newer, StormByte-BuildMaster >= 2.1.5, StormByte Base 3.0.0, StormByte-System 3.0.0 and StormByte-Logger 3.0.0. Configure downloads those release archives. There are no StormByte source submodules. BuildMaster is the installed package when it is >= 2.1.5, otherwise the buildmaster submodule.

git clone --recursive https://github.com/StormByte-Suite/StormByte-Buffer.git
cd StormByte-Buffer
cmake -S . -B build
cmake --build build

Shared vs static follows CMake BUILD_SHARED_LIBS (declared in lib/, default ON). A plain configure builds the shared library. -DBUILD_SHARED_LIBS=OFF builds a static archive; on Windows the headers then do not use dllimport. The downloaded archives are the same mode.

A shared build keeps this library as its own .so / .dll. Under the LGPL that is usually the simpler way to ship: the user can replace that file. A static archive is folded into your binary. The LGPL still applies to this code; you must give the recipient a way to relink your product with a different build of this library. If that does not fit how you distribute the final product, a commercial license is available from the copyright holder (see License).

For installed Base, Logger, System and Buffer prefixes, add each prefix's share/cmake/Modules directory to CMAKE_MODULE_PATH and the prefixes to CMAKE_PREFIX_PATH, then use the component finder:

find_package(StormByte REQUIRED COMPONENTS Base Buffer)
target_link_libraries(app PRIVATE StormByte::Buffer)

The Buffer finder is installed in share/cmake/Modules and discovers Logger and System as dependencies. You can also link the library as StormByte-Buffer. Headers: #include <StormByte/buffer/….hxx>.

Bytes

These types move octets. The payload is StormByte::Safe::Binary. A length is StormByte::ByteSize. FIFO is the plain buffer. SharedFIFO and Ring add concurrency. Producer and Consumer are the write and read tips of one ring. Pipeline transforms those tips. Bridge copies octets from a read tip to a write tip, and can steal an IO leaf. Pumper owns that Bridge and runs it. None of them stores a T.

Namespace root is StormByte::Buffer.

FIFO

int main() {
FIFO fifo;
fifo.Write("Hello World");
Binary data;
fifo.Read(5, data); // "Hello", still in the buffer
fifo.Seek(6, Position::Absolute);
Binary extracted;
fifo.Extract(5, extracted); // "World"
}
Byte-oriented FIFO buffer with grow-on-demand and close/error support.
Definition fifo.hxx:86
Position
Forward declaration of the WriteOnly interface.
Definition typedefs.hxx:74

FIFO is not thread-safe. Concurrent ends use SharedFIFO or Ring. Read / Peek keep data; Extract consumes it.

Producer and Consumer

A Producer yields a Consumer over the same ring. The Consumer is a ReadOnly; the Producer is a WriteOnly. Either tip can be passed to a Bridge. Ring ownership uses opaque StormByte::Safe::Owner callbacks implemented in Buffer, not std::shared_ptr<Ring> in the public ABI. The owner and any Producer/Consumer copy can outlive its handle, but the Buffer and Base modules must stay loaded until all handles are destroyed. This conditional contract does not certify arbitrary cross-module payloads or compiler/STL ABI compatibility.

The threads in this example are the caller's. They are not part of the Buffer ABI.

#include <thread>
int main() {
Producer producer;
Consumer consumer = producer.Consumer();
std::thread writer([producer]() mutable {
producer.Write("Data chunk 1");
producer.Write("Data chunk 2");
producer.Close();
});
std::thread reader([consumer]() mutable {
while (!consumer.EoF()) {
Binary data;
if (consumer.Extract(0, data) && !data.empty()) {
// process
}
}
});
writer.join();
reader.join();
}
Read-oriented handle over a shared Ring.
Definition consumer.hxx:80
Write-only handle over a shared Ring.
Definition producer.hxx:71

Pipeline

Pipeline transforms stream buffers (ReadOnly / WriteOnly). It does not take IO leaves. A file or device is attached later with a Bridge.

A Pipe is a nonpolymorphic value owning a copyable Safe::Function. Construct it from a copyable lambda or callable taking const PipeInput&, const PipeOutput& and const Safe::Shared<Logger::Log>&. Mutable value captures are supported. Add(const Pipe&) deep-copies the callable context without changing the caller; Add(Pipe&&) transfers it. Copying or assigning a Pipeline independently copies its stages, not its buffers or workers. Reference captures and shared handles retain their normal sharing semantics. Do not copy a running pipeline while callbacks mutate their captures; externally serialize owner operations and callback-state access.

PipeInput borrows a live ReadOnly; PipeOutput borrows a live WriteOnly. Their const functions forward stream operations to those endpoints. Read(count, bytes) advances the input cursor, with zero meaning all currently available bytes; Write(bytes) copies or moves binary data and Write(text) uses a length-aware string view, including Base Safe::String's view conversion. Close or SetError the output before returning. Pipeline::Process(buffer, log, mode) keeps mode last and supplies a scoped logger when present.

The endpoint facades, all their copies, and the logger reference are synchronous borrows valid only until invocation returns. Never retain them in captures, background work or handles. Their exact-type MaybeSafe declarations admit the explicit facade contract; they do not automatically certify the lifetime of an arbitrary raw pointer or underlying endpoint. Const forwarding is not a thread-safety guarantee. Pipeline keeps its endpoints alive for invocation, including on background workers. The callback provider, Buffer, Base and Logger must remain loaded with compatible C++/STL ABI until all callbacks and workers are released.

The callable factory constructs captures with Safe::Heap::Allocate in the creator module and attaches matching creator-side invoke, clone and release callbacks. For explicitly controlled DLL providers, construct Pipe::Callback(context, invoke, clone, release) and move it into Pipe. Clone must independently copy context in that provider and return null on failure; release must destroy captures there and free storage with Safe::Heap::Free. Template instantiations and externally defined callable types still require a provider-locality audit; this is not compatibility across arbitrary runtimes. Run returns Safe::Status: Failure, moved-from Missing, Safe exceptions and foreign exceptions cause Pipeline to mark every output errored and wake waiters. Nothing escapes Process's noexcept boundary.

To migrate an old derived stage, replace its Run implementation with a callable using the facades, remove inheritance and virtual Clone/Move, and construct a Pipe from that callable. Replace endpoint Extract calls with a Read loop and Write(count, bytes) with Write(bytes) when writing the whole chunk. Move-only captures are intentionally rejected to preserve Pipeline copy semantics.

Pipe MakeStripCrPipe() {
return Pipe([](const PipeInput& in, const PipeOutput& out,
while (!in.EoF()) {
Binary raw;
if (!in.Read(0, raw))
break;
Binary unix_newlines;
unix_newlines.reserve(raw.size());
for (const std::byte octet : raw) {
if (octet != std::byte{'\r'})
unix_newlines.push_back(octet);
}
out.Write(std::move(unix_newlines));
}
out.Close();
});
}
int main() {
Producer src;
src.Write("a\r\nb\r\n");
src.Close();
Pipe crlf = MakeStripCrPipe();
Pipeline pipe;
pipe.Add(crlf);
Consumer result = pipe.Process(src.Consumer(), {}, ExecutionMode::Sync);
(void)result;
}
Synchronous, non-owning stage input borrow.
Definition pipe.hxx:71
Synchronous, non-owning stage output borrow with const mutation forwarding.
Definition pipe.hxx:115
One transformation in a Pipeline.
Definition pipe.hxx:199
Multi-pipe byte transformation over stream buffers.
Definition pipeline.hxx:95
ExecutionMode
Bitmask controlling how Pipeline::Process schedules work.
Definition typedefs.hxx:106

Sync runs on the caller thread. Async returns at once. Parallel is one thread per pipe. Combine with |. The pipeline worker is a Safe::Thread created and joined in this module.

Bridge

Bridge is a manual transfer. Bytes move only when you call Bridge::Passthrough. There is no worker and no occupancy cap here. Continuous transfer is Pumper.

  • In-memory tips: ReadOnly& / WriteOnly&. Those buffers must outlive the Bridge.
  • IO tips: stolen by move as the concrete leaf (BufferedFileReader, BufferedFileWriter, …).
  • Passthrough(n, Operation) is atomic. Write TryAgain is retried until that call completes.
  • Operation applies to the read tip only: Blocking waits for n or EoF; NonBlocking takes what is available now, up to n.
  • n == 0 is current contents (Available() on IO does not touch the origin).
  • State is Open, Closed or Failed. Failed() is only a real tip fault. A moved-from Bridge is Closed, not Failed.
  • Close() ends the session. Borrowed in-memory tips are detached and stolen IO leaves are released. After Close, a consumed source EoF, or a move-from, the instance is Closed and cannot be re-armed. Construct a new Bridge to transfer again. Close() is idempotent and does not set Failed.
  • A NonBlocking Passthrough that returns 0 is not the end of the session.
  • Telemetry: the Bridge always keeps a Shared copy of the read and write counters. Non-IO tips use a basic telemetry owned and updated by the Bridge. IO tips donate the leaf handle at attach; the leaf updates that object. After Close the same handles remain valid. A Shared you already copied stays valid when the Bridge dies.

On Windows an IO leaf keeps the origin handle until that leaf is destroyed. While the Bridge still owns a BufferedFileReader or a BufferedFileWriter, the path stays locked: DeleteFile / std::filesystem::remove fail. In-memory tips do not lock a file. Call Bridge::Close() when you are done so the stolen IO origin is closed and the path can be unlinked. The destructor does the same work; Close is for when *this must stay alive.

#include <filesystem>
int main() {
FIFO in;
in.Write("payload");
in.Close();
BufferedFileWriter out("out.bin");
out.Open();
Bridge bridge(in, std::move(out));
while (!bridge.EoF() && !bridge.Failed()) {
if (bridge.Passthrough(64 * 1024) == 0 && !bridge.EoF())
break;
}
auto read = bridge.ReadTelemetry();
bridge.Close();
(void)read; // still valid
std::filesystem::remove("out.bin");
}
Private pump for StormByte::Buffer::Bridge.
Manual bridge between two StormByte-Buffer ends.
Definition bridge.hxx:129
Final BufferedLocationWriter over a filesystem file.
Definition buffered_file_writer.hxx:101
virtual Result Write(const FIFO &src) final
Write every unread byte of src.

A Pipeline is not a Bridge tip. Pipeline::Process is. It returns a Consumer, and a Consumer is ReadOnly, so it can be the source of a Bridge. The stage factory MakeStripCrPipe is the one from Pipeline.

int main() {
Producer capture;
capture.Write("line 1\r\nline 2\r\n");
capture.Close();
auto crlf = MakeStripCrPipe();
Pipeline pipe;
pipe.Add(crlf);
Consumer normalized = pipe.Process(capture.Consumer(), {}, ExecutionMode::Sync);
BufferedFileWriter out("session.log");
out.Open();
Bridge to_disk(normalized, std::move(out));
while (!to_disk.EoF() && !to_disk.Failed()) {
if (to_disk.Passthrough(4 * 1024) == 0 && !to_disk.EoF())
break;
}
to_disk.Close();
}

session.log holds line 1\nline 2\n. For a long capture, wrap to_disk in a Pumper instead of the Passthrough loop. The file stays locked while that Pumper is alive; ~Pumper joins and then releases the owned Bridge.

Pumper

Pumper takes a Bridge by move and starts a worker immediately. The destructor joins and may block until the current cycle ends. It does not Cancel: it lets the worker finish. That worker is a Safe::Thread.

  • Chunk — bytes asked of Passthrough each cycle. 0 is automatic chunking, not Bridge “current contents”.
  • HighWater — input cap only. Omitted: 0 if the source is IO, otherwise a backend default (constexpr in the PIMPL). Explicit 0: no Pumper cap. Use 0 when the IO source already limits itself. Non-IO sources are unbounded by design.
  • Toggle pauses and resumes. No-op if Failed or Canceled.
  • Cancel is terminal (Canceled, no restart). It does not set Failed. It Close()s the owned Bridge so Windows can unlink an IO path.
  • Failed() is only a real Bridge fault. A moved-from Pumper is empty (EoF), not Failed and not Canceled.
  • Telemetry handles are copied from the Bridge at construction. They stay valid after Cancel. A Shared you already copied stays valid when the Pumper dies.

Parameters::Chunk() and Parameters::HighWater() return StormByte::Safe::Optional<StormByte::ByteSize>. An empty optional means omitted, not an engaged zero. Omitted or zero Chunk selects automatic chunking; omitted HighWater selects the input-kind default, while an engaged zero explicitly disables the cap.

An IO path stays locked while the Pumper is alive. After Cancel (or after ~Pumper joins and destroys the Bridge) the handle is released.

int main() {
BufferedFileReader in("in.bin");
BufferedFileWriter out("out.bin");
in.Open();
out.Open();
Pumper pump(Bridge(std::move(in), std::move(out)), { Chunk{1 << 20} });
while (!pump.EoF() && !pump.Failed() && !pump.Canceled()) {
// work elsewhere; pump runs on its thread
}
auto read = pump.ReadTelemetry();
(void)read; // still valid after ~Pumper
// ~Pumper joins
}
Private worker for StormByte::Buffer::Pumper.
Pumper cycle size.
Definition pumper.hxx:82
Input occupancy cap for Pumper.
Definition pumper.hxx:114
Final BufferedLocationReader over a filesystem file.
Definition buffered_file_reader.hxx:98
Owns a Bridge and moves bytes until EoF, failure or Cancel.
Definition pumper.hxx:176

finish = !pump.Failed() && !pump.Canceled() && pump.EoF(). active = !pump.Failed() && !pump.Canceled() && !pump.EoF().

IO

IO is the buffered file-like hierarchy. It still moves octets, but the public type is not a FIFO. A leaf is a reader or a writer with a page cache, a logical cursor and an origin hook.

The chain is fixed:

  • BufferedReader is the read engine. Cache, prefetch, logical Seek, Tell and telemetry live here. A custom origin implements the Origin* hooks.
  • BufferedLocationReader sits on that engine. A location is named, always seekable and sized. Device() returns a Safe::Shared<System::Device> so a subclass is not sliced.
  • BufferedFileReader is the final file leaf. It opens a path, probes the device and uses the location layer.
  • BufferedWriter, BufferedLocationWriter and BufferedFileWriter are the write side of the same three levels.

A Pipeline cannot take one of these. A Bridge can, by move.

Parameters

IO leaves and Pumper take a nested Parameters object. Omitted knobs keep the office default. Brace-init and a named bag are both valid. Bags are header-only and store StormByte::Safe::Optional values; ownership and release use the storage provider's callbacks, not an exposed STL optional. IO constructors resolve the knobs into numbers, durations and probe flags. Pure knob and bag functions use ordinary inline definitions.

IO byte-count getters return const Safe::Optional<ByteSize>&; BackPressure() returns const Safe::Optional<std::size_t>&. The bag's MaxWait() returns const Safe::Optional<std::chrono::milliseconds::rep>&, a signed millisecond count. Resolve it with std::chrono::milliseconds{p.MaxWait().value_or(0)}. MaxWait{std::chrono::milliseconds{...}} still accepts durations, and live reader/writer MaxWait() getters still return durations. Negative counts are preserved; an absent knob remains distinct from an engaged zero (unlimited wait).

Bag construction, copying and knob assignment may allocate and throw. The empty file-leaf Parameters constructor is not noexcept. Copy operations are not noexcept; moves and destruction remain nonthrowing and transfer or release Safe-owned state. Keep the storage provider loaded until all bags and copies are released.

BufferedFileReader in("in.bin"); // probe at Open
BufferedFileReader in2("in.bin", { ReadAhead{1 << 20} }); // one knob
BufferedFileReader::Parameters p{ ReadAhead{1 << 20}, MaxMemory{8 << 20} };
BufferedFileReader in3("in.bin", p);
Cache / dirty-page budget knob.
Definition parameters.hxx:304
Prefetch length knob.
Definition parameters.hxx:276

Explicit knobs do not request a device probe (except that ReadChunk alone does not disable the existing ReadAhead probe). ReadChunk{0} selects its 64 KiB default; the other zero meanings are listed below.

Transfer limits and live configuration

These are Buffer origin-hook request sizes, not physical disk operations. No device profiles or new detection are involved.

Knob Omitted / zero Meaning
ReadChunk 64 KiB / 64 KiB Maximum OriginPull request, including direct reads and prefetch. With a nonzero page budget the effective limit is min(ReadChunk, MaxMemory).
ReadAhead Existing base/location/file defaults / disables prefetch Bytes to anticipate, independent of transfer size.
WriteTransfer Unlimited / unlimited Maximum OriginPush request in direct, ring, page, eviction, Flush and Close paths. Short writes are retried using requests no larger than this maximum.
WriteChunk Existing defaults / disables ring Legacy ring unit and drain trigger, not a universal transfer maximum.
BackPressure Existing defaults / disables ring Logical ring admission capacity is WriteChunk * BackPressure, checked for overflow.
WriteThreshold 0 / memory-pressure drains only Page high watermark, clamped to MaxMemory. At or above it, drain in offset order to at most half that watermark.
MaxMemory Existing defaults / no page cache Payload-byte budget of reader pages or writer dirty pages, not an RSS limit. Pages take priority over the ring.
MaxWait 0 ms / unlimited Wait limit; negative or overflowing durations are rejected by configuration. A blocking origin hook must implement its own timeout/cancellation.

Writer pages accumulate contiguous small writes up to min(MaxMemory, WriteTransfer) (64 KiB in place of unlimited WriteTransfer). They are bounded segments rather than an ever-growing contiguous allocation. Drain pages in the calling thread, with the state mutex released during origin I/O. Only the legacy ring uses a background writer. A request larger than the page budget is processed incrementally; it never waits for the entire request to fit. Memory pressure drains without waiting for the threshold; Flush, Close and reconfiguration drain all tails. Sparse offsets and overlapping writes retain their exact byte contents. Fragmented or short contiguous ranges can produce transfers smaller than the maximum; the maximum is not a promise of full-sized transfers.

using namespace StormByte::Buffer::IO;
BufferedFileReader in("input.bin",
{ReadChunk{256 * 1024}, ReadAhead{1024 * 1024}, MaxMemory{4 * 1024 * 1024}});
BufferedFileWriter out("output.bin",
{WriteTransfer{256 * 1024}, WriteThreshold{1024 * 1024},
MaxMemory{4 * 1024 * 1024}});
// These numbers are caller policy, not an NVMe/SATA/HDD profile.
auto policy = out.Configuration();
policy.WriteTransfer = StormByte::ByteSize{128 * 1024};
policy.MaxMemory = StormByte::ByteSize{2 * 1024 * 1024};
const auto changed = out.Configure(policy);
if (changed.status != Status::Ok) {
// Handle the error: policy is unchanged, but completed origin I/O is not rolled back.
}
const auto effective = out.Transfers(); // no origin I/O
WriterConfiguration Configuration() const noexcept
Read the complete policy without I/O.
Maximum OriginPull request.
Definition parameters.hxx:78
Page drain high watermark.
Definition parameters.hxx:182
Maximum OriginPush request in every mode.
Definition parameters.hxx:130
Buffered binary sources and sinks.
@ Ok
The call completed as requested (see Result::count).
Definition typedefs.hxx:80
StormByte::ByteSize WriteTransfer
Universal push maximum; zero is unlimited.
Definition parameters.hxx:249

ReaderConfiguration / WriterConfiguration are complete, non-owning, by-value policies; Configuration() returns a coherent requested-policy snapshot. MaxWaitMilliseconds is a scalar millisecond count, not a clock time point or an owning STL value crossing the DLL. Start from Configuration() when preserving unspecified fields: a default-constructed complete policy does not inherit a file/location's constructor defaults. Configure() validates first, waits for active session operations, cancels reader prefetch or flushes writer data with the old policy, and publishes all fields together. A replacement ring really adopts the new capacity. Allocation or flush failure leaves the old policy installed (the origin may already contain part of the old pending data). An I/O failure faults the session; it is not reported as a successful configuration. Legacy void setters and the new scalar setters use this operation and throw Buffer::Exception on failure. They may block; use Configure() for an explicit result and multi-field changes. Writer changes, including MaxWait, flush the old policy. Reader configuration does not pull.

Session-mutating calls on one coordinator are serialized; a per-session operation gate can wait across I/O, but the state mutex is never held across origin hooks. Configuration(), Transfers(), cursor/dirty queries and existing counters do not take that gate or perform I/O. Do not reenter session operations from an origin hook. Moving/destroying an instance concurrently with its users remains unsupported.

Transfers() reports real hook calls (including zero-byte retries), actual returned/transferred bytes, last and maximum requested size, effective transfer limit, page budget, physical ring storage capacity, logical queue admission capacity, effective page watermark, pending bytes, and requested bytes in an origin operation. Counters are lifetime cumulative; maxima can therefore exceed a subsequently reduced configuration. They are distinct from consumer-delivered and writer-accepted bytes. Existing wait/try-again telemetry remains available.

Memory accounting: writer dirty payload, including a moved page being materialized, never exceeds its page budget; no whole-page block-writing copy is made. Page container metadata, allocator spare capacity, Safe ownership allocations, caller buffers and origin/library buffers are extra. Page mode allocates no dormant ring. Ring mode has a separate logical cap; its storage includes a spare slot and rounds up to a power of two (minimum 16 bytes), exposed separately. Draining borrows its storage without a second snapshot allocation. During transactional ring replacement both rings may coexist; Reconfiguration reports the replacement reservation until publication. FIFO writes use a bounded scratch window (page budget / transfer limit, or 64 KiB fallback). Reader assembly and span adapters may allocate up to the public request size in addition to pages and a transfer temporary; nonseekable Peek or a partially completed TryAgain retains unread data outside the page budget, reported as Retained. Prefetch only requests the remaining window, never the entire configured chunk unnecessarily. These are payload accounting limits, not a hard total-allocation, physical-memory or RSS guarantee.

Breaking: scalar-policy constructors of the buffered IO bases/location bases and the old scalar constructor symbols of the file leaves are removed. Derived classes use Parameters or complete configurations; location leaves resolve knobs in the caller. Policy getters/setters are now nonvirtual so they cannot bypass coherent configuration through overrides. IO parameter bags, location knobs and Hopper/Sink template layouts also change. Ownership still uses the existing provider/Safe mechanisms. Neither aggregation nor these knobs eliminates read/write contention on a shared disk. No timers, fsync policy or cross-file scheduling are provided.

Telemetry

Every IO office and every Bridge exposes const StormByte::Safe::Shared<ReadTelemetry> / WriteTelemetry. The handle is the same object for the life of the office. IO types add cache / origin / seek / wait counters. Non-IO Bridge tips use the basic type.

MeanRate is octets per second of requested user operations, including cache hits. It is not a disk benchmark. A cached write can look like GiB/s. Worker, GC and internal flushes enter the rate only when they delay the caller. Explicit Flush / Close Flush pull it back.

Each measured operation owns an independent Base clock sample, so concurrent and nested operations do not share a start/stop state. A sample borrows its Telemetry owner and must not outlive it.

Reader Cap is the MaxMemory snapshot. Writer Cap is the ring cap at that snapshot: WriteChunk * BackPressure, or 0 when the ring is off. It is not the page-map budget.

Flatten with operator StormByte::Safe::String or operator std::string() (the latter is FORCE_INLINE so the std::string lives in your TU):

auto tel = reader.Telemetry();
if (tel)
log << Level::Info << *tel << std::endl;

BufferedReader

Public base for a binary origin. Leaves implement OriginOpen, OriginClose, OriginPull, OriginCanSeek, OriginSeek, OriginHasSize, OriginSize. Construction is Unavailable; a successful Open is Idle.

Size() and the virtual OriginSize() const noexcept return StormByte::Safe::Optional<StormByte::ByteSize>, not std::optional. Empty means unknown or unavailable size; an engaged ByteSize{0} means a known empty origin. Custom readers must update the override signature and preserve this distinction. Downstream Network reader overrides must be updated in the Network repository too; this repository does not implement that migration.

Exceptions from OriginOpen, OriginClose, OriginPull and OriginSeek are converted to Status::Error. Exceptions from Setup or CreateTelemetry make Open return false. The public Device() accessor rethrows StormByte exceptions and translates foreign exceptions to StormByte::Buffer::Exception.

Available() is cached bytes at Tell. It does not call the origin.

Prefetch pulls at most 64 KiB per origin call, further clamped to the requested read-ahead and memory budget. Small budgets are not rounded up. This amortizes sequential I/O and worker notifications without letting a large prefetch request delay cancellation for a full read-ahead window.

Seek is logical. A cache hit does not move the device. Tell never lies. Consumed bytes remain in RAM up to MaxMemory. Garbage collection evicts farthest from Tell.

BufferedLocationReader

Sits between the engine and a file leaf. A location is file-like: named by Location() (StormByte::Safe::String, owned by Base), always seekable and sized. Path-only Setup() lives here.

Device() and the pure OriginDevice() return StormByte::Safe::Shared<StormByte::System::Device>, not System::Device by value. A leaf may hand out a subclass and the dynamic type survives, so overridden Throughput() / Window() are honoured. Build the owner with StormByte::Safe::Shared<StormByte::System::Device>::MakePointer<Leaf>(…).

OriginDeviceUsable(const Safe::Shared<System::Device>&) const noexcept defaults to a non-empty owner whose operator bool() is true. A leaf whose identifier is not a filesystem path overrides it. An empty owner is always unusable. Setup() is final: it calls OriginDevice() once, asks OriginDeviceUsable() and only then applies Window().

BufferedFileReader

Final file leaf. It passes Location::Local, keeps the plain System::Device and the real path probe. CreateDevice() is gone. Path-only construction probes at Setup. Explicit knobs stay explicit. The message constructor of Buffer::Exception takes string_view. A null const char* is not a message.

int main() {
BufferedFileReader in("in.bin", { ReadAhead{1 << 20}, MaxMemory{8 << 20} });
in.Open();
FIFO dest;
(void)in.Read(64 * 1024, dest);
auto tel = in.Telemetry();
in.Close();
}
bool Read(const StormByte::ByteSize &count, StormByte::Safe::Binary &outBuffer) const noexcept override
Read into owned storage and advance the position.
virtual bool Open() final
Arm the origin.

BufferedWriter

Public base for a binary sink. Leaves implement OriginOpen, OriginClose, OriginPush, OriginFlush, OriginTruncate. Writes are lazy up to MaxMemory. Public Flush and the Flush inside Close count toward MeanRate. Internal drains do not.

Seek is logical. Materializing a page (eviction, flush or close) moves the origin. Nearby corrections and far-future islands are supported while memory allows. Eviction prefers the oldest dirty page behind the origin cursor.

Exceptions from origin hooks are converted to Status::Error; Open returns false when setup or telemetry creation throws, and Close returns false when closing the origin fails.

BufferedLocationWriter

The write-side location layer. Same contract as the reader: a named, seekable, sized location, Device() as Safe::Shared<System::Device>, and OriginDeviceUsable before the window is applied. Path-only Setup() lives here. A socket on a lower layer can pass Location::Remote. This repository's file leaf does not.

BufferedFileWriter

Final file leaf. It passes Location::Local and probes the device at Setup when no knobs are set. Explicit WriteChunk, BackPressure and MaxMemory stay explicit. Truncate overwrites. The path stays locked on Windows until the leaf is destroyed.

Items

Items are not byte buffers and not IO. Hopper<T> and Sink<T> carry C++ objects. T must be an unqualified StormByte::Type::SafeValue, default-constructible and movable, with nonthrowing default construction, move construction, move assignment and destruction. alignof(T) must not exceed __STDCPP_DEFAULT_NEW_ALIGNMENT__, the alignment operator new gives Base's heap. That is 16 on the supported platforms, so Safe::String is admitted. alignof(std::max_align_t) is not the limit: it is 8 on macOS. Counts are StormByte::Size. A Safe::Binary, a std::string or a raw pointer is not admitted just because it can be moved. Use int, Safe::String, Safe::Shared<int> or another value that already meets the contract.

The wait path is Safe::Mutex, Safe::ConditionVariable and Safe::Atomic<std::size_t>. It is not a public std::mutex, std::condition_variable or std::atomic.

Hopper

Hopper<T> is an SPSC queue. Queue storage uses Base's heap. Extended alignment is rejected at compile time, with no standard-allocator fallback. Front additionally requires nonthrowing copying.

Capacity 0 is unbounded. Push blocks when a bounded hopper is full. Eof() ends production; queued items can still be drained. Nullable pointer items discard nulls.

Notify(cv) borrows a Safe::ConditionVariable. Replacing or unregistering an observer synchronizes with in-flight notifications; Unnotify() must return before that condition variable dies. This does not make concurrent destruction of the Hopper safe: stop and join its users first.

Notify(cv) does not store events: the caller must coordinate predicate checks, waits and producer operations through the consumer's wait mutex. For consumers with an independent mutex, use Notify(cv, generation), where generation is a borrowed Safe::Atomic<std::size_t>. Push and EOF increment it with release ordering and call notify_all() after publishing queue state. Load the counter with acquire ordering before checking readiness; if false, call generation.wait(captured) and repeat. Both the condition variable and the counter must remain alive until Unnotify() returns. Explicit stop/failure events must publish their state and increment/notify this same counter; do not reset it while consumers can wait on it.

#include <thread>
int main() {
std::thread producer([&hopper]() {
for (int i = 0; i < 10; ++i)
hopper.Eof();
});
std::thread consumer([&hopper]() {
while (!hopper.Empty() || !hopper.EoF()) {
auto item = hopper.Pop();
(void)item;
}
});
producer.join();
consumer.join();
}
Synchronized typed item queue with shared-writer support.
Definition hopper.hxx:166
static Shared< T > MakePointer(Args &&... args)

Sink

Sink<T> maps integer keys to Hopper<T> buckets. It takes the same items as Hopper. It does not take octets. Wire with To(key) >> dest, dest << producer or dest << producer.To(key). To returns a lane. It does not take the destination.

Keys() returns an independent, ascending StormByte::Safe::Vector<int> snapshot that remains valid after the Sink is destroyed. Copies are independent; keep Base and the snapshot's creator module loaded until all copies are released.

The coordinator is Safe::Owner. Wired hoppers are Safe::Shared. Buckets are Safe::Map<int, Safe::Shared<Hopper<T>>>, writers are Safe::Set of that same shared hopper, and the round-robin order is Safe::Vector. Over-aligned payloads are rejected, not routed to a standard allocator. The wait path is the Safe mutex, condition variable and atomic named above.

Select is StormByte::Safe::Function<StormByte::Size(StormByte::Size)>. Pop(const Select&) borrows it synchronously and never retains it. The provider callback has the form Safe::Status(void* context, Size* output, Size count): write the index through output and report Success. Missing, Failure or an exception returns default T without consuming queued items. The callback provider owns and releases its context and must remain loaded through callback destruction. The caller-side callable overload borrows the original callable, including mutable or move-only captures, only for that Pop call.

Conditions passed to Notify are borrowed Safe::ConditionVariable objects. Unnotify unregisters this Sink's matching hopper registrations and waits for in-flight notifications before returning; finish it before destroying the condition variable. Stop and join all queue users before destroying the Sink.

Sink::Notify(cv, generation) uses the same stored-event protocol as Hopper on current and future buckets, including keyed wiring, all-bucket wiring and fan-in. Wiring publishes an event after updating the consumer, so already queued data or EOF also wakes it. Sink closure publishes an event even with no buckets. Shared hoppers have one observer: the latest registration replaces its condition variable, counter and owner identity. An older Sink's Unnotify() cannot remove that newer registration.

consumer_sink.Notify(wake, generation);
for (;;) {
const auto captured = generation.load(StormByte::Safe::MemoryOrder::Acquire);
if (!consumer_sink.Ready()) {
generation.wait(captured);
continue;
}
if (consumer_sink.EoF())
break;
(void)consumer_sink.Pop();
}
consumer_sink.Unnotify();
T load(MemoryOrder order=MemoryOrder::SeqCst) const noexcept
void wait(T old, MemoryOrder order=MemoryOrder::SeqCst) const noexcept

Stored notifications change the private Hopper/Sink template implementation layouts in 2.0.0. Rebuild all providers and consumers that instantiate them before replacing the library; do not mix older and newer instantiations across DLLs. The borrowed condition variable and counter are Safe types. Their providers must remain loaded until notification removal completes. The counters add no ownership callbacks or changes to queue limits, blocking or element release.

#include <thread>
int main() {
producer_sink.To(1) >> consumer_sink;
producer_sink.To(2) >> consumer_sink;
std::thread writer([&producer_sink]() {
producer_sink.Push(1, StormByte::Safe::String("ch1"));
producer_sink.Push(2, StormByte::Safe::String("ch2"));
producer_sink.Eof();
});
std::thread reader([&consumer_sink]() {
while (!consumer_sink.EoF())
(void)consumer_sink.Pop();
});
writer.join();
reader.join();
}
Forward declaration of the keyed typed queue.
Definition sink.hxx:97
bool EoF(int key) const noexcept
Whether producers marked Eof on bucket key.
Lane To(int key) noexcept
Redirect of one hopper key.
void Push(int key, T item) noexcept
Pushes an item into the bucket specified by key.
void Eof() noexcept
Closes the Sink and releases writer holds on its hoppers.
T Pop() noexcept
Pops one item from the Sink using default round-robin order.

Weighted queues and shared byte budgets

Hopper<T> and existing Sink<T> buckets add generic weighted admission. Buffer never inspects T, estimates a frame size or runs a weight callback. The caller supplies each entry's logical byte weight.

  • QueueLimits{optional<Size>, optional<ByteSize>} uses Safe::Optional: absent ({} or std::nullopt) is unlimited; an engaged zero is an active zero limit. Both bounds apply independently.
  • Limits(limits) / Limits(key, limits) replace local ceilings without dropping entries. Lowering a ceiling below current usage prevents further positive admission until enough entries are popped. Capacity(key, 0) still removes the item limit, not the byte limit. The legacy getter cannot distinguish explicit zero from unlimited; use Snapshot().Limits.
  • Configure(limits, budget) / Configure(key, limits, budget) attach one shared budget atomically to an empty, idle hopper. Active admissions or queued entries return Invalid, leaving both limits and budget unchanged. Sink keys must already be wired; no implicit bucket creation occurs.
  • TryPush(item, bytes) and PushFor(item, bytes, timeout) (with key first on Sink) accept a mutable lvalue reference. They move it only on Ok. Oversize weights or zero item ceilings return TooLarge immediately. Local/aggregate pressure returns WouldBlock for the immediate operation, or waits until capacity, closure, cancellation or the bounded timeout. Other explicit results are Closed, Cancelled, Timeout, Missing, Empty for empty nullable payloads, Invalid and recoverable storage Error. Negative/overflowing timeouts are invalid; zero is an immediate attempt.
  • Legacy Push stays void noexcept, waits for the legacy item ceiling and treats its entry as weight zero. It cannot infer byte usage. Convert producers to weighted admission wherever byte-accounting guarantees matter; mixing unweighted entries understates payload bytes by design.
using namespace StormByte::Buffer;
ByteBudget shared(ByteSize{64 * 1024 * 1024});
Sink<int> producer, consumer;
producer.To(1) >> consumer;
const QueueLimits local{
Optional<Size>{Size{8}}, Optional<ByteSize>{ByteSize{16 * 1024 * 1024}}};
const auto configured = producer.Configure(1, local, shared);
if (configured == QueueStatus::Ok) {
int item = 42;
const auto admitted = producer.PushFor(1, item, ByteSize{2 * 1024 * 1024},
std::chrono::milliseconds{100});
if (admitted == QueueStatus::Ok)
(void)consumer.Pop(1); // returns this entry's aggregate charge
// On any non-Ok result item remains owned by this caller.
}
Shared logical byte accounting for queues and concurrent work.
Definition byte_budget.hxx:256
Independent local bounds; absent means unlimited, engaged zero means zero capacity.
Definition byte_budget.hxx:125
Buffer module of the StormByte suite.
@ Ok
Accepted.
Definition byte_budget.hxx:81

ByteBudget{} / ByteBudget{std::nullopt} are unlimited; ByteBudget{ByteSize{0}} allows only zero-byte reservations. Copies share the same provider-owned state, not independent per-bucket limits. TryReserve(bytes, lease) / ReserveFor(bytes, lease, timeout) reserve before work/queue insertion. Their output lease changes only on success. A lease is RAII: final release, Reset, rollback, Abort or destruction returns its charge and wakes waiters.

For jobs outside queues, retain a lease while the job retains its accounted payload. To share one payload across queues, reserve once, then use TryPush(item, lease) / PushFor(item, lease, timeout) with copies of that lease. Its budget must match the hopper's configured budget. Each entry counts lease.Bytes() against its local byte limit, while the aggregate budget charges that lease just once:

ByteBudget::Lease payload_charge;
if (shared.TryReserve(ByteSize{4096}, payload_charge) == QueueStatus::Ok) {
// Copy this lease to every queue entry/job referencing the SAME payload.
// Queue shared-charge overloads retain a copy; they do not consume this one.
payload_charge.Reset(); // charge persists if other copies still retain it
}
Shared charge that can explicitly follow one payload across queues or work.
Definition byte_budget.hxx:261
void Reset() noexcept
Release this reference.

No automatic physical-allocation deduplication is promised: caller-supplied weights and explicitly shared leases define accounting. Independently reserved leases are separate logical charges, even if payload pointers happen to match. Queued payload/entry storage, owner control blocks, allocator overhead, stacks, unweighted entries and unaccounted external work are not an RSS guarantee. Normal Pop releases the queue's lease before returning T; processing is not charged unless the caller deliberately keeps a shared lease alongside it. EOF preserves queued entries and their charges until Pop/destruction; final writer EOF semantics remain unchanged. Abort discards entries and releases queue-owned charges, but cannot release a lease retained elsewhere.

CancelPending cancels existing weighted waits, not future admissions: on a hopper/key it affects that shared hopper; on a budget it affects all currently waiting participating queues/jobs. ByteBudget::Close wakes waiters and rejects further weighted reservations/admissions, leaving live charges valid. Closing one Sink cancels its own weighted waits even if another co-writer keeps the hopper open. Unlike weighted calls, a legacy Push already handed to a shared hopper retains its original last-writer EOF behavior.

Concurrency: the lock order is Sink registration, then hopper state, then budget state; waiting releases all of them. Budget code never calls back into a hopper. A stored generation is captured before admission predicates are checked, preventing lost wakeups across queue Pop, limit changes, EOF, cancellation and aggregate release. Payload processing and I/O are outside locks; no user-supplied callback is added. Queue entry allocation and short nonthrowing Safe payload moves still serialize admission. Topology changes (rewiring) must be finished before concurrent admission/Abort. Stop/close or cancel, then join all users before destruction; handle destruction is not concurrent-call cancellation.

All bounded waiters are notified when capacity changes. There is no strict FIFO fairness guarantee among variable-weight producers; a larger admission may time out while smaller ones fit. Bounded deadlines and explicit results avoid an indefinite wait for an impossible admission. Timeout is not a real-time scheduling guarantee.

Snapshot(key) / Hopper::Snapshot() return independent local limits, current entries/weighted bytes, lifetime peaks, pressure attempts, waits and WaitNanoseconds. ByteBudget::Snapshot() reports its single aggregate Reserved, limit, peak, pressure and reservation-wait counters. Queue waits are reported on the queue; external ReserveFor waits on the budget. A shared lease can therefore yield larger summed queue bytes than aggregate Reserved.

Weighted Hopper::PushFor(item, bytes_or_lease, timeout, observer) and Sink::PushFor(key, item, bytes_or_lease, timeout, observer) accept an optional AdmissionWaitObserver{context, callback} (default {}). Its borrowed callback has signature void(void*, AdmissionWaitReason, bool entering) noexcept. It runs synchronously on the calling thread, outside all Buffer locks, immediately before each actual budget wait and after it returns, with paired reasons: LocalCapacity when local element/byte bounds deny admission before reservation, or SharedBudget when TryReserve returns WouldBlock. Local bounds take precedence if both are exhausted. Retrying after a wake may produce another pair, possibly for a different reason; an already-signalled event may return immediately. There are no callbacks on fast paths or outcomes that do not reach a wait. The observer owns nothing, allocates nothing, and is not retained. Keep its context and callback alive until the call returns; callbacks should be brief and must not throw. Snapshot inspection is safe; concurrent calls sharing a context require caller-provided synchronization. A null callback disables observing. This does not change admission policy, reservations, or existing snapshot accounting.

ByteBudget::SharesIdentity(other) compares shared state without reserving, locking or changing counters, before or after Close(). Copies share identity; independent budgets (even with equal limits) and moved-from handles do not. No pointer or global ID is exposed. Comparing handles concurrently is safe if none is moved or assigned meanwhile.

Queries perform no I/O, callbacks or capacity waits, but take short mutexes: they are not lock-free and do not promise zero scheduling latency.

DLL compatibility: budget state, mutexes and charge destructors stay in Buffer; handles use Base Safe::Shared provider callbacks. Budget exported wait functions take scalar durations, not steady_clock::time_point or STL owners. Chrono convenience wrappers execute in the caller and pass scalar counts. All involved modules must remain loaded with compatible C++/STL ABI; this is conditional Safe ownership, not arbitrary compiler compatibility. The new private Hopper state layout is a Breaking ABI change. The installed-prefix and extracted-ZIP consumer tests also use a separate shared-library budget/hopper provider to exercise cross-DLL handles.

Support

Questions and bugs: GitHub issues on this repository. Sponsorship: github.com/sponsors/StormBytePP.

Contributing

Open an issue before large work. Conventional Commits. Public headers need Doxygen. Do not send patches that reintroduce raw new for StormByte pointer types.

License

Dual license: GNU Lesser General Public License v3.0 or later, or a commercial license from the copyright holder. See [LICENSE](LICENSE), COPYING.LGPLv3 and https://www.gnu.org/licenses/lgpl-3.0.html. Third-party trees under thirdparty/ keep their own licenses.

Binary releases add THIRD-PARTY beside LICENSE. That file is a notice for what was compiled into the shared library, not another grant. The texts it names are the COPYING.* files in the same archive.