StormByte-Multimedia 3.3.2
C++26 multimedia processing module of the StormByte suite
Loading...
Searching...
No Matches

Contributor guides: Contributing and Coding Style.

Platform C++26 CMake License: LGPL v3 / Commercial CI Sponsor

StormByte-Multimedia brings media processing and real-time preview into a C++ application without making that application manage FFmpeg's C API. The complexity stays inside the library: asynchronous workers, stream routing, decoding, filtering, encoding, muxing and presentation timing. Outside, applications describe a job or connect a player to their own display.

There are two independent workflows:

Goal Starting point What the application supplies
Copy, encode or filter media into an output Pipeline::Transcoder Input/output locations, output-track mapping and settings
Build a specialized processing graph Pipeline::Plan and Step stages The same job description, explicit connections and graph lifetime
Preview video, audio and subtitles in a UI Multimedia::SimplePlayer A buffered input and a consumer-implemented Display

Transcoder is a complete facade over the processing pipeline, not a reduced encoder. Manual wiring exposes the same machinery when a graph needs more control. SimplePlayer handles real-time playback independently; it is a preview service, not a replacement for a professional media player, an editor or a bundled Qt/SDL widget.

It depends on StormByte Base, StormByte Logger and StormByte System at version 3.0.0, and StormByte Buffer at version 4.0.0. Buffer must be the retagged 4.0.0 build exposing ByteBudget::SharesIdentity in both its headers and library; older artifacts with the same version are incompatible. Each is a release archive. Public headers live under StormByte/multimedia/.

The suite is split on purpose. Base, Buffer, Config, Crypto, Database, Logger, Network and System are other repositories. This one does not implement them.

The rest of the suite

Module Role API
Base Exceptions, Expected, serialization, strings, UUID, concepts /StormByte
Buffer FIFO, SharedFIFO, Ring, Producer/Consumer and multi-stage pipelines /StormByte-Buffer
Config Human-readable text and versioned binary documents (groups, lists, raw bytes) /StormByte-Config
Crypto Hash, compress, encrypt, sign and key agreement — Crypto++ never leaves the private tree /StormByte-Crypto
Database One API over SQLite, PostgreSQL and MariaDB /StormByte-Database
Logger Stream logging, levels, headers, redaction, ThreadedLog /StormByte-Logger
Multimedia This repository /StormByte-Multimedia
Network Framed packets, Client/Server, IPv4/IPv6 TCP and Buffer pipelines (compress/encrypt) /StormByte-Network
System Processes, pipes and environment variables across Linux, Windows and macOS /StormByte-System

Table of Contents

Documentation

This README explains the workflows, integration choices and important contracts. It is not a method-by-method API reference.

Inspect media and discover capabilities

Use File to inspect streams, typed audio/video properties, metadata and attachments before presenting choices to a user. File::Streams() contains ordinary tracks; File::Attachments() exposes pictures and other attachments separately. An album cover is not an ordinary video track.

The registry provides stable codec/container identities, aliases and the implementations available in the configured build. A name being registered does not imply an encoder, decoder or suitable output container is available.

using namespace StormByte::Multimedia;
auto hevc = Registry::Instance().FindCodec("H.265");
if (hevc) {
const Codec& codec = hevc->get(); // Borrowed registry identity.
const bool writable = codec.HasAccess(Operation::Write);
const bool hdrWritable = codec.HasAccess(Operation::Write, Feature::HDR10);
const bool doviReadable = codec.HasAccess(Operation::Read, Feature::DOVI);
}
Immutable codec identity owned by Registry.
Definition codec.hxx:94
bool HasAccess(Access access) const noexcept
Tests Read/Write flags.
static Registry & Instance()
Process-wide instance.
Multimedia module of the StormByte suite.
@ Write
At least one encoder / muxer exists.
Definition type.hxx:103
@ Read
Decode / demux is available.
Definition type.hxx:102
@ HDR10
Static HDR10 metadata on the frame.
Definition feature.hxx:99
@ DOVI
Parsed Dolby Vision metadata decoding or native RPU encoding.
Definition feature.hxx:110

HasAccess(Access, Features) requires a covering available implementation in each requested direction. An empty feature mask is the generic access query. This is a compiled-capability check, not a device probe or a guarantee that every setting, profile and container combination will open.

Codec::Encoders(features) lists concrete implementations and their supported controls; EncoderInfo::Option(name) describes option types, ranges, defaults and choices. This lets a UI expose settings the selected encoder can actually apply.

Codec::SubtitleKind() classifies subtitle codecs as text, bitmap or unknown independently of decoder/encoder availability. ASS, SRT, WebVTT and mov_text are text; PGS, DVD/DVB and XSUB are bitmap. Classification does not imply OCR or rendering support.

Encoding and remuxing

Processing creates an output, not a timed presentation. Both entry points use the same demux/decode/filter/encode/mux stages and explicit failure handling:

  • Start with Transcoder for ordinary jobs and product integration.
  • Wire the stages yourself only when the graph requires explicit routing or lifetime control.

Quick start with Transcoder

Remux the first video stream into Matroska. The output suffix selects the container; the application does not create FFmpeg contexts or connect workers.

#include <chrono>
#include <iostream>
#include <thread>
int main(int argc, char** argv) {
if (argc != 3) {
std::cerr << "usage: " << argv[0] << " <input> <output.mkv>\n";
return 1;
}
auto logger = StormByte::Safe::MakeShared<ThreadedLog>(std::clog, Level::Notice, "[%L] %T %c");
Transcoder job{String{argv[1]}, String{argv[2]}, logger};
job.Video(0).Remux();
if (job.Failed()) {
std::cerr << job.Error().value_or(String{"configure failed"}) << '\n';
return 1;
}
job.Run();
for (;;) {
const auto status = job.Status();
if (status == Status::Done)
return 0;
if (status == Status::Error || status == Status::Aborted) {
std::cerr << job.Error().value_or(String{"media job failed"}) << '\n';
return 1;
}
std::this_thread::sleep_for(std::chrono::milliseconds(20));
}
}
Facade that maps tracks and runs one reader-to-writer job.
Definition transcoder.hxx:315
Coordinator behind StormByte::Multimedia::Pipeline::Transcoder.
Status
Lifecycle of a Transcoder instance.
Definition transcoder.hxx:151
@ String
Free text.
Definition encoder_info.hxx:93
Shared< T > MakeShared(Args &&... args)

Queue buffering policy

ResourceUsage continues to select the CPU profile and automatic queue caps. Use BufferingPolicy to override those element caps independently, including while keeping ResourceUsage::Full. QueueLimit::Automatic() follows the selected profile, QueueLimit::Bounded(n) requests a strict item limit, and QueueLimit::Unlimited() disables that item limit. A video-frame cap must be at least two; audio-frame, packet and demux-pending caps must be at least one.

queues.QueueBytes(StormByte::ByteSize{64 * 1024 * 1024});
Transcoder job{source, destination, logger};
job.Resources(ResourceUsage::Full);
job.Buffering(queues);
Queue element-cap overrides shared by Plans and Transcoders.
Definition buffering.hxx:99
constexpr QueueLimit DemuxPending() const noexcept
Definition buffering.hxx:104
constexpr QueueLimit Packets() const noexcept
Definition buffering.hxx:103
const StormByte::Buffer::ByteBudget & Budget() const noexcept
Definition buffering.hxx:110
constexpr QueueLimit AudioFrames() const noexcept
Definition buffering.hxx:102
constexpr QueueLimit VideoFrames() const noexcept
Definition buffering.hxx:101
const StormByte::Safe::Optional< StormByte::ByteSize > & QueueBytes() const noexcept
Definition buffering.hxx:106
static constexpr QueueLimit Bounded(StormByte::Size value) noexcept
Definition buffering.hxx:58
@ Full
Keep native codec threading and all requested VMAF workers, with bounded queues.
Definition resource_usage.hxx:76
Demux / decode / filter / encode / mux types.

The same value can be set on a manually created Plan before connecting or starting its stages with plan->Buffering(queues). Invalid minima are rejected without changing the previous policy. These overrides affect queue element counts and logical payload-byte admission; they do not change codec CPU threading. Byte weights are estimates of queued frame or packet payload, not process RSS or uniquely owned physical memory.

Newly wired track queues attach their byte limits and shared budget before producers can enqueue. Inputs wired before Plan binding receive the Plan's policy before workers start; initial element ceilings are installed too. A stage may refine its element ceiling during startup without rebinding that budget or discarding packets already waiting in its input, including clone lanes and inputs shared by multiple producers.

Manually mounted stages expose the effective limits and live queue accounting for an input track through Step::QueueSnapshot(track) and Filter::FFmpeg::QueueSnapshot(track). Read the requested mode from the Plan's BufferingPolicy, the applied limits and occupancy from the queue snapshot, and shared reservations from Plan::Buffering().Budget().Snapshot(). A snapshot with Exists == false means that track's input queue is not wired yet.

The muxer uses a positive 0.25-1 second media-DTS window in FFmpeg's interleaver, including encoded-video/remux combinations. An empty or sparse subtitle track therefore cannot select the former 60-second window; queue depth and header-startup timestamp span do not enlarge it. This is not a wall-clock delivery deadline: encoder startup, ordered subtitle processing, container clustering, AVIO and the output writer can still delay bytes.

The muxer's own audio/video readiness guard is separate. Its existing 60-second media-DTS skew fallback protects delayed encoders; reducing it to one second lets remuxed audio run ahead of video with long lookahead. Encoded subtitle progress watermarks still protect cue ordering. Payloads held by this guard, codec internals and FFmpeg's interleaver are not covered by the queue ByteBudget; neither time window guarantees a byte/RSS cap. Buffer's output telemetry counts bytes only once the writer receives them, not packets held inside the muxer.

Configure output tracks

Map the streams you want, in output order, and choose Remux() or Codec(...) for each. Omitted streams are dropped. These calls append outputs: Audio(1) twice means two independent destination tracks, not another lookup of the first one.

For example, after constructing a job and checking the registry identities:

// hevc/eac3 are successful Registry::FindCodec results; logger is shared.
job.Video(0)
.Codec(hevc->get())
.Implementation(StormByte::Safe::String{"libx265"})
.CRF(20);
job.Audio(1).Remux();
job.Audio(1).Codec(eac3->get());
job.Subtitle(3).Remux();
job.Attachments();
// Check Failed()/Error() before Run().

This can preserve the compressed original audio and add an E-AC3 compatibility track alongside it. Each output has independent settings, filters and metadata overrides. Save a returned Track handle when further configuring that particular output.

Choice Meaning
Remux() Forward compressed media with destination adaptation; no separate “Copy” stage
Codec(identity) Decode and encode with a registered destination codec
Implementation(name) Pin the encoder; unavailable or mismatched pins fail without fallback
Implementation(ImplementationSide::Decoder, name) Independently pin the source decoder on an encode lane
CRF, BitRate, MaxBitRate, Preset, Tune, FineTune Configure supported quality/rate/implementation options
Language, Title Override metadata on that output
Ignore(n) Remove every mapped output of origin stream n
Attachments() / Attachments(mime) Select general attachments, otherwise dropped; automatic audio artwork is described below
Explicit registered Container constructor argument Override output-suffix inference; the identity is borrowed

Unsupported settings fail rather than being ignored. Audio quality-based VBR replaces bitrate when selected; without a quality or bitrate setting, the encoder default applies, not the source bitrate. FineTune accepts individual keys and packs them into implementation options where necessary.

Encoder preparation preserves supported representations or performs exact storage repacking, such as yuv420p10le/p010le. Depth reduction, chroma resampling and color conversion require an explicit filter. Hardware encoding and decoding are independent: decode can be hardware-assisted while encoding is software, and vice versa.

Video decoders use DecodeAcceleration::Auto by default on preview and encode lanes. Auto checks FFmpeg's hardware configurations for the selected codec, tries the platform backend, then opens a software decoder if device creation or hardware decoder initialization fails. Use SimplePlayer::Acceleration(...) or Track::Acceleration(...) to force CPU decoding for preview or an individual transcode lane. Decoder pins and EnforceFeatures remain independent of this setting; remux tracks do not open a decoder or device. Hardware-decoded frames are transferred to software storage with FFmpeg frame properties preserved before entering preview conversion or the filter pipeline, so this path is not zero-copy.

MPEG-4 Part 2 (DivX/Xvid, not H.264 in an MP4 container) always uses software decoding, even in Auto mode. Hardware decoding can corrupt GMC (global motion compensation) frames, including when they first appear later in the stream. This policy avoids an additional compatibility scan and applies to preview and conversion with both bundled and system FFmpeg. Hardware encoding, including NVENC, remains available; other video codecs retain Auto acceleration.

The bundled FFmpeg configuration enables VideoToolbox on macOS, probes VA-API on Linux, enables NVDEC where the NVIDIA codec-header plugin is available, and probes D3D11VA/DXVA2 on Windows. Device drivers and codec/profile support are runtime-dependent; Auto falls back only during device/decoder initialization, not after a stream has started failing.

Audio encode adaptation can automatically reduce 7.1 to 5.1 for AC-3/E-AC3. MP3 requires an explicit downmix for inputs above two channels. These are encoding policies, not SimplePlayer output behavior: preview preserves decoded channels.

Filters and analytics

Attach typed filters to an output with Track::Filter<T>(...); job-level Filter<Analytics>(...) attaches global analysis. The facade creates the filter graph, so a simple job can use the same filters as manual wiring.

Use Examples and considerations
Geometry Scale, crop and pad; registration order determines the transformation order
Watermark PNG, JPEG, WebP, BMP or self-contained SVG from a path or Safe::Binary
Restoration and cadence Denoising, sharpening, decimation and experimental two-pass Degrain
Color Explicit conversions and tone mapping; incompatible HDR metadata is removed from SDR output
Audio Typed sample-rate, layout and processing filters
Quality analysis VMAF compares decoded reference and destination pictures without modifying the output

Watermark's designedFor size is the video reference size, not the requested logo size. SVG is rendered at the final paint resolution; external resources and system-font discovery are disabled. Convert text to paths and embed images. LunaSVG is not a browser SVG engine; validate the artwork visually. Invalid SVG dimensions warn and pass media through without a logo.

The three long-gap mov_text regression tests are skipped with WITH_FFMPEG=SYSTEM when the runtime FFmpeg release is 9.0.2 or older: those releases lack the custom patch applied to bundled FFmpeg. They still run for bundled FFmpeg and SYSTEM releases newer than 9.0.2. This test exclusion does not fix the subtitle timestamp limitation in an unpatched system library.

Degrain measures regional grain before filtering, preserves timing/geometry and has bounded look storage. It is an experimental heuristic, not motion-compensated restoration or a guarantee of perceptual improvement. Do not stack it with another grain-denoising filter.

VMAF reports scores and scored-frame counts per output. Wait for Done and inspect report status and counts, not just the mean. Resource modes govern workers and queue budgets; process memory still depends on resolution, concurrent contexts and retained pictures. A queue limit is not a whole-job RAM limit.

To implement a filter, follow the existing Scale/Watermark classes and the Doxygen contracts. Frame filters use Save for a one-for-one replacement, or Drop/EmitGraph/EmitFrame for buffering and multiple outputs, including EOF output. Analytics cannot mutate media. Use the library's RAII frame/packet types; the libav-backed graph surface is for operations not covered by the typed wrappers. Retain a scaling context for repeated direct conversions and use it from one thread at a time.

Progress, reports and application hooks

Run() is asynchronous; Pause() / Resume() / Cancel() control the coordinator. Success comes from Status() == Done, not a progress percentage or muxer EOF.

Progress()->Snapshot() reads the progress axes atomically:

Axis Purpose
Duration calculation Source scan, only when neither a positive supplied duration nor a usable container hint exists
Main Actual destination-writing percent; 100 when the trailer is written
Measure Optional first pass for a two-pass filter
Analytics Optional destination analysis, which can continue after main output finishes
All Weighted overall display, below 100 until all mounted phases complete

Show Main as the transcode bar, with measure/analytics separately when present. All is useful for a one-line overall indicator, not as the encode percentage. File::DurationHint() does not scan; File::Duration() retains full-scan behavior. Duration estimates do not limit how much media is processed. Observed scans support pause/cancel and rewind on completion or interruption.

After Done, Reports() contains analytics results. Capture Telemetry() before running if the final snapshot must outlive the job. It retains stage counts, setup/processing/wait times and sampled memory. Memory minimum/current/peak are process-wide RSS, not stage allocations; sampling can miss short peaks.

Pipeline topology and queue telemetry

Telemetry()->Topology() returns an owned TopologySnapshot of the real stages and StormByte::Buffer queues of the latest run. It is thread-safe while running, paused, after cancel/failure and after the Transcoder was destroyed; the snapshot keeps no stage alive and holds no raw pointers.

auto telemetry = job.Telemetry(); // retain before or during Run()
const auto topo = telemetry->Topology();
for (const auto& stage : topo.Stages)
if (stage.Kind == StormByte::Multimedia::Pipeline::TopologyStageKind::Filter && stage.FilterPosition)
std::cout << "filter #" << *stage.FilterPosition << " on track " << stage.Track.value_or(-1) << '\n';
for (const auto& queue : topo.Queues) {
const auto& q = queue.Snapshot; // Buffer::QueueSnapshot
const bool unlimited = !q.Limits.ElementsLimited(); // limited with Elements() == 0 is a zero cap
std::cout << "queue " << queue.Id << (q.Exists ? "" : " absent") << (unlimited ? " unlimited" : "")
<< " waits=" << q.Waits << (q.Closed ? " closed" : "") << '\n';
}
for (const auto& budget : topo.Budgets) // Buffer::BudgetSnapshot, once per shared budget
std::cout << "budget " << budget.Id << " charged by " << budget.Queues.size() << " queues\n";
@ Filter
User or built-in filter leaf (see TopologyFilterKind).
Definition telemetry.hxx:395
  • Stages carry Kind (demuxer, decoder, filter, encoder, remuxer, muxer, inspection decoder), Filter family, source Track, OutputTrack, Media, FilterPosition within the track filter list, FilterName, Order (longest producer hop count, ties by Sequence) and Queues references with role Input, Output or Pending. Use these fields, never the display Name.
  • Queues appear once even when several producers share them (Producers lists them all). Kind is Pipe (Buffer::Sink hopper) or DemuxPending (demuxer park list: elements only, BytesAccounted == false). Snapshot.Exists == false means the key is not present (yet), not empty. Absent Limits means unlimited; an engaged limit of 0 is a zero cap. Input–output relations come only from registered wiring, not inferred.
  • Budgets are listed once per shared ByteBudget (Id is stable across reads and after the run); each queue names its Budget. Identity comparisons also work when a budget is closed before it is first observed; they do not reserve bytes or alter accounting. An invalid budget handle cannot be identified and makes the topology incomplete instead of creating a misleading budget entry.
  • RecorderFailed == true means the latest run could not allocate its topology recorder: nothing was recorded, so Available == false and Complete == false. It is never confused with a job that has not run yet (Available == false, Complete == true). The job itself is unaffected.
  • Complete == false means telemetry bookkeeping lost a stage, producer link, budget or pending list to an allocation failure, or encountered an invalid budget handle; the job itself is unaffected.
  • State is Live while stages exist and Final after teardown: final values never change. Right after Status::Done the run may still be finishing and report Live. If the final copy cannot be allocated at teardown, the run reports Unfrozen with Complete == false: queue values are their own immutable per-queue captures (Final), while budgets stay Live and may still change through other users.
  • Bytes are logical admission weights, not RSS. Counters are per queue and per budget in their own units; do not derive a batch average. A full queue or pressure/waits are normal backpressure, not an error. Different queues are read one after another, so a snapshot is not an atomic cut across queues.
  • Encoder/muxer-internal codec buffers are not Buffer queues and are not reported.

Per-worker activity

Each topology stage also exposes an owned Activity sample: independent worker ids/generations, preparation, exclusive processing, input wait, local output wait, shared-budget wait, cooperative pause and unclassified time. Open intervals are included in live snapshots and terminal times freeze. Demux feeders are separate workers, not an exclusive aggregate stage state. Coverage and missing instrumentation are explicit; these are wall times, not CPU utilization. Existing counters and queue topology are unchanged. See the activity API and Alchemist delta example for retention, pause semantics, limitations and validation. This requires Buffer 4.0.0 retagged with its optional admission-wait observer.

Construction, graph attachment, owning metadata access and report snapshots may throw StormByte::Safe::AllocationError or std::bad_alloc; they do not promise allocation-free operation. Worker lifecycle callbacks retain their noexcept contracts and convert allocation failures into the stage/job failure state, waking blocked peers. After allocation failure the diagnostic text may be empty: check the failure state rather than treating an empty message as success. Shutdown and destruction do not depend on allocating formatted diagnostics.

Transcoder is designed to be inherited, but inheritance is not required for a normal job. Override OnProgress, OnDone and OnError to integrate a UI, scheduler or batch runner. Override EmptyPlan() / EmptySettled() for application-owned payloads while retaining the processing machinery and provider-safe ownership.

Advanced: wire the pipeline yourself

For a specialized graph, construct a shared Plan, configure its output tracks, create the stages and connect them with operator>>. A recode lane has this shape:

control/setup
Demuxer -------------> Muxer
|
Packet
v
Decoder
|
Frame
v
[filters/analytics]
|
Frame
v
Encoder
|
Packet
v
Muxer

A remux lane substitutes Remuxer for decode/filter/encode. Muxer reserves output slots and interleaves destination packets; it does not trust worker arrival order. Fan-out permits independent routes from the same demuxer.

The wiring sequence below assumes already-constructed compatible stages and a checked Safe::Shared<Plan>:

demux >> mux;
demux >> decode;
decode >> encode;
encode >> mux;
plan >> demux; // Last: binding publishes Running and starts job setup.

Important differences from the facade:

  1. Configure before binding. Plan owns input/output locations and output-track intentions. It is noncopyable/nonmovable; bind the shared owner, not a value. Check() validates formation. Container and codec support are also checked during setup/run.
  2. Finish every connection first. Workers exist from stage construction but wait for the Plan to run. For a filter graph, use shared endpoints, register leaves, call Filters::Close(), then bind.
  3. Observe, do not publish, state. Plan states are Created, Running, Finished, Failed and Aborted. Intentional cancellation publishes Aborted; a genuine in-flight failure still publishes Failed and preserves the first error. Finished requires every participating stage to complete; muxer completion alone is insufficient.
  4. Own the lifetime. Keep stages, graph, Plan and borrowed identities alive through completion and notifications. Wait for graph Idle() where applicable. Stop every peer before destroying any of them and quiesce concurrent controllers; Step::Stop() is nonblocking, not a join barrier.
  5. Keep errors distinct from EOF. Normal EOF drains queued media; failure/cancellation wakes blocked peers. Queue capacities bound individual queues, not total memory.

Packet is compressed media and Frame is decoded media. Their serial is a lineage identifier, not a frame count; timing is not set through public Frame setters. Transform through filters and supported adapters.

Plan::Policy::EnforceFeatures is enabled by default: plans reject encode routes without an implementation capable of keeping source HDR10, HDR10+ or Dolby Vision. Disabling it permits feature loss; it does not manufacture metadata support. Configuration, state/error and telemetry are public; private source snapshots and reader/writer accessors are not an application inspection surface.

Metadata, subtitles and artwork

Encoding and remuxing copy the supported typed music tags; the destination chooses their native representation. This does not preserve every arbitrary ID3 frame or a byte-identical metadata block. Muxed output advertises StormByte-Multimedia <version> where application tags exist; technical muxer/bitstream identifiers remain truthful, and raw streams may carry no branding.

HDR and Dolby Vision

Feature checks distinguish static HDR10, HDR10+ dynamic metadata and Dolby Vision. They are not interchangeable with ten-bit pixels or hardware acceleration.

  • HEVC Dolby Vision decoding is listed for hevc; AV1 for av1/libdav1d. Listed encoders are compatible builds of libx265, libsvtav1 and libaom-av1.
  • HDR10+ paths include native HEVC/AV1 and dav1d decoding, and x265/libaom encoding. VP9 metadata preservation is container-conditional through Matroska side data, not a general VP9 bitstream promise.
  • Actual support depends on the selected FFmpeg/dependency build, compatible input metadata and encoder settings. An encoder cannot create a Dolby Vision grade from ordinary pixels.

Pixel transformations can make metadata stale. DOVI-only content must not acquire an invented HDR10 fallback. Validate or regenerate metadata in a mastering workflow after grading, geometry or color changes. Successful processing and metadata preservation are not visual correctness, Dolby certification or calibrated presentation.

Subtitle conversion

Remux compatible subtitles or select an output subtitle codec to convert them. Bitmap-to-text conversion uses OCR; the source language selects the model, and missing models fail explicitly. OCR is not translation. This belongs to the processing workflow; SimplePlayer instead delivers decoded cues for the application's renderer.

Embedded audio artwork

Audio-only outputs automatically copy supported embedded artwork, even without Transcoder::Attachments(). Pictures are separate from ordinary tracks and retain their PNG/JPEG/BMP bytes without recompression.

Output Native representation
MP3 ID3 APIC pictures
FLAC FLAC PICTURE blocks
Ogg Opus, Vorbis, FLAC and Speex METADATA_BLOCK_PICTURE comments
M4A / audio-only MP4 MP4 cover metadata
AIFF / AIFC ID3 pictures
Matroska audio Image attachments

Multiple pictures are retained without duplication. Picture type is not retained in the public snapshot: destinations requiring it receive the first as front cover and the rest as other. Ogg picture blocks must be smaller than 16 MiB including MIME/description.

WAV, CAF, ASF/WMA, WavPack, AU and raw AAC/AC-3/E-AC-3 have no implemented native artwork-writing route here; ordinary audio output remains usable. APE is not writable by the bundled muxer. General attachments such as subtitle fonts still require explicit selection; unsupported selected attachments fail, and WebM rejects attachments.

Preview playback with SimplePlayer

SimplePlayer is in StormByte::Multimedia, not Pipeline. It owns decoding, real-time scheduling, synchronization, playback state and track switching. The consumer owns presentation and device integration through one abstract Display.

This division lets Alchemist-like applications add a play button without implementing a media clock or another decoder pipeline, while keeping their existing widget toolkit and audio integration.

Open an input and control playback

// MyDisplay derives from Multimedia::Display; logger is optional here.
StormByte::Multimedia::SimplePlayer player(display, logger);
if (!player.Open("clip.mkv"))
ReportError(player.Error());
else if (!player.Play())
ReportError(player.Error());
// Later: application controls call Pause(), Seek(seconds), Play() or Stop().
// Poll State()/Error() too: decode and presentation continue asynchronously.
Real-time asynchronous file preview, not a professional media player.
Definition simple_player.hxx:230

The first ordinary video and audio tracks are selected initially; subtitles are opt-in. Use Tracks() to build selectors and pass each track's stream index to SelectTrack(Type, index). Index -1 disables that media kind. Attached pictures are not active video tracks; the UI chooses its audio-only placeholder.

Controls return bool; check failures and monitor PlaybackState::Error while playing. An accepted seek can still fail asynchronously. Ended means decoding and presentation have drained, not merely input EOF. After Ended, use Seek or Stop before Play.

Seek uses absolute media seconds and preserves playing/paused intent. Paused seeks reconstruct the covering image and active subtitle cues without sending audio; resuming starts from that media position. Position() is the playback timeline, while Duration() is zero when unknown. Playback and absolute seeks do not require a known duration; a UI must not turn unknown duration into an upper seek limit of zero.

Build a Display adapter

Include display.hxx and derive one application class from StormByte::Multimedia::Display. Implement all eight callbacks, even for an audio-only consumer. Construct it with Safe::MakeShared<MyDisplay>() and pass the shared owner to the player; the dynamic type is preserved. No Qt, SDL, FFmpeg or operating-system headers/handles are required by the public player contract.

Think of the adapter as four small cooperating pieces:

  1. A current-generation token and bounded, thread-safe delivery queues.
  2. A video/subtitle surface on the application's GUI thread.
  3. An audio device adapter with format/layout negotiation and a truthful clock.
  4. Pause/flush/drain bookkeeping connecting those pieces.

The player already schedules presentation. The adapter should not build a second independent playback timeline or decode the input again.

Threading and invalidation come first

Callbacks run on worker or control-caller threads, not necessarily the GUI thread. Do not reenter any player method from a callback, including getters. Keep callbacks bounded and marshal widget work to the UI thread. Throw on delivery/device failure so the player can expose it; do not silently discard current-generation output.

Invalidate(epoch, active_video) synchronously publishes the new generation, flushes audio, clears cues/images and cancels pending presentation timers. Deferred GUI work must compare its epoch again when executed, not just when queued. Otherwise an old image or subtitle can reappear after a seek, stop, close or track switch. When active_video is false, draw the application's chosen black/cover/placeholder; the library does not create dummy pixels.

Pause(paused, epoch) freezes/resumes the audio device and subtitle timers for the current generation. Queued pause operations must also respect invalidation.

Optionally override InvalidateAudio(epoch) to synchronously flush PCM and device timing without clearing video, subtitle cues or changing the epoch, and return true. This enables live audio-track changes without a demux seek, video-decoder reset or global invalidation. New audio starts at the demux cursor; decoded audio/video read-ahead is bounded to about two seconds of presentation time (extended only while another selected stream stored later in the file still needs data, within the existing queue limits), so the player bridges that silence, plus any such storage skew, with wall timing until the new audio clock catches up, then reapplies Pause before delivering new PCM. Callbacks remain bounded and serialized; cancel deferred audio work using an adapter-local audio generation if necessary. Return false without side effects when unsupported (the default); selection then uses the existing global seek. Paused and end-of-input changes still use seek reconstruction, and selecting the already-active track is a no-op. Rebuild consumers against the new Display interface when adopting this unreleased API.

Deliver pictures, samples and cues

Callback Payload Adapter work
Video(image) Safe-owned packed RGBA8, dimensions, media position/duration and epoch Present or enqueue the scheduled picture; recheck epoch on GUI execution
Audio(audio) Safe-owned actual decoded PCM, rate, scalar format, planarity, ordered channel identities, sample count, position and epoch Queue without waiting for playback; negotiate device format and map channels
Subtitle(cue) Text, ASS events/style/fonts or RGBA8 bitmap regions, coordinates, cue identity and timing Render the appropriate representation and preserve overlapping cues

For audio, m_planes contains one plane per channel when planar, otherwise one interleaved plane. Bytes are native-endian, tightly packed and exclude decoder padding. The player performs no downmix, remapping or resampling. Signed integer and floating formats, mono, stereo and multichannel sources remain explicit.

The adapter owns scalar conversion, interleaving, sample-rate conversion and device negotiation. Interpret m_layout and ordered m_channels, not just their count: BL/BR and SL/SR are different speakers. Unspecified, unknown/custom and ambisonic identities are not permission to guess a speaker map. Reject unsupported layouts or make an explicit application policy; do not silently force stereo. Reconfigure safely if the decoded format changes.

Subtitle decoding is not subtitle drawing. ASS needs styling and embedded font handling; bitmap cues need canvas placement. Position(position, epoch) advances the subtitle clock and expires finite cues at m_end. Infinite cues remain until cleared. Empty m_regions clears the given m_id, or all cues when ID is zero. Invalidation clears every old-generation cue.

Report what has actually played

AudioClock(epoch) returns current-generation AudioPresentation:

  • m_valid says meaningful device feedback is available.
  • The audible cursor is m_position - m_latency, in media seconds; latency is nonnegative.
  • The cursor must remain frozen during pause and account for samples still unheard in device buffers.
  • m_drained concerns audio only, not queued GUI work.

Without usable audio feedback, the player uses its steady clock, including audio gaps and video/subtitle tails. Do not report queued samples as already heard.

Drained(epoch) returns true only when all pending audio and GUI deliveries for that generation are finished. This is how the player distinguishes accepted callbacks from completed presentation at EOF.

Bound memory in the adapter too. By default, audio arrives at most 500 ms ahead plus one decoded block; keep enough device headroom without retaining the whole stream. Keep video and GUI queues short. Playback targets a 750 ms decoded cushion before starting or resuming from a reconstructed seek, without advancing the media clock during that wait. It starts earlier at EOF or when queue limits prevent further filling. Pending events are bounded to 256 units/128 MiB during playback; individual units and paused previews retain the 64 MiB limit. Storage is allocated on demand, not reserved upfront. In-flight conversion, decoder scratch, codec reference frames and adapter queues are additional storage. This absorbs temporary scheduling stalls, not sustained decoding slower than real time.

These are defaults, not fixed limits for every application. Pass a PreviewBuffering as the third constructor argument (SimplePlayer(display, logger, buffering)) to choose Preload, ReadAhead and AudioAhead as std::chrono::milliseconds, plus MaxBytes and MaxEvents. The policy is copied and immutable for that instance; the existing two-argument constructor uses the defaults. Zero preload disables the initial wait. Caps and read-ahead must be positive, and preload/audio advance must be nonnegative and no greater than read-ahead, otherwise construction throws. A decoded unit larger than the configured byte budget fails playback explicitly instead of waiting forever. Custom audio advance must fit the application's Display/device queues. Paused previews retain the smaller of the configured byte budget and 64 MiB. Embedded fonts, codec reference buffers and conversion scratch are separate from the pending-payload budget.

Video color conversion and HDR tone mapping run on a separate worker so they do not serialize audio decoding. That worker retains at most two pending decoded frame references plus the frame being converted, outside the pending-payload budget. When conversion falls behind the playing timeline and the initial cushion is exhausted, overdue pending images may be skipped to let audio decoding continue. Paused seek previews and initial prebuffering do not use that policy. The player reports the number skipped when workers stop. This prioritizes audio continuity under processing pressure; it does not guarantee every video frame under sustained CPU starvation.

Actual backend and measurements

player.Diagnostics() returns a thread-safe PreviewDiagnostics snapshot. Use m_decoder, m_backend and m_device to log actual decoder output. PreviewDecodeBackend::Unknown means no decoded frame has established the backend yet; Auto is only a policy and never proof of hardware acceleration. Changing Acceleration takes effect when a decoder is next opened. An audio-only source has no video decoder identity.

m_skipped_video counts pressure drops before conversion; m_shown_video counts successful Display/HDRDisplay callbacks, not compositor presentations. m_audio_underruns counts distinct valid AudioClock drained episodes while audio production is active, excluding preload, pause, audio switching and EOF drain. This is sampled starvation, not a device xrun counter; zero with m_audio_clock_samples == 0 means unavailable feedback, not verified continuity. Counters reset on Open, survive seeks/stop/close; decoder identity clears on decoder close. All snapshots contain public scalars and Safe strings only.

m_converted_video and m_conversion_fps are separate from these counters. Throughput divides converted pictures by the sum of measured conversion-stage wall seconds (no queue waits). Stage fields include setup and internal worker scheduling, not summed CPU time: download, HDR input color/linearization, tone mapping, HDR output color, and RGBA allocation/packing. For SDR, swscale fuses color conversion and packing; only packing is charged. m_peak_queue_bytes counts presentation payload only, not RSS, codec references or scratch. Application OpenGL uploads and real device/compositor behavior are outside these measurements.

Build MultimediaSimplePlayerTests with ENABLE_TEST=ON, then run:

bash test/measure_preview.sh build/test/MultimediaSimplePlayerTests \
/path/to/input.mkv 4 10 build/test-output/preview-measurements/my-source

The script saves three baseline and three contention CSV/log pairs, using the unchanged buffering defaults and SDR RGBA8 output. Contention runs a separate process with nproc busy threads for two seconds, starting one second after launch. Startup is Open-to-first-delivery; seek is Seek-to-first-delivery. Peak RSS comes from getrusage on Linux/macOS (-1 when unavailable). For forced software, invoke the runner directly:

build/test/MultimediaSimplePlayerTests --measure /path/to/input.mkv 4 10 software

Reproduce the generated SDR/20 ms Opus signals used in the software baseline:

ffmpeg -f lavfi -i testsrc2=size=1920x1080:rate=24:duration=12 \
-f lavfi -i sine=frequency=440:sample_rate=48000:duration=12 \
-c:v libx264 -preset ultrafast -threads "$(nproc)" -pix_fmt yuv420p \
-color_trc bt709 -color_primaries bt709 -colorspace bt709 \
-c:a aac -shortest build/test-output/preview-sdr1080.mp4
ffmpeg -f lavfi -i sine=frequency=440:sample_rate=48000:duration=12 \
-c:a libopus -frame_duration 20 build/test-output/preview-opus20.opus

See CHANGELOG.md for results and pending hardware measurements.

Reader ownership and transfer to Transcoder

Open(std::string_view) constructs a BufferedFileReader, copying the path into its Safe::String. Open(Safe::Unique<Buffer::IO::BufferedLocationReader>) takes exclusive ownership of a compatible buffered reader, preserving its subtype. It opens a not-yet-open reader once or rewinds a healthy already-open one. Failed Open retains the reader.

Operation Workers/presentation Reader
Stop() Cancel/join and invalidate Rewind the actual buffered reader; retain it open
Close() / destructor Join and invalidate before releasing input Close/release without rewind
ReleaseReader() Stop/join and invalidate Rewind, then transfer the same open dynamic reader at cursor zero

Stop/transfer failures are explicit and retain ownership. An empty transfer result means failure, not a closed replacement reader. Destruction is noexcept and does not close the shared Display/widget.

auto input = player.ReleaseReader();
if (!input)
ReportError(player.Error());
else {
std::move(input), destination, logger);
// Configure output tracks, check errors, then Run().
}

This matters for Network's BufferedRemoteFileReader: closing permanently releases its mount token, so that object cannot simply reopen for transcoding. Transfer the healthy reader instead and keep its client/transport alive. The player rejects closed/faulted rewind paths rather than using Close+Open as recovery. A closed/faulted remote source requires a new reader.

Scope and limitations

The common engine and abstract contract support Windows, Linux and macOS. Consumers provide their platform integration; no native output backend is bundled into Multimedia. Decode availability depends on the configured codecs. Cross-platform design is not a claim that every native backend or platform was runtime-tested.

SimplePlayer::Acceleration(DecodeAcceleration::Software) forces CPU video decoding; the default Auto tries an available backend (VideoToolbox, D3D11VA/DXVA2, CUDA/NVDEC or VA-API as built) and safely opens the software decoder when initialization is unavailable. The same choice is configurable independently for each encode lane through Pipeline::Transcoder::Track::Acceleration(...) or the lower-level Pipeline::Config::Implementation::Acceleration field; it does not pin the decoder implementation or change encoder policy.

Preview defaults to tightly packed RGBA8 for an SDR-oriented Display. SimplePlayer::OutputColor(PreviewColorMode::HDR) opts into an additional HDRDisplay interface; PQ/HLG frames arrive as tightly packed RGBA16LE signal values with transfer, primaries and optional mastering/content-light metadata, without SDR tone mapping. An HDR sink must implement both interfaces; requesting HDR mode without it returns false and logs an error. SDR sources continue through the existing RGBA8 callback. This is not a native swapchain, display-capability negotiation, Dolby Vision rendering, or guarantee that the consumer's OS/display is in HDR mode. With the default SDR mode, PQ/HLG frames are converted through linear light, tone-mapped and converted to BT.709 before presentation. There is no surround bitstream passthrough or automatic subtitle compositor. Subtitle seeks reconstruct cue state from the beginning, so long inputs can take time; reconstruction is bounded and fails explicitly if it exceeds storage limits.

Buffering permits local or network-backed playback without loading the whole file. Reads/probing/seeks and worker joining can wait for bounded origin I/O: configure network timeouts. Actual seek support depends on the reader and demuxer, not on having a duration value.

Interactive output demo

The optional SDL2 demo is a real consumer-side Display example, not an installed library backend. Its implementation is in test/preview_demo.cxx; SDL linkage stays outside Multimedia's public API.

cmake -S . -B build -DENABLE_PREVIEW_DEMO=ON
cmake --build build --target MultimediaPreviewDemo -j"$(nproc)"
build/test/MultimediaPreviewDemo "/path/to/video.mkv"

Controls: Space play/pause, Left/Right seek 30 seconds, Home stop/rewind, A/V/S cycle audio/video/subtitles including disabled, Esc close. It preserves image aspect ratio and shows text cues in the window title, but does not compose ASS styling or bitmap subtitles.

Use --smoke for three-second segments or --listen for thirty-second segments: initial playback, seeks to 50%, 90%, 10%, then Stop/verify zero. Unknown-duration input uses absolute targets 30, 120, 10 seconds instead.

The audio adapter requests the source rate and all channels, converts decoded PCM to device float32 and maps supported speaker sets to SDL order. It rejects unsupported/unknown/ambisonic layouts rather than falling back to stereo. SDL/hardware conversion and physical speaker routing remain external; an obtained six-channel spec is not proof of physical six-channel hardware or exact latency.

Reports include actual post-seek video/audio positions, image hashes, callback short reads, missing frames, inserted silence and sample peaks. These help diagnose delivery; clean counters do not prove audible correctness. Add a diagnostic directory after either mode to capture bounded per-epoch PCM with format/layout metadata sidecars.

Shared integration rules

DLL boundaries and lifetimes

Public owning data crosses modules through StormByte::Safe values, collections and owners. Create polymorphic owners with Safe::MakeShared<T>(...) or the appropriate Safe unique factory; preserve dynamic type and provider-local destruction rather than slicing. Borrowed registry identities must outlive their users.

Safe ownership is not arbitrary-ABI compatibility or a safe-unload guarantee. Consumers/providers need compatible C++/STL ABIs, and every provider supplying live values, release operations or callbacks must remain loaded until those objects are released. STORMBYTE_DECLARE_MAYBE_SAFE is the provider's boundary-safety promise, not an automatic audit.

Derived Transcoder providers create their own Plan/TrackSettled payloads in EmptyPlan()/EmptySettled() using provider-safe factories. Plans remain shared and do not implement cloning/moving; derived settled payloads must implement their required Clone()/Move() operations in the creating module.

Logging

Stages and filters accept Safe::Shared<Logger::Log>; Transcoder requires a nonempty logger, while SimplePlayer's logger is optional. Prefer ThreadedLog for concurrent workers and keep its underlying stream alive through every retaining job/stage.

The application controls logging levels, headers and redaction. Processing stages scope their messages under StormByte/Multimedia/<stage>. Detailed per-unit logging has a real cost; use it for diagnosis rather than ordinary UI progress. Observe status/error explicitly instead of treating a quiet log as success.

Build options and distribution

Option Values Meaning
WITH_FFMPEG BUNDLED (default) / SYSTEM Bundled libav engine or compatible system installation
WITH_VMAF BUNDLED / SYSTEM Must match FFmpeg mode; a differing cache value is overridden with a status message
WITH_OCR BUNDLED (default) / SYSTEM Tesseract/Leptonica; Windows forces bundled OCR
WITH_TESSDATA BUNDLED (default) / SYSTEM Embedded selected models or installed runtime models, independent of OCR mode
WITH_ZIMG BUNDLED / SYSTEM Image-processing dependency
WITH_LUNASVG BUNDLED (default) / SYSTEM SVG watermark renderer; SYSTEM requires LunaSVG 3.5 or newer. Bundled builds disable system font loading and the font-face cache
BUILD_SHARED_LIBS ON (default) / OFF Shared/static Multimedia and matching StormByte archives
ENABLE_TEST ON / OFF (default) Build/register CTest; PGO enables tests in the instrumented tree
ENABLE_CI_FIXTURES ON / OFF (default) CI-only package discovery fixtures; explicitly enabled by CI, not needed for local tests or PGO
ENABLE_PREVIEW_DEMO ON / OFF (default) Optional SDL2 output demo; requires SDL2 development files
ENABLE_LTO ON (default) / OFF Link-time optimization
ENABLE_PGO ON / OFF (default) Instrument/train in pgo-instr, then build the profiled library
ENABLE_ASAN ON / OFF (default) Debug ASan/UBSan on supported non-Windows builds; disabled for Release
WITH_GPL ON / OFF Permit GPL bundled components, such as x265/Rubberband; OFF also omits Multimedia Rubberband/VagueDenoiser sources and headers in system mode
WITH_NONFREE ON / OFF Permit nonfree bundled components such as FDK-AAC and dependent filters

These flags select dependencies/capabilities, not the license of original Multimedia source. System installations determine their own codec availability and licensing. Review the enabled components when distributing an application.

CI/release CPU baselines are selected by the shared build action: x86-64-v3 on Linux amd64, ARMv8.2-A on Linux arm64, AVX on Windows x64 and Apple M1 on Apple Silicon. Direct CMake builds preserve the compiler's defaults and caller-provided CPU flags. Bundled specialized SIMD kernels retain runtime selectors without raising the global baseline; bundled x86 FFmpeg requires NASM.

Linux releases cover amd64/arm64, glibc/musl and GCC/libstdc++ or Clang/libc++. Their archive names are StormByte-Multimedia-<tag>-linux-<arch>-<libc>-<stdlib>.zip. macOS uses macos-arm64 and Windows uses windows-x64, without a runtime suffix. Each release build uploads its own archive directly; documentation deployment is independent.

For GCC/Linux builds with LTO, configure checks for the Alpine fortify-headers/LTO bug. When the toolchain is affected, Multimedia adds -fno-builtin-vsnprintf to its C++ build flags, including PGO training and bundled CMake dependencies such as Tesseract. LTO and fortify remain enabled. Consumers compiling their own code may need the same flag until their toolchain is fixed.

Bundled FFmpeg is pinned to 42c25af15f51 on meson-9.0. Its upstream Meson build includes the clang-cl, VA-API, intrinsics and native AArch64 generator fixes merged in !208; the corresponding five local patches have been removed.

Bundled Opus presumes NEON on ARM64 when the compiler supports its intrinsics; 32-bit ARM does not assume NEON. Bundled FFmpeg includes the upstream AArch64 swscale fix for signed intermediate samples and uses upstream Meson NEON/SSE2 intrinsics probes, building the NEON MPEG unquantizer only when NEON intrinsics are detected.

Clang/Linux AArch64 PGO training disables value profiling to avoid the concurrent profiler hang tracked in llvm/llvm-project#160968. Execution-count profiling, LTO, SIMD and codec multithreading remain enabled. The workaround is removed from the optimized build.

Bundled tessdata embeds the models selected in the language configuration, including available vertical variants. System models are loaded from the configured installation; use TESSDATA_DIR, TESSDATA_PREFIX or Tessdata_ROOT to select it and deploy the required files. Partial installations are accepted, but an absent requested language fails without fallback. OCR receives decoded grayscale subtitle pixels, not external image files.

Bundled SVG rendering uses LunaSVG 3.5.0/PlutoVG 1.3.1. Binary packages must retain THIRD-PARTY, COPYING.LunaSVG and COPYING.FTL alongside the existing notices.

Installation

Requires a C++26-capable compiler/standard library, CMake with CXX_STANDARD 26 support (3.25+) and StormByte-BuildMaster 2.1.5 or newer. BuildMaster downloads matching Base/Buffer/Logger/System release archives by architecture, Linux C library and C++ runtime; these are not submodules. Bundled media dependencies need recursive checkout and NASM on x86.

git clone --recursive https://github.com/StormBytePP/StormByte-Multimedia.git
cd StormByte-Multimedia
cmake -S . -B build \
-DWITH_FFMPEG=BUNDLED -DWITH_VMAF=BUNDLED \
-DWITH_GPL=OFF -DWITH_NONFREE=OFF
cmake --build build -j"$(nproc)"
cmake --install build --prefix <prefix>

On platforms without nproc, pass the detected available CPU count to -j.

The in-tree target is StormByte::Multimedia, the library is StormByte-Multimedia, and installed headers use #include <StormByte/multimedia/...>. Deploy runtime dependencies with a compatible ABI; public headers do not expose the platform or media-engine implementation dependencies.

Installed prefixes and extracted release ZIPs provide FindStormByteMultimedia.cmake. Put both the Multimedia prefix and its matching StormByte dependency prefix in CMAKE_PREFIX_PATH, and their share/cmake/Modules directories in CMAKE_MODULE_PATH, then consume the imported target:

find_package(StormByte REQUIRED COMPONENTS Multimedia)
target_link_libraries(my_application PRIVATE StormByte::Multimedia)

The finder brings in Base, Buffer, Logger and System transitively and propagates the shared-library definitions required by Windows consumers.

Tests

cmake -S . -B build-tests -DENABLE_TEST=ON \
-DWITH_FFMPEG=BUNDLED -DWITH_VMAF=BUNDLED \
-DWITH_OCR=BUNDLED -DWITH_TESSDATA=BUNDLED
cmake --build build-tests -j"$(nproc)"
ctest --test-dir build-tests/test --output-on-failure

Processing tests cover registries, File, facade/manual graphs, codecs, filters, remux/encode, mux policies, interleaving, OCR, artwork and HDR metadata. Outputs are reopened and decoded through library APIs; tests do not require external media command-line tools. HDR tests compare metadata and frame counts, not merely successful file creation. Hardware cases preflight device/profile support; unavailable implementations skip, but an available implementation's processing failure does not.

For shared builds with bundled StormByte dependencies, PackageDiscoveryTests installs Multimedia, creates and extracts a ZIP, then configures, links and runs a separate consumer against each prefix. The consumer probes the stereo WAV fixture and verifies audio properties and the codec registry. OptimizationTests checks that the common LTO module does not inject x86 flags into ARM builds or overwrite caller-supplied flags.

Preview tests use a clocked fake Display, synthetic inputs and a remote-like reader with irreversible close. They cover exact decoded audio, playback timing, pause/seek/track changes, subtitles, errors, ownership and transfer into Transcoder without another origin Open. Run the focused group with:

cmake --build build-tests --target MultimediaSimplePlayerTests -j"$(nproc)"
ctest --test-dir build-tests/test -R '^preview\.' --output-on-failure

The skewed-interleave software-playback tests bound video lateness to 100 ms in ordinary builds and 250 ms under PGO instrumentation or ASan, where shared-runner scheduling and instrumentation can delay delivery. The audio underrun (50 ms), video lead (100 ms), delivery gap (500 ms), and timestamp-continuity checks remain unchanged.

The live-audio cases also cover disabled audio, repeated/rapid selections, full PCM queues, sink errors and pause/seek/EOF reconstruction. For a local multitrack file (not redistributed with the tests), the same registered case cycles every audio track and checks increasing video timestamps, bounded delivery gaps and unchanged video epoch:

STORMBYTE_MM_LIVE_INPUT=/path/to/local-multitrack.mkv \
ctest --test-dir build-tests/test -R '^preview.test_live_audio_tracks$' --output-on-failure

See fixture documentation for provenance and redistribution notices. Test success is not certification of every codec/platform, visual restoration quality or long-running workload. Real-output listening uses the optional demo separately.

Contributing

Issues only on this repository. Fork and open a pull request against master.

Public API does not grow “because the coordinator needs it”. No new friends. Doxygen on a header is part of the file: update it so it does not lie, do not delete it. Commits are English, one topic, feat(pipeline): … / fix(watermark): ….

License

StormByte-Multimedia original source is dual-licensed:

  1. GNU Lesser General Public License v3.0 (or later)
    Redistribute and/or modify the original source under the LGPL v3 or any later version.
    Full text: [LICENSE](LICENSE), COPYING.LGPLv3, https://www.gnu.org/licenses/lgpl-3.0.html.
  2. Commercial license
    The same original source may be used under a commercial agreement with the copyright holder (David C. Manuelda Storm.nosp@m.Byte.nosp@m.@gmai.nosp@m.l.co.nosp@m.m).
    That option requires a written agreement. Without it, the LGPL applies.

Both licensing options cover original StormByte-Multimedia source only. Third-party components — including FFmpeg, libvmaf and embedded trained data — keep their own licenses and are not covered by the commercial grant. See [NOTICE](NOTICE) and thirdparty/.

A written StormByte commercial agreement may license the original StormByte-Multimedia code on terms other than the LGPL, including specific use and linking arrangements such as static linking, as stated in that agreement. It does not grant rights to dependencies or waive their license conditions. In particular, enabling WITH_GPL or WITH_NONFREE can add separate obligations to a binary or other work that uses those components, including for modification, linking (static or dynamic) and redistribution. The packager and final user are responsible for determining and meeting all applicable license and distribution requirements and obtaining any needed patent permissions. A StormByte license does not provide those permissions for GPL or nonfree components and grants no patent rights.

SPDX: LGPL-3.0-or-later OR LicenseRef-StormByte-Commercial.

The headers of the public and private trees repeat this grant. When in doubt, those headers and LICENSE win over this README.

Supporting the project

If this saved you from building a separate media pipeline inside your application, a star is the polite nod. A well-aimed issue beats a vague “it broke”. Pull requests that keep the public API focused — Plan, Step, Transcoder, SimplePlayer, Display and composable filters — are the ones that land.

I wrote this because the alternative was another private transcoder in every product. Maintaining that difference takes evenings.

Sponsor StormBytePP on GitHub

Use it. Break it on purpose. Tell me which sentence in this file lied.