Contributor guides: Contributing and Coding Style.
StormByte-Multimedia brings media processing and real-time preview into a C++ application without making that application manage FFmpeg's C API. The complexity stays inside the library: asynchronous workers, stream routing, decoding, filtering, encoding, muxing and presentation timing. Outside, applications describe a job or connect a player to their own display.
There are two independent workflows:
| Goal | Starting point | What the application supplies |
|---|---|---|
| Copy, encode or filter media into an output | Pipeline::Transcoder | Input/output locations, output-track mapping and settings |
| Build a specialized processing graph | Pipeline::Plan and Step stages | The same job description, explicit connections and graph lifetime |
| Preview video, audio and subtitles in a UI | Multimedia::SimplePlayer | A buffered input and a consumer-implemented Display |
Transcoder is a complete facade over the processing pipeline, not a reduced encoder. Manual wiring exposes the same machinery when a graph needs more control. SimplePlayer handles real-time playback independently; it is a preview service, not a replacement for a professional media player, an editor or a bundled Qt/SDL widget.
It depends on StormByte Base, StormByte Logger and StormByte System at version 3.0.0, and StormByte Buffer at version 4.0.0. Buffer must be the retagged 4.0.0 build exposing ByteBudget::SharesIdentity in both its headers and library; older artifacts with the same version are incompatible. Each is a release archive. Public headers live under StormByte/multimedia/.
The suite is split on purpose. Base, Buffer, Config, Crypto, Database, Logger, Network and System are other repositories. This one does not implement them.
| Module | Role | API |
|---|---|---|
| Base | Exceptions, Expected, serialization, strings, UUID, concepts | /StormByte |
| Buffer | FIFO, SharedFIFO, Ring, Producer/Consumer and multi-stage pipelines | /StormByte-Buffer |
| Config | Human-readable text and versioned binary documents (groups, lists, raw bytes) | /StormByte-Config |
| Crypto | Hash, compress, encrypt, sign and key agreement — Crypto++ never leaves the private tree | /StormByte-Crypto |
| Database | One API over SQLite, PostgreSQL and MariaDB | /StormByte-Database |
| Logger | Stream logging, levels, headers, redaction, ThreadedLog | /StormByte-Logger |
| Multimedia | This repository | /StormByte-Multimedia |
| Network | Framed packets, Client/Server, IPv4/IPv6 TCP and Buffer pipelines (compress/encrypt) | /StormByte-Network |
| System | Processes, pipes and environment variables across Linux, Windows and macOS | /StormByte-System |
This README explains the workflows, integration choices and important contracts. It is not a method-by-method API reference.
Use File to inspect streams, typed audio/video properties, metadata and attachments before presenting choices to a user. File::Streams() contains ordinary tracks; File::Attachments() exposes pictures and other attachments separately. An album cover is not an ordinary video track.
The registry provides stable codec/container identities, aliases and the implementations available in the configured build. A name being registered does not imply an encoder, decoder or suitable output container is available.
HasAccess(Access, Features) requires a covering available implementation in each requested direction. An empty feature mask is the generic access query. This is a compiled-capability check, not a device probe or a guarantee that every setting, profile and container combination will open.
Codec::Encoders(features) lists concrete implementations and their supported controls; EncoderInfo::Option(name) describes option types, ranges, defaults and choices. This lets a UI expose settings the selected encoder can actually apply.
Codec::SubtitleKind() classifies subtitle codecs as text, bitmap or unknown independently of decoder/encoder availability. ASS, SRT, WebVTT and mov_text are text; PGS, DVD/DVB and XSUB are bitmap. Classification does not imply OCR or rendering support.
Processing creates an output, not a timed presentation. Both entry points use the same demux/decode/filter/encode/mux stages and explicit failure handling:
Remux the first video stream into Matroska. The output suffix selects the container; the application does not create FFmpeg contexts or connect workers.
ResourceUsage continues to select the CPU profile and automatic queue caps. Use BufferingPolicy to override those element caps independently, including while keeping ResourceUsage::Full. QueueLimit::Automatic() follows the selected profile, QueueLimit::Bounded(n) requests a strict item limit, and QueueLimit::Unlimited() disables that item limit. A video-frame cap must be at least two; audio-frame, packet and demux-pending caps must be at least one.
The same value can be set on a manually created Plan before connecting or starting its stages with plan->Buffering(queues). Invalid minima are rejected without changing the previous policy. These overrides affect queue element counts and logical payload-byte admission; they do not change codec CPU threading. Byte weights are estimates of queued frame or packet payload, not process RSS or uniquely owned physical memory.
Newly wired track queues attach their byte limits and shared budget before producers can enqueue. Inputs wired before Plan binding receive the Plan's policy before workers start; initial element ceilings are installed too. A stage may refine its element ceiling during startup without rebinding that budget or discarding packets already waiting in its input, including clone lanes and inputs shared by multiple producers.
Manually mounted stages expose the effective limits and live queue accounting for an input track through Step::QueueSnapshot(track) and Filter::FFmpeg::QueueSnapshot(track). Read the requested mode from the Plan's BufferingPolicy, the applied limits and occupancy from the queue snapshot, and shared reservations from Plan::Buffering().Budget().Snapshot(). A snapshot with Exists == false means that track's input queue is not wired yet.
The muxer uses a positive 0.25-1 second media-DTS window in FFmpeg's interleaver, including encoded-video/remux combinations. An empty or sparse subtitle track therefore cannot select the former 60-second window; queue depth and header-startup timestamp span do not enlarge it. This is not a wall-clock delivery deadline: encoder startup, ordered subtitle processing, container clustering, AVIO and the output writer can still delay bytes.
The muxer's own audio/video readiness guard is separate. Its existing 60-second media-DTS skew fallback protects delayed encoders; reducing it to one second lets remuxed audio run ahead of video with long lookahead. Encoded subtitle progress watermarks still protect cue ordering. Payloads held by this guard, codec internals and FFmpeg's interleaver are not covered by the queue ByteBudget; neither time window guarantees a byte/RSS cap. Buffer's output telemetry counts bytes only once the writer receives them, not packets held inside the muxer.
Map the streams you want, in output order, and choose Remux() or Codec(...) for each. Omitted streams are dropped. These calls append outputs: Audio(1) twice means two independent destination tracks, not another lookup of the first one.
For example, after constructing a job and checking the registry identities:
This can preserve the compressed original audio and add an E-AC3 compatibility track alongside it. Each output has independent settings, filters and metadata overrides. Save a returned Track handle when further configuring that particular output.
| Choice | Meaning |
|---|---|
| Remux() | Forward compressed media with destination adaptation; no separate “Copy” stage |
| Codec(identity) | Decode and encode with a registered destination codec |
| Implementation(name) | Pin the encoder; unavailable or mismatched pins fail without fallback |
| Implementation(ImplementationSide::Decoder, name) | Independently pin the source decoder on an encode lane |
| CRF, BitRate, MaxBitRate, Preset, Tune, FineTune | Configure supported quality/rate/implementation options |
| Language, Title | Override metadata on that output |
| Ignore(n) | Remove every mapped output of origin stream n |
| Attachments() / Attachments(mime) | Select general attachments, otherwise dropped; automatic audio artwork is described below |
| Explicit registered Container constructor argument | Override output-suffix inference; the identity is borrowed |
Unsupported settings fail rather than being ignored. Audio quality-based VBR replaces bitrate when selected; without a quality or bitrate setting, the encoder default applies, not the source bitrate. FineTune accepts individual keys and packs them into implementation options where necessary.
Encoder preparation preserves supported representations or performs exact storage repacking, such as yuv420p10le/p010le. Depth reduction, chroma resampling and color conversion require an explicit filter. Hardware encoding and decoding are independent: decode can be hardware-assisted while encoding is software, and vice versa.
Video decoders use DecodeAcceleration::Auto by default on preview and encode lanes. Auto checks FFmpeg's hardware configurations for the selected codec, tries the platform backend, then opens a software decoder if device creation or hardware decoder initialization fails. Use SimplePlayer::Acceleration(...) or Track::Acceleration(...) to force CPU decoding for preview or an individual transcode lane. Decoder pins and EnforceFeatures remain independent of this setting; remux tracks do not open a decoder or device. Hardware-decoded frames are transferred to software storage with FFmpeg frame properties preserved before entering preview conversion or the filter pipeline, so this path is not zero-copy.
MPEG-4 Part 2 (DivX/Xvid, not H.264 in an MP4 container) always uses software decoding, even in Auto mode. Hardware decoding can corrupt GMC (global motion compensation) frames, including when they first appear later in the stream. This policy avoids an additional compatibility scan and applies to preview and conversion with both bundled and system FFmpeg. Hardware encoding, including NVENC, remains available; other video codecs retain Auto acceleration.
The bundled FFmpeg configuration enables VideoToolbox on macOS, probes VA-API on Linux, enables NVDEC where the NVIDIA codec-header plugin is available, and probes D3D11VA/DXVA2 on Windows. Device drivers and codec/profile support are runtime-dependent; Auto falls back only during device/decoder initialization, not after a stream has started failing.
Audio encode adaptation can automatically reduce 7.1 to 5.1 for AC-3/E-AC3. MP3 requires an explicit downmix for inputs above two channels. These are encoding policies, not SimplePlayer output behavior: preview preserves decoded channels.
Attach typed filters to an output with Track::Filter<T>(...); job-level Filter<Analytics>(...) attaches global analysis. The facade creates the filter graph, so a simple job can use the same filters as manual wiring.
| Use | Examples and considerations |
|---|---|
| Geometry | Scale, crop and pad; registration order determines the transformation order |
| Watermark | PNG, JPEG, WebP, BMP or self-contained SVG from a path or Safe::Binary |
| Restoration and cadence | Denoising, sharpening, decimation and experimental two-pass Degrain |
| Color | Explicit conversions and tone mapping; incompatible HDR metadata is removed from SDR output |
| Audio | Typed sample-rate, layout and processing filters |
| Quality analysis | VMAF compares decoded reference and destination pictures without modifying the output |
Watermark's designedFor size is the video reference size, not the requested logo size. SVG is rendered at the final paint resolution; external resources and system-font discovery are disabled. Convert text to paths and embed images. LunaSVG is not a browser SVG engine; validate the artwork visually. Invalid SVG dimensions warn and pass media through without a logo.
The three long-gap mov_text regression tests are skipped with WITH_FFMPEG=SYSTEM when the runtime FFmpeg release is 9.0.2 or older: those releases lack the custom patch applied to bundled FFmpeg. They still run for bundled FFmpeg and SYSTEM releases newer than 9.0.2. This test exclusion does not fix the subtitle timestamp limitation in an unpatched system library.
Degrain measures regional grain before filtering, preserves timing/geometry and has bounded look storage. It is an experimental heuristic, not motion-compensated restoration or a guarantee of perceptual improvement. Do not stack it with another grain-denoising filter.
VMAF reports scores and scored-frame counts per output. Wait for Done and inspect report status and counts, not just the mean. Resource modes govern workers and queue budgets; process memory still depends on resolution, concurrent contexts and retained pictures. A queue limit is not a whole-job RAM limit.
To implement a filter, follow the existing Scale/Watermark classes and the Doxygen contracts. Frame filters use Save for a one-for-one replacement, or Drop/EmitGraph/EmitFrame for buffering and multiple outputs, including EOF output. Analytics cannot mutate media. Use the library's RAII frame/packet types; the libav-backed graph surface is for operations not covered by the typed wrappers. Retain a scaling context for repeated direct conversions and use it from one thread at a time.
Run() is asynchronous; Pause() / Resume() / Cancel() control the coordinator. Success comes from Status() == Done, not a progress percentage or muxer EOF.
Progress()->Snapshot() reads the progress axes atomically:
| Axis | Purpose |
|---|---|
| Duration calculation | Source scan, only when neither a positive supplied duration nor a usable container hint exists |
| Main | Actual destination-writing percent; 100 when the trailer is written |
| Measure | Optional first pass for a two-pass filter |
| Analytics | Optional destination analysis, which can continue after main output finishes |
| All | Weighted overall display, below 100 until all mounted phases complete |
Show Main as the transcode bar, with measure/analytics separately when present. All is useful for a one-line overall indicator, not as the encode percentage. File::DurationHint() does not scan; File::Duration() retains full-scan behavior. Duration estimates do not limit how much media is processed. Observed scans support pause/cancel and rewind on completion or interruption.
After Done, Reports() contains analytics results. Capture Telemetry() before running if the final snapshot must outlive the job. It retains stage counts, setup/processing/wait times and sampled memory. Memory minimum/current/peak are process-wide RSS, not stage allocations; sampling can miss short peaks.
Telemetry()->Topology() returns an owned TopologySnapshot of the real stages and StormByte::Buffer queues of the latest run. It is thread-safe while running, paused, after cancel/failure and after the Transcoder was destroyed; the snapshot keeps no stage alive and holds no raw pointers.
Each topology stage also exposes an owned Activity sample: independent worker ids/generations, preparation, exclusive processing, input wait, local output wait, shared-budget wait, cooperative pause and unclassified time. Open intervals are included in live snapshots and terminal times freeze. Demux feeders are separate workers, not an exclusive aggregate stage state. Coverage and missing instrumentation are explicit; these are wall times, not CPU utilization. Existing counters and queue topology are unchanged. See the activity API and Alchemist delta example for retention, pause semantics, limitations and validation. This requires Buffer 4.0.0 retagged with its optional admission-wait observer.
Construction, graph attachment, owning metadata access and report snapshots may throw StormByte::Safe::AllocationError or std::bad_alloc; they do not promise allocation-free operation. Worker lifecycle callbacks retain their noexcept contracts and convert allocation failures into the stage/job failure state, waking blocked peers. After allocation failure the diagnostic text may be empty: check the failure state rather than treating an empty message as success. Shutdown and destruction do not depend on allocating formatted diagnostics.
Transcoder is designed to be inherited, but inheritance is not required for a normal job. Override OnProgress, OnDone and OnError to integrate a UI, scheduler or batch runner. Override EmptyPlan() / EmptySettled() for application-owned payloads while retaining the processing machinery and provider-safe ownership.
For a specialized graph, construct a shared Plan, configure its output tracks, create the stages and connect them with operator>>. A recode lane has this shape:
A remux lane substitutes Remuxer for decode/filter/encode. Muxer reserves output slots and interleaves destination packets; it does not trust worker arrival order. Fan-out permits independent routes from the same demuxer.
The wiring sequence below assumes already-constructed compatible stages and a checked Safe::Shared<Plan>:
Important differences from the facade:
Packet is compressed media and Frame is decoded media. Their serial is a lineage identifier, not a frame count; timing is not set through public Frame setters. Transform through filters and supported adapters.
Plan::Policy::EnforceFeatures is enabled by default: plans reject encode routes without an implementation capable of keeping source HDR10, HDR10+ or Dolby Vision. Disabling it permits feature loss; it does not manufacture metadata support. Configuration, state/error and telemetry are public; private source snapshots and reader/writer accessors are not an application inspection surface.
Encoding and remuxing copy the supported typed music tags; the destination chooses their native representation. This does not preserve every arbitrary ID3 frame or a byte-identical metadata block. Muxed output advertises StormByte-Multimedia <version> where application tags exist; technical muxer/bitstream identifiers remain truthful, and raw streams may carry no branding.
Feature checks distinguish static HDR10, HDR10+ dynamic metadata and Dolby Vision. They are not interchangeable with ten-bit pixels or hardware acceleration.
Pixel transformations can make metadata stale. DOVI-only content must not acquire an invented HDR10 fallback. Validate or regenerate metadata in a mastering workflow after grading, geometry or color changes. Successful processing and metadata preservation are not visual correctness, Dolby certification or calibrated presentation.
Remux compatible subtitles or select an output subtitle codec to convert them. Bitmap-to-text conversion uses OCR; the source language selects the model, and missing models fail explicitly. OCR is not translation. This belongs to the processing workflow; SimplePlayer instead delivers decoded cues for the application's renderer.
Audio-only outputs automatically copy supported embedded artwork, even without Transcoder::Attachments(). Pictures are separate from ordinary tracks and retain their PNG/JPEG/BMP bytes without recompression.
| Output | Native representation |
|---|---|
| MP3 | ID3 APIC pictures |
| FLAC | FLAC PICTURE blocks |
| Ogg Opus, Vorbis, FLAC and Speex | METADATA_BLOCK_PICTURE comments |
| M4A / audio-only MP4 | MP4 cover metadata |
| AIFF / AIFC | ID3 pictures |
| Matroska audio | Image attachments |
Multiple pictures are retained without duplication. Picture type is not retained in the public snapshot: destinations requiring it receive the first as front cover and the rest as other. Ogg picture blocks must be smaller than 16 MiB including MIME/description.
WAV, CAF, ASF/WMA, WavPack, AU and raw AAC/AC-3/E-AC-3 have no implemented native artwork-writing route here; ordinary audio output remains usable. APE is not writable by the bundled muxer. General attachments such as subtitle fonts still require explicit selection; unsupported selected attachments fail, and WebM rejects attachments.
SimplePlayer is in StormByte::Multimedia, not Pipeline. It owns decoding, real-time scheduling, synchronization, playback state and track switching. The consumer owns presentation and device integration through one abstract Display.
This division lets Alchemist-like applications add a play button without implementing a media clock or another decoder pipeline, while keeping their existing widget toolkit and audio integration.
The first ordinary video and audio tracks are selected initially; subtitles are opt-in. Use Tracks() to build selectors and pass each track's stream index to SelectTrack(Type, index). Index -1 disables that media kind. Attached pictures are not active video tracks; the UI chooses its audio-only placeholder.
Controls return bool; check failures and monitor PlaybackState::Error while playing. An accepted seek can still fail asynchronously. Ended means decoding and presentation have drained, not merely input EOF. After Ended, use Seek or Stop before Play.
Seek uses absolute media seconds and preserves playing/paused intent. Paused seeks reconstruct the covering image and active subtitle cues without sending audio; resuming starts from that media position. Position() is the playback timeline, while Duration() is zero when unknown. Playback and absolute seeks do not require a known duration; a UI must not turn unknown duration into an upper seek limit of zero.
Include display.hxx and derive one application class from StormByte::Multimedia::Display. Implement all eight callbacks, even for an audio-only consumer. Construct it with Safe::MakeShared<MyDisplay>() and pass the shared owner to the player; the dynamic type is preserved. No Qt, SDL, FFmpeg or operating-system headers/handles are required by the public player contract.
Think of the adapter as four small cooperating pieces:
The player already schedules presentation. The adapter should not build a second independent playback timeline or decode the input again.
Callbacks run on worker or control-caller threads, not necessarily the GUI thread. Do not reenter any player method from a callback, including getters. Keep callbacks bounded and marshal widget work to the UI thread. Throw on delivery/device failure so the player can expose it; do not silently discard current-generation output.
Invalidate(epoch, active_video) synchronously publishes the new generation, flushes audio, clears cues/images and cancels pending presentation timers. Deferred GUI work must compare its epoch again when executed, not just when queued. Otherwise an old image or subtitle can reappear after a seek, stop, close or track switch. When active_video is false, draw the application's chosen black/cover/placeholder; the library does not create dummy pixels.
Pause(paused, epoch) freezes/resumes the audio device and subtitle timers for the current generation. Queued pause operations must also respect invalidation.
Optionally override InvalidateAudio(epoch) to synchronously flush PCM and device timing without clearing video, subtitle cues or changing the epoch, and return true. This enables live audio-track changes without a demux seek, video-decoder reset or global invalidation. New audio starts at the demux cursor; decoded audio/video read-ahead is bounded to about two seconds of presentation time (extended only while another selected stream stored later in the file still needs data, within the existing queue limits), so the player bridges that silence, plus any such storage skew, with wall timing until the new audio clock catches up, then reapplies Pause before delivering new PCM. Callbacks remain bounded and serialized; cancel deferred audio work using an adapter-local audio generation if necessary. Return false without side effects when unsupported (the default); selection then uses the existing global seek. Paused and end-of-input changes still use seek reconstruction, and selecting the already-active track is a no-op. Rebuild consumers against the new Display interface when adopting this unreleased API.
| Callback | Payload | Adapter work |
|---|---|---|
| Video(image) | Safe-owned packed RGBA8, dimensions, media position/duration and epoch | Present or enqueue the scheduled picture; recheck epoch on GUI execution |
| Audio(audio) | Safe-owned actual decoded PCM, rate, scalar format, planarity, ordered channel identities, sample count, position and epoch | Queue without waiting for playback; negotiate device format and map channels |
| Subtitle(cue) | Text, ASS events/style/fonts or RGBA8 bitmap regions, coordinates, cue identity and timing | Render the appropriate representation and preserve overlapping cues |
For audio, m_planes contains one plane per channel when planar, otherwise one interleaved plane. Bytes are native-endian, tightly packed and exclude decoder padding. The player performs no downmix, remapping or resampling. Signed integer and floating formats, mono, stereo and multichannel sources remain explicit.
The adapter owns scalar conversion, interleaving, sample-rate conversion and device negotiation. Interpret m_layout and ordered m_channels, not just their count: BL/BR and SL/SR are different speakers. Unspecified, unknown/custom and ambisonic identities are not permission to guess a speaker map. Reject unsupported layouts or make an explicit application policy; do not silently force stereo. Reconfigure safely if the decoded format changes.
Subtitle decoding is not subtitle drawing. ASS needs styling and embedded font handling; bitmap cues need canvas placement. Position(position, epoch) advances the subtitle clock and expires finite cues at m_end. Infinite cues remain until cleared. Empty m_regions clears the given m_id, or all cues when ID is zero. Invalidation clears every old-generation cue.
AudioClock(epoch) returns current-generation AudioPresentation:
Without usable audio feedback, the player uses its steady clock, including audio gaps and video/subtitle tails. Do not report queued samples as already heard.
Drained(epoch) returns true only when all pending audio and GUI deliveries for that generation are finished. This is how the player distinguishes accepted callbacks from completed presentation at EOF.
Bound memory in the adapter too. By default, audio arrives at most 500 ms ahead plus one decoded block; keep enough device headroom without retaining the whole stream. Keep video and GUI queues short. Playback targets a 750 ms decoded cushion before starting or resuming from a reconstructed seek, without advancing the media clock during that wait. It starts earlier at EOF or when queue limits prevent further filling. Pending events are bounded to 256 units/128 MiB during playback; individual units and paused previews retain the 64 MiB limit. Storage is allocated on demand, not reserved upfront. In-flight conversion, decoder scratch, codec reference frames and adapter queues are additional storage. This absorbs temporary scheduling stalls, not sustained decoding slower than real time.
These are defaults, not fixed limits for every application. Pass a PreviewBuffering as the third constructor argument (SimplePlayer(display, logger, buffering)) to choose Preload, ReadAhead and AudioAhead as std::chrono::milliseconds, plus MaxBytes and MaxEvents. The policy is copied and immutable for that instance; the existing two-argument constructor uses the defaults. Zero preload disables the initial wait. Caps and read-ahead must be positive, and preload/audio advance must be nonnegative and no greater than read-ahead, otherwise construction throws. A decoded unit larger than the configured byte budget fails playback explicitly instead of waiting forever. Custom audio advance must fit the application's Display/device queues. Paused previews retain the smaller of the configured byte budget and 64 MiB. Embedded fonts, codec reference buffers and conversion scratch are separate from the pending-payload budget.
Video color conversion and HDR tone mapping run on a separate worker so they do not serialize audio decoding. That worker retains at most two pending decoded frame references plus the frame being converted, outside the pending-payload budget. When conversion falls behind the playing timeline and the initial cushion is exhausted, overdue pending images may be skipped to let audio decoding continue. Paused seek previews and initial prebuffering do not use that policy. The player reports the number skipped when workers stop. This prioritizes audio continuity under processing pressure; it does not guarantee every video frame under sustained CPU starvation.
player.Diagnostics() returns a thread-safe PreviewDiagnostics snapshot. Use m_decoder, m_backend and m_device to log actual decoder output. PreviewDecodeBackend::Unknown means no decoded frame has established the backend yet; Auto is only a policy and never proof of hardware acceleration. Changing Acceleration takes effect when a decoder is next opened. An audio-only source has no video decoder identity.
m_skipped_video counts pressure drops before conversion; m_shown_video counts successful Display/HDRDisplay callbacks, not compositor presentations. m_audio_underruns counts distinct valid AudioClock drained episodes while audio production is active, excluding preload, pause, audio switching and EOF drain. This is sampled starvation, not a device xrun counter; zero with m_audio_clock_samples == 0 means unavailable feedback, not verified continuity. Counters reset on Open, survive seeks/stop/close; decoder identity clears on decoder close. All snapshots contain public scalars and Safe strings only.
m_converted_video and m_conversion_fps are separate from these counters. Throughput divides converted pictures by the sum of measured conversion-stage wall seconds (no queue waits). Stage fields include setup and internal worker scheduling, not summed CPU time: download, HDR input color/linearization, tone mapping, HDR output color, and RGBA allocation/packing. For SDR, swscale fuses color conversion and packing; only packing is charged. m_peak_queue_bytes counts presentation payload only, not RSS, codec references or scratch. Application OpenGL uploads and real device/compositor behavior are outside these measurements.
Build MultimediaSimplePlayerTests with ENABLE_TEST=ON, then run:
The script saves three baseline and three contention CSV/log pairs, using the unchanged buffering defaults and SDR RGBA8 output. Contention runs a separate process with nproc busy threads for two seconds, starting one second after launch. Startup is Open-to-first-delivery; seek is Seek-to-first-delivery. Peak RSS comes from getrusage on Linux/macOS (-1 when unavailable). For forced software, invoke the runner directly:
Reproduce the generated SDR/20 ms Opus signals used in the software baseline:
See CHANGELOG.md for results and pending hardware measurements.
Open(std::string_view) constructs a BufferedFileReader, copying the path into its Safe::String. Open(Safe::Unique<Buffer::IO::BufferedLocationReader>) takes exclusive ownership of a compatible buffered reader, preserving its subtype. It opens a not-yet-open reader once or rewinds a healthy already-open one. Failed Open retains the reader.
| Operation | Workers/presentation | Reader |
|---|---|---|
| Stop() | Cancel/join and invalidate | Rewind the actual buffered reader; retain it open |
| Close() / destructor | Join and invalidate before releasing input | Close/release without rewind |
| ReleaseReader() | Stop/join and invalidate | Rewind, then transfer the same open dynamic reader at cursor zero |
Stop/transfer failures are explicit and retain ownership. An empty transfer result means failure, not a closed replacement reader. Destruction is noexcept and does not close the shared Display/widget.
This matters for Network's BufferedRemoteFileReader: closing permanently releases its mount token, so that object cannot simply reopen for transcoding. Transfer the healthy reader instead and keep its client/transport alive. The player rejects closed/faulted rewind paths rather than using Close+Open as recovery. A closed/faulted remote source requires a new reader.
The common engine and abstract contract support Windows, Linux and macOS. Consumers provide their platform integration; no native output backend is bundled into Multimedia. Decode availability depends on the configured codecs. Cross-platform design is not a claim that every native backend or platform was runtime-tested.
SimplePlayer::Acceleration(DecodeAcceleration::Software) forces CPU video decoding; the default Auto tries an available backend (VideoToolbox, D3D11VA/DXVA2, CUDA/NVDEC or VA-API as built) and safely opens the software decoder when initialization is unavailable. The same choice is configurable independently for each encode lane through Pipeline::Transcoder::Track::Acceleration(...) or the lower-level Pipeline::Config::Implementation::Acceleration field; it does not pin the decoder implementation or change encoder policy.
Preview defaults to tightly packed RGBA8 for an SDR-oriented Display. SimplePlayer::OutputColor(PreviewColorMode::HDR) opts into an additional HDRDisplay interface; PQ/HLG frames arrive as tightly packed RGBA16LE signal values with transfer, primaries and optional mastering/content-light metadata, without SDR tone mapping. An HDR sink must implement both interfaces; requesting HDR mode without it returns false and logs an error. SDR sources continue through the existing RGBA8 callback. This is not a native swapchain, display-capability negotiation, Dolby Vision rendering, or guarantee that the consumer's OS/display is in HDR mode. With the default SDR mode, PQ/HLG frames are converted through linear light, tone-mapped and converted to BT.709 before presentation. There is no surround bitstream passthrough or automatic subtitle compositor. Subtitle seeks reconstruct cue state from the beginning, so long inputs can take time; reconstruction is bounded and fails explicitly if it exceeds storage limits.
Buffering permits local or network-backed playback without loading the whole file. Reads/probing/seeks and worker joining can wait for bounded origin I/O: configure network timeouts. Actual seek support depends on the reader and demuxer, not on having a duration value.
The optional SDL2 demo is a real consumer-side Display example, not an installed library backend. Its implementation is in test/preview_demo.cxx; SDL linkage stays outside Multimedia's public API.
Controls: Space play/pause, Left/Right seek 30 seconds, Home stop/rewind, A/V/S cycle audio/video/subtitles including disabled, Esc close. It preserves image aspect ratio and shows text cues in the window title, but does not compose ASS styling or bitmap subtitles.
Use --smoke for three-second segments or --listen for thirty-second segments: initial playback, seeks to 50%, 90%, 10%, then Stop/verify zero. Unknown-duration input uses absolute targets 30, 120, 10 seconds instead.
The audio adapter requests the source rate and all channels, converts decoded PCM to device float32 and maps supported speaker sets to SDL order. It rejects unsupported/unknown/ambisonic layouts rather than falling back to stereo. SDL/hardware conversion and physical speaker routing remain external; an obtained six-channel spec is not proof of physical six-channel hardware or exact latency.
Reports include actual post-seek video/audio positions, image hashes, callback short reads, missing frames, inserted silence and sample peaks. These help diagnose delivery; clean counters do not prove audible correctness. Add a diagnostic directory after either mode to capture bounded per-epoch PCM with format/layout metadata sidecars.
Public owning data crosses modules through StormByte::Safe values, collections and owners. Create polymorphic owners with Safe::MakeShared<T>(...) or the appropriate Safe unique factory; preserve dynamic type and provider-local destruction rather than slicing. Borrowed registry identities must outlive their users.
Safe ownership is not arbitrary-ABI compatibility or a safe-unload guarantee. Consumers/providers need compatible C++/STL ABIs, and every provider supplying live values, release operations or callbacks must remain loaded until those objects are released. STORMBYTE_DECLARE_MAYBE_SAFE is the provider's boundary-safety promise, not an automatic audit.
Derived Transcoder providers create their own Plan/TrackSettled payloads in EmptyPlan()/EmptySettled() using provider-safe factories. Plans remain shared and do not implement cloning/moving; derived settled payloads must implement their required Clone()/Move() operations in the creating module.
Stages and filters accept Safe::Shared<Logger::Log>; Transcoder requires a nonempty logger, while SimplePlayer's logger is optional. Prefer ThreadedLog for concurrent workers and keep its underlying stream alive through every retaining job/stage.
The application controls logging levels, headers and redaction. Processing stages scope their messages under StormByte/Multimedia/<stage>. Detailed per-unit logging has a real cost; use it for diagnosis rather than ordinary UI progress. Observe status/error explicitly instead of treating a quiet log as success.
| Option | Values | Meaning |
|---|---|---|
| WITH_FFMPEG | BUNDLED (default) / SYSTEM | Bundled libav engine or compatible system installation |
| WITH_VMAF | BUNDLED / SYSTEM | Must match FFmpeg mode; a differing cache value is overridden with a status message |
| WITH_OCR | BUNDLED (default) / SYSTEM | Tesseract/Leptonica; Windows forces bundled OCR |
| WITH_TESSDATA | BUNDLED (default) / SYSTEM | Embedded selected models or installed runtime models, independent of OCR mode |
| WITH_ZIMG | BUNDLED / SYSTEM | Image-processing dependency |
| WITH_LUNASVG | BUNDLED (default) / SYSTEM | SVG watermark renderer; SYSTEM requires LunaSVG 3.5 or newer. Bundled builds disable system font loading and the font-face cache |
| BUILD_SHARED_LIBS | ON (default) / OFF | Shared/static Multimedia and matching StormByte archives |
| ENABLE_TEST | ON / OFF (default) | Build/register CTest; PGO enables tests in the instrumented tree |
| ENABLE_CI_FIXTURES | ON / OFF (default) | CI-only package discovery fixtures; explicitly enabled by CI, not needed for local tests or PGO |
| ENABLE_PREVIEW_DEMO | ON / OFF (default) | Optional SDL2 output demo; requires SDL2 development files |
| ENABLE_LTO | ON (default) / OFF | Link-time optimization |
| ENABLE_PGO | ON / OFF (default) | Instrument/train in pgo-instr, then build the profiled library |
| ENABLE_ASAN | ON / OFF (default) | Debug ASan/UBSan on supported non-Windows builds; disabled for Release |
| WITH_GPL | ON / OFF | Permit GPL bundled components, such as x265/Rubberband; OFF also omits Multimedia Rubberband/VagueDenoiser sources and headers in system mode |
| WITH_NONFREE | ON / OFF | Permit nonfree bundled components such as FDK-AAC and dependent filters |
These flags select dependencies/capabilities, not the license of original Multimedia source. System installations determine their own codec availability and licensing. Review the enabled components when distributing an application.
CI/release CPU baselines are selected by the shared build action: x86-64-v3 on Linux amd64, ARMv8.2-A on Linux arm64, AVX on Windows x64 and Apple M1 on Apple Silicon. Direct CMake builds preserve the compiler's defaults and caller-provided CPU flags. Bundled specialized SIMD kernels retain runtime selectors without raising the global baseline; bundled x86 FFmpeg requires NASM.
Linux releases cover amd64/arm64, glibc/musl and GCC/libstdc++ or Clang/libc++. Their archive names are StormByte-Multimedia-<tag>-linux-<arch>-<libc>-<stdlib>.zip. macOS uses macos-arm64 and Windows uses windows-x64, without a runtime suffix. Each release build uploads its own archive directly; documentation deployment is independent.
For GCC/Linux builds with LTO, configure checks for the Alpine fortify-headers/LTO bug. When the toolchain is affected, Multimedia adds -fno-builtin-vsnprintf to its C++ build flags, including PGO training and bundled CMake dependencies such as Tesseract. LTO and fortify remain enabled. Consumers compiling their own code may need the same flag until their toolchain is fixed.
Bundled FFmpeg is pinned to 42c25af15f51 on meson-9.0. Its upstream Meson build includes the clang-cl, VA-API, intrinsics and native AArch64 generator fixes merged in !208; the corresponding five local patches have been removed.
Bundled Opus presumes NEON on ARM64 when the compiler supports its intrinsics; 32-bit ARM does not assume NEON. Bundled FFmpeg includes the upstream AArch64 swscale fix for signed intermediate samples and uses upstream Meson NEON/SSE2 intrinsics probes, building the NEON MPEG unquantizer only when NEON intrinsics are detected.
Clang/Linux AArch64 PGO training disables value profiling to avoid the concurrent profiler hang tracked in llvm/llvm-project#160968. Execution-count profiling, LTO, SIMD and codec multithreading remain enabled. The workaround is removed from the optimized build.
Bundled tessdata embeds the models selected in the language configuration, including available vertical variants. System models are loaded from the configured installation; use TESSDATA_DIR, TESSDATA_PREFIX or Tessdata_ROOT to select it and deploy the required files. Partial installations are accepted, but an absent requested language fails without fallback. OCR receives decoded grayscale subtitle pixels, not external image files.
Bundled SVG rendering uses LunaSVG 3.5.0/PlutoVG 1.3.1. Binary packages must retain THIRD-PARTY, COPYING.LunaSVG and COPYING.FTL alongside the existing notices.
Requires a C++26-capable compiler/standard library, CMake with CXX_STANDARD 26 support (3.25+) and StormByte-BuildMaster 2.1.5 or newer. BuildMaster downloads matching Base/Buffer/Logger/System release archives by architecture, Linux C library and C++ runtime; these are not submodules. Bundled media dependencies need recursive checkout and NASM on x86.
On platforms without nproc, pass the detected available CPU count to -j.
The in-tree target is StormByte::Multimedia, the library is StormByte-Multimedia, and installed headers use #include <StormByte/multimedia/...>. Deploy runtime dependencies with a compatible ABI; public headers do not expose the platform or media-engine implementation dependencies.
Installed prefixes and extracted release ZIPs provide FindStormByteMultimedia.cmake. Put both the Multimedia prefix and its matching StormByte dependency prefix in CMAKE_PREFIX_PATH, and their share/cmake/Modules directories in CMAKE_MODULE_PATH, then consume the imported target:
The finder brings in Base, Buffer, Logger and System transitively and propagates the shared-library definitions required by Windows consumers.
Processing tests cover registries, File, facade/manual graphs, codecs, filters, remux/encode, mux policies, interleaving, OCR, artwork and HDR metadata. Outputs are reopened and decoded through library APIs; tests do not require external media command-line tools. HDR tests compare metadata and frame counts, not merely successful file creation. Hardware cases preflight device/profile support; unavailable implementations skip, but an available implementation's processing failure does not.
For shared builds with bundled StormByte dependencies, PackageDiscoveryTests installs Multimedia, creates and extracts a ZIP, then configures, links and runs a separate consumer against each prefix. The consumer probes the stereo WAV fixture and verifies audio properties and the codec registry. OptimizationTests checks that the common LTO module does not inject x86 flags into ARM builds or overwrite caller-supplied flags.
Preview tests use a clocked fake Display, synthetic inputs and a remote-like reader with irreversible close. They cover exact decoded audio, playback timing, pause/seek/track changes, subtitles, errors, ownership and transfer into Transcoder without another origin Open. Run the focused group with:
The skewed-interleave software-playback tests bound video lateness to 100 ms in ordinary builds and 250 ms under PGO instrumentation or ASan, where shared-runner scheduling and instrumentation can delay delivery. The audio underrun (50 ms), video lead (100 ms), delivery gap (500 ms), and timestamp-continuity checks remain unchanged.
The live-audio cases also cover disabled audio, repeated/rapid selections, full PCM queues, sink errors and pause/seek/EOF reconstruction. For a local multitrack file (not redistributed with the tests), the same registered case cycles every audio track and checks increasing video timestamps, bounded delivery gaps and unchanged video epoch:
See fixture documentation for provenance and redistribution notices. Test success is not certification of every codec/platform, visual restoration quality or long-running workload. Real-output listening uses the optional demo separately.
Issues only on this repository. Fork and open a pull request against master.
Public API does not grow “because the coordinator needs it”. No new friends. Doxygen on a header is part of the file: update it so it does not lie, do not delete it. Commits are English, one topic, feat(pipeline): … / fix(watermark): ….
StormByte-Multimedia original source is dual-licensed:
Both licensing options cover original StormByte-Multimedia source only. Third-party components — including FFmpeg, libvmaf and embedded trained data — keep their own licenses and are not covered by the commercial grant. See [NOTICE](NOTICE) and thirdparty/.
A written StormByte commercial agreement may license the original StormByte-Multimedia code on terms other than the LGPL, including specific use and linking arrangements such as static linking, as stated in that agreement. It does not grant rights to dependencies or waive their license conditions. In particular, enabling WITH_GPL or WITH_NONFREE can add separate obligations to a binary or other work that uses those components, including for modification, linking (static or dynamic) and redistribution. The packager and final user are responsible for determining and meeting all applicable license and distribution requirements and obtaining any needed patent permissions. A StormByte license does not provide those permissions for GPL or nonfree components and grants no patent rights.
SPDX: LGPL-3.0-or-later OR LicenseRef-StormByte-Commercial.
The headers of the public and private trees repeat this grant. When in doubt, those headers and LICENSE win over this README.
If this saved you from building a separate media pipeline inside your application, a star is the polite nod. A well-aimed issue beats a vague “it broke”. Pull requests that keep the public API focused — Plan, Step, Transcoder, SimplePlayer, Display and composable filters — are the ones that land.
I wrote this because the alternative was another private transcoder in every product. Maintaining that difference takes evenings.
Use it. Break it on purpose. Tell me which sentence in this file lied.