PipeWire: Latency support

PipeWire 1.6.4

Loading...
Searching...
No Matches
Latency support

This document explains how the latency in the PipeWire graph is implemented.

# Use Cases

## A node port has a latency

Applications need to be able to query the latency of a port.

Linked Nodes need to be informed of the latency of a port.

## dynamically update port latencies

It needs to be possible to dynamically update the latency of a port and this should inform all linked ports/nodes of the updated latency.

## Linked nodes add latency to the signal

When two nodes/ports have a latency, the signal is delayed by the sum of the nodes latencies.

## Calculate the signal delay upstream and downstream

A node might need to know how much a signal was delayed since it arrived in the node.

A node might need to know how much the signal will be delayed before it exists the graph.

## Detect latency mismatch

When a signal travels through 2 different parts of the graph with different latencies and then eventually join, there is a latency mismatch. It should be possible to detect this mismatch.

# Concepts

## Port Latency

The fundamental object for implementing latency reporting in PipeWire is the Latency object.

It consists of a direction (input/output) and min/max latency values. The latency values can express a latency relative to the graph quantum, the samplerate or in nanoseconds. There is a mininum and maximum latency value in the Latency object.

The direction of the latency object determines how the latency object propagates.

- SPA_DIRECTION_OUTPUT Latency objects move from output ports downstream and contain the latency from all nodes upstream. An output latency received on an input port should instruct the node to update the output latency on its output ports related to this input port. This corresponds to the JackCaptureLatency.
- SPA_DIRECTION_INPUT Latency objects move from input ports upstream and contain the latency from all nodes downstream. An input latency received on an output port should instruct the node to update the input latency on its input ports related to this output port. This corresponds to the JackPlaybackLatency.

PipeWire will automatically propagate Latency objects from ports to all linked ports in the graph. Output Latency objects on output ports are propagated to linked input ports and input Latency objects on input ports are propagated to linked output ports.

If a port has links with multiple other ports, the Latency objects are merged by taking the min of the min values and the max of the max values of all latency objects on the other ports.

This way, Output Latency always describes the aggragated total upstream latency of signal up to the port and Input latency describes the aggregated downstream latency of the signal from the port.

## Node ProcessLatency

This is a per node property and applies to the latency introduced by the node logic itself.

This mostly works if (almost) all data processing ports (input/output) participate in the same data processing with the same latency, which is a common use case.

ProcessLatency is mostly used to easily calculate Output and Input Latency on ports. We can simply add the ProcessLatency to all latency values in the ports Latency objects to obtain the corresponding Latency object for the other ports.

# Latency updates

Latency params on the ports can be updated as needed. This can happen because some upstream or downstream latency changed or when the ProcessLatency of a node changes.

When the ProcessLatency changes, the procedure to notify of latency changes is:

- Take output latency on input port, add new ProcessLatency, set new latency on output port. This propagates the new latency downstream.
- Take input latency on output port, add new ProcessLatency, set new latency on input port. This propagates the new latency upstream.

PipeWire will automatically aggregate latency from links and propagate the new latencies down and upstream.

# Async nodes

When a node has the node.async property set to true, it will be considered an async node and will be scheduled differently, see scheduling.dox.

A link between a port of an async node and another port (async or not) is called an async link and will have the link.async=true property.

An async link will add 1 quantum of latency between the nodes it links. A special exception is made for the output ports of the driver node, which do not add latency.

The Latency param will be updated with 1 extra quantum when they travel over an async link.

# Examples

## A source node with a given ProcessLatency

When we have a source with a ProcessLatency, for example, of 1024 samples:

+----------+ + Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
| FL ---+ Latency: [{ "direction": "input", "min-rate": 0, "max-rate": 0 } ]
| source +
| FR ---+ Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
+----------+ + Latency: [{ "direction": "input", "min-rate": 0, "max-rate": 0 } ]
^
|
ProcessLatency: [ { "quantum": 0, "rate": 1024, "ns": 0 } ]

Both output ports have an output latency of 1024 samples and no input latency.

## A sink node with a given ProcessLatency

When we have a sink with a ProcessLatency, for example, of 512 samples:

Latency: [{ "direction": "output", "min-rate": 0, "max-rate": 0 } ]
Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
^
| +----------+
+---- FL |
| sink | <- ProcessLatency: [ { "quantum": 0, "rate": 512, "ns": 0 } ]
+---- FR |
| +----------+
v
Latency: [{ "direction": "output", "min-rate": 0, "max-rate": 0 } ]
Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]

Both input ports have an input latency of 512 samples and no output latency.

## A source and sink node linked together

With the source and sink from above, if we link the FL channels, the input latency from the input port of the sink is propagated to the output port of the source and the output latency of the output port is propagated to the input port of the sink:

Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
^
| Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
| Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
| |
+----------+v v+--------+
| FL ------------ FL |
| source + | sink |
| FR --+ FR |
+----------+ | +--------+
v
Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
Latency: [{ "direction": "input", "min-rate": 0, "max-rate": 0 } ]

## Insert a latency node

If we place a node with a 256 sample latency in the above source-sink graph:

Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 768 } ]
^
| Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
| Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 768 } ]
| ^
| | Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
| | Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
| | ^
| | |
+----------+v v+--------+v +-------+
| FL ------------ FL FL --------- FL |
| source + | node | ^ | sink |
| . | . | . |
+----------+ +--------+ | +-------+
v
Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]

See how the output latency propagates and is increased going downstream and the input latency is increased and traveling upstream.

## Link a port to two port with different latencies

When we introduce a sink2 with different input latency and link this to the node FL output port, it will aggregate the two input latencies by taking the min of min and max of max.

Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 2304 } ]
^
| Latency: [{ "direction": "output", "min-rate": 1024, "max-rate": 1024 } ]
| Latency: [{ "direction": "input", "min-rate": 768, "max-rate": 2304 } ]
| ^
| | Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
| | Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 2048 } ]
| | ^
| | | Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
| | | Latency: [{ "direction": "input", "min-rate": 512, "max-rate": 512 } ]
| | | ^
| | | |
+----------+v v+--------+v v +-------+
| FL ------------ FL FL --------- FL |
| source + | node | \ | sink |
| . | . \ . |
+----------+ +--------+ \ +-------+
\
\ +-------+
+--- FL |
^ | sink2 |
| . .
| +-------+
v
Latency: [{ "direction": "output", "min-rate": 1280, "max-rate": 1280 } ]
Latency: [{ "direction": "input", "min-rate": 2048, "max-rate": 2048 } ]

The source node now knows that its output signal will be delayed between 768 amd 2304 samples depending on the path in the graph.

We also see that node.FL has different min/max-rate input latencies. This information can be used to insert a delay node to align the latencies again. For example, if we delay the signal between node.FL and FL.sink with 1536 samples, the latencies will be aligned again.

## An async output stream and sink node linked together

The sink has 1 quantum of Input latency. The stream has no output latency. When the Input latency travels over the async link 1 quantum of latency is added and the Input latency on the stream is now 2 quanta. Similar for the stream Output latency that receives an additional 1 quantum of latency when it arrives in the sink over the async link.

Latency: [{ "direction": "output", "min-quantum": 0, "max-quantum": 0 } ]
Latency: [{ "direction": "input", "min-quantum": 2, "max-quantum": 2 } ]
^
| Latency: [{ "direction": "output", "min-quantum": 1, "max-quantum": 1 } ]
| Latency: [{ "direction": "input", "min-quantum": 1, "max-quantum": 1 } ]
| |
+----------+v v+--------+
| async FL ------------ FL |
| stream + | sink |
| FR --+ FR |
+----------+ | +--------+
v
Latency: [{ "direction": "output", "min-quantum": 0, "max-quantum": 0 } ]
Latency: [{ "direction": "input", "min-quantum": 0, "max-quantum": 0 } ]