xk6-SIP: Audio Quality Checks in Load and Automated Tests

How audio quality checks work in xk6-sip, an extension of the k6 load testing tool for VoIP/SIP calls: recording a call, scoring what the callee heard against a reference, and how it behaves in functional tests and under load.

200 OK means the PBX connected the call; it does not mean the people can hear each other. Exhausted media server ports, NAT errors, crossed media streams, and transcoding at the CPU limit all produce a call with successful signaling and silence, a stranger's voice, or crackle in the handset. SIPp and most telephony load tools check signaling, and treat RTP as a packet count at best. For a contact center such a call is a lost customer — and in the load test report it is green.

Functional telephony tests usually solve this with references: record what the callee should hear in each case, and compare it with the call recording in CI. The approach works, but a reference per case has to be maintained, and under load you cannot run it. In xk6-sip the same comparison lives inside the k6 script and runs on thousands of calls.

In this article:

1. What we check: three levels of audio checking.
2. Call recording: why audio is laid out by RTP timestamp.
3. How the compareAudio() score is computed.
4. ViSQOL from Google: how it works and why it is not built into the generator.
5. Checks in numbers.
6. A functional test for CI.
7. Under load: how much memory and CPU the audio check costs, and why the dashboard showed a score of 0.958 instead of 0.993.
8. Limitations and what to do next.

What we check

"Audio quality" in telephony is three different questions, and each catches its own defects.

The first level already existed in xk6-sip: isHeard() looks at the signal level rather than at packet arrival, so a stream of silence does not fool it. But it cannot tell your peer's voice from a stranger's: if the PBX connected the media to the wrong subscriber, there is sound, and isHeard() returns true. The other two levels need a reference, and to compare against one the audio has to be recorded first.

The main difference from the reference-per-case approach: you already know what each subscriber says — it is that subscriber's audio option in the script. The reference for "what B heard" is what A sent. Recorded references are needed only where the PBX itself generates the sound.

Call recording

Recording is enabled with the record option at any level — for the whole test, a subscriber, or a single call. It is off by default and costs nothing.

sip.options({ record: 'onFailure', recordDir: 'records' }); // whole test: keep failed calls
const B = new sip.Device({ /* ... */ record: true });      // one subscriber
A.call({ callee: B, record: true });                       // one call

inc.saveRecording('records/b.wav'); // left channel — what B heard, right — what was said

Stereo, not mono. The received sound goes into the left channel, the sent sound into the right, on one time scale. In an audio editor you immediately see echo (your own phrase coming back into the left channel) and the delay between turns.

Layout by RTP timestamp. The simplest approach is to append packets to a buffer in arrival order, but then the recording lies: a lost packet shifts everything after it by 20 ms, and packets reordered by the jitter buffer swap places. So every frame is placed at the position of its own timestamp:

• the first frame of a new SSRC is pinned to its arrival time relative to the start of the recording;
• each following frame is offset from it by ts − ts0, including overflow of the 32-bit counter;
• a hole where a packet was lost stays as silence.

As a result, drop-outs sound in the file exactly where they happened, and the comparison algorithm gets an honest timeline.

This is what a recording with 20% packet loss looks like — the same case as in the table below (score 0.754, drop-outs 560 ms). The left channel lags the right by the network delay, and every loss is a hole in its own place rather than a shift of everything after it.

Modes. record: true keeps the recording in memory for saveRecording() and compareAudio(), and with recordDir it also saves every call. record: 'onFailure' writes to disk only calls with a failed expectation (expect*, isHeard) or calls cut short by an error. After a CI run, the artifacts hold exactly the calls worth listening to.

How the score is computed

compareAudio(ref) answers the question "how close is what the callee heard to the reference" and returns a number from 0 to 1 with diagnostics:

const q = inc.compareAudio(hello);
// { score: 0.993, offset: 1042, compared: 3000, gaps: 0, clippedStart: 0, gain: -0.3 }

The algorithm has four steps, all in plain Go with no external libraries.

1. Coarse alignment. The reference has to be found in the recording first: the network delay is unknown, and the phrase loops from an arbitrary point. Both signals are cut into 10 ms blocks, the level in dB is computed for each, and that loudness envelope of the reference is searched for in the recording by the maximum Pearson correlation coefficient. The speech envelope — syllables and pauses — survives the codec and gain changes, and costs 80 times less than a search over samples.
2. Fine alignment. Within ±10 ms of the found point, the shift is refined to a single sample by normalized correlation of the waveform over the first second of the reference.
3. Spectrograms. The reference and the aligned piece of the recording are turned into spectrograms: 32 ms windows with a 16 ms hop, Hann window, 256-point FFT, power in dB over the 300–3400 Hz telephone band. Only frames where the reference has speech are compared: not quieter than -55 dBFS and not more than 40 dB below the loudest frame.
4. Correlation of changes. Each frequency band has its mean over time subtracted, and the two spectrograms are correlated as two large vectors. The result is score.

After alignment the path splits: the score is computed from the spectrograms, while drop-outs and loudness are computed from the 10 ms blocks.

Step 4 is the key decision. If the spectra are correlated as they stand, two different phrases in a similar voice get a high score: any speech has a similar average timbre — more energy in the lows, less in the highs. After subtracting the mean, what gets compared is how the spectrum changes over time: when a syllable starts, where the pitch goes. A stranger's phrase has a different pattern, and the score drops below 0.4.

Three more properties follow from this scheme:

• Loudness does not matter. Gain in dB is a shift of the whole spectrogram, and subtracting the mean removes it. The level difference is reported separately as gain.
• Drop-outs are penalized. Silence in the recording is a drop to the noise floor in every band where the reference has sound. Drop-outs (gaps) are counted separately on 10 ms blocks: a 32 ms spectrogram frame is wider than a lost packet and would hide isolated losses.
• A clipped start (clippedStart) — drop-outs from the start of speech to the first heard block. A classic telephony defect: media connects with a delay after the answer, and the callee does not hear "Hello".

If the reference is stationary (the sine xk6-sip sends by default), nothing remains of it after subtracting the mean. For that case the spectral shape is compared: a 1 kHz tone differs from 440 Hz, but a meaningful quality score comes only from speech.

ViSQOL from Google and why it is not built into the generator

The idea behind compareAudio() comes from ViSQOL (Virtual Speech Quality Objective Listener), Google's open algorithm under the Apache-2.0 license. It works with a reference (full-reference) and outputs a MOS from 1 to 5 — a score on the scale as if live listeners had rated the call.

How ViSQOL works:

1. Alignment — first a global shift, then refinement over pieces.
2. Auditory spectrogram on a gammatone filter bank — a model of the cochlea of the inner ear: bands are narrow at the lows and wide at the highs, the way humans resolve them.
3. Patches. The reference is cut into short pieces, and for each one the most similar nearby piece is looked for in the recording. This way the algorithm does not penalize local shifts and stretches from the jitter buffer.
4. NSIM (Neurogram Similarity Index Measure) — the SSIM analogue from image processing: it compares patches by brightness and structure in each band.
5. Conversion to MOS — by a function calibrated on ratings from live listeners.

In xk6-sip the same idea in simplified form: linear FFT bands instead of gammatones, one alignment for the whole phrase instead of a patch search, one correlation instead of NSIM, and no calibration against listeners. So score is not a MOS but a similarity measure for comparing calls, runs, and PBX versions against each other. The threshold comes from a clean reference run.

Why not build in the original. The license allows it; the cost sits in integration:

There are also two technical nuances. ViSQOL's speech mode is designed for 16 kHz, while G.711 is an 8 kHz narrow band: the recording would have to be resampled, and the score ceiling even with perfect transmission would be lower than for wideband audio. And the auditory model is noticeably heavier than the ~15 ms per comparison here, so it cannot run on every call under load.

The resulting division of labor: a fast Go score inside the generator — for load and quick CI checks — and reference ViSQOL as a separate step over saved recordings when MOS is what you need.

Checks in numbers

The score separates clean audio, losses, and foreign speech far apart, and drop-outs are counted exactly to the block. Synthetic cases come from the unit tests: a speech-like signal (syllables with a floating fundamental and harmonics, noisy consonants, pauses) passed through the G.711 codec. Real calls go through a test PBX (registrar and B2BUA) over a network.

Two observations.

Drop-outs are not counted accurately at first. In the first version they were counted by spectrogram frames, and 20% loss produced only 112 ms of drop-outs: a 32 ms frame is wider than a 20 ms packet, and an isolated loss only mutes it. After switching to 10 ms blocks the test compares gaps with the speech actually lost instead of merely checking gaps > 0. The discrepancy is now zero.

The reverse check is needed as much as the forward one. A high score proves nothing by itself: it could just as well assign 0.99 to anything. A score of 0.113 for B's own phrase on the same call shows that the algorithm distinguishes speech rather than guessing that "sound exists".

A functional test for CI

The scenario audio-quality.js: A and B each say their own phrase, and each must clearly hear the peer's phrase and must not hear their own.

const phraseA = sip.audio(open('./refs/phrase-a.wav', 'b'));
const phraseB = sip.audio(open('./refs/phrase-b.wav', 'b'));
const minScore = Number(__ENV.MIN_SCORE || 0.9);

sip.options({ record: 'onFailure', recordDir: 'recordings' });

const A = device('A', 1, { audio: phraseA });   // A says phrase A
const B = device('B', 2, { audio: phraseB });   // B says phrase B

function hears(who, leg, ref, whose) {
  const q = step(`${who} audio compared`, leg.compareAudio(ref));
  audioStep(leg, `${who} hears ${whose} clearly (score ${q.score})`, q.score >= minScore);
  audioStep(leg, `${who}: no drop-outs (${q.gaps} ms)`, q.gaps < 100);
  audioStep(leg, `${who}: first word not clipped (${q.clippedStart} ms)`, q.clippedStart < 60);
}

export default function () {
  const [out, inc] = call(A, B);     // call, answer, isHeard both ways
  sleep(4);                          // a 3-second phrase plays in full

  hears('B', inc, phraseA, "A's phrase");
  hears('A', out, phraseB, "B's phrase");

  const self = step('B own phrase compared', inc.compareAudio(phraseB));
  audioStep(inc, `B does not hear itself (score ${self.score})`, self.score < 0.5);

  inc.hangup();
  step('A disconnected', out.expectDisconnected('5s'));
}

How this works in CI:

• step() records a check under a readable name and stops the scenario at the first failure. k6 exits with a non-zero code and writes a JUnit report.
• The numbers are baked into the check name, so the report shows not only "failed" but also "score 0.71".
• audioStep() on failure first saves the WAV for that leg, then stops the test. The built-in onFailure only reacts to the extension's expectations, and the q.score >= 0.9 check is script logic the extension knows nothing about.

On the test PBX the scenario passes 16 of 16 checks. Raise the threshold to 0.999 and it fails with exit code 99, leaving the call WAV in recordings/. Against a real PBX the subscribers are passed with -e REGISTRAR=... -e A_USER=..., and with transcoding the threshold is lowered with -e MIN_SCORE=0.8.

In GitHub Actions, recordings of failed calls are collected into the build artifacts along with JUnit reports — a step with if: always() so it runs on a red build too:

- name: keep reports
  if: always()
  uses: actions/upload-artifact@v7
  with:
    name: functional-reports
    path: reports/          # JUnit and reports/recordings/*.wav
    if-no-files-found: ignore

Under load: memory, CPU, and Prometheus histograms

The audio check also works under load, but it has a price, and the main cost is memory, not CPU.

Memory. Recording one leg is two tracks of 16-bit samples at 8 kHz, that is 32 KB per second of call, and it is held in memory for the whole call.

If both sides are recorded, 1,000 calls are 2,000 legs and about 2 GB. So under load recording is enabled for a sample of subscribers and saved in onFailure mode. Without record none of this exists: per RTP packet there is a single pointer-to-nil check left.

CPU. One comparison of a 3-second reference against a 30-second recording takes ~15 ms on a desktop CPU (the BenchmarkCompare benchmark). Time grows with the length of the whole recording: coarse alignment tries every shift. A thousand comparisons per second would take about 15 cores, so it is worth comparing a sample of calls right after the phrase rather than at the end of a long conversation.

The native histogram trap. Every compareAudio() call writes its score into the rtp_audio_score metric, and a threshold in k6 is set the usual way: rtp_audio_score: ['p(95)>0.9']. With a Grafana dashboard it got more interesting. In a trial run every call scored exactly 0.993, yet Prometheus showed a median of 0.958 and a 5th percentile of 0.921.

The reason is how native histograms are built. Their buckets grow exponentially, and with the schema k6 emits, adjacent boundaries differ by about 9%. Around 1.0 a single bucket covers the whole 0.917–1.0 range, and histogram_quantile interpolates inside it blindly. For response times that is invisible; for a score, where the entire difference between "good" and "bad" sits between 0.9 and 1.0, it is critical.

The fix is a second metric, rtp_audio_mismatch = 1 − score. Near zero the exponential buckets are small, and the dashboard computes the score back:

# median and worst 5% of calls
1 - histogram_quantile(0.5,  sum(rate(k6_rtp_audio_mismatch{testid=~"$testid"}[$window])))
1 - histogram_quantile(0.95, sum(rate(k6_rtp_audio_mismatch{testid=~"$testid"}[$window])))

# share of comparisons below 0.9
histogram_fraction(0.1, +Inf, sum(rate(k6_rtp_audio_mismatch{testid=~"$testid"}[$window])))

A trial run: 4 VU, 70 s, every fifth comparison made with a foreign reference. On the left the median and the worst 5% of scores with the 0.9 line; on the right the share of comparisons below 0.9 and the number of comparisons per second.

The same panels on a clean CI run: 20 VU, 2 minutes, audio comparison in every call. The score holds at 0.993 above the threshold line, the share of comparisons below 0.9 is zero, and comparisons run at 3–4 per second. The score axis shows 0.8–1.0 and expands downward on its own when scores fall, as in the previous screenshot.

After that the dashboard median matched k6 to three decimal places: 0.993. In the run where every fifth comparison was deliberately made with a foreign reference, the "below 0.9" panel showed 17–25% and the worst 5% came out around 0.02. The rule that stayed with the project: a quantity that piles up against an upper bound is fed into a native histogram as the distance to that bound.

Limitations and what to do next

What to know before relying on the score:

• It is not MOS. score is not calibrated against listener ratings. It is for comparing calls and versions against each other, and the threshold comes from a clean run.
• G.711 only. Transcoding to G.729 or Opus changes the waveform and lowers the score. Spectral comparison is more tolerant of this than sample-by-sample, but for that path the threshold has to be lowered.
• The references in the example are synthetic — a speech-like signal with no licensing questions. For production tests it is better to replace them with your own 3–5 second recordings.
• Deliberate reverse checks also land in rtp_audio_score, such as "B does not hear itself". In scripts with thresholds on this metric it is better not to mix them.
• The recording is held in memory until the end of the iteration; a single track is limited to 15 minutes.

Next:

1. ViSQOL in CI as a separate step over saved WAVs — real MOS where it is needed.
2. Tone signatures for "who hears whom". Each subscriber sends its own combination of frequencies, and a Goertzel detector checks it in fractions of a millisecond. A cheap crossed-stream check on every call at full load.
3. Speech recognition for IVR — checking prompt content without a recorded reference.
4. Mono and compressed recordings — half the disk space and, at worst, tens of times less.

Code, documentation, and the dashboard live in the repository github.com/Dmitry-Fedotov-Dev/xk6-sip:

• compareAudio()
• saveRecording()
• functional test