From Microphone and System Audio to Speech Segments

Meetily’s audio problem is not simply how to press a record button. It must accept microphone and system sound that arrive through different operating-system devices, turn them into a coherent session recording, and create speech-sized inputs that a local recognizer can process quickly. This chapter follows that path from startup preflight to voice-activity detection, so the reader can see where local control is created, where latency is introduced, and where the implementation remains platform-dependent.

The distinction matters because the audio needed for a faithful recording is not identical to the audio needed by speech recognition. A recording should preserve the meeting as a continuous mixed artifact. A recognizer benefits from receiving speech rather than long stretches of silence. Meetily’s current design addresses those goals with one shared pipeline and two downstream purposes.

Recording starts before audio flows

The first step is a preflight decision: can the application produce a useful transcript at all? Meetily checks whether the configured local transcription engine can initialize and whether a usable model is available before it creates the active recording resources. If that check fails, the application reports the problem instead of capturing a session that cannot reach its intended result. This ordering also prevents a half-started recording from owning a microphone while the user is still waiting for a model error.

Next, Meetily loads recording preferences and resolves the devices. A requested microphone is used when it is still present; if a saved choice is stale or unavailable, the start path can fall back to the system default and tell the interface that the device changed. If no microphone is usable, the code allows a system-audio-only attempt. System audio is optional in the stream manager, so failure to create that stream does not necessarily cancel a microphone recording. The hard failure is the case in which no stream can be created at all.

This is more than convenience behavior. Device selection is part of the information boundary. The microphone determines what a person says, while system audio may contain the voices, videos, or notifications played through the computer. Choosing a different device can change what the meeting record contains. A fallback that keeps the session alive is useful, but it should not be mistaken for transparent continuity: the acoustic source may have changed, and the user needs to know which one is now active.

Permissions are another part of the same contract. On macOS, Meetily has to deal with both microphone access and system-audio capture. The code verifies microphone access by creating a brief test stream and waiting for its callback. That matters because an operating system can allow a stream object to be created while delivering no samples after permission is denied. For direct Core Audio capture, the repository similarly notes that creating a tap is not by itself proof that usable audio will arrive; a denied permission can present as silence. The practical test is not “did the API return a handle?” but “did the expected data path become active?”

System audio also differs by platform. The current code has an active CPAL-based stream path, an optional direct Core Audio path on macOS, Windows branches built around WASAPI output devices for loopback use, and Linux branches that look for PulseAudio monitor sources. These are different ways of representing “sound the computer is playing.” macOS can use Core Audio or ScreenCaptureKit, Windows relies on the WASAPI device graph, and Linux may expose a monitor source as an input-like device.

The repository itself shows why broad platform language needs care. The active stream and device modules contain non-macOS paths for selecting and opening system-audio devices, while a separate convenience function named for system capture is explicitly unimplemented outside macOS. Documentation describes Windows loopback and Linux audio support, but the repository does not include a cross-platform runtime test matrix proving that every route works under every desktop configuration. A defensible conclusion is that Meetily has platform-specific capture designs and setup dependencies, not one uniform system-audio abstraction.

Once preflight succeeds, the recording manager creates the channels and state that connect the devices to the pipeline. The manager starts the processing pipeline before starting the audio streams, initializes the recording saver, and then begins monitoring device availability. This order gives incoming callbacks somewhere to send their samples and gives the application a place to report a failure. It also establishes who owns what: the manager owns the session, the pipeline owns transformation and routing, and the recognizer will later own speech-segment processing.

One captured signal, two purposes

Audio devices do not necessarily agree on sample rate, number of channels, or callback timing. A microphone might provide stereo samples at one rate while a system source arrives at another. If those streams are combined without normalization, the result can be out of sync, incorrectly paced, or unnecessarily difficult for a model to interpret.

Meetily converts multichannel input to mono when required and moves captured audio toward a 48 kHz pipeline target. The live capture object uses a persistent resampler for devices whose native rate differs from that target. Persistence matters: a resampler created afresh for every callback would not retain the state needed to preserve continuity across arbitrary callback sizes. The code instead buffers variable-size input until the resampler can process a stable chunk, then carries leftover samples into the next call. When the buffer has not reached that point, the unprocessed samples are held rather than sent as if they were already valid 48 kHz audio.

Microphone audio receives additional preparation because it is usually the less controlled source. The current path can remove low-frequency rumble with a high-pass filter, optionally run RNNoise suppression, and apply stateful loudness normalization with an EBU R128 target and true-peak limiting. These stages are ordered so that low-frequency noise is removed before later gain changes can make it more prominent. System audio is treated as comparatively clean and is not sent through the same microphone enhancement chain.

These operations do not magically make two streams identical. They make their differences manageable enough for the next decision: alignment and mixing. Microphone and system samples arrive asynchronously, so Meetily places them in separate ring buffers and extracts corresponding windows. If one side has fewer samples, the current code pads that side with silence rather than repeating its last sample. Repetition would create an artificial tone or echo; silence preserves a more honest gap while the streams catch up.

The mixed signal then branches. One copy goes to the recording path, where it is accumulated for the session artifact. The other goes through voice-activity detection, or VAD. VAD is a speech detector: it classifies short portions of audio as speech or non-speech and groups adjacent speech portions into segments. Only completed segments are sent onward to recognition.

flowchart LR N01["Device resolution and permission preflight"] --> N02["Microphone stream"] N01 --> N03["System-audio stream"] N02 --> N04["Mono, resample, and microphone enhancement"] N03 --> N05["System-audio preparation"] N04 --> N06["Align and mix asynchronous streams"] N05 --> N06 N06 --> N07["Mixed recording output"] N06 --> N08["Voice-activity detection"] N08 --> N09["Timestamped speech segments"] N09 --> N10["Transcription worker"]

The separation solves a real tension. If the recognizer received every sample in a long meeting, it would spend work on silence and background sound. If the saved file contained only the fragments classified as speech, it would no longer be a faithful record of pauses, playback, or the relationship between speakers and system sound. Meetily keeps the mixed recording path continuous while making the recognition path selective.

The exact behavior of the mixer should be described carefully. Some repository documentation uses language about intelligent or RMS-based ducking, which usually means lowering one source when another becomes prominent. The current mix_window implementation is more modest: it sums the microphone and system samples and proportionally scales a sum that would exceed the unit range. The microphone scaling variable is not applied, and the system scaling expression is currently unity. That is useful clipping prevention, but it is not evidence of a separately measured ducking algorithm. The ring-buffer comments also do not line up cleanly with all of the literal window values. The design intent is clear—align and combine asynchronous streams—but exact latency and ducking claims require runtime measurement, not a stronger reading of the comments.

VAD supplies the second major control point. Internally, the detector works at 16 kHz in short frames, with thresholds for entering and leaving speech, padding before and after detected speech, and a minimum speech duration. Padding gives the recognizer some context around a word; a minimum duration avoids sending tiny fragments that are unlikely to be useful. The segment is also timestamped relative to the beginning of the recording, so the text can later be associated with the corresponding place in the saved audio.

Meetily deliberately uses different pause policies for live and batch work. In the live path, a silence must qualify for roughly half a second before the detector closes a speech segment. A shorter wait reduces the delay before the interface can show text, but it also means that a natural pause can divide one thought into separate recognition requests. In import and retranscription, the code uses a two-second redemption period because it can process the entire file without a live feedback requirement. A longer pause bridges more conversational gaps and generally creates larger, more coherent segments.

That difference becomes visible in a simple case. Imagine someone explaining a design and pausing for a second to choose the next word. Batch processing can keep that pause inside one segment. Live processing may keep waiting only if the pause does not meet its shorter closure rule; otherwise it emits earlier but divides the explanation. The two settings are not contradictory. They optimize different outcomes.

There is also a boundary under continuous sound. If system audio is a podcast or a video with no qualifying silence, the live detector can keep one speech interval open and delay a transcript update. The current code explicitly treats bounded live segmentation during uninterrupted speech as an unresolved issue. Batch processing handles long intervals differently: after VAD, segments over about 25 seconds are split near low-energy regions rather than cut at arbitrary sample positions. That protects words near a boundary, but it is a batch quality strategy, not a guarantee that live output will arrive at a fixed cadence.

A useful way to see the hidden coupling is to follow one short exchange through the system. Suppose a participant speaks into a microphone whose native sample rate differs from the 48 kHz pipeline target while a video call contributes system audio at another rate. The microphone path is converted to mono if necessary, resampled with state carried across callbacks, and given the microphone-specific cleanup stages. The system stream is prepared through its own device path. The two streams are then aligned into a mixed window. That window is eligible for the recording output even if its energy is later classified as non-speech, because the recording’s job is to preserve the session. Only the speech portion selected by VAD becomes a recognition request.

This sequence also shows where time can accumulate. Device callbacks may not arrive together, so alignment waits for enough corresponding samples. A persistent resampler may hold a partial input until its processing window is complete. VAD then waits for a qualifying silence before it knows that a segment has ended. Finally, the recognition worker must process the segment. Those waits serve different purposes: alignment protects synchronization, resampling protects signal continuity, and redemption protects conversational context. Removing any one of them can make the interface feel faster while making the audio less reliable or the transcript less coherent.

The same example explains why system audio is not simply a second microphone. On Windows, the selected output is represented through a loopback-oriented device path; on Linux, the source may be a PulseAudio monitor; on macOS, the application can use a direct Core Audio tap or another capture backend. Each representation has its own permission and failure behavior. If the operating system exposes no usable monitor or tap, the local pipeline cannot reconstruct the missing sound from the transcript model. Locality gives the application control over the samples it receives; it does not give it privileged access to samples the operating system withholds.

By the time a speech segment reaches the next stage, several decisions have already shaped it: which device was accepted, whether permission delivered samples, how rates and channels were normalized, how microphone noise was treated, how the two streams were mixed, and how much silence was allowed around speech. The transcription model receives a local, timestamped request because this pipeline has prepared one—not because a model can independently recover every lost or misrouted signal.

That is the chapter’s central implication for configurable locality. Keeping capture local gives Meetily control over the raw audio path, but local control does not eliminate the engineering conditions that make the path accurate and timely. The next stage is therefore not a generic model tour. It is the question of what Whisper and Parakeet do with these segments, what their common interface hides, and how model size and hardware determine the cost of keeping recognition on the machine.

References