voice-activity-detection

by benchflow-ai

Install this Skill

curl -fsSL -o voice-activity-detection.zip https://skill.seembox.com/api/v1/skills/voice-activity-detection/download && unzip voice-activity-detection.zip

Download voice-activity-detection.zip

SKILL.md About

---
name: Voice Activity Detection (VAD)
description: Detect speech segments in audio using VAD tools like Silero VAD, SpeechBrain VAD, or WebRTC VAD. Use when preprocessing audio for speaker diarization, filtering silence, or segmenting audio into speech chunks. Choose Silero VAD for short segments, SpeechBrain VAD for general purpose, or WebRTC VAD for lightweight applications.
---

# Voice Activity Detection (VAD)

## Overview

Voice Activity Detection identifies which parts of an audio signal contain speech versus silence or background noise. This is a critical first step in speaker diarization pipelines.

## When to Use

- Preprocessing audio before speaker diarization
- Filtering out silence and noise
- Segmenting audio into speech chunks
- Improving diarization accuracy by focusing on speech regions

## Available VAD Tools

### 1. Silero VAD (Recommended for Short Segments)

**Best for:** Short audio segments, real-time applications, better detection of brief speech

```python
import torch

# Load Silero VAD model
model, utils = torch.hub.load(
    repo_or_dir='snakers4/silero-vad',
    model='silero_vad',
    force_reload=False,
    onnx=False
)
get_speech_timestamps = utils[0]

# Run VAD
speech_timestamps = get_speech_timestamps(
    waveform[0],  # mono audio waveform
    model,
    threshold=0.6,  # speech probability threshold
    min_speech_duration_ms=350,  # minimum speech segment length
    min_silence_duration_ms=400,  # minimum silence between segments
    sampling_rate=sample_rate
)

# Convert to boundaries format
boundaries = [[ts['start'] / sample_rate, ts['end'] / sample_rate]
              for ts in speech_timestamps]
```

**Advantages:**
- Better at detecting short speech segments
- Lower false alarm rate
- Optimized for real-time processing

### 2. SpeechBrain VAD

**Best for:** General-purpose VAD, longer audio files

```python
from speechbrain.inference.VAD import VAD

VAD_model = VAD.from_hparams(
    source="speechbrain/vad-crdnn-libriparty",
    savedir="/tmp/speechbrain_vad"
)

# Get speech segments
boundaries = VAD_model.get_speech_segments(audio_path)
```

**Advantages:**
- Well-tested and reliable
- Good for longer audio files
- Part of comprehensive SpeechBrain toolkit

### 3. WebRTC VAD

**Best for:** Lightweight applications, real-time processing

```python
import webrtcvad

vad = webrtcvad.Vad(2)  # Aggressiveness: 0-3 (higher = more aggressive)

# Process audio frames (must be 10ms, 20ms, or 30ms)
is_speech = vad.is_speech(frame_bytes, sample_rate)
```

**Advantages:**
- Very lightweight
- Fast processing
- Good for real-time applications

## Postprocessing VAD Boundaries

After VAD, you should postprocess boundaries to:
- Merge close segments
- Remove very short segments
- Smooth boundaries

```python
def postprocess_boundaries(boundaries, min_dur=0.30, merge_gap=0.25):
    """
    boundaries: list of [start_sec, end_sec]
    min_dur: drop segments shorter than this (sec)
    merge_gap: merge segments if silence gap <= this (sec)
    """
    # Sort by start time
    boundaries = sorted(boundaries, key=lambda x: x[0])

# Remove short segments
    boundaries = [(s, e) for s, e in boundaries if (e - s) >= min_dur]

# Merge close segments
    merged = [list(boundaries[0])]
    for s, e in boundaries[1:]:
        prev_s, prev_e = merged[-1]
        if s - prev_e <= merge_gap:
            merged[-1][1] = max(prev_e, e)
        else:
            merged.append([s, e])

return merged
```

## Choosing the Right VAD

| Tool | Best For | Pros | Cons |
|------|----------|------|------|
| Silero VAD | Short segments, real-time | Better short-segment detection | Requires PyTorch |
| SpeechBrain VAD | General purpose | Reliable, well-tested | May miss short segments |
| WebRTC VAD | Lightweight apps | Fast, lightweight | Less accurate, requires specific frame sizes |

## Common Issues and Solutions

1. **Too many false alarms**: Increase threshold or min_speech_duration_ms
2. **Missing short segments**: Use Silero VAD or decrease threshold
3. **Over-segmentation**: Increase merge_gap in postprocessing
4. **Missing speech at boundaries**: Decrease min_silence_duration_ms

## Integration with Speaker Diarization

VAD boundaries are used to:
1. Extract speech segments for speaker embedding extraction
2. Filter out non-speech regions
3. Improve clustering by focusing on actual speech

```python
# After VAD, extract embeddings only for speech segments
for start, end in vad_boundaries:
    segment_audio = waveform[:, int(start*sr):int(end*sr)]
    embedding = speaker_model.encode_batch(segment_audio)
    # ... continue with clustering
```