Deep Learning Practice · Quiz 2 · May 2025

In a speaker diarization pipeline combining Whisper (for ASR transcription) and a speaker embedding model (such as x-vector / ECAPA-TDNN followed by clustering), if the transcribed text is completely accurate but a single speaker's utterances are split across multiple speaker labels (`SPEAKER 1`, `SPEAKER 3`, `SPEAKER 5`), the issue lies in the **speaker embedding / clustering stage**, not the ASR stage. Specifically, if the speaker's voice changes pitch, tone, emotion, or speaking rate significantly across segments, the embedding model produces highly dissimilar x-vector embeddings for the same person, causing the clustering algorithm to assign those segments to different speaker clusters. **Answer: D (The speaker's voice changed pitch and tone significantly during their speech, causing the embedding model to generate very different x-vectors for the same person.)** A: **Incorrect:** Configuring `AgglomerativeClustering` with too *few* clusters would merge different speakers into the same label (under-clustering), not split one speaker into many labels. B: **Incorrect:** Inaccurate timestamps would misalign words at speaker boundaries rather than creating multiple distinct speaker clusters for a single speaker. C: **Incorrect:** The prompt states that the Whisper transcription is perfectly accurate, ruling out Whisper confusion. D: **Correct:** Large variations in pitch and tone cause the speaker embedding model to output distant x-vectors for the same speaker, which clustering separates into multiple speaker IDs.

In Hugging Face `transformers`, audio models like `Wav2Vec2ForCTC` expect a **2D batched tensor** of shape `(batch_size, sequence_length)` as `input_values`. In the given snippet, `input_values = torch.randn(16000)` creates a **1D tensor** of shape `(16000,)` without a batch dimension (it should be `torch.randn(1, 16000)` or `input_values.unsqueeze(0)`). **Answer: C (The model expects batched input; `input_values` must have shape `(batch_size, sequence_length)`, but currently has shape `(sequence_length,)`.)** A: **Incorrect:** PyTorch `Wav2Vec2ForCTC` models require `torch.Tensor` inputs, not NumPy arrays. B: **Incorrect:** While `processor.batch_decode` can decode logits after the forward pass, the immediate runtime error in the forward pass is the missing batch dimension. C: **Correct:** `Wav2Vec2ForCTC` expects `input_values` of shape `(batch_size, sequence_length)`, whereas `torch.randn(16000)` has shape `(16000,)`. D: **Incorrect:** PyTorch models run on CPU by default unless explicitly moved to GPU.

Let us evaluate each statement about the ASR (`DataCollatorCTCWithPadding`) and TTS (`TTSDataCollatorWithPadding`) data collators: - **A is correct**: In ASR with CTC loss, padded positions in the target `labels` are replaced with `-100` so that PyTorch loss functions ignore padded tokens during loss computation. - **B is incorrect**: The TTS data collator also masks/handles padded target positions (e.g., replacing padded label/mel positions or creating an attention/stop mask) so padding does not corrupt the spectrogram loss. - **C is correct**: In `SpeechT5ForTextToSpeech`, when `reduction_factor > 1`, the model predicts `reduction_factor` spectrogram frames per decoder step, so the collator must pad/adjust target spectrogram lengths to be a multiple of the model's `reduction_factor`. - **D is correct**: Both collators dynamically pad the variable-length sequence inputs in each batch to the length of the longest sequence in that batch to form rectangular tensors. - **E is incorrect**: `DataCollatorCTCWithPadding` is used for ASR (audio-to-text) and does not generate speaker embeddings (which are used in multi-speaker TTS). **Answer: A, C, D** A: **Correct:** `DataCollatorCTCWithPadding` sets padded target label tokens to `-100` so they are ignored in CTC loss calculation. B: **Incorrect:** Padded positions in TTS targets must also be masked/padded properly rather than left unmasked in loss calculation. C: **Correct:** `TTSDataCollatorWithPadding` adjusts target mel-spectrogram lengths to be divisible by the SpeechT5 `reduction_factor`. D: **Correct:** Both collators perform dynamic batch padding to the maximum sequence length within the batch. E: **Incorrect:** Speaker embeddings are used in TTS, not generated by the ASR CTC data collator.

Let us evaluate each statement about fine-tuning `SpeechT5ForTextToSpeech` on a new, non-English language dataset: - **A is correct**: Pretrained x-vector speaker embedding models (like `speechbrain/spkrec-xvect-voxceleb`) capture acoustic voice characteristics (timbre/pitch) rather than language semantics, so they can be used directly without training a new x-vector model on the target language. - **B is correct**: If the pretrained SpeechT5 character vocabulary lacks characters from the new language's script, transliterating/converting input text into phonetic units (phonemes/IPA) or replacing unsupported characters is a standard preprocessing step. - **C is correct**: During inference, `SpeechT5ForTextToSpeech` requires a `speaker_embeddings` tensor as a conditional input alongside `input_ids` to synthesize speech. - **D is incorrect**: `SpeechT5ForTextToSpeech` outputs **log-mel spectrograms**, not raw audio waveforms; a separate vocoder (such as `SpeechT5HifiGan`) converts the predicted spectrogram into a waveform. - **E is correct**: Because the HiFi-GAN vocoder (`microsoft/speecht5_hifigan`) maps acoustic mel-spectrograms to waveforms, it is largely language-agnostic and can be reused directly for the new language. **Answer: A, B, C, E** A: **Correct:** A pretrained x-vector model captures language-independent speaker voice characteristics and works without language-specific retraining. B: **Correct:** Converting text to phonemes or cleaning unsupported characters resolves out-of-vocabulary script issues. C: **Correct:** `SpeechT5ForTextToSpeech` requires a conditional `speaker_embeddings` vector during inference. D: **Incorrect:** SpeechT5 predicts mel-spectrograms, which are then converted to waveforms by a HiFi-GAN vocoder. E: **Correct:** The `speecht5_hifigan` vocoder is acoustic rather than linguistic and works across languages.

**Step 1: Recall the formula for Word Error Rate (WER).** \[\text{WER} = \frac{S + D + I}{N}\] where \(S\) is the number of substitutions, \(D\) is the number of deletions, \(I\) is the number of insertions, and \(N\) is the number of words in the `ground_truth` reference. **Step 2: Align the `ground_truth` and `prediction` word sequences.** - `ground_truth`: `"i like to eat apples and bananas"` (\(N = 7\) words) - `prediction`: `"i like eating apple and banana"` (6 words) Word-by-word alignment: 1. `"i"` \(\to\) `"i"` (Match) 2. `"like"` \(\to\) `"like"` (Match) 3. `"to"` \(\to\) deleted (\(D = 1\)) 4. `"eat"` \(\to\) `"eating"` (Substitution, \(S = 1\)) 5. `"apples"` \(\to\) `"apple"` (Substitution, \(S = 2\)) 6. `"and"` \(\to\) `"and"` (Match) 7. `"bananas"` \(\to\) `"banana"` (Substitution, \(S = 3\)) **Step 3: Compute WER.** \[\text{WER} = \frac{3 + 1 + 0}{7} = \frac{4}{7} \approx 0.5714\] Rounded to two decimal places, the WER is **0.57**. **Answer: 0.57**

**Step 1: Compute the total number of audio segments across all speakers.** \[\text{Total segments} = 150\text{ speakers} \times 20\text{ segments/speaker} = 3000\text{ segments}\] **Step 2: Multiply by the embedding dimension (512 floating-point values per segment).** \[\text{Total floating-point values} = 3000 \times 512 = 1,536,000\] **Answer: 1536000**

In the Wav2Vec2 pipeline, `Wav2Vec2Processor` wraps a feature extractor (`Wav2Vec2FeatureExtractor`) that normalizes the raw audio waveform (zero mean, unit variance) and converts it into a PyTorch tensor (`return_tensors="pt"`) of shape `(batch_size, sequence_length)` ready for the model. **Answer: B (To normalize the raw audio waveform and prepare it in the correct tensor format for the model.)** A: **Incorrect:** Contextualized hidden states are produced by `Wav2Vec2Model` (the CNN + Transformer encoder), not the processor. B: **Correct:** `Wav2Vec2Processor` normalizes the raw waveform and formats it as a PyTorch tensor. C: **Incorrect:** Passing `sampling_rate=16000` only informs the processor that the input is already at 16 kHz; it does not resample from another rate. D: **Incorrect:** Raw audio is continuous and is not tokenized into discrete text-like tokens by the processor.

**Step 1: Identify the input audio length and output feature sequence length.** - Input audio length = \(16000\) samples - Output `features` sequence length = \(49\) frames **Step 2: Calculate the approximate downsampling factor.** \[\text{Downsampling factor} = \frac{16000}{49} \approx 326.5306\] Rounded to the nearest integer, this is **327** (note that the 7-layer CNN feature encoder in Wav2Vec2 has a stride product of \(5 \times 2^6 = 320\) with receptive field 400, yielding \(\lfloor(16000 - 400)/320\rfloor + 1 = 49\) frames, so the ratio \(16000 / 49 \approx 327\)). **Answer: 327**

Let us evaluate each statement about `features = outputs_model.last_hidden_state` of shape `(1, 49, 768)`: - **A is correct**: The last dimension `768` is the Transformer hidden size (`config.hidden_size`), representing a 768-dimensional feature vector for each of the 49 time frames. - **B is incorrect**: The sequence length (`49` for 1 second of audio) scales linearly with the duration of the input audio waveform. - **C is correct**: `last_hidden_state` is the output of passing the local CNN latent representations through the stack of Transformer encoder layers. - **D is correct**: For sequence-level tasks such as audio/speaker/intent classification, applying mean pooling over the time-step dimension (`dim=1`) collapses `(batch_size, seq_len, 768)` into a single fixed-size vector `(batch_size, 768)` per audio clip. - **E is incorrect**: `Wav2Vec2Model` is the base encoder without a CTC language modeling head, so `last_hidden_state` contains continuous hidden representations rather than vocabulary probability distributions. **Answer: A, C, D** A: **Correct:** `768` is the hidden embedding dimension at each output time step. B: **Incorrect:** The number of output frames depends on the input audio duration (roughly 50 frames per second). C: **Correct:** `last_hidden_state` is the contextualized representation output by the Transformer encoder stack. D: **Correct:** Mean pooling along `dim=1` aggregates the 49 frame vectors into a single 768-D clip representation. E: **Incorrect:** Vocabulary probability distributions require a `Wav2Vec2ForCTC` projection head and softmax, not the base `Wav2Vec2Model`.

Setting `group_by_length=True` in Hugging Face `TrainingArguments` groups training examples of roughly similar input sequence lengths into the same mini-batch. Because dynamic padding pads every sample in a batch to the length of the longest sample in that batch, grouping similar-length audios together drastically reduces wasted padding tokens and speeds up training. **Answer: C (To create batches with samples of similar input length, which minimizes the amount of padding required and improves training efficiency.)** A: **Incorrect:** It does not permanently sort the entire dataset monotonically from shortest to longest across the whole epoch without randomization; it buckets similar lengths while preserving randomness. B: **Incorrect:** Samples in a batch have *similar* (not identical) lengths, so dynamic padding is still used. C: **Correct:** Bucketing samples of similar lengths minimizes dynamic padding overhead and significantly accelerates training. D: **Incorrect:** For `Wav2Vec2ForCTC`, `group_by_length` groups by the primary model input (`input_values` audio length), not text transcription length.

Let us examine the given `TrainingArguments`: - **A is correct**: `evaluation_strategy="steps"` and `eval_steps=500` cause the Trainer to evaluate the model on the validation set every 500 training steps. - **B is correct**: `save_total_limit=2` and `load_best_model_at_end=True` ensure that at most 2 checkpoints are kept on disk during training (plus the final best model loaded at the end). - **C is correct**: `learning_rate=1e-4` and `warmup_steps=1000` (with default `lr_scheduler_type="linear"`) linearly warm up the learning rate from `0` to `1e-4` over the first 1000 steps and then linearly decay it. - **D is incorrect**: If the dataset size is not evenly divisible by `8` or if multi-GPU (`n_gpu > 1`) is used, the number of steps per epoch is `ceil(num_train_samples / (8 * n_gpu))`, not necessarily exact division `(num_train_samples / 8) * 7`. - **E is correct**: `fp16=True` enables 16-bit mixed-precision training, which reduces memory usage and accelerates training on GPUs with Tensor Cores. **Answer: A, B, C, E** A: **Correct:** `evaluation_strategy='steps'` with `eval_steps=500` runs validation every 500 steps. B: **Correct:** `save_total_limit=2` restricts the number of saved checkpoint folders in `output_dir` to at most 2. C: **Correct:** `warmup_steps=1000` with `learning_rate=1e-4` linearly warms up LR to `1e-4` over 1000 steps before decaying. D: **Incorrect:** Batch size rounding (`ceil`), drop-last settings, or multi-GPU device counts mean total steps is not strictly `(num_train_samples / 8) * 7`. E: **Correct:** `fp16=True` enables mixed-precision FP16 training.