❯ Harshith.
Back to work

Run 08ML for Signal Processing · Johns Hopkins · Fall 2024

Noise-Resilient Emotion Recognition

Speech + NLP · MELD

noisy baseline
18%
after denoise
51%
multimodal
+20%

PyTorch · Wav2Vec2 · HuBERT · BERT · MetricGAN+

Problem

Speech emotion recognition collapses under additive noise. A model that looks fine on studio audio is not a model you can put on a real call.

Approach

Staged integration on the MELD dataset:

  1. MetricGAN+ enhancement under additive noise.
  2. Wav2Vec2 features as a robust speech encoder.
  3. HuBERT + BERT fusion for joint audio-text classification.

Noisy-audio accuracy moved from 18% to 51% after denoising and Wav2Vec2, then gained another 20% with multimodal fusion.

The lesson I kept: do not ask the classifier to undo a corrupted front-end. Fix the signal, then fuse modalities, and measure each stage on its own.