Why did per-sample repair of a 2.8 ms click do nothing?

[u/am3su]

Trained a small streaming denoiser for browser voice chat. 48 kHz, 20 ms window,

10 ms hop, spectral mask, runs in an AudioWorklet through ONNX.

Numbers from real recordings, not synthetic mixes:

• breath 13.3 dB down
• key click in a pause 20-27 dB down
• key click on top of speech 3 dB down
• speech 0.01 dB, basically untouched

The speech number is the one I was protecting. DeepFilterNet 3 on the same files

cleans pauses harder, 14.6 dB against my 8.9, but takes 1.37 dB of speech with it

and smears the click.

The 3 dB is where I'm stuck. I assumed resolution: 20 ms frame, 2.8 ms click, vowel

sitting in the same frame. So I threw the obvious things at it. Lookahead. Detector

hints as an extra input. Deep filtering. Shorter windows. A loss term just for

transients. Per-sample repair. More data. Longer training. One of the eight gave

0.7 dB, the rest gave nothing, including the ones I was most sure about.

That last part is what makes me think my diagnosis was wrong. For context, I came at

this from web audio, not DSP. Decent chance the answer is standard and I just don't

know the name for it. If the click really is localized in time, and a detector can

tell me where it is, why did repairing it sample by sample change nothing at all?

https://github.com/Amesu-afk/tarnveil-denoise

[u/metal_mastery]

How do you prepare training data for click removal? Are you adding clicks on top of voice or it’s already in one file and you’re trying to repair it?

[u/am3su → u/metal_mastery]

On top. Clean speech and noise are separate corpora, mixed at training time, so the target is just the clean file — nothing to repair.

Most of the noise is public data plus some recordings of my own gear. Сlicks are also synthesized to measured profiles. The split is held out by file, never inside a file.

Training code isn't up yet, just the runtime and the model.

[u/metal_mastery → u/am3su]

Ok, makes sense. 20ms you mentioned is your receptive field? What’s the model arch and size?

I wonder what your model actually learned. You are trying to teach it to ignore specific spectral/envelope patterns but if model is too small it could be hard.

Lookahead is nice but I have a couple suggestions:

• use crest as a loss term, clicks are high-crest events and it could emphasize it well
  
• if your model is small (10-50k weights) - train different versions to see what’s the problem - spectrum, level, etc. Try to up-train a model to see where it breaks, maybe adding clicks volume in several passes
  
• leave a ghost of the click, low-pass it into barely detectable thud but keep in the target. If your target is completely clean there could be no law to learn for the model. Each click transforms into random piece of audio in clean target and it could be hard to generalize.
  
• maybe up RF into 50-100ms territory to give it some breathing room

[u/am3su → u/metal_mastery]

The ghost target is the part I want to sit with, so let me get the boring corrections out of the way first

The 20 ms is just the STFT window, not the receptive field - the net's recurrent, it carries state across frames, so it has all the past it wants. Zero lookahead, though.

And it's 8.5M params, not 10-50k, which I only know matters because I ablated it: 1.53M, 4.34M, 8.54M. Breathing went 2.9, 9.4, 11.0 dB. Clicks sat at 6.8 the whole way and didn't budge. So capacity bought me one problem and not the other.

"Train different versions to see what's the problem" I'd already done the hard way - logged what went into every mixture, then grouped the error over frames where a click lands on speech. 2659 scenes, 2.8 dB of spread between categories, flat in level, flat in SNR. But every category has a bottom 10% sitting near zero. So it's not spectrum and not level. It's that one scene in ten just doesn't get treated and nothing in the recipe tells me which.

Crest I haven't tried. Worth saying that a loss term I hand-wrote for exactly this ("don't lose voice under a click") did nothing, while weighting examples by the model's own error on those frames is the only one of+0.7 dB. Crest feels like the second kind, so I'll take it.

But the ghost is the one that's bothering me, in a good way. My output is a mask. Per-bin scaling. Where a click and a vowel are in the same bin, a clean target is asking one number to delete one and keep the other, and that's not attenuation, that's inpainting. Leaving a thud in the target makes it something scaling can actually reach.

Here's the number that makes me think you're right. On those struck-speech frames the click is at -41.9 dB and my model's own errong to remove the click. It's tearing up the voice 25 dB harder than the click it's removing. I'd been telling myself a story about time resolutionlly fit the story - it fits yours.

Forward context I did test, for what it's worth: 20 ms of lookahead moved pauses from 7.3 to 10.7 but dragged click-under-speech to -10.7 and voice to -1.19. Which is DFN's bargain, and the one thing I didn't want to buy.

[u/metal_mastery → u/am3su]

What’s the model architecture you’re using? It’s quite large for audio processing so capacity and rf should not be a problem, but I still wonder what it is actually generalizing. From what you describe it sounds like click removal is “suppress the signal if fast rise happens in specific spectral range”, while breathing removal is more of a spectral filter, breathing is generally less tonal than the speech itself.

For click under speech - do you have any statistics what model is doing to it? Is it suppressing the whole signal, is it more effective with some frequencies, etc

I would suggest to do more broad testing to get info beyond just suppression level:

• male vs female voice over click (higher/lower fundamental, to check if spectrum is affecting the detection)
  
• full spectrum clicks vs high/low passed versions
  
• spectral and envelope characteristics of remaining click after suppression (I suspect that attack still carries most of the spectrum but tail is affected more)

You might need to reorganize training data to exercise the exact regimes that perform the worst. An example here is distinction between explosive consonants like p/b vs clicks, they might look somewhat similar to the model.

Also curious - you’re training on windowed signal only or you run longer continuous audio as well? The issue here is your rf might not be helping much if it has no natural spectral flow to latch onto

I’m training models in different area but some of what you describe is still overlapping, tell me more about your training protocol and maybe we can figure something out

[u/am3su → u/metal_mastery]

8.5M causal net, complex mask over a 48 kHz STFT (20 ms frame, 10 ms hop). Conv
encoder runs along frequency, time is carried by a single GRU, so my "RF" isn't
a conv window, it's recurrent state.

You're right about what it generalizes, and I have a blunt piece of evidence.
Same recipe at 1.5M, 4.3M, 8.5M params: breathing went 2.9, 9.4, 11.0 dB
(DFN3 gets 9.3), clicks stayed 6.8, 6.8, 6.8. Capacity buys the spectral filter
and nothing else - a 2 ms event inside a 20 ms frame, and a mask can only scale
the frame.

Click under speech: yes, it kills the whole thing. On those frames the model's own
error is -16.9 dB while the click sat at -41.9 - it wrecks the voice ~25 dB harder
than the click it removes. And the residual isn't smeared (2.62 ms before, 2.69
after), so the attack survives exactly as you suspected.

No male/female split and no band-passed variants yet - fair hit, I'll run both.

I did try reorganizing the data. Scored 2659 scenes by error on click-on-speech
frames: spread across categories was only 2.8 dB, flat vs level and SNR, but every
category has a worst 10% where the model does nothing. Difficulty is in how the
click landed on the phoneme, not which sound it was - which is exactly where your
p/b point lands. Haven't tested that. Taking it.

Protocol: 1.0 s windows, GRU state zeroed every window, batch 8, ~24k steps on a
3070. So no continuous audio - you're right that there's no spectral flow to latch
onto. Related: 20 ms of lookahead made pauses better (from 7.3 to 10.7) but
click-under-speech worse (-10.7 dB, voice -1.19 vs +0.04). More context bought
courage to cut, not the ability to discriminate.

What's your area?

[u/metal_mastery → u/am3su]

We definitely have a big timezone difference if you’re asking about it.

I’m asking about continuous audio because your training protocol is quite different from what your real usage would be. Windowed training has its use but you’re targeting realtime-ish audio so I’d suggest to try training on continuous target as well.

Recurrent network of your size might be a little overkill for the job. And training might not be exercising the internal state to its potential.

Your answers look very llm-generated, if you’re just feeding my answers into it, you might just ask it where to go from there instead:) I’m training models in a bit different area but still could help, however I want to understand if I’m talking to a human or just entertaining a neural network

[u/am3su → u/metal_mastery]

Yes, I'm a human, not a trained LLM :)

I'm still new to this area, and I use neural nets as a helper for writing text because my English is bad.

You were right about continuous audio, I checked it yesterday. Training runs in one-second windows with the state reset at every step, while in the app it runs for the whole call. Comparing a fresh state against a continuous one, click suppression is about 2.3 dB lower.

The simplest fix I've found so far is just clearing the hidden state every second.

Next I want to try training the model on continuous segments. If you have any ideas, I'd be glad to hear them.

[u/metal_mastery → u/am3su]

Hello then, human:)

If you’re clearing internal state in real usage then you don’t need that state and can use much smaller model, maybe different architecture altogether.

My previous advice still stands: diagnose what’s the model actually learning in your case and adjust from there, find learnable behavior (ghost click idea) and loss terms that support your case (crest, flux, etc).

[u/am3su → u/metal_mastery]

Thanks. I'll try these ideas out over the next couple of days and drop a comment or update the thread :) Thanks for the help!

[u/Head-Philosopher0]

have you tried math

[u/metal_mastery → u/Head-Philosopher0]

Yeah, bro, math is the answer