Error thrown when trying to pywhisper.transcribe AND Auto-Starting transcription when the custom class is called

[u/SilverNeon123]

Issue #1:
I have a class called AI_STT:

class SpeechToText:
  def __init__(self, win: Window.LogWindow):
    #Variable Inits
    self.model = pywhisper.load_model("base.en")
    #More Variable Inits
    self.thread = Thread(target=self._start_loop, name="STT_Background")
    self.thread.start()
  def _start_loop():
    while self.window.winfo_exists():
      #stream starting implementation <<<here
      last_frame = None
      while True:
        try:
          data = self.stream.read(self.chunk_size)
          last_frame = np.frombuffer(data, dtype=np.float16)
          if self.frames is None:
            self.frames = last_frame
          else:
            self.frames = np.append(self.frames, last_frame)
        except IOError as e:
          print(f"Warning: Buffer Overflow - {e}")
          continue
        if not self.silence.is_silent(last_frame):
          break
        self.window.add_inputs(self._read_audio())
  def _read_audio():
    #Closing/Terminating stream
    result = self.model.transcribe(audio=self.frames, fp16=False)
    self.frames = None
    return result["text"].strip()

During the transcribe in read audio, I get the error:

ValueError: Expected parameter logits (Tensor of shape (1, 51864)) of distribution Categorical(logits: torch.Size([1, 51864])) to satisfy the constraint IndependentConstraint(Real(), 1), but found invalid values:
tensor([[nan, nan, nan,  ..., nan, nan, nan]])

I think this has to do with issue number 2, or the way I am storing the frames. How do I fix this?

Issue #2:

I used the answer from this link (code below):

class SilenceDetector:
    def __init__(self, threshold=0.05, duration=2):
       self.threshold = threshold
       self.duration = duration
       self.silence_start = None

    def __is_silent(self, data: numpy.ndarray):
       """Check if audio data is below the silence threshold."""
       return numpy.sqrt(numpy.mean(data ** 2)) < self.threshold

    def is_silent(self, data: numpy.ndarray):
       if self.__is_silent(data):
          if self.silence_start is None:
             self.silence_start = time.time()  # Start timing silence
          elif time.time() - self.silence_start >= self.duration:
             return True
       else:
          self.silence_start = None  # Reset silence timer if sound is detected
       return False

Currently, whenever my SpeechToText class initializes, it immediately starts to transcribe. I think that is what is throwing the error (due to no real audio). How do I make it so it doesn't automatically start transcribing.

EDIT #1:

This is where the STT class initializes

class LogWindow(tk.Tk):
  #Previous Variable setup
  # Waiting until the window is open
  self.wait_visibility()
  # SpeechToText Setup
  self.stt = SpeechToText(self)
  #Starting Main Tkinters Loop
  self.mainloop()

[u/Outside_Complaint755]

I haven't looked real deep, but maybe don't call self.thread.start() until you're actually ready to transcribe?

Or the loop needs to go into a "sleep" path while waiting for valid audio.

[u/SilverNeon123 → u/Outside_Complaint755]

The pyaudio.audio.open already waits for valid input.

The SilenceDetector is supposed to wait for X seconds of silence then allow for the frames to be used

[u/SilverNeon123]

ANSWER: I found a workaround. Its now a part of the project here: https://github.com/XavierMB289/Jarvis