Realtime OpenAI Model Hearing Itself

[kavimathur]

I've found that without using headphones, the realtime model will hear itself and get caught in conversation loops as it keeps interrupting itself. This is especially true when using .semanticVAD, and a makeshift patch that ameliorates the problem is by using .serverVAD with a high threshold.

I know a lot of work has been put into this already, and I probably won't be able to work on a solution for it right away, but I do want to highlight that this still seems to be a problem and document it here.

This was found while testing on an iPhone 16 Pro, iOS 26.0.1

[tominisx-1]

Hello, is there any solution or workaround for this issue? I'm facing the same problem and not sure how it can be used if it’s now just talking to itself :/

[lzell]

Dropping some context here from a chat with Kavi I had.

I'm sorry to say I don't have the bandwidth to really dive into this right now, as realtime hasn't seen the demand that I was hoping for with AIProxy. There is actually a bunch of work to be done to get the SDK off of the beta version of realtime and onto the general release. I think there is a chance that doing that alone would help, because presumably the model has improved to not hear itself.

For the AudioUnit / AudioEngine implementations of getting mic data, I believe setting setVoiceProcessingEnabled and kAudioUnitSubType_VoiceProcessingIO are the only tools that Apple gives us to prevent the mic stream from containing the speaker output. If you search for those in the codebase, you'll find the two different implementations.

There is also a docstring on AudioController that shows which implementation (AudioEngine or AudioToolbox) is used on each platform: https://github.com/lzell/AIProxySwift/blob/main/Sources/AIProxy/AudioController.swift#L21-L30

We could possibly try swapping the platform that you are on to use the other implementation to see if that helps. But then I think the headphone use will be wonky. It took a lot of fiddling to get this right across platforms. I will say, though, that iOS is the most important platform so if we need to sacrifice one of the others in the short term, we could

[tominisx-1]

Thanks for the explanation. I’m testing on iOS 26.1 with no headphones connected, and this problem appears every time.
When the AI speaks first, it reliably trips on the first few audio packets, causing the model to interrupt itself and effectively talk to itself for first ~20 seconds or so.

I’m not sure this is the ideal fix, but in my testing the issue disappears completely if the change below is made in AudioController.

        try? AVAudioSession.sharedInstance().setCategory(
            .playAndRecord,
            mode: .voiceChat,
            options: [.defaultToSpeaker, .allowBluetooth]
        )

to

        try? AVAudioSession.sharedInstance().setCategory(
            .playAndRecord,
            mode: .voiceChat,
            options: [.allowBluetooth]
        )

Keen to hear your thoughts on this.

[lzell]

Whoa, really?! Is the playback audio still loud enough for use without headphones?

[tominisx-1]

You're right - when I removed .defaultToSpeaker, the volume is lower because it no longer routes through the speaker. I spent some time yesterday / today trying to figure out why using the speaker causes this issue, but I wasn’t able to pinpoint it :(

With the fix above, the volume is relatively low (still usable at max volume, but not as loud as some might expect). On the plus side, this approach makes the realtime functionality actually usable in prod - otherwise, it’s a really poor user experience if the model always just interrupts itself at the start of each conversation :(

If you have any other ideas or workarounds in mind that would at least make it work reliably on iOS without headphones, I’d really appreciate it! Also, let me know if you cannot reproduce the issue - I'd be happy to share some evidence / sample code.

[lzell]

It's weird that I can't repro this. I just tried on my phone with 26.1 and the speaker volume cranked to max, the model does not hear itself. I'll put up a gist tonight of my little sample project, but in the meantime if you have a snippet that reliably reproduces the model talking to itself, it would be helpful in a gist.

I'm wondering now if it has something to do with the initialization order of the mic PCM sample vendor and the audio player.

Did you use the RealtimeManager sample snippet here as a starting point:
https://github.com/lzell/AIProxySwift?tab=readme-ov-file#how-to-use-realtime-audio-with-openai

[lzell]

@kavimathur @tominisx-1 when you can, please see if the model hears itself in this sample app: https://github.com/lzell/OpenAIRealtimeSample

[tominisx-1]

Hi @lzell . Unfortunately, the issue is still occurring in your sample app. I did not modify the code in any way (just put my unprotectedAPIKey in code).

I recorded the current behavior directly from the sample app to demonstrate the problem. During the recording, I was in a completely silent room and did not make any noise that could have triggered an interruption (input audio).

I've tested in iOS 26.1 iPhone 16 and iOS 26.1 iPhone 11. Same problem on both devices.

See recording bellow:
https://www.transfernow.net/dl/202601013NK2L0QR

For some reason, .inputAudioBufferSpeechStarted is being triggered even in a completely silent environment. My current hypothesis is that this may be caused by audio echo from the model’s own speech.
To investigate further, I used .inputAudioTranscriptionCompleted to log what the model was detecting as input in this silent environment. The transcriptions appear to be nonsensical (not even the words that model is saying), which suggests the model may be producing hallucinated output when it receives only low-level echo from speaker rather than actual speech.
After the interruptions .inputAudioTranscriptionCompleted outputs examples such as:
"Thank you."
"Absolutely."
"Bye-bye."
"All right."

[lzell]

Thank you for the repro vid! I am def perplexed at this one, because I am running on an iPhone 12 mini and iPhone 16 Pro and neither exhibit this behavior with volume all the way up. I guess the room acoustics must have a meaningful effect on Apple's kAudioUnitSubType_VoiceProcessingIO. I'll try a few different areas and see if I can find a spot to match your experience.

The fact that it's switching to Spanish and your observations about the transcription being nonsensical is making me think Apple's voice processing unit is working, because when kAudioUnitSubType_VoiceProcessingIO is not used, the model hears itself really clearly and just keeps repeating "hello" back to itself.

So I like your theory about the low grade background echo, stuff that Apple's framework doesn't fully eliminate (or what's left over when it tries to negate the speaker signal from the mic input). I wonder if applying a low pass filter or band pass filter would help.

[lzell]

One idea to quantify what we are seeing. writePCM16IntValuesToFile is a debug function that I used when I was developing the audio parts of the lib for macOS. I would dump the pcm16 values from the mic to a file, and then plot the data points in the file (I used python's pandas for this part). I had to do this because there were periods were the model would get really confused, and it turned out it was because sample rate conversion (from the native hardware's rate to the rate OpenAI expects) I was introducing some audio blips.

But I think we could use that helper here too, and see how much of the speaker audio is being snipped out if we are silent and let the model speak to itself.