How to write generated ogg data into a webrtc audio track?

[shellfly]

I'm using Text to Speech service to generate streaming OGG data, I want to send the streaming output into a WebRTC audio track. I checked the play-from-disk example in the repo and the below code is what I have written so far.

The problem is that I can send data to the client but I can't hear the audio on the client.

Other things I have tested:

1. If I save the data into a ogg file, it's a valid file and I can play it.
2. Change the oggreader.NewWith(data) to oggreader.NewWith(f) where f is a file opened from disk. There is sound in the client but the speed is weird.

It seems it's something about the ogg options, but I'm not familiar with it. Any idea how can I get this code to work?

func sendAudioData(data io.Reader, audioTrack *webrtc.TrackLocalStaticSample) error {
    ogg, _, oggErr := oggreader.NewWith(data)
    if oggErr != nil {
        return oggErr
    }
    var lastGranule uint64
    for {
        pageData, pageHeader, oggErr := ogg.ParseNextPage()
        if errors.Is(oggErr, io.EOF) {
            fmt.Printf("All audio pages parsed and sent")
            return nil
        }

        if oggErr != nil {
            return oggErr
        }
        slog.Info("ParseNextPage", "header", pageHeader)
        // The amount of samples is the difference between the last and current timestamp
        sampleCount := float64(pageHeader.GranulePosition - lastGranule)
        lastGranule = pageHeader.GranulePosition
        sampleDuration := time.Duration((sampleCount/48000)*1000) * time.Millisecond

        slog.Info("Write Sample", "duration", sampleDuration, "length", len(pageData))
        if oggErr = audioTrack.WriteSample(media.Sample{Data: pageData, Duration: sampleDuration}); oggErr != nil {
            return oggErr
        }
    }
}

[breadface]

@shellfly I'm facing a similar problem. Were you able to figure this out?

[shellfly]

@breadface Unfortunately, I didn't get a valid way to do it. Currently, I'm doing webrtc through a third party software like LiveKit and Agora which provide official integration to OpenAI Realtime API. The integration code are all open source, the code might be helpful for the research of this issue, but I don't have time to dig into it yet.

[breadface]

Thanks for the quick turnaround @shellfly

I'll look into this approach.

[Sean-Der]

Sorry I didn't respond to this sooner @shellfly @breadface can you share a file that has problems?

My guess is non-20ms page sizes? I will fix and get the changes landed in Pion!

[breadface]

Thanks for your response @Sean-Der

My guess is non-20ms page sizes?

Most probable, but I do not know for sure.

received_ca5ade96-20a8-45f9-b47a-e78cd8d08387.ogg.zip

[shellfly]

@Sean-Der I got a similar answer about 20-ms page size on StackOverflow, but I didn't find a way to modify the page size in Go, so I didn't try it.

[tyohan]

From my experiences working with Text To Speech output, it will be better to receive the output in PCM format. With this you can resample it to sample rate 48Khz with stereo format, and stream it in 20ms pace. Without following this requirement, the audio won't play in the browser. If you receive the audio output in ogg format, most likely the sample rate is in 24Khz and mono format. To resample it, you will decode and encode it again.

I made a PR to add the example of TTS to WebRTC app for your reference.

cc @Sean-Der might need a review to merge. Thanks