Using Bungee with FMOD (Unity3D)

[HTML-Earth]

I'm trying to integrate Bungee with FMOD inside a Unity project. I've managed to make a C# wrapper class and I am able to modify FMOD's audio buffer with it, but I must be doing something wrong, because the result sounds kind of like a robot filter, even with pitch and speed set to 1.

This is the code that modifies the buffer:

[AOT.MonoPInvokeCallback(typeof(FMOD.DSP_READ_CALLBACK))]
static FMOD.RESULT CaptureDSPReadCallback(ref FMOD.DSP_STATE dsp_state, IntPtr inbuffer, IntPtr outbuffer, uint length, int inchannels, ref int outchannels)
{
    FMOD.DSP_STATE_FUNCTIONS functions = (FMOD.DSP_STATE_FUNCTIONS)Marshal.PtrToStructure(dsp_state.functions, typeof(FMOD.DSP_STATE_FUNCTIONS));

    IntPtr userData;
    functions.getuserdata(ref dsp_state, out userData);

    var objHandle = GCHandle.FromIntPtr(userData);
    var instance = objHandle.Target as DSPCapture;

    // Copy the incoming buffer for processing
    var lengthElements = (int)length * inchannels;
    Marshal.Copy(inbuffer, instance.dataBuffer, 0, lengthElements);
    
    if (bungeeWrapper != null)
    {
        var request = new Bungee_Request
        {
            position = 0,
            speed = instance.voiceRate,
            pitch = instance.voicePitch,
            reset = false
        };
        
        var concatenatedOutput = new List<float>();
        
        for (var done = false; !done;)
        {
            //_bungeeWrapper.Preroll(request);
            
            var inputChunk = bungeeWrapper.SpecifyGrain(request);
            
            bungeeWrapper.AnalyseGrain(instance.dataBuffer, new IntPtr());
            
            var outputChunk = new Bungee_OutputChunk();
            bungeeWrapper.SynthesiseGrain(ref outputChunk);
            
            var outputData = new float[outputChunk.frameCount];
            Marshal.Copy(outputChunk.data, outputData, 0, outputChunk.frameCount);
            
            concatenatedOutput.AddRange(outputData);
            if (concatenatedOutput.Count >= length)
                done = true;
            
            bungeeWrapper.Next(request);
        }
        
        concatenatedOutput.CopyTo(instance.dataBuffer);
    }
    
    // Copy to the outbuffer so we can hear the effect
    Marshal.Copy(instance.dataBuffer, 0, outbuffer, lengthElements);

    return FMOD.RESULT.OK;
}

AnalyseGrain takes a channelStride parameter, but I'm working with a mono channel group, so I assume the value doesn't matter here.
I don't know what the log2SynthesisHopOverride value does, but I've kept it at 6 when instantiating the wrapper. Changing it seems to make it worse.

[HTML-Earth]

Here's a video that shows what it sounds like at the moment.
The original buffer is on top and the processed version is below.

Screen.Recording.2024-11-29.133659.mp4

[kupix]

I've never used C# but will try to help.

How are you using this function? Are you calling this function only once per audio clip that you need processing? Or are you calling it repeatedly as you progress through an audio source?

Either way, the most obvious issue is that inputChunk is never used. It's how the stretcher tells you which audio input range it needs to process the current grain. You need to look at the start and end markers in inputChunk and provide only that range of audio for analysis. This is done by helper class Processor in the command line example here.

[HTML-Earth]

Thanks, I'll try using the input chunk after the weekend and see if it works.

The function is called repeatedly by FMOD with 1024 samples from the audio buffer. It works as a live DSP effect.

[HTML-Earth]

In my case the inputChunk is giving ranges like -256, 256 which I can't provide, but I see that the Processor ignores the negative values like this:

if (inputChunk.begin < 0)
{
	inputChunk.begin -= inputChunk.begin;
	inputChunk.end -= inputChunk.begin;
}

• Should I also just ignore the negative values, or offset the whole thing to fit?
• If I keep calling specify, analyse, synthesise and next, can I be sure that I'll end up with a matching amount of output samples?

I should probably also clarify that I'm only trying to change the pitch, and not the speed.

[kupix]

Yes you could offset in the same way as Processor does. Probably it's more accurate though to zero pad. So, in your example, you'd provide a buffer whose first 256 values are 0.f and the next 256 values are from the beginning of your input audio.

But it may make sense to use the Push adapter for your use case. This will buffer streaming input for you and provide overlapping InputChunks to the stretcher. In your app, you'd set pushFrameCount = 1024. You'll have to buffer the output though if you need 1024 frames output each time. As is, OutputChunk can have any number of frames: and you'll need to iterate until you have 1024 frames to output. There's an example of the Push adapter in use here: https://github.com/kupix/bungee/blob/b0189ddded8210711780cc4f91fd2b710408f63e/cmd/main.cpp#L32

Let me know how you get on!

[HTML-Earth]

I've tried porting the InputBuffer and following the Push example. Getting all of the different buffers (FMOD+Bungee+my own debug visualisation) to work together was tricky, but it's almost working now.

My only issue right now is that each 1024 sample output from Bungee seems to fade in and out, which makes the audio sound noisy/stuttery.

Here is the input audio compared with the combined output from FMOD

This is what the waveforms from Bungee look like (each row is one concatenatedOutput)

In my examples the pitch and speed were set to 1, but changing the pitch does seem to work despite the other issue.

For reference, this is my current implementation (the part that interfaces with Bungee)

static List<float> Process(ref Queue<float> inputSamples, int pushFrameCount, float speed, float pitch)
{
    var concatenatedOutput = new List<float>();
    var request = new Bungee_Request
    {
        position = 0,
        speed = speed,
        pitch = pitch,
        reset = false
    };
    
    var inputChunk = bungeeWrapper.SpecifyGrain(ref request);
    
    var inputBuffer = new Push_InputBuffer(bungeeWrapper.MaxInputFrameCount(),1);
    inputBuffer.Grain(inputChunk);
    
    var done = false;
    for (int position = 0; !done; position += pushFrameCount)
    {
        // Here we loop over segments of input audio, each with pushFrameCount audio frames.
        
        // First get pushFrameCount frames of audio from the input
        //processor.getInputAudio(inputBuffer.InputData(), inputBuffer.Stride(), position, pushFrameCount);
        for (int i = 0; i < pushFrameCount; i++)
        {
            if (inputSamples.Count > 0)
                inputBuffer.InputData()[i] = inputSamples.Dequeue();
            else break;
        }
        
        // The following function and loop delivers pushFrameCount to Bungee
        // Zero or more output audio chunks will be emitted and we concatenate these.
        inputBuffer.Deliver(pushFrameCount);
        while (inputBuffer.InputFrameCountRequired() <= 0)
        {
            bungeeWrapper.AnalyseGrain(inputBuffer.OutputData(), new IntPtr());
            
            var outputChunk = new Bungee_OutputChunk();
            bungeeWrapper.SynthesiseGrain(ref outputChunk);
            
            bungeeWrapper.Next(ref request);
			
            var outputData = new float[outputChunk.frameCount];
            Marshal.Copy(outputChunk.data, outputData, 0, outputChunk.frameCount);
            
            inputChunk = bungeeWrapper.SpecifyGrain(ref request);
            inputBuffer.Grain(inputChunk);
            
            concatenatedOutput.AddRange(outputData);
            if (concatenatedOutput.Count >= pushFrameCount)
            {
                done = true;
                break;
            }
        }
    }
    return concatenatedOutput;
}

[kupix]

In your example, do you call Process multiple times with segments of audio that will be joined?

[HTML-Earth]

Yes, each time I get a 1024 segment from fmod, I call process with it and use the result for output.

I was creating a new request with position 0 each time. I have now changed it to reuse the same request, and have also reworked the loops.

static List<float> Process(ref Queue<float> inputSamples, int pushFrameCount, ref Push_InputBuffer inputBuffer, ref Bungee_Request request)
{
    var concatenatedOutput = new List<float>();
    
    // First get pushFrameCount frames of audio from the input
    for (int i = 0; i < pushFrameCount; i++)
    {
        if (inputSamples.Count > 0)
            inputBuffer.InputData()[i] = inputSamples.Dequeue();
        else break;
    }
    
    // The following function and loop delivers pushFrameCount to Bungee
    // Zero or more output audio chunks will be emitted and we concatenate these.
    inputBuffer.Deliver(pushFrameCount);
    while (concatenatedOutput.Count < 1024)
    {
        bungeeWrapper.AnalyseGrain(inputBuffer.OutputData(), new IntPtr());
        
        var outputChunk = new Bungee_OutputChunk();
        bungeeWrapper.SynthesiseGrain(ref outputChunk);
        
        bungeeWrapper.Next(ref request);
        var outputData = new float[outputChunk.frameCount];
        Marshal.Copy(outputChunk.data, outputData, 0, outputChunk.frameCount);
        
        var inputChunk = bungeeWrapper.SpecifyGrain(ref request);
        inputBuffer.Grain(inputChunk);
        
        concatenatedOutput.AddRange(outputData);
    }
    return concatenatedOutput;
}

It seems that the segments I get now are shifted by 256. So the output starts with samples 256-512 (but fading in), followed by the second half of the segment (normal) and then 256 samples of silence to fill the rest.

[HTML-Earth]

It seemed like part of the output was normal. So I tried to work around it by keeping an extra buffer shifted by 512, calling Process on it with a separate InputBuffer+Request and then splicing half of each result together, but it seems like the output that I thought was normal still has some sort of noise on it.

Screen.Recording.2024-12-11.140826.mp4

Here's what the waveforms look like. Blue is the original input and yellow is half of each output concatenated together (the left half is the end of the shifted input and the right half is the end of the non-shifted output)

[kupix]

I think that you need to make sure that the input to Bungee is as it expects. To do this, I would create an audio file, or hack the audio input in your app, so that frame 0 has value 0.0, frame 1 has value 1.0, frame 2 has value 2.0 and so on. Effectively an audio signal f(t) = fs * t where fs is your sample rate and t is time in seconds.

Now, when specifyGrain asks for a particular range of samples, you can look in your debugger at what you provide back to analyseGrain. If, say, inputChunk.begin == 2000 and inputChunk.end == 3000, then you would expect the chunk of 'audio' that you pass to analyseGrain to have sample values ramping from 2000 to 2999.

If you do this, make sure to turn off any connected speakers or headphones: this ramp signal is only for debugging.

You should find that the same ramp is propagated to Bungee's output (give or take some arithmetic errors). This method will help ensure that the right pieces of audio go to the right places.

[HTML-Earth]

Since the last post, I managed to get it to work and was satisfied with how it sounded given that it was Bungee Basic.
We then decided to switch to Bungee Pro and after reworking my C# wrapper, I got the latest version of Pro to work today.

But now I suspect there's another issue with my implementation, since it sounds very robotic compared to the demo page:

Screen.Recording.2025-01-13.162012.mp4

The first speech has a rate of 0.75 and the second has 0.6
Both have the pitch set to 1.12
The dll returns "Pro" and "v1.2.89"

Any ideas of what could cause this?

[HTML-Earth]

It might help to know that in order to use the push method, I tried to recreate the InputBuffer code from Push.h in C#, and it's possible that I've implemented something incorrectly there:

public class Push_InputBuffer
{
    float[] vector;
    int maxInputFrameCount;
    int begin = 0;
    int end = -1;
    int endRequired = 0;

    public Push_InputBuffer(int maxInputFrameCount, int channelCount)
    {
        vector = new float[maxInputFrameCount * channelCount];
        this.maxInputFrameCount = maxInputFrameCount;
    }
    
    public void Grain(InputChunk inputChunk) // readonly inputChunk
    {
        bool firstCall = end - begin < 0; // readonly firstCall
        if (firstCall)
        {
            begin = inputChunk.begin;
            end = 0;
        }

        int overlap = end - inputChunk.begin; // readonly overlap
        if (overlap <= 0)
        {
            begin = end = inputChunk.begin;
        }
        else
        {
            int offset = inputChunk.begin - begin; // readonly offset

            // loop over channels, move lapped segment to start of buffer
            for (int x = 0; x < vector.Length; x += Stride())
                Array.ConstrainedCopy(vector, x+offset, vector, x, overlap);
                //std::move(
                //    &vector[x + offset],
                //    &vector[x + offset + overlap],
                //    &vector[x]);

            begin = inputChunk.begin;
        }
        endRequired = inputChunk.end;

        if (InputFrameCountRequired() > InputFrameCountMax())
            throw new Exception("InputFrameCountRequired() > InputFrameCountMax()");
        if (InputFrameCountMax() < 0)
            throw new Exception("InputFrameCountMax() < 0");
        //assert(InputFrameCountRequired() <= InputFrameCountMax());
        //assert(InputFrameCountMax() >= 0);
    }

    public void Deliver(int frameCount)
    {
        if (frameCount < 0)
            throw new Exception("frameCount < 0");
        if (frameCount > InputFrameCountMax())
            throw new Exception($"frameCount ({frameCount}) > InputFrameCountMax() ({InputFrameCountMax()})");
        //assert(frameCount >= 0);
        //assert(frameCount <= InputFrameCountMax());
        end += frameCount;
    }

    public ref float[] InputData()
    {
        return ref vector; //&vector[end - begin];
    }

    public int InputFrameCountRequired()
    {
        return endRequired - end;
    }

    public int InputFrameCountMax()
    {
        return Stride() - (end - begin);
    }

    public float[] OutputData()
    {
        return (float[])vector.Clone();
    }

    public int Stride()
    {
        return maxInputFrameCount;
    }
}

And this is how it is being used:

static List<float> Process(ref List<float> inputSamples, int pushFrameCount, ref Push_InputBuffer inputBuffer, ref Request request, ref InputChunk inputChunk)
{
    var concatenatedOutput = new List<float>();
    
    var maxPasses = pushFrameCount / 32;
    var pass = 0;
    
    while (concatenatedOutput.Count < pushFrameCount)
    {
        if (pass > maxPasses)
            break;
        
        var frameCountMax = inputBuffer.InputFrameCountMax();
        var inputSize = inputBuffer.InputData().Length;
        
        var frameCount = 0;
        for (var i = inputChunk.begin; i < inputChunk.end; i++, frameCount++)
        {
            if (frameCount >= inputSize)
                break;
            if (i >= inputSamples.Count)
                break;
            
            if (i < 0)
                inputBuffer.InputData()[frameCount] = 0;
            else
                inputBuffer.InputData()[frameCount] = inputSamples[i];
        }
        
        if (frameCountMax >= pushFrameCount)
            inputBuffer.Deliver(pushFrameCount);
        
        var output = inputBuffer.OutputData();
        
        bungeeWrapper.AnalyseGrain(output, new IntPtr());
        
        var outputChunk = new OutputChunk();
        bungeeWrapper.SynthesiseGrain(ref outputChunk);
        
        bungeeWrapper.Next(ref request);
        
        var outputData = new float[outputChunk.frameCount];
        Marshal.Copy(outputChunk.data, outputData, 0, outputChunk.frameCount);
        
        inputChunk = bungeeWrapper.SpecifyGrain(ref request);
        inputBuffer.Grain(inputChunk);
        
        concatenatedOutput.AddRange(outputData);
        pass++;
    }
    
    return concatenatedOutput;
}

[HTML-Earth]

Here are some logs from the new bungee version. Playing the same audio, first at 1.0 and then 0.5 speed

bungee_log.txt