Tutorial 29. Buffering and storing audio

Recording audio into MEAP.

In addition to passing audio through MEAP, we have the capability of storing buffers of audio for playback or analysis. In this tutorial, we will tackle the example of recording a short sample of audio and playing it back using mSample. Of course, once you have this audio sample in memory, you could also analyze it or manipulate it however you want.

If we want to store audio within MEAP, we need to give it a place in memory to live, so we are going to create an empty buffer of ints. Specifically, we are going to use the datatype int16_t which specifies that these integers will be 16-bit integers (which is the format that our audio codec chip uses). The size of the array we create will determine how much audio we can record. Our sampling rate is 32768 Hz, so if we created a buffer of 32768 int16_t's, we would be able to record one second of audio. In our example we'll create and array of 327680 int16_t's, which corresponds to a maximum of ten seconds of audio.

The ESP32's internal memory is relatively limited, capable of only storing up to about four seconds of audio. Luckily, our ESP32 chip also comes with an SPI RAM chip with an additional 8 megabytes of RAM (enough to store several minutes of audio). Because this memory is in an external chip, it is not available to the program at compile time. This means that we can't define regular arrays in the external RAM chip; we need to dynamically allocate memory to the RAM chip at runtime.

In the global variable section we'll add the following code:

#define BUFFER_LENGTH 327680 
int16_t *input_buffer; // instead of something like int16_t input_buffer[BUFFER_LENGTH]

mSample<BUFFER_LENGTH, AUDIO_RATE, int16_t> my_sample(input_buffer);

float default_freq;

int record_index = 0;
int sample_length = 0;
bool recording = false;

First, we create a pointer to an array of 16-bit integers. This pointer just says that we are going to create an array later, it doesn't contain an array just yet.

Next we create our mSample object. Unfortunately, when we pass input_buffer to my_sample's constructor, we are passing an empty reference. The actual input_buffer array hasn't been created yet so we'll need to point my_sample to it again in setup() later.

Next, we'll create a few variables to help us record our sample. Then way our system will work is as follows:

1. When pad 0 is pressed, start recording a buffer.
2. When pad 0 is released, or when four seconds have elapsed, stop recording and store the length of the recorded buffer.
3. When pad 1 is pressed, play back the buffer.

The variable recording just keeps track of if we are currently recording or not, record_index keeps track of how far we are through the buffer while recording and sample_length is used to store how long the sample is once we are finished recording it.

In setup() we need to allocate our input buffer array and point my_sample to it.

  input_buffer = (int16_t *)calloc(BUFFER_LENGTH, sizeof(int16_t)); 
  my_sample.setTable(input_buffer);                                 

The first line dynamically allocates our input_buffer array at runtime. The calloc allocates a certain amount of memory and ties it to the pointer input_buffer. In this case, we want BUFFER_LENGTH samples in our array, and each is of size sizeof(int16_t). The ESP32 will automatically store large buffers of dynamically allocated memory on the SPI RAM chip rather than internal memory (where this buffer would be too large to fit).

Now that we've created this buffer, we can access it just like a normal array using square brackets.

The second line points my_sample to input_buffer. With normal samples we could just do this in the global section through the constructor, but remember that my_sample didn't actually exist until the previous line so we need to explicitly do this in setup.

In updateTouch(), we use pads 0 and 1 to trigger recording and playback respectively.

case 0:
  if (pressed) {  // Pad 0 pressed
    Serial.println("t0 pressed ");
    record_index = 0;
    recording = true;
  } else {  // Pad 0 released
    Serial.println("t0 released");
    sample_length = record_index;
    my_sample.setEnd(sample_length);
    recording = false;
  }
  break;
case 1:
  if (pressed) {  // Pad 1 pressed
    Serial.println("t1 pressed");
    my_sample.start();
  } else {  // Pad 1 released
    Serial.println("t1 released");
  }
  break;

When pad 0 is pressed, we set record_index to the start of the buffer and enable recording. When it is released, we stop recording, store the length of the sample we recorded, and pass that information to our mSample object using the setEnd function.

When pad 1 is pressed, we simply trigger the mSample object to play our sample.

updateAudio is where the brunt of the code is for this tutorial.

if(recording){
  input_buffer[record_index++] = meap_input_frame[0];
  if(record_index >= BUFFER_LENGTH){
    sample_length = BUFFER_LENGTH;
    my_sample.setEnd(sample_length);
    recording = false;
  }
}
int64_t out_sample = my_sample.next();
return StereoOutput::fromNBit(16, (out_sample * meap.volume_val)>>12, (out_sample * meap.volume_val)>>12);

There are two separate processes going on here: recording and then playback.

If we are recording, we do the following:

1. Grab a sample from our line input using meap_input_frame[0] and place it into our input buffer. We place it at index record_index and then increment that index. Note that we are only grabbing meap_input_frame[0], which is the left channel of the line input, and ignoring the right channel. If we wanted to record stereo, we would need two input buffers.
2. Check if we have filled up the input buffer: record_index >= BUFFER_LENGTH. If we have, we stop recording and store the length of the sample as if we had released pad 0.

For playback, we just grab a sample from our mSample object and send it to the output. Remember that mSample by default only plays through the sample once, so we will only hear audio after we press pad 1 to trigger the sample.

---

FULL CODE HERE

---