Luasynth

Audio framework in Lua

Luasynth is a small audio framework, written in Lua, emphasizing clean, modular, declarative code. For a taste, read the source of its panning effect or delay. Both are very short.

The plan: allow sound generators (oscillator, noise generator, sampler…) and effects (gain, distortion, reverb, delay…) to be written, chained together, and embedded in a host program. It’s like a miniature version of the VST or LV2 plugin interfaces.

Why use Lua? Unlike other high-level languages, it’s extremely simple to embed and LuaJIT provides excellent performance for this type of thing.

Check out the code on GitHub. Luasynth is MIT-licensed and contributions are welcome.

Notes

The rest of this page is devoted to my thoughts while developing the project. For most people, this will be too much information, but I post it publicly in the name of learning and sharing.

Original plan (Feb 16th, 2013)

Adapt Truesynth to use Lua-written generators and effects.

Arrays (Feb 16th, 2013)

Based on the benchmarks on the LuaJIT FFI’s image example (see LuaJIT’s doc/ext_ffi.html), memory usage will probably be excessive using tables for sample arrays, and speed will be slow. In the example, the FFI gives a 35x improvement in memory usage and performs 20x faster.

What I may want to do is create an opaque object (provided by the host) that acts like a table, but provides a more efficient 32- (not 64-)bit float array. Send that to the Lua code.

In the JavaScript implementation, maybe use typed arrays or whatever the new speedy JS hotness is for that kind of thing. For the native implementations, use the LuaJIT FFI functions.

I’m not sure actually if the Lua code has to change to accommodate this. It seems not. In fact, as long as the Lua implementation uses functions that only processOneSample() at a time, obviously nothing will need to change because they never see arrays to begin with… but some effects will need internal arrays of their own, e.g. a delay line or reverb. So for that I think I need to provide an FFI- or JS-whatever-backed array object the Lua code can create.

Lua beats C in benchmark (Feb 17th, 2013)

time amp -dB -6 <test_input.f32 >/dev/null
Average of 6 trials:  real 0.540s

time luajit main.lua amp -gain -6 <test_input.f32 >/dev/null
Average of 6 trials:  real 0.509s

Wow. With no optimization or buffering, and loading every pair of samples individually into a table (which seems like it’d be inefficient and dumb), the LuaJIT version is faster than the C version!

And except for small rounding differences, the output files are identical. Redirecting the Lua version to lua_out.f32 and the C version to c_out.f32:

$ pmix 'cat lua_out.f32' 'amp -vol -1 <c_out.f32' | replaygain -fB32
+64.54 dB
0.00000006
$ rattodb .00000006
-144.436975

So the max amplitude difference between the two is less than FS-144dB. For all practical purposes it’s absolutely identical. Differences are due to the Lua version using doubles instead of 32-bit floats, so it’s potentially more accurate.

Ideas for next steps (Feb 18th, 2013)

Mono effects (Feb 18th, 2013)

If I had a special “mono” flag machines could set, then we could keep the whole chain in mono, e.g.

mono generator -> mono effect -> mono effect -> mono effect -> stereo effect

…converting it to stereo at the last possible moment and thus making the mono effects only do ½ as much work.

This seems way more efficient, but is it premature optimization? Probably. But keep this in mind for later. For now, all stereo all the time = simpler, so…

Stereo effects and testing (Feb 19th, 2013)

Today I implemented a panning effect. In the process, I revised the API a bit to support stereo effects. For now, if your unit defines a processOneSample() function, it’s assumed to be a stateless mono effect (like amp), and if it instead defines processSamplePair(), that gets called and passed a separate left and right value.

If you define neither of the two valid types of processing function, an error is thrown. But the error isn’t thrown as soon as it should be. Wrapping the unit with wrapMachineDefs works fine, as if nothing’s wrong. It’s only when you then call .new() that you get the error.

I caught this behavior when I wrote my first test with Busted, an Rspec-inspired Lua testing tool. My first impression is that adopting TDD, even partially, will be a huge boon to my coding.

However, I’m a little disappointed in Busted’s output so far. I’m calling assert.has_error with a specific error message:

assert.has_error(function() wrapDefs(noProcessingUnit).new() end,
                 "Unit `Bad Unit` has no processing function")

Here’s the output if an error isn’t raised at all:

Failure: spec/wrap_spec.lua @ 5
should require a processing function
spec/wrap_spec.lua:10: Expected error to be thrown.

Here’s the output if an error is raised, but the message doesn’t match:

Failure: spec/wrap_spec.lua @ 5
should require a processing function
spec/wrap_spec.lua:10: Expected error to be thrown.

In other words, it’s identical! I would like it to say “expected the error message to be ‘foo’, but error ‘bar’ was raised instead”. Very confusing.

Also, the test is wrapped inside a describe block, like this:

describe("Unit definition wrapper", function()
    it("should require a processing function", function()
        ...
    end)
end)

In the test failure output, where is the describe block? I want it to say “Unit definition wrapper” somewhere, so I know what it was that should (but doesn’t) require a processing function.

Option knobs (Feb 20th, 2013)

Today, in the process of writing a filter, I implemented option knobs. Unlike the existing numeric knobs, an option knob is a set of strings:

defs.knobs.filtType = {
    label    = 'Filter type',
    options  = {'Lowpass', 'Highpass', 'Bandpass', 'Notch'},
    onChange = function(state, newVal) updateCoefs(state) end
}

Is this efficient? Passing strings around? Not really, but we’ll see if it’s a problem. If so, maybe a set constructor could be used, like:

options = OptionSet {'Lowpass', 'Highpass'}

which automatically yields an internal representation of:

{Lowpass = 1, Highpass = 2}

or something. We’ll see. I think option knobs, being discrete, can be expected to change infrequently, and certainly not continuously.

I added a bunch of tests for knobs. In the process I discovered a couple more gotchas with Busted. The first is that for tests like assert.are.same and assert.are.equal, Busted wants the expected value first, the value being tested second. This was the opposite of how I had it, leading to confusing test failures. I guess the original Rspec avoids this problem with its arguably better syntax: expect(foo).toEqual(bar) is pretty clear on treating bar as the expected value.

The second gotcha was that any code sitting loose in a describe block is all executed before the tests are run. So if I have a test that mutates an object, that object needs to be created from scratch and modified as needed within the it block. This seems obvious in retrospect.

History arrays (Feb 20th, 2013)

For the past day or so I’ve been hacking on a filter effect, implementing four of the filter types from Robert Bristow-Johnson’s EQ cookbook. The routine to calculate the coefficients is done, but not the actual processing function.

A slightly simplified version of the cookbook’s formula is:

y[n] = c1*x[n] + c2*x[n-1] + c3*x[n-2]
               - c4*y[n-1] - c5*y[n-2]

where the c’s are coefficients. In this, x[] is the array of input values, y[] is the array of output values, x[n] is the input sample that just came in, and y[n] is the output sample that you are calculating.

The simplest way you could possibly implement this is to keep all old values indefinitely:

x={0,0} y={0,0} n=3
while moreInputWaiting() do
    x[n] = readSample()
    y[n] = c1*x[n] + c2*x[n-1] + c3*x[n-2]
                   - c4*y[n-1] - c5*y[n-2]
    writeSample(y[n])
    n=n+1
end

But if you’re processing an hour of audio, by the end you’re wasting more than a gigabyte of RAM on old samples that will never be accessed again because you only need the last two. Clearly this is silly.

My goal is to enable something very close to the above simplicity while only keeping as many values as needed, no more. My answer is history arrays. Meet HistArray:

x = HistArray.new(2)
y = HistArray.new(2)
n = 1
while moreInputWaiting() do
    x[n] = readSample()
    y[n] = c1*x[n] + c2*x[n-1] + c3*x[n-2]
                   - c4*y[n-1] - c5*y[n-2]
    writeSample(y[n])
    n=n+1
end

The loop is the same, but a HistArray has a metatable with custom __index and __newindex methods. You call HistArray.new with the history size, 2 in the above example.

Indexes you set in a HistArray must be monotonically increasing:

y[1] = 1  -- ok
y[2] = 1  -- ok
y[4] = 1  -- not necessarily recommended, but ok
y[4] = 0  -- ERROR: can't reset a value
y[3] = 0  -- ERROR: can't set 3 after 4

And the indexes you can get include the current one, and .histSize old ones:

y = HistArray.new(2)
print(y.histSize)  -- 2
y[1]=10 y[2]=20 y[3]=30 y[4]=40
print(y[4])  -- current value:      40
print(y[3])  -- one before current: 30
print(y[2])  -- two before current: 20
print(y[1])  -- ERROR: too old!

I have not implemented HistArray yet, but have written a battery of tests, in the spirit of behavior-driven development, illustrating how it should work.

I’m hopeful that this array will satisfy not only the filter use case above, but also work for delay lines, which may be much longer (imagine a 10-second delay line, with 441,000 sample pairs). Also, it should be amenable to optimizing with LuaJIT’s FFI when running on a native (non-JavaScript) host.

Filter effect (Feb 23rd, 2013)

I implemented history arrays on the 21st, and today I got the filter effect working — lowpass, highpass, bandpass and notch filters, to be specific. Here’s a benchmark, versus my old C synth version, along the lines of the amp benchmark.

The average after 6 trials of

time filter -type hp -cutoff 5000 <begin.f32 >/dev/null

was 0.539sec “user”. The average after 6 trials of

time luajit main.lua filter -filtType Highpass -center 5000 <begin.f32 >/dev/null

was 1.085sec “user”. As before, the two outputs are identical except for negligible rounding differences, proving I implemented the filter correctly (or, the same as before, anyway). The peak difference between the two outputs is -132dB.

So this time, Lua did not beat C. It’s taking 2.01x as long to process. But again, this is unoptimized, and I feel it may still be possible to attain more speed. In addition, the Lua code is using double-width arithmetic (64-bit floats) as opposed to 32-bit floats in the C code, which I would assume puts Lua at a disadvantage. Besides, for a dynamic language, a factor of 2 difference from C is pretty darn good, and is still 155x realtime on my MacBook Air.

(The input, as before, is the song “51st State” by New Model Army, ripped from CD, encoded into Ogg Vorbis at ~160kbps, decoded and converted to 32-bit float. No particular reason for using this song. I just had an .ogg file of it handy. So far, for both the effects I’ve tested, the actual content of the input audio should have basically no effect on performance, whether it’s white noise, silence, music or anything in between.)

Soft saturator (Feb 25th, 2013)

Today I implemented a delay effect and a soft saturator.

The soft saturator mimics the transfer function of a tube amp. Given a hardness value g, 0 <= g < 1:

f(x) = { x,                                  when x <= g }
       { g + (x-g) / (1 + ((x-g)/(1-g))**2), when x > g  }

There are two complications. One, if x is negative, take -f(-x) instead. Two, to normalize the amplitude, compute r * f(x/r) where r = 2 + (1/g).

The formula is taken from, originally, a code snippet on MusicDSP.org. I’ve been using the resulting effect, implemented in C, to mix and master my tracks for years.

But I implemented it incorrectly. I left out the x > 1 case from the original snippet, so instead of hard-clipping at 1, the amplitude gradually decays toward g as the input value gets higher. This seems to be my own (accidental) innovation.

I left it in the new version, even if I’m not convinced why or how (or if) it improves the sound.

For my next trick, I want to do something a lot less mathy than fiddling with formulas. Something eminently practical, like this bullet point from Feb 18th:

Lua.js and modules (Feb 26th, 2013)

I ran into a frustrating roadblock trying to get a tiny example (not Luasynth) working in Lua.js. For background, I prefer to define modules in Lua using the table method:

-- stuff.lua
local M = {}
function M.getAnswer()
    return 42
end
return M

-- example.lua
local stuff = require "stuff"
print("The answer, my friend, is " .. stuff.getAnswer())

Lua.js instead seems designed to support the Lua 5.1 module() function, which is considered harmful. The above code fails in Lua.js, partly because require doesn’t return a value, partly because it can’t find the module stuff, and partly because even after I got it to find stuff through a horrible hack:

<script type="text/javascript" src="stuff.js"></script>
<script type="text/javascript">
  lua_createmodule(lua_script, "stuff", []);
</script>
<script type="text/javascript" src="example.js"></script>

(This lua_createmodule call has to be wedged in here because the compiled versions stuff.js and example.js both define a value called lua_script that holds their contents.)

Even then, the field getAnswer inside stuff is a nil value. I’m not sure what’s going on here. The Lua.js author’s comment in a related issue hints that there may be a way to get this to work, but offers no suggestions.

Init behavior (Feb 26th, 2013)

At a momentary impasse with Lua.js, I went back to the effect wrapper in Luasynth, improving the init behavior.

Let’s say you have an effect with two different knobs. For example, from softsat.lua:

defs.knobs.range = {
    min     = 0.01,
    max     = 2.0,
    default = 1.0,
    label   = 'Range',
    onChange = function(state, newRange)
        state.normalizedRange = newRange * 2 / (state.public.hardness + 1)
    end
}

defs.knobs.hardness = {
    min     = 0.0,
    max     = 0.99,
    default = 0.5,
    label   = 'Hardness',
    onChange = function(state, newHardness)
        state.normalizedRange = state.public.range * 2 / (newHardness + 1)
    end
}

The callback for hardness uses range to compute something, and the callback for range uses hardness to compute something.

Until before, the way this worked was that in a new unit, knobs would get set to their defaults one at a time, calling the onChange callback. So if range got set before hardness, then range’s onChange callback would get called with state.public.hardness == nil, making the computation fail. I had a workaround which returned early from the callback if state.public.hardness was falsy, to avoid this problem. Similar workarounds littered the filter effect’s code.

The new behavior is to set all knobs to their default values first, through the private interface, without calling callbacks; then call each of the onChange callbacks in turn. Thus, when a new unit is created, every onChange callback gets called, and every other knob is sanely set to its default by the time the callback runs. So I can (and did) replace this:

onChange = function(state, newHardness)
    -- XXX abort if not all knobs are set yet
    if not state.public.range then return end
    
    state.normalizedRange = state.public.range * 2 / (newHardness + 1)
end

with this:

onChange = function(state, newHardness)
    state.normalizedRange = state.public.range * 2 / (newHardness + 1)
end

I added three new init- and callback-related tests to the spec.