Beijaflor's blog RSS feed

cmakefmt

Thu, 18 Jan 2024 00:00:00 GMT

I have written an initial prototype of cmakefmt, a CMake files auto-formatter.

The source code is available on GitHub at (https://github.com/yamadapc/cmakefmt).

Installing from source

Using cargo:

cargo install cmakefmt

Examples

These are interactive examples using a WASM version of the tool.

Long commands

Long lines are broken-up into separate lines.

set_property( TARGET test PROPERTY INTERFACE_LINK_LIBRARIES "${DIR}/lib64/libcublas_static.a")

Key-value pair behaviour

Arguments with all upper-case letters are grouped together with the next non-uppercase argument. So if there is space, upper-case arguments will be printed as key-value pairs as follows.

juce_add_gui_app( StoryBook PRODUCT_NAME "BF StoryBook" COMPANY_NAME "Pedro Tacla Yamada" BUNDLE_ID "com.beijaflor.juce_storybook")

Indentation

Nested blocks are indented by 2 spaces.

function(foo) bar( x y z ) endfunction()

Conditions

Condition nodes are grouped together.

if(true AND (true AND false OR NOT true) OR true OR (true AND NOT (false OR true)) OR (true AND false OR NOT true) OR true OR (true AND NOT (false OR true))) message(STATUS ok) endif()

Implementation

cmakefmt is written in Rust using nom and pretty combinator libraries. It parses the CMake document onto an AST using nom parser combinators and then serialises it into a string using pretty, which are combinators in the same style of “A pretty-printer”.

Playground

Please try it out here or check-out the GitHub repository.

Drawing knobs

Sat, 08 Oct 2022 00:00:00 GMT

This is an exploration on knobs UIs. It uses JavaScript to render knobs on your web browser and shows code, however it is intended to be a more general look at knobs. Therefore, some concerns specific to the web platform may be ignored.

Knobs are a common control on music and audio applications, such as virtual instruments, DAW (digital audio workstations) or audio effects plugins.

This is done in resemblance to real world musical objects, which have knobs and also because they are a very compact way of representing many different parameters’ state in a way that can be glanced at quickly.

Drawing knobs

Let there be a canvas:

const canvas = document.createElement("canvas");
document.body.appendChild(canvas);

const size = 80;
const canvasSize = size * scaleFactor;
const center = canvasSize / 2;
const radius = (canvasSize - 20) / 2;

canvas.width = canvasSize;
canvas.height = canvasSize;
canvas.style = `width: ${size}px; height: ${size}px`;

const ctx = canvas.getContext("2d");

These simple knob controls above are just arcs.

function renderKnobArc({ color, value }) {
  const revolution = 2 * Math.PI;
  const totalSweepAngle = 0.75 * revolution;

  const sweepAngle = value * totalSweepAngle;

  const startAngle = 0.375 * revolution;
  const endAngle = startAngle + sweepAngle;

  ctx.beginPath();
  ctx.strokeStyle = color;
  ctx.lineWidth = 10 * scaleFactor;
  ctx.arc(center, center, radius, startAngle, endAngle);
  ctx.stroke();
}

Reacting to input

We must react to mouse input. The most natural would be to react to the angle of the mouse relative to the knob:

For this, we’ll use atan2:

const revolution = Math.PI * 2.0;
const startAngle = 0.375 * revolution;
const sweepAngle = 0.75 * revolution;

let angle =
  Math.atan2(
    mouse.y - center.y,
    mouse.x - center.x,
    // Offset the angle so 0 is at startAngle
  ) - startAngle;

// This fixes the angle so it goes from 0 to 2.pi radius
if (angle < 0) {
  angle = 2 * Math.PI + angle;
}

// This truncates values under the knob so they either snap
// to start or end. In reality we should cancel the gesture
// so that there're no jumps.
if (angle > sweepAngle) {
  angle = 0;
}

Above, we also rotate our reference points and clamp the results.

By “clamping”, I mean what to do when the drag/mouse is outside of the expected track. In the code above, the angle resets to zero. However that’s akward in practice and it’s best to ignore drags that are outside of the track.

You can see how clamping to 0 reacts to input here:

If we fix that and also add drag tracking and calculate a value that corresponds to the rotation:

We now have an usable knob:

Knobs that are sliders

One alternative UI for changing numeric parameters, would be to use sliders:

These have the advantage of being more generally available and intuitive for mouse users, but will take more space in one axis (horizontal / vertical).

And interesting interaction pattern is a “slider knob”. That is a knob which is interacted with as a slider:

In this case, the knob is simply a visual representation of the control. Clicking it reveals a secondary interaction layer, which contains a slider. The slider is positioned using the value and mouse position to show the user what their next gestures will cause.

This has the advantage that the knob may be really small but still usable:

The choice between horizontal and vertical backing sliders can be done on a case by case basis, but most of the time the vertical slider would make the most sense due to being analogue to mixer sliders.

However, I find this interaction is a bit akward. There’s a level of indirection between the visual representation and how to manipulate its value, which I think makes it harder to learn.

Expanding knobs

I’d like to explore expanding knobs GUI, where knobs would expand with interactions on another document.

That is it

Pretty short, thanks for reading.

You should find Swift, Rust (Skia), Flutter and JS implementations for knobs on https://github.com/yamadapc/augmented-audio/

Wisual Logger

Mon, 20 Jun 2022 00:00:00 GMT

Logging is an important strategy for building observable software systems. The practices and set-up for application logging often consist of a program printing some of its operations onto a file, console or network such that later on they can be inspected by a human or automated tools.

This is particularly useful when the software mis-behaves and requires debugging that would otherwise be hard to conduct on an opaque system.

Often on back-end systems, logs will end-up, temporarily, on some tool that can archive, search or visualise the output (Splunk, ELK).

Logs should be complimentary to metrics or analytics and will also be useful during the software development process in order to quickly provide some feedback of internal behaviour.

When developing back-end web services, logging is essential in order to allow the software to be operated by teams at scale. This is partially due to such services running on closed environments, which can not be easily probed otherwise.

I believe this is in contrast with GUI applications and, in particular, browser based GUIs. While analytics, metrics and some form of automated application monitoring are wide-spread amongst front-end application, logging from the front-end does not seem to be fully exploited.

@wisual/logger

@wisual/logger on GitHub

I wrote a very simple logger for a side-project which tries to bring hierarchy into browser logs.

Usage is as follows:

import { LoggerFactory } from "@wisual/logger";

const logger = LoggerFactory.getLogger("MyLogger");

logger.info("Hello world", { data: 10 });

This will print:

2022-06-20T10:48:47.470Z [app>MyLogger] (info) Hello world [Object >]

Where ‘object’ is a table in Chrome inspector or a JSON string on Node.js.

Hierarchy

Loggers can have contextual data attached to them:

const usersServiceLogger = logger.child("UsersService", {
  dbBackend: "psql",
  flushTimeout: 1000,
});
const usersCacheLogger = usersServiceLogger.child("UsersCache", {
  cacheBackend: "redis",
});

The context will be merged with data of each log message.

React integration

There are hooks provided for react integration:

import React from "react";
import { useLogger } from "@wisual/logger";

function MyComponent({ userId, data }) {
  // Creating our logger and properly setting its context
  const logger = useLogger("MyComponent", { userId });

  // Will log on render with proper context / names
  logger.info("rendering");

  // This will `useEffect` to log & log everytime the argument object's values change
  logger.onInfo("Data has changed", { data });

  // This is wrapping the children with this logger's context provider
  // calling `useLogger` on a child will use `MyComponent` logger as the parent.
  return logger.wrap(<div />);
}

Logger sinks

Sinks are implementors of the LoggerSink interface.

Tree sinks are provided:

PrettyConsoleSink - Pretty print messages on Node.js
PrettyBrowserSink - Pretty print messages on browser consoles
DelegatingSink - Send messages to multiple sinks

There’s no filtering currently implemented.

Configuring the sink

Use LoggerFactory.setSink.

Why named and hierarchical loggers?

Large applications will benefit from being able to filter out logs based on their place in the hierarchy. For React applications the UI hierarchy can be indicative of ownership or package boundaries.

What else?

That is it. This is a very simple package for now and should grow with the needs of my project.

All the best

Parsing and serializing MIDI files in Rust

Tue, 03 May 2022 00:00:00 GMT

This is a super simple post about augmented-midi, an unpublished crate for MIDI event and file parsing and serialization.

The library:

Parses standard MIDI files and messages
Serializes MIDI messages
Performs no allocations on parse (results are either owned or input references)

It is part of the augmented-audio repository and is used by audio-processor-standalone to feed MIDI input into generated processor CLIs.

You implement the AudioProcessor and MIDIEventHandler traits and wrap it with audio-processor-standalone.

Now you have a CLI that can process audio and MIDI, both in real-time and offline.

When rendering offline, your CLI will be able to be process MIDI input:

cd ./crates/apps/synth
cargo run \
    --release -- \
    --input-file=../../../input-files/synthetizer-loop.mp3 \
    --output-file=synth.wav \
    --midi-input-file=../../augmented/data/augmented-midi/bach_846.mid

This outputs synth.wav (⚠️ level are off so this is loud):

I wrote about what I’m trying with these abstractions in more detail on “Generic AudioProcessors in Rust”.

Usage example

Using the TryFrom trait implementation:

let input_buffer = [0x9_8, 0x3C, 0x44];
let message = MIDIMessage::<&[u8]>::try_from(input_buffer).unwrap();
assert_eq!(
   midi_message,
   MIDIMessage::NoteOn(MIDIMessageNote {
      channel: 8,
      note: 60,
      velocity: 68
   })
);

Using the nom functions, which are more verbose:

use augmented_midi::{MIDIMessage, MIDIMessageNote, MIDIParseResult, parse_midi_event, ParserState};

// Initialize parser state. This is here to support rolling status on MIDI files.
// Not relevant for single events.
let mut state = ParserState::default();

// We'll parse this &[u8] buffer
let input_buffer = [0x9_8, 0x3C, 0x44];

// We parse a message borrowing from the input buffer. We could use `MIDIMessage<Vec<u8>>`
// to allocate owned messages.
//
// This will determine how variable size messages (SysEx) will be allocated.
//
// Parsing is otherwise only using the stack.
let parse_result: MIDIParseResult<MIDIMessage<&[u8]>> =
    parse_midi_event(&input_buffer, &mut state);
let (_remaining_input, midi_message) = parse_result.unwrap();

assert_eq!(midi_message, MIDIMessage::NoteOn(MIDIMessageNote { channel: 8, note: 60, velocity: 68 }));
match midi_message {
    MIDIMessage::NoteOn(MIDIMessageNote { channel, note, velocity }) => {
        println!("\o/");
    },
    // MIDIMessage::... variants contain all the messages
    _ => {}
}

Parser combinators: `nom`

The parser is implemented using nom parser combinators.

nom not required for MIDI, which is a simple protocol, but made it easier to go with a good structure from the start.

We have some type aliases:

pub type Input<'a> = &'a [u8];
pub type MIDIParseResult<'a, Output> = IResult<Input<'a>, Output>;

Out inputs are slices of bytes and our outputs are nom results, these are Result types where success is a tuple of the remaining input and the parsed type and failure is an error, that will reference the input position for the failure.

Here’s the function to parse the MIDI file format from MIDI files’ header chunk as an example:

fn parse_file_format(input: Input) -> MIDIParseResult<MIDIFileFormat> {
    let (input, format) = be_u16(input)?;
    let format = match format {
        0 => Ok(MIDIFileFormat::Single),
        1 => Ok(MIDIFileFormat::Simultaneous),
        2 => Ok(MIDIFileFormat::Sequential),
        _ => Ok(MIDIFileFormat::Unknown),
    }?;
    Ok((input, format))
}

Our function takes Input, which is just &[u8] and returns Result<(&[u8], MIDIFileFormat), Error>.

It takes a stream of bytes, parses a MIDIFileFormat enum and returns it with the remaining stream of bytes. Otherwise, it returns an error.

MIDI events/files are very simple and compact, but there’s a bit of bit fiddling. For example, most events have 3 bytes, with the first byte being both the type of the event and sometimes what channel it refers to. The first 4 bits of this status byte are the status and the last 4 bits might contain the channel information.

The crate is useful to avoid repeating this on application code. It uses bit-masks and returns nice ‘rusty’ types other code can pattern match over.

The entry-points to the library are the parse_midi_event and parse_midi_file functions and the MIDIMessage or MIDIFile types.

The functions and types are generic: MIDIMessage<Buffer: Borrow<[u8]>>.

Most messages have static sizes and will be stack-allocated. For example, note-on messages always have 3 bytes while program change messages always have 2 bytes. The generic argument is used to determine how the variable size messages are handled (just SysEx).

If MIDIMessage<&'a [u8]> is parsed, the sys-ex messages will point to the input buffer. On the other hand, if MIDIMessage<Vec<u8>> is parsed, the sys-ex messages will be owned and the buffers copied into vecs.

The same goes for parts of the MIDI file spec which require strings such as chunks (this will only be used for unknown chunks found). Though these strings will always only have 4 bytes so maybe this could be removed.

Some nice Rust, here is SysEx parsing:

pub fn parse_sysex_event<'a, Buffer: Borrow<[u8]> + From<Input<'a>>>( // [1]
    input: Input<'a>,
) -> MIDIParseResult<'a, MIDISysExEvent<Buffer>> {                    // [2]
    let (input, _) = alt((tag([0xF7]), tag([0xF0])))(input)?;
    let (input, bytes) = take_till(|b| b == 0xF7)(input)?;
    let (input, _) = take(1u8)(input)?;
    Ok((
        input,
        MIDISysExEvent {
            message: bytes.into(),                                    // [3]
        },
    ))
}

The type is generic over a Buffer type argument, which will be the output storage a. we require this type to be borrow-able as a [u8] slice, so that we can read it back in a standard way as a slice of bytes b. we require that this type be convertible from our input type, which is &[u8]
We return the input/output tuple result as before, lifetime annotations are carried with the input type
Conversion from the input slice into just a reference to it (no conversion) or into a Vec<u8> is handled by .into() calling the Into trait

Serialization: cookie_factory

cookie-factory is used for serializing MIDI events. Currently, only MIDI events can be serialized, rather than the full MIDI standard files specification.

This maps closely to the nom parser combinators. I remember something about isomorphic parser/pretty-printer code, but this is a very tiny serializer.

Performance

augmented-midi is very fast, these are benchmarks..

Generally:

Performance isn’t an issue for this
No allocation is 1 order of magnitude faster than yes allocation 🤷‍
My computer can parse billions of MIDI messages a second; so it doesn’t matter

Conclusion

That is all! Thank you for reading and give me feedback on Twitter @yamadapc.

Continuous Looper

Tue, 12 Apr 2022 00:00:00 GMT

This is an April update on Augmented Audio Libraries, my hobby libraries for audio programming in Rust (check previous updates).

It is a special one because today I have released an initial version of Continuous Looper on the macOS App Store (iOS version coming ~~soon~~ next).

There’s a lot to do and fix, but I think publishing this post (and the app) to get some feedback will help me.

You can download the app now here

In this post, I’ll discuss how I have written this app, the approaches I have been trying and challenges faced.

Continuous Looper

Continuous Looper is a fully open-source 8-track live-looper and performance sampler, written in Rust/Swift which features:

8-track looping/sequencing
Step sequencing live after recording
2 quantization modes
Scenes and parameter locks
Click and twist MIDI mapping to any parameter & the record-button
2 LFOs + 1 ADSR envelope per track
Automatic slicing based on transient detection
Basic pitch shifting (potentially dated phase-vocoder)
Variable start/end/speed

Since the scope got large, there are many rough edges! But it has worked well.

The app/GUI source-code is available under the AGPLv3 license at augmented-audio/crates/apps/looper/Sequencer and the engine source-code is available under the MIT license at augmented-audio.

Continuous Looper is inspired by the great Drambo iOS modular synth and by Octatrack.

Demo video

Here is a tiny demo video.

(Note: this is an iPhone video and audio is very low)

Requirements
Architecture
Rust and audio
Conclusion

Some requirements for the app

It’s good to mention a couple of requirements that guide the decisions I have been taking and greatly increase complexity.

Real-time audio

The app must render audio in real-time. At fixed rate intervals, our software’s audio callback will be called with a buffer of numbers representing an audio signal. Our software will have a capped amount of time (roughly between 1ms-10ms) to finish. It must finish in this time or else playback will audibly glitch.

The code that runs in the audio-thread:

should avoid variable size work
should avoid locks
should avoid allocation/de-allocation
should not perform IO

Some good resources on this subject:

Interactive GUI

Second, we want the GUI to be highly interactive. The GUI should render visualisations of the operations taking place in the audio side whenever possible.

(Note: this is a screen recording and much louder than the previous video)

The following data needs to constantly update on the UI:

Global play-head position
- The position of the track in beats at the center-top
Sequencer position
- The current active step flashes in the sequencer view
Looper play-head position
- The position of each looper within its recording, represented by the green line over the rendered audio wave and by the green fill over the track’s button
Last MIDI messages
- This isn’t shown, but a monitor of previous events is displayed when the MIDI button on the top-right is clicked

Looper quantization

This is the main reason I wanted to write a looper.

Quantization in live-loopers refers to automatic tempo correction of the presses to the record/stop/play buttons. This is to aid musicians to record loops in sync with some tempo.

For example, there may be a drum track playing or another looped track. Whenever a new loop is recorded it should have a length that makes sense in the current tempo and time-signature (subject to configuration).

Most software loopers I’ve tried only correct the musician being early, but don’t correct them being late. I have hardware looper pedals which do both.

If there’s a tempo track playing and a musician presses to start a recording at beat 3.7, the looper will wait until the 1st beat of the next bar. This is correcting for when the musician is early.

However, if the musician is late, we’d also like to correct the tempo, by assuming that the button had already been pressed (in the past) and then continuing to operate.

These two quantization strategies are implemented and selectable on Continuous Looper and there is some more around implementation later on.

Architecture

IMPORTANT: I am not a professional audio developer

Please give me feedback! :)

Continuous Looper’s engine is built using the Rust crates I have been building on augmented-audio. It benefits from several crates in the rust ecosystem such as petgraph, cpal, basedrop, actix and more (ringbuf, etc. the list of dependencies is very long).

The GUI is built using Swift, AppKit (and UIKit at some point) and SwiftUI.

Originally a simpler looper GUI was built using iced:

I will likely still use iced to ship a simplified VST.

Swift/AppKit/UIKit/SwiftUI are more mature and allow iOS support, so this version of the app uses them.

Containers or components

Here’s a “Container diagram” (I’m likely misusing the C4 model in my context):

User
- Some musician that wants to use the app
Continuous macOS/iOS App
- GUI, which users will interact with
LooperEngine
- Audio engine, which communicates with the app through a C API
MIDI controllers
System Audio/MIDI APIs

Next, I will go over details, starting with high-level notes, then talking about the GUI and moving on to details of specifics of the rust implementation.

System Audio/MIDI APIs - API wrappers

For 5, the system APIs are wrapped by cpal and midir for audio and MIDI support, respectively.

Both of these are wrapped by the higher-level AudioProcessor trait abstraction:

These higher-level abstractions allow the looper engine (3) to work drop-in in a hosted scenario, such as within a VST. They currently use traits for AudioBuffer and MidiMessageLike so that buffers and messages could be compatible with both rust and VST C API representations.

The ‘application code’ uses:

Then, the crates in crates/augmented/application provide standalone and hosted helpers to run our processors as an online CLI, offline file processor or VST plugin.

The gist is audio-buffers and MIDI messages are abstract and so are audio processing nodes. However, I now have the impression converting into a common type before doing audio processing could have been a better option than making all implementations support abstract types.

LooperEngine/Swift integration - Rust & Swift integration

Most of the Continuous Looper integration issues are in the bridge between Continuous macOS/iOS App 2 and LooperEngine 3.

Communication is done through a C bridge. C API bindings are generated with cbindgen. Universal binaries are built with cargo-lipo.

The bridge is a good candidate for review, because it uses too many strategies at the same time.

Data is exchanged between Rust/Swift in 4 ways:

Polling: GUI poll/queries for state from the audio engine
- for example: play-head state or shared read-only references to audio buffers
Push from GUI: GUI dispatches changes to some object
- for example: a float/int/binary/enum parameter
Dispatch from GUI: GUI dispatches a command
- for example: “start recording” when the record button is clicked
Dispatch from engine: LooperEngine dispatches an event
- for example: MIDI/IO events happened

At the moment, most data is copied in both Swift and Rust sides. These two must sync and things would be a lot simpler if this wasn’t the case.

I believe this will start to be a bigger problem when persistence is added, which allow certain more complex structures to be changed from the Rust side. These changes will need to be pushed/queried somehow.

Additionally, there’s shared memory in the case of audio-buffers, where Swift just references the same audio-buffer used in the looper. This is fine thanks to reference counting, some safer wrappers around the C API and because the looper buffer will be pre-allocated to a maximum capacity at the start.

Polling for changes

For some data, the GUI poll the audio-thread every 1/60 seconds. Though the data could be pushed from the rust side instead I think generally the strategy of polling audio-thread state makes sense.

The play-head for example (our position within the song or loop measured in samples/beats/seconds) updates at least every audio callback (1-10ms). Since this may update at much higher frequency than our GUI and represents time ticking, we can poll for it at the UI refresh rate we want.

Some data can be polled at lower frequencies, such as the CPU indicator on the top-right (there’s a section about the CPU meter later).

One downside of SwiftUI in this context is that since our data will change on every frame, I imagine a lot of work performed by the library to optimise performance by avoiding invalidating a view is wasted.

Push from GUI

SwiftUI provides an ObservableObject class, which allows views to update automatically when some properties change. This is similar to mobx and really nice to use.

The bridging code benefits by listening to changes to these objects and pushing them into the rust engine.

Pushing observable object changes out of the GUI into Rust could look something like this:

class StartParameter: ObservableObject {
    @Published var value: Double = 0.0
}

struct StartParameterView: View {
    // this view will update whenever StartParameter::value changes
    @ObservedObject var startParameter: StartParameter
    var body: View { /**/ }
}

// Set-up a subscription to the value field and push it into rust
// on changes
func syncStart(parameter: StartParameter) -> AnyCancellable {
    return parameter.$value.sink(receiveValue: { value in
        looper_engine__set_start_parameter(value)
    })
}

However, this is not super manageable for a larger nº of parameters, so the solution used is moving towards a slightly more generic binding.

This also stops working once there’s communication in the other direction (Rust -> Swift). The users may map MIDI controllers to parameters, so at any point a MIDI event could be changing a parameter without the GUI knowing.

We want the GUI to update when a MIDI controller’s knob is mapped to a parameter, and it’s moved.

At the moment this is addressed with polling the Rust looper state from the GUI. There’s an EntityId, which all GUI controls and parameters have and all entities* are pushed into and polled from the rust side.

*all entities that are MIDI/LFO/Step/Scene mappable (knobs and some buttons)

SwiftUI Performance

I’ve found a number of bottlenecks related to the GUI and this set-up (but I think it is just generally really hard to build this sort of GUI and be performant whatever the technology).

The iced GUI, though simpler and ‘inefficient’ (iced must re-layout and draw everything if anything wants to update), consumed a lot less CPU. I’m afraid my app might be too CPU intensive for some lower-end computers or iPads - this might go into another post.

In particular:

Setting a @Published property will trigger updates even if the value doesn’t change, this is a massive CPU sink, so we can’t do it
- Instead, we read the property before setting it
Reading the @Published property in an ObservableObject is super expensive compared to reading a normal property
- Instead, I am using @FastPublished, a custom property wrapper that copies the value to a normal field, so it’s faster to read
Updating SwiftUI views at a high rate is a major performance bottleneck, so the views that will update frequently (step sequencer buttons and knobs) use AppKit instead of SwiftUI

I have profiled my SwiftUI issues and found some accessibility code + forced layout are the culprit. Setting a fixed size for the window improved things partially, but using AppKit was the better escape hatch.

There is still a lot of room to improve the GUI performance.

Callbacks from rust to swift

Regarding how to call into Swift from Rust, Nick Wilcox has written great documentation on how to achieve this here:

https://www.nickwilcox.com/blog/recipe_swift_rust_callback/

That is pretty much the set-up used. You can check the source code here and here.

Swift frameworks, packages and apps

The following is a diagram of some views / components in the Swift GUI side.

Just some tiny things worth mentioning.

Store: We hold all state under a ‘store’ object which makes it a bit easier to pass it around or mock it out.

SwiftSequencerEngine: There’s no code calling into native Rust directly in the GUI itself. There are multiple packages in the XCode project:

the GUI exists in a cross-platform Swift package SequencerUI
the rust library is linked against a separate framework

The reason for this is to be able to support multiple platforms (iOS/macOS). Also, having a simple SequencerUI swift package that doesn’t link to anything fancy allows normal XCode tooling such as previews or Swift specific unit-tests to work normally and faster.

SequencerUI declares a SequencerEngine protocol/interface, which the SequencerEngine.framework frameworks for macOS/iOS (which are exactly the same source, but different build configurations) implement in SequencerEngineImpl.

As mentioned, most data is copied from the engine into the Store objects, except for audio buffers which are shared.

For audio buffers, another protocol is declared in SequencerUI and the engine implements this protocol for some C struct pointer wrappers, which are essentially wrappers on top of reference counted AudioBuffer rust objects.

SequencerUI and SequencerEngine.framework are then consumed by the application projects themselves for iOS/macOS.

I have not released the iOS app, because there are currently issues with certain IO channel configurations; but the app runs (with Rust!) fine on iOS, and it is going to be great.

Rust components

All the audio processing code I’m writing is following a “shared handle” pattern:

Every audio processor is a struct with a “process buffer” method
This method takes a mutable reference to the processor
The audio-thread owns the processor
Any state that must be changed from other threads is done via a handle
- A handle is a reference counted shared thread-safe struct (using atomics) that both the processor and GUI can read/write from

In code, this looks like this:

let processor = MultiTrackLooper::default();
let processor_handle: Shared<MultiTrackLooperHandle> =
        processor.handle().clone();

// send the processor to the audio-thread
let _audio_handles = audio_processor_start_with_midi(
    processor,
    audio_garbage_collector::handle(),
);

// the handle can change things and dispatch events
processor_handle
    .set_source_parameter(LooperId(0), SourceParameter::Speed, 2.0);
// open the GUI with the handle now

This introduces indirection to access anything on the handle (which is all processor state a lot of the time). I haven’t measured the performance impact in a larger example such as the looper.

That’s how the looper is structured: there is a LooperEngine struct created by the Swift GUI that starts a couple of different background workers, among which is the audio_processor_standalone instance. audio_processor_standalone is the audio/MIDI threads.

The audio/MIDI threads own the processor and reference shared state the GUI can read and update (through special APIs).

MIDI events might arrive in a different thread and are pushed via a lock-free queue to the audio-thread. I think we could handle MIDI events out of the audio-thread, but that’s how things are done right now.

MultiTrackLooper and MultiTrackLooperHandle are where all the magic happens, which leads us to…

Rust and audio

Multi-track looper audio graph

MultiTrackLooper processing has the following topology:

Each track has, in series, a single-track looper, pitch-shifter, ADSR envelope, effects (which are not in the published version of the app, as they have some issues) and gain. This graph of processors is built using audio-processor-graph which is a partial audio graph implementation in the repository.

The looper is by far the largest processor in this chain, since a lot of the other nodes are very simple. Currently, the “effects” node is a sub-graph.

Looper states

The looper can be in different states:

Empty
Recording
Playing
Paused

Three additional states are used to represent overdubbing and the cases where playing/recording are scheduled.

Each of these needs special treatment of both the actions (what transition to trigger on interaction) and more importantly on how they render/handle the audio callback.

Always recording buffer

We want to support “snap closest” behaviour:

And, in fact, we also want to support this when starting to record/play.

In this case, the looper will receive a start recording command after it was supposed to have started.

For this, we need to have an always recording buffer (the ScratchPad). The scratch-pad will be a circular buffer for the last few seconds which will always be recording.

Then, when recording is triggered, we can either:

store what our “write cursor” position within the scratch-pad is
or copy the last few seconds into another buffer

Currently, Continuous Looper has a pretty small 10s limit on loop length, so it uses the scratch-pad to record and copies to another buffer only when the recording is ended.

Quantization implementation

Keeping track of time: Play-head

I didn’t know of Vlad Voina’s “Loopers and bloopers” talk when I implemented this, but gives some ideas of things I could be fixing next.

In order to quantize we first need to be able to keep track of the play-head position, as represented in previous tempo diagrams.

The play-head is our position within the song. This should be a value we can convert into/from samples, beats and seconds and that represents the timestamp within the song of the play-head (the playback).

In augmented-audio, I’ve implemented a basic hosted PlayHead data structure in augmented_playhead.

The gist is that we keep track of the current sample and then calculate our position in seconds/beats based on the tempo and sample-rate:

let elapsed_samples = get_elapsed_samples_from_somewhere_pass_it_in();
let elapsed_secs = (1.0 / sample_rate) * (num_samples as f64);
let beats_per_second = tempo_in_bpm as f64 / 60.0;

let elapsed_beats = beats_per_second * elapsed_secs;

All units can be converted to one another. One important thing to do is to use double precision, because there will be serious inaccuracies if f32 is used after a certain song length.

Vlad Voina’s talk linked above mentions the size of the elapsed_samples cursor, as just incrementing it will eventually overflow. Generally it’s not an issue for this use-case.

In order to tempo-sync with a VST host, the Continuous Looper engine is using a TimeInfoProvider abstraction, which wraps both a ‘stand-alone’ play-head and a ‘hosted’ implementation. The source code is here.

The idea is we will tick and control the tempo when running in an app, but when running inside a VST we will call back into the host via the VST API to get the current play-head. I will extract this out of the looper application for re-use later.

Implementing quantization

Once we know the play-head position, quantization is some calculations, which depend on the time-signature, our position in beats and (for snap closest) a mistake correction threshold.

// This code is subject to the MIT LICENSE terms in https://github.com/yamadapc/augmented-audio/blob/master/crates/apps/looper/looper-processor/src/audio/loop_quantization.rs#L89-L114

// This returns a positive nº to the next 1 beat
fn snap_next_beat(quantization_beats: usize, position_beats: f32) -> f32 {
    let f_beats = quantization_beats as f32;
    (position_beats / f_beats).ceil() * f_beats
}

// This returns either a positive or negative nº to the closest 1 beat
fn snap_closest_beat(
    quantization_beats: usize,
    tempo: f32,
    threshold_ms: f32,
    position_beats: f32,
) -> f32 {
    let beats_per_ms = tempo / 60_000.0;
    let threshold_beats = beats_per_ms * threshold_ms;

    let f_beats = quantization_beats as f32;
    let ratio = position_beats / f_beats;
    let lower = ratio.floor() * f_beats;
    let upper = ratio.ceil() * f_beats;

    #[allow(clippy::float_equality_without_abs)]
    if ((lower - position_beats).abs() - threshold_beats) < f32::EPSILON {
        lower
    } else {
        upper
    }
}

Since we have our scratch-pad, the looper can handle negative quantization offsets. I imagine similar quantization logic would work for other applications.

Testing audio

In order to test audio processors I am trying a couple of approaches.

Again, I’m not an audio developer (and there’s a lot to fix in the repo!), but maybe it can be useful.

Normal unit-testing with mock buffers

For looper logic which just records and reproduces samples, we can unit-test using small made-up (1,2,3) buffers and check that the output makes sense after certain operations.

This is regular unit-testing as on any domain.

Property based testing

On some small pieces, I’ve experimented with tests such as “feed a sine-wave into this DSP code and check that the output has the same RMS as the input”.

I imagine with more analysis properties this would be more meaningful.

I have drafted some analysis code on audio-processor-analysis, but there isn’t a lot of usage in tests, and I’d need to experiment more.

Visualization

This isn’t a proper test, but very often, it’s helpful to visualise what the signal is doing, so I used plotters and sometimes piet to generate charts.

The analysis crate above is a good example of doing this, but there are several in the repository.

audio-processor-testing-helpers exposes a function to make it dead simple to plot a vector of floats and have a look at the signal:

// this macro returns a relative path to the CARGO_MANIFEST_DIR
let pth = relative_path!("src/lib.rs");
// will create `src/lib.rs--chart_name.png
draw_vec_chart(pth, "chart_name", my_vector);

Snapshot testing

Since we get free CLIs from our abstraction code, the repository has a couple of snapshot tests which run in CI and are documented in SNAPSHOT_TESTING.md.

This is just rendering some audio through each effect and checking hashes match what had been committed.

Maybe comparing properties or visualisations here would be more meaningful and less brittle.

However, this approach and the others increase my level of confidence I’m not breaking things when making changes.

Testing allocations

All audio-thread code is trying to avoid (de)allocations, which increases complexity quite a lot. This is very hard to avoid in the Rust ecosystem with many 3rd party libraries that provide lock-free or immutable data-structures allocating on reads.

In order to unit-test that allocations aren’t being made the assert_no_alloc crate is of great help. It replaces the global allocator so that sections of the source can disable allocations (causing the program to print a warning or crash).

On unit-tests this can be used to test sections are allocation free.

Sequencer implementation

My sequencer implementation is based on detecting when the play-head has changed from one beat to another. A “step-tracker” struct keeps track of the last position in beats. When a new position is received, it checks if there was a trigger.

let prev_position = 0.9; // <- prev beat is 0
let curr_position = 1.1; // <- curr beat is 1
// trigger beat 1.0

The active steps, along with parameter lock information are kept on a shared vector pointer.

When the GUI wants to add a step, it’ll copy the vector, make changes and then atomically swap the old pointer with the new. This is done using basedrop::SharedCell.

When both threads are done with the old data it’ll be released on a background thread.

I wanted to use immutable data-structures for this case, but there should be negligible gain at this point because the sequence can only be 16 steps.

Parameter storage - LFOs, parameter locks, sequencing

Parameters, LFO mapping and parameter locks work in a similar way.

There’s a “ParametersMap” struct, which is an array of all parameters and their values, combined with a HashMap that locates a typed (enum) parameter ID into its position.

These positions and array are pre-calculated because we know all possible parameters ahead of time. This likely won’t work moving forward.

For each step and scene there’s a similar map. The steps only hold parameters that have been locked.

On each audio callback we apply the parameter locks, scenes and so on.

We have a scratch buffer of parameters, onto which we load ‘user parameters’ the user has set through the GUI. Next, we override these with the scenes (performing linear interpolation according to the slider position). Third, we apply parameter locks if there’s an active step with them. Forth, we apply LFO modulation.

Finally, all parameters are “flushed” from the scratch parameters buffer to their corresponding handles (currently this is just the looper handle). Once there are effects and more mappable processors they would be flushed to each of these.

This scratch stage is necessary because scenes, locks and modulation introduce a mismatch between what the user has set on the GUI (the ‘user value’) and what the value should be at this current point in time.

I think this has room for improvement because:

The current approach uses a HashMap for a pre-determined set of items
- We could be using just an index and a Vec
We have to manually flush parameters from this storage into the handles, which centralises a lot of mapping code in the main MultiTrackLooper processor

Conclusion

Thank you for reading my post!

I really hope you got something out of it.

I will continue to develop Continuous Looper and libraries in augmented-audio with goal #1 being to learn and have fun. (augmented-audio/crates/apps/looper/Sequencer)

If you have feedback, please reach out, and I will be super happy to chat.

All the best, @yamadapc.

Reddit post

Augmented Audio - Generic AudioProcessors in Rust

Thu, 24 Feb 2022 00:00:00 GMT

This is a February update on Augmented Audio Libraries, my hobby libraries for audio programming in Rust.

Note: Most of this is unpublished to crates.io.

This is 3rd in a series:

Overview

In this post I’ll try to write about some ideas around generating CLIs and GUIs.

I’ll share the AudioProcessor and AudioProcessorHandle traits and some of audio_processor_standalone usage in conjunction with these.

AudioProcessor: Implementing a BitCrusher

We’d like to implement a bit-crusher, so some sample-rate reduction distortion effect.

I’m using an algorithm I found on the internet, which works by implementing sample-and-hold of the input signal.

The algorithm is:

// src/lib.rs
fn process(
  // Given sample_rate as samples per second
  sample_rate: f32,
  // Given bit_rate as output samples per second
  bit_rate: f32,
  // Given buffer as the mono input/output buffer of samples
  buffer: &mut [f32]
) {
  let buffer_size = buffer.len();
  // We'll samples every `sample_rate / bit_rate` indexes
  let step_size = (sample_rate / bit_rate) as usize;

  let mut sample_index = 0;

  // For each index in the buffer
  while sample_index < buffer_size {
    // Take the first sample
    let first_sample = *buffer[sample_index];
    let limit_index = (sample_index + step_size).min(buffer_size);

    // Hold it for `sample_rate / bit_rate` steps
    while sample_index < limit_index {
      *sample = first_sample;
      sample_index += 1;
    }
  }
}

Cool. We have our processor which is super simple, but it has a few minor issues:

It only supports mono buffers
Multichannel buffers would need to not be interleaved (where each multichannel frame happens one after the other with [l1, r1, l2, r2, l3, r3])
We can’t hear it
We can’t play with the parameters on a GUI
We can’t load it in a DAW and play with it

We’ll address each of these.

Implementing `AudioProcessor` - Generic buffers

The AudioProcessor trait aims to provide a way to implement audio-processors against a general interface.

There are two methods to implement:

prepare - Here we’ll receive audio-settings like sample-rate and prepare our processor for playback
process - Here we’ll receive an audio buffer and process it

First let’s make this into a struct and declare some imports we’ll need:

// src/lib.rs
use audio_processor_traits::{AudioProcessor, AudioProcessorSettings, AudioBuffer};

pub struct BitCrusherProcessor {
  sample_rate: f32,
  bit_rate: f32,
}

impl Default for BitCrusherProcessor {
  fn default() -> Self {
    Self { sample_rate: 44100.0, bit_rate: 11025.0 }
  }
}

We’ll store parameters in the struct, so they can be changed externally (this will change slightly later). Also, we need imports for AudioProcessor, AudioProcessorSettings and AudioBuffer.

We can now declare the AudioProcessor impl:

impl AudioProcessor for BitCrusherProcessor {
  type SampleType = f32;

  fn prepare(&mut self, settings: AudioProcessorSettings) {
    self.sample_rate = settings.sample_rate();
  }

  fn process<BufferType: AudioBuffer<SampleType = Self::SampleType>(
    &mut self,
    data: &mut BufferType
  ) {
    let buffer_size = data.num_samples();
    let step_size = (self.sample_rate / self.bit_rate) as usize;

    let mut sample_index = 0;

    while sample_index < buffer_size {
      let first_index = sample_index;
      let limit_index = (sample_index + step_size).min(buffer_size);

      while sample_index < limit_index {
        // **************************************************
        // **************************************************
        // **************************************************
        // HERE - This is the only real change
        // **************************************************
        for channel_index in 0..data.num_channels() {
          let out = *data.get(channel_index, first_index);
          data.set(channel_index, sample_index, out);
        }
        // **************************************************
        // **************************************************
        // **************************************************

        sample_index += 1;
      }
    }
  }
}

In our prepare method we use the sample_rate provided, which will match whatever output device we’ll use this processor with.

On process we receive an AudioBuffer. The AudioBuffer is an abstract multichannel buffer, and it may be several types.

The audio-processor-traits crate provides implementations for:

Interleaved samples
Non-interleaved
Helpers for VST interop

Additionally, the SampleType = f32 is declared as part of the processor specification. If we wanted we could declare both f32 and f64 processors with generics.

The AudioBuffer provides a handful of methods, here we use get, set, num_samples and num_channels which should have clear names for what they do.

I’ve found Rust can do better optimisations with iterators, so AudioBuffer provides frames and frames_mut which is an iterator over the multichannel frames and will work for different layouts at 0 cost in the cases where one should benefit from these. They are the preferred method of processing, but in this case things are simpler with get / set as we want to index into the buffer.

Processing code has only changed in an inner for loop; we’ll do the sample-and-hold for each channel in the input buffer.

Listening to our `AudioProcessor` - Generic CLI

At this point, we can listen to our code. We’ll create an examples directory and add audio_processor_standalone:

⚠️ Don’t run this without reading on ⚠️

// examples/bitcrusher.rs
use bitcrusher::BitCrusherProcessor;

fn main() {
  let processor = BitCrusherProcessor::default();
  audio_processor_standalone::audio_processor_main(processor);
}

What audio_processor_standalone::audio_processor_main will do is run a little CLI program with our processor.

If I run this: cargo run --example bitcrusher -- --help, I’ll get the following:

audio-processor-standalone

USAGE:
    bitcrusher [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -i, --input-file <INPUT_PATH>      An input audio file to process
    -o, --output-file <OUTPUT_PATH>    If specified, will render offline into this file (WAV)

If I run it without options, it’ll run “online” processing on the system’s default audio input/output (this may cause feedback issues)

We also get --input-file and --output-file which we can use to process audio files offline.

These are two independent methods of running the processor:

CPAL online rendering
- We can run the processor on default input/output & plug in a guitar if we want
CLI offline rendering
- We can generate output from input files and run analysis or tests over it

Listening to our `AudioProcessor` - Generic VST

If you wanted to just plug this into a DAW now you can do it.

Add another examples/bitcrusher_vst.rs:

// examples/bitcrusher_vst.rs

audio_processor_standalone::standalone_vst!(BitCrusherProcessor);

examples/bitcrusher_vst should be configured to compile as:

[[example]]
name = "bitcrusher_vst"
crate-type = ["cdylib"]

In the augmented-audio repository (this tooling isn’t installable otherwise) we can generate a proper VST bundle for macOS (the Info.plist and .vst directory structure) by marking this crate as having a VST example:

[package.metadata.augmented]
vst_examples = ["bitcrusher_vst"]
private = true

And then running dev.sh:

./scripts/dev.sh build --crate ./path-to-bitcrusher

This will output a VST under target.

Generic GUI

Now we want a GUI for the processor, so we can twist a knob and change our parameter. In order to do this, we’ll need to introduce an “audio processor handle”.

The audio processor handle is a pattern I’ve introduced which consists of the following idea:

Our audio processor struct is a mutable stateful object owned by the audio-thread
If we want a GUI thread, the GUI thread will hold a “handle” for the processor
The handle will be a shared reference counted struct that uses atomics for thread-safety

Extract parameters into a handle

So first we’ll extract the parameters from processor onto handle:

// src/lib.rs

use audio_garbage_collector::{Shared, make_shared};
use audio_processor_traits::{AtomicF32, AudioBuffer, AudioProcessor, AudioProcessorSettings};

pub struct BitCrusherHandle {
  sample_rate: AtomicF32,
  bit_rate: AtomicF32,
}

impl Default for BitCrusherHandle {
  fn default() -> Self {
    Self {
      sample_rate: AtomicF32::from(44100.0),
      bit_rate: AtomicF32::from(11025.00)
    }
  }
}

pub struct BitCrusherProcessor {
  handle: Shared<BitCrusherHandle>,
}

impl Default for BitCrusherProcessor {
  fn default() -> Self {
    Self { handle: make_shared(BitCrusherHandle::default()) }
  }
}

There are 3 parts I want to clarity in this snippet:

audio_processor_traits::AtomicF32

AtomicF32 and AtomicF64 are conveniently exported from audio_processor_traits. On release builds these compile to the same code as using normal floats, and they’re an unfortunate need to make rust compiler happy with changing floats in multiple threads.

Shared

Shared is re-exported from basedrop.

basedrop provides a couple of smart pointers that will not de-allocate on the current thread. The idea here is that if we drop BitCrusherProcessor on the audio-thread (in the case of a dynamic processor graph) we’d not want any de-allocations to happen on the audio-thread.

basedrop will instead push the de-allocation onto a queue.

audio_garbage_collector is a wrapper on top of basedrop which provides a standard background thread for running the collection.

make_shared

make_shared will create a smart Shared pointer using the default global GC thread. So audio_garbage_collector will start a global GC thread & this Shared is associated with it. With raw basedrop we’d have to set this up manually on different places.

Implement a generic handle

I won’t go over the changes required to the process / prepare methods as these changes are mechanical.

In order to generate a GUI we need to have a way of introspecting into the available parameters at runtime.

This will be done by implementing AudioProcessorHandle. The way to do this is to create a newtype around our smart-pointer:

// src/generic_handle.rs
use audio_garbage_collector::Shared;
use audio_processor_traits::parameters::{
  AudioProcessorHandle, FloatType, ParameterSpec, ParameterType, ParameterValue,
};

use super::BitCrusherHandle;

pub struct BitCrusherHandleRef(Shared<BitCrusherHandle>);

impl BitCrusherHandleRef {
  pub fn new(inner: Shared<BitCrusherHandle>) -> Self {
    BitCrusherHandleRef(inner)
  }
}

impl AudioProcessorHandle for BitCrusherHandleRef {
  fn parameter_count(&self) -> usize {
    1
  }

  fn get_parameter_spec(&self, _index: usize) -> ParameterSpec {
    ParameterSpec::new(
      "Bit rate".into(),
      ParameterType::Float(FloatType {
        range: (100.0, self.0.sample_rate()),
        step: None,
      }),
    )
  }

  fn get_parameter(&self, _index: usize) -> Option<ParameterValue> {
    Some(ParameterValue::Float {
      value: self.0.bit_rate(),
    })
  }

  fn set_parameter(&self, _index: usize, request: ParameterValue) {
    if let ParameterValue::Float { value } = request {
      self.0.set_bit_rate(value);
    }
  }
}

This is some basic boilerplate that can be generated at some point.

Once we have this, by convention, we’ll declare a generic_handle method on our processor:

// src/lib.rs

impl BitCrusherProcessor {
  pub fn generic_handle(&self) -> impl AudioProcessorHandle {
    BitCrusherHandleRef::new(self.handle.clone())
  }
}

And now we can add a GUI example

// examples/bitcrusher_gui.rs
fn main() {
  let handle: AudioProcessorHandleRef = Arc::new(processor.generic_handle());

  // This actually starts the audio-thread and audio
  let _audio_handles = audio_processor_standalone::audio_processor_start(processor);

  // This opens the GUI
  audio_processor_standalone::gui::open(handle);
}

The VST can also change:

use bitcrusher::BitCrusherProcessor;
audio_processor_standalone::generic_standalone_vst!(BitCrusherProcessor);

With all of this we should get the following on the screen and processing the default input/output device audio:

⚠️ This runs online, control your volume & so on, there may be mistakes above ⚠️

That’s it

For the proper bitcrusher source-code see crates/augmented/audio/audio-processor-bitcrusher

Custom GUI

Custom GUI can be done easily as well and is done for other processors in the repository.

See the looper for an example

Its GUI looks like this:

All the best

Simple Metronome

Tue, 11 Jan 2022 00:00:00 GMT

On this post, I’ll discuss some work on the Augmented Audio Libraries and particularly go over a “metronome” app written using Rust and Flutter.

Goals primer

Before anything else:

These are hobby projects
I’m not a professional audio software developer
- take information with a grain of salt
- any feedback is great

Augmented Audio Libraries

While I would like to go over a metronome, it’s built around the Augmented Audio Libraries. They are a set of building blocks for audio development in Rust, building on top of great tooling from the RustAudio GitHub organization, such as cpal and vst-rs.

The initial driver for them was the Test Plugin Host, a VST host for development with support for hot-reload of in-development plugins. You can read about it here: “Test Plugin Host: An early VST plugin host to aid quick prototyping of audio plugins written in Rust”.

I’ve done some explorations with web and native GUI which I walk over above and was now trying out Flutter for the audio application space.

It’s still missing a lot of real-world testing and features, but there are quite a few utilities which could deserve posts of their own.

For example:

Audio Processors
Synthesizer building blocks: Envelope, Filters & Oscillators
Data structures: Lock-free queue, MIDI, play-head

Most of the repository is MIT licensed, but hasn’t been published into crates.io yet as it’s alpha quality. There are quite a few interesting discussions to tackle in terms of how to build GUIs or how to best represent audio processors (and audio buffers) as abstract objects in Rust.

The Metronome

Simple Metronome is the app above and should be published soon. You can find the entire source code at the metronome subdirectory in augmented audio.

A metronome is a practice aid for musicians. To practice timing, a musician will set a certain BPM on the metronome and start it. It should then produce some kind of sound on every beat the musician can use as a reference. In Simple Metronome this sound is a synthesized beep.

The app is written using Flutter and Rust. The GUI and business logic (state/persistence) are written in Dart and drive the Rust audio back-end.

The Rust side of things exposes methods using flutter_rust_bridge, a high-level codegen solution for Dart/Rust interop. A static library is compiled from the Rust side, exposing functions to dart. The library is linked onto the macOS and iOS apps (and other platforms would work similarly, but haven’t been tested).

Since this is a very simple audio app, only 3 pieces of data need to be communicated between the GUI and Audio Thread:

tempo parameter (a float)
current beat (a float)
volume parameter (a float)

For this, Rust exposes callbacks which set atomic values onto the audio processor handle.

Playing back a Sine Wave with Augmented

Let me show what a sine wave generator looks like using augmented.

First, we declare our audio processor struct.

use augmented_oscillator::Oscillator;

#[derive(Default)]
struct OscillatorProcessor {
  oscillator: Oscillator<f32>,
}

Then, implement the AudioProcessor trait:

use audio_processor_traits::{AudioProcessor, AudioBuffer, AudioProcessorSettings};

impl AudioProcessor for OscillatorProcessor {
  type SampleType = f32;

  fn prepare(&mut self, settings: AudioProcessorSettings) {
    self.oscillator.set_sample_rate(settings.sample_rate());
  }

  fn process<BufferType: AudioBuffer<SampleType = Self::SampleType>>(
    &mut self,
    buffer: &mut BufferType
  ) {
    for frame in buffer.frames_mut() {
      let out = self.oscillator.next_sample();

      for sample in frame {
        *sample = out;
      }
    }
  }
}

The AudioProcessor trait requires two methods:

prepare will receive the audio settings such as sample rate, so the processor can set up internal state
process will receive a mutable, generic audio buffer implementation, which allows processing decoupled from sample layout

The 2nd detail is up to discussion. I’ve found that some audio buffer indirection may cause serious performance cost due to missed optimisations, and it might not be worth to have it instead of simple sample slices.

However, some kind of ‘process callback context’ is required, as the processor needs to know the nº of input/output channels.

AudioBuffer::frames_mut returns an iterator of the channel chunks; each frame is a sample pair (for stereo audio). We simply tick the oscillator struct and set it as the output.

Main function

Once this is set up, a command-line app can be generated with audio_processor_standalone.

fn main() {
  let processor = OscillatorProcessor::default();
  audio_processor_standalone::audio_processor_main(processor);
}

The command-line supports both “online” (using your input/output devices) and “offline” (into a file) rendering.

audio-processor-standalone

USAGE:
    my-crate [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -i, --input-file <INPUT_PATH>              An input audio file to process
        --midi-input-file <MIDI_INPUT_FILE>    If specified, this MIDI file will be passed through the processor
    -o, --output-file <OUTPUT_PATH>            If specified, will render offline into this file (WAV)

It can also read an input audio file and feed it into the processor as well as read an input MIDI file.

In order to process MIDI audio_processor_main_with_midi should be used and the struct needs to implement the MidiEventHandler trait.

Embedding into an app

On top of a command-line, audio_processor_standalone can also be used to embed an audio-thread into another app. For this, the standalone_start function is used. The library will return a set of “handles”, objects that should be held onto (the audio thread will stop when they are dropped).

Audio Processor Handle Pattern

I found a little pattern for sharing state between GUI and audio thread, which is the “audio processor handle” pattern.

Each AudioProcessor declares its shared state as a handle, which is thread-safe (uses atomics or other real-time safe synchronization primitives). This handle uses a shared pointer which is reference counted & de-allocated on a background thread.

The GUI thread will copy the handle & keep a reference to it, while the audio processor will be moved into the audio thread.

use audio_processor_traits::AtomicF32;
use audio_garbage_collector::{Shared, make_shared};

struct MyProcessor {
  handle: Shared<MyProcessorHandle>,
}

struct MyProcessorHandle {
  volume: AtomicF32,
}

fn example() {
  let processor = MyProcessor {
    handle: make_shared(MyProcessorHandle {
      volume: AtomicF32::new(0.0)
    })
  };

  let handle = processor.handle.clone();

  // now the processor can be moved to the audio thread.
  // the handle can be shared between many threads & must be thread safe
}

Conclusion

Simple Metronome will soon be available for free on macOS and iOS stores.

Its source code and building blocks will continue to be available, and I intend to write more simple/complex apps in order to stress them.

Please reach out with any feedback, questions or comments on Twitter - @yamadapc.

All the best.

Test Plugin Host

Sat, 24 Jul 2021 00:00:00 GMT

Over the last several weeks, I’ve been locked-down at home. A new cluster of the Delta variant has resulted in a lockdown of Sydney, as well as other parts of Australia.

With nothing else to do, I started exploring building audio apps with Rust.

My main goal was to learn and have fun, but I had a specific product in mind driving my exploration, which has now reached a minimally useful version.

All the code I’m writing is MIT licensed and available on GitHub here.

It still needs a lot of polish and testing & I’m just learning as I go, as I’m not a professional audio developer.

However, I want to share its existence with the world and also share some thoughts from my first development attempts.

Plugin Host

The product I had in mind was plugin-host and plugin-host-gui.

Plugin Host is a testing utility for VST plug-ins.

Screenshot of Plugin Host's iced based GUI

It comes in two flavours: as a Command-line Tool and as a GUI.

The workflow is as follows:

Start plugin-host
Open an input audio file with it
Open your plugins dylib with it
Develop your plugin

Plugin host will host your plug-in and loop the audio file over it. Whenever you recompile your dylib, plugin host will reload your VST and its editor and restart audio playback with the new version.

Doing proper “hot-reload” with fade-ins/outs and without restarting the audio-thread may come at some point soon.

Very quick demo

Here’s a 1-minute demo. Imagine I’m working on a new looper VST, and I’m trying to build a nice audio visualization chart.

I started with a blue color, but now I’m not quite sure. So I make some changes and re-compile:

For this small VST, most builds should complete in a couple of seconds, which is an awesome dev-loop for a Rust based plug-in. I could try things out, see how they feel and continue.

More features I’d want to add

Input & Output

For now, the host only supports looping over a file, but I’ll add support for processing normal input soon. Then, one should be able to write some audio-processing code, a GUI, maybe plug their guitar in and see how things feel.

The CLI version of the host supports generating an output .wav file and outputting some basic performance diagnostics, but that has not been added to the GUI yet.

Protecting your ears and speakers

It’s quite a bad idea to hot-reload audio-processing code with your speakers turned up as you may damage your ears or speakers depending on what type of mistake you make.

In order to mitigate this, I’d like to add a limiter and similar protections onto the host, so we could iterate without fear.

Visualization

For now, the host only draws a very basic RMS chart at the bottom & a broken (not using dBs) volume indicator. It’d be great to provide more tools that’d be commonly useful for developing audio processors, such as a spectrogram.

Volume control & fixes

This is a basic thing that is missing.

Augmented Audio

I’m pushing these things to the augmented-audio repository on GitHub.

Ultimately, I’d very much like to have some kind of framework-y selection of solutions for common audio application problems in Rust.

Some examples of “solutions” that need to be selected (or built):

Testing tools - Such as plugin-host, but also unit-testing approaches like snapshot testing or property testing over the audio
Reference counting background GC - I’m using a basedrop wrapper to avoid de-allocation on the audio-thread & have a WIP alternative. New things that I think are missing in both: metrics, more rigid scheduling of collection, more tests.
GUI toolkit
Audio processor abstraction & audio graph implementation
Stand-alone & multiple plug-in format packaging
Filters & common effects
Guidelines of how to handle shared audio state - e.g. “ProcessorHandle” structs?

There’s a ton of amazing work on the RustAudio GitHub organisation which is super helpful for me as well as other framework-y projects such as baseplug.

augmented-audio is my own exploration on this space, leveraging some tooling from RustAudio & several crates I’m trying to write or port from C++.

Web GUI: Earlier steps

The initial idea for these libraries and for Plugin Host GUI was to explore using Web GUI for the audio apps.

I’ve now deprecated this work and will replace all web bits with native GUI. Notes on it will continue to live on WEB_GUI.md and the legacy-web-gui branch of the repository.

However, it’s still an interesting little experiment, as I did get it to work and perform acceptably.

There were two parts of this:

plugin-host-gui
- Equivalent to the GUI you saw above, but built using tauri
tasv2
- A tremolo plug-in built with a React.js front-end hosted on a WkWebView and communicating through messages with the Rust audio processing parts

I won’t discuss the tauri app, though you may check the markdown file above for notes including how I was trying to get around bottlenecks for the volume meter.

Web front-ends for VST plug-ins

Screenshot of the tremolo's React.js based GUI

The incomplete web based plug-in idea would work in the following manner.

VST layer

Rust exposes VST APIs for an audio host to call using the rust-vst crate.

It exposes callbacks to process audio, as well as callbacks to create its “editor”.

For the very simple web plugin prototype I built, a shared AudioParameterStore object held all parameter states and definitions (value & instructions to render on the GUI).

pub trait PluginParameterLike {
    fn name(&self) -> String;
    fn label(&self) -> String;
    fn text(&self) -> String;
    fn value(&self) -> f32;
    fn set_value(&self, value: f32);
    fn can_be_automated(&self) -> bool;
    fn value_range(&self) -> (f32, f32);
    fn value_type(&self) -> ParameterType;
    fn value_precision(&self) -> u32;
}

pub type ParameterRef = Arc<dyn PluginParameterLike>;

struct ParameterStoreInner {
    parameters: HashMap<ParameterId, ParameterRef>,
    parameter_ids: Vec<ParameterId>,
}

GUI Thread

The host will call into the VST passing a pointer for the platform-specific view the plugin GUI should attach itself to.

The GenericParametersEditor struct creates a webview and a transport layer to communicate with it. Internally this uses webview-holder and darwin-webkit crates which are just low-level wrappers to the WebKit Objective-C API.

Transport layer

In order to exchange messages with JavaScript, there’s an abstract transport layer. Two WebviewTransport implementations were available.

WkMessageHandlerTransport uses WkScriptMessageHandlers to exchange messages with JavaScript. This is a webkit API for attaching native callbacks for JavaScript to post messages into. The ‘Rust to JavaScript’ side is achieved by evaluating a JavaScript function call that broadcasts messages onto a global EventEmitter.

WebSocketsTransport uses websockets to send/receive messages into JavaScript.

The reason I implemented both transports is WebSockets is more flexible, since it allows for the plugin front-end to run in Chrome or over the network. At the same time, I expected that WkMessageHandlerTransport would be more efficient.

However, I never benchmarked one transport against the other, so I don’t know if there are performance benefits from one or another. It’d be interesting to see if WkMessageHandlers are more efficient or the other way around.

I think, in theory, one benefit of using the native API is one could avoid serialization of the messages and communicate via JavaScript objects. However, in my transport implementation messages are always serialised to JSON.

Another benefit is not having to run a websockets server, which does bring concerns around security and bloat.

JavaScript side

JavaScript has counter-parts to the transport layer which handle the different message transport options.

As mentioned, we may develop the front-end in Google Chrome or, for example, use it as a remote control, but when packaged we may use the native transport.

The tremolo is a super simple front-end written in React.js. It subscribes-to and publishes parameter changes as well as rendering knobs dynamically based on what parameters the Rust side sends to it.

Command handler

When messages are received from JavaScript they’re pushed into a queue. A tokio task pops messages and actually performs the state updates or broadcast.

The command handler keeps a reference to the parameters-store and handles messages to set parameters as well as broadcasting initial parameter state for the GUI to render when it’s created.

When the React.js front-end is mounted on the DOM, it’ll publish a message signaling it’s ready for which the command handler will respond with definitions of all the knobs it should render.

This might be a bit more complex than it had to, but it allowed for very loosely coupled components (transport is just something that pushes messages onto a channel, command handler is application specific). Even though I won’t be using the web front-end bits for now, the building blocks could be adapted for other use-cases.

Issues & benefits with web front-ends

There are several issues and benefits of web front-ends for VST plug-ins.

Here’s a quick list of some down-sides and comments:

Much more complexity
- Runtime & build-time infrastructure to support it
Worse performance
- All messages must be serialized and some rendering tasks are less efficient on the Web
- However, there are several rendering tasks which are very efficient on the web, and it’s a very featured platform
Worse memory usage?
- Having a completely separate GUI would mean a lot of data is duplicated
- It could be the case comparing with an optimised native GUI that memory usage is worse with a browser, but it’s not really the case comparing my simple tremolo with iced based plug-ins
- The React.js plug-in actually consumes significantly less memory (several times less) than a similar iced counterpart
Not cross-platform (yet)
- Since I’m just using raw WkWebView bindings, this is not cross-platform at all (though once would use tauri’s building blocks if they support the plugin use-case at some point - currently, their libraries require that the event-loop is owned by tao)

Overall, the reason I’m not going to continue pushing this side is because I’d really like to do basic audio-visualisation in the plug-ins, and I’ve found that iced has quite good performance at doing that & would be able to integrate onto a custom rendering pipeline if I need to.

I think for plug-ins that do not intend to do audio-visualisation, if there was a cross-platform wrapper supporting event callbacks in/out of JavaScript it could be a potential option.

For a professional application, I’d really consider building on top of the web platform a safer bet than adopting Rust native GUI toolkits, as unfortunate as that may be. As long as the project can survive the short-comings of message passing and the extra complexity, it’d be something to consider.

It really depends on the application and its constraints.

`iced` and `druid`

I did a very lousy benchmark of iced and druid drawing the input signal as a line.

For this, iced could stay with only around 5% CPU usage to render (not at 60fps), while druid saturated the CPU to do a similar thing.

I sure that this may be due to my incorrect usage of druid.

I’m trying to use the widget API to draw charts (and doing it without much performance considerations). When profiling, most of the time is spent within CoreGraphics calls, so if drawing back-ends are switchable (I have not checked) this could potentially be worked around.

Even so, the results of this spike, combined with existing support from iced_baseview to render iced apps inside plug-ins led me to pick iced.

iced performance is not 100% free on plugin-host as I had to change some internal lyon calls to get the plugin-host-gui2 audio chart to perform better. I’ve taken notes here: https://github.com/yamadapc/augmented-audio/wiki/Volume-Charts---iced-lyon-performance.

Plugin host `iced` GUI notes

So far I’ve found iced a very nice library to use, it’s very easy to use and solves several problems out-of-the-box with good documentation and working examples.

Some things that I’d put on a wish-list:

NOTE iced is a great open-source project, that I’m sure has been built with care (who knows if not by someone else in lock-down!), and these are things I think need to be solved, but not complaints or demands or anything like that.

Documentation on how to structure a larger app
Support for avoiding re-rendering/re-drawing
Accessibility
Context menus
Multiple windows
Gradients & shadows - I started drafting gradient support here
High memory usage

Conclusion

Check-out the plugin-host-gui2 source code if you’re interested.

I’ve started a discourse topic here as well for discussion.

Reach out to me on the Rust Audio chat if you’d like information, have feedback or just want to try these things out. I’m going as @yamadapc there.

I’m also on https://twitter.com/yamadapc.

I’ll try to write some thoughts next on the implementation details of plugin-host as well as just keep pushing forward with it.

All the best.

Rust Audio Experiments: 1 - Logging

Fri, 16 Jul 2021 00:00:00 GMT

This is (hopefully) the first in a series of posts I want to write while I try to use Rust for audio development. In this post I’d like to share some thoughts around building Rust audio plugins locally.

I’ll show:

The set-up & libraries for logging from a VST to a file

Logging from the VST

Audio plug-ins will be loaded into a host application as a dynamic library and because of this, logging to standard output will not be effective (as the host’s standard output is not available while doing manual testing).

In order to log from the VST, we’ll log to a file.

Rust logging facade

Rust has the standard log crate to be used as a logging facade. Logs can be added to the code as follows:

fn main() {
    log::error!("Error msg");
    log::warn!("Warning msg");
    log::info!("Info msg");
    log::debug!("Debug msg");
    log::trace!("Trace msg");
}

Adding log statements to the code will not output anywhere and needs a logger implementation to be configured.

log4rs set-up to log to a file

For this use-case I’ve found log4rs provided a simple enough set-up. The plug-in will call a configure_logging() function, which will try to initialize the logger.

fn configure_logging(root_config_path: &Path, name: &str) { /* ... */ }

Log destination

We’ll write logs to two destinations:

Standard output
A log file

We’ll rotate the log file when it reaches a certain size and ensure the destination directory exists before starting.

Creating the root configuration/logging directory

Some other part of our application will give us the logging directory, which may be in a . directory of the user’s HOME or, for example, in /Library/Application Support.

The ensure_logging_directory function will make sure the root path exists:

fn ensure_logging_directory(root_config_path: &Path) -> Result<PathBuf> {
    std::fs::create_dir_all(root_config_path)
        .map_err(LoggingSetupError::CreateLogDirectoryError)?;
    Ok(root_config_path.to_path_buf())
}

It returns a Result, which is a type I’ll define soon.

Configuring logging

To configure logging we’ll set up two Appenders which log to stdout & the file:

pub fn configure_logging(root_config_path: &Path, name: &str) -> Result<()> {
    let log_dir = ensure_logging_directory(root_config_path)?;
    let log_path = log_dir.join(name);
    let logfile = RollingFileAppender::builder()
        .encoder(Box::new(PatternEncoder::new(
            "{d} [{l}] {M}:{L} - {m} - tid:{T}:{t} pid:{P}\n",
        )))
        .build(
            log_path,
            Box::new(CompoundPolicy::new(
                Box::new(SizeTrigger::new(1024 * 1024 * 10)),
                Box::new(DeleteRoller::new()),
            )),
        )
        .map_err(LoggingSetupError::FileAppenderError)?;

    let config = Config::builder()
        .appender(Appender::builder().build("logfile", Box::new(logfile)))
        .appender(Appender::builder().build("stdout", Box::new(ConsoleAppender::builder().build())))
        .build(
            Root::builder()
                .appender("logfile")
                .appender("stdout")
                .build(LevelFilter::Info),
        )?;

    log4rs::init_config(config)?;

    Ok(())
}

Errors

We’ll use thiserror to handle errors. It’ll join the 4 types of errors that may happen during this process into one type:

#[derive(thiserror::Error, Debug)]
pub enum LoggingSetupError {
    #[error("Failed to create logging directory")]
    CreateLogDirectoryError(std::io::Error),
    #[error("Failed to set-up log-file appender")]
    FileAppenderError(std::io::Error),
    #[error("Failed to set-up logging configuration")]
    LogConfigError(#[from] ConfigErrors),
    #[error("Failed to set logger configuration")]
    SetLoggerError(#[from] SetLoggerError),
}

pub type Result<T> = std::result::Result<T, LoggingSetupError>;

Source code

See augmented-audio/crates/audio-plugin-logger.

FileSaver - v1.1.1

Mon, 08 Jun 2020 00:00:00 GMT

FileSaver is now “Parallel Disk Scanner” and is available for sale on the app store.

FileSaver logo

Since I got my latest MacBook, I can only go for a couple of months before it starts running out of free disk space. It’s a pretty bad situation where to save a bit on SSD size, I have to spend my time cleaning-up large files like music samples, VMs, node_modules and so on.

So then I decided to save a lot of time 😂 and instead of buying one of the available “disk cleaning” solutions, I spent some time trying some open-source software and eventually decided to write FileSaver, a fast multithreaded disk usage scanner.

On my MacBook, scanning 5.3 million files takes 100 seconds, with around 51.000 files/second of throughput. The compromise is high memory usage of around 2GB.

What I used to do with coreutils

I used to do the following command combination to find sizes:

# Find file sizes
du -h ./directory | \
    # Sort by size
    # (just sort on linux, requires gnu coreutils on OSX)
    gsort -h      | \
    # Put on a pager
    less

This works fine except that:

It’s slow
It’s not interactive

`duc`

I’ve also tried a great alternative to the above, called duc.

duc is a great open-source utility written in C that performs this task a bit faster and with interactive options. It can also generate an index that can be re-used, so we don’t need to scan all files when looking for places to clean.

Even so, what I found was that on my filled-up disk, it was really slow to generate the index for the whole thing, including system directories like /Library and /usr/local. As a developer, a lot of the trash was around those.

Parallel size scanning

I profiled duc and found that it’s single threaded nature was a bit bottleneck. Writes to the index blocked scanning file sizes and the calculations to aggregate sizes also probably compromised throughput.

I ran over this issue:

Some thoughts about indexing, filtering, parallelization, etc https://github.com/zevv/duc/issues/205

The idea that I could write a multithreaded program and get a massive performance boost stuck with me.

I considered contributing to duc, but it had two major flaws for me:

It’s GPL licensed
It’s written in C

The 2nd point is just because, knowing a bit of C++ I didn’t see the point of going to C.

I then designed and iterated on a parallel scanning architecture / aggregation algorithm that achieves pretty good results in terms of throughput.

Problem domain and aggregation overview

The problem we have to solve is that of finding directory sizes based on all their children’s sizes.

Given a certain path to scan, we want to lstat it. This will give us its size and file type. If this path is a directory, we then want to scan all its children.

This path will have “completed” once all its children have completed or if lstat says it isn’t a directory.

So we have to continuously scan children and as they come in, update the parent path’s completed children count (and do this recursively up).

To support better UX and performance, whenever a path is completed, its delta update in size is carried up each of its parents.

For example, imagine we have the following directory structure:

content
└── blog
    └── filesaver-freeing-up-space
        ├── index.md
        └── logo.png

We’ll start the scan at /content
- Since it’s a directory we read its children
- Its size and 1 child count is stored
We do the same at /content/blog and then /content/blog/filesaver-freeing-up-space
At this point our state looks like:
- /content - 1 pending child path
- /content/blog - 1 pending child path
- /content/blog/filesaver-freeing-up-space - 2 pending child paths
We scan index.md
- /content/blog/filesaver-freeing-up-space/index.md - is completed + 100KB
- /content/blog/filesaver-freeing-up-space - 1 pending child paths (+ 100KB)
- Other paths’ pending counts are kept, but their total sizes are updated
- /content/blog - 1 pending child paths (+ 100KB)
- /content - 1 pending child paths (+ 100KB)
We scan logo.png
- /content/blog/filesaver-freeing-up-space/logo.png - is completed + 300KB
- /content/blog/filesaver-freeing-up-space - is completed (+ 300KB)
- /content/blog - is completed (+ 300KB)
- /content - is completed (+ 300KB)

Architecture

FileSaver is a simple program, but it’s structured. The architecture is as follows.

Components

Lock based queue
Worker Threads
Reader Thread
UI Thread

Queue

We pass data between threads using an uninteresting lock based queue, plus one extra lock on the size/pending state, since the UI thread will read it while its being processed.

For this we have filesaver::data::WorkQueue

Worker Threads

The workers work with two queues:

workQueue provides incoming paths to scan
resultQueue has entries pushed onto it, with their sizes, types and child paths attached

The worker will enqueue any children it finds.

Reader thread

The reader thread performs the aggregation as described in the aggregation overview. The state gets stored in a giant hash-map (std::unordered_map) mapping paths to their sizes and number of pending children.

UI

The UI uses AppKit and is written in Objective-C++. It polls the state for the directories that are visible on the screen.

Distribution

For now, FileSaver is available from its GitHub page. The project has structure that should make it convenient to include it as a library onto a different codebase. Though there could still be improvements for this use-case like organizing public headers and the build system, it goes most of the way and should have good DoxyGen docs.

The code is distributed under the MIT license.

Download FileSaver

Beijaflor's blog RSS feed

cmakefmt

Installing from source

Examples

Long commands

Key-value pair behaviour

Indentation

Conditions

Implementation

Playground

Drawing knobs

Drawing knobs

Reacting to input

Knobs that are sliders

Expanding knobs

That is it

Wisual Logger

@wisual/logger

Hierarchy

React integration

Logger sinks

Configuring the sink

Why named and hierarchical loggers?

What else?

Parsing and serializing MIDI files in Rust

Usage example

Parser combinators: nom

Serialization: cookie_factory

Performance

Conclusion

Continuous Looper

Continuous Looper

Demo video

Table of contents

Some requirements for the app

Real-time audio

Interactive GUI

Looper quantization

Architecture

Containers or components

System Audio/MIDI APIs - API wrappers

LooperEngine/Swift integration - Rust & Swift integration

Polling for changes

Push from GUI

SwiftUI Performance

Callbacks from rust to swift

Swift frameworks, packages and apps

Rust components

Rust and audio

Multi-track looper audio graph

Looper states

Always recording buffer

Quantization implementation

Keeping track of time: Play-head

Implementing quantization

Testing audio

Normal unit-testing with mock buffers

Property based testing

Visualization

Snapshot testing

Testing allocations

Sequencer implementation

Parameter storage - LFOs, parameter locks, sequencing

Conclusion

Other links

Augmented Audio - Generic AudioProcessors in Rust

Overview

AudioProcessor: Implementing a BitCrusher

Implementing AudioProcessor - Generic buffers

Listening to our AudioProcessor - Generic CLI

Listening to our AudioProcessor - Generic VST

Generic GUI

Extract parameters into a handle

Implement a generic handle

That’s it

Custom GUI

Simple Metronome

Goals primer

Augmented Audio Libraries

The Metronome

Parser combinators: `nom`

Implementing `AudioProcessor` - Generic buffers

Listening to our `AudioProcessor` - Generic CLI

Listening to our `AudioProcessor` - Generic VST

`iced` and `druid`

Plugin host `iced` GUI notes

`duc`