Realtime Convolution FULLTEXT01

Real-time Convolution of Two Unknown Signals for Use in a Musical Context

Antoine Henning BardozLars Eri Myhre

Master of Science in Electronics

Supervisor: Jan Tro, IETCo-supervisor: Tor A. Ramstad, IET

Sigurd Saue, IMyvind Brandtsegg, IM

Department of Electronics and Telecommunications

Submission date: June 2013

Norwegian University of Science and Technology

NORWEGIAN UNIVERSITY OF SCIENCE AND TECHNOLOGY

AbstractFaculty of Information Technology, Mathematics and Electrical Engineering

Department of Electronics and Telecommunications

Master of Science

Cross Convolution of Live Audio Signals for Musical Applications

by Antoine Henning Bardoz

Lars Eri Myhre

This thesis proposes a method for convolution of two real-time audio signals, for

use in live performances or post-production. In contrast to traditional convolu-

tion techniques, which require a predefined impulse response as one of the input

signals, our method allows for convolution of two continuously updated, and un-

known, signals, allowing two musicians to shape each others timbral and temporal

contributions.

The aim was to create an effect that sounded like convolution, offered low output

delay, as well as giving satisfying feedback to musicians. To achieve this, a hybrid

of time- and frequency domain techniques has been used, offering the low output

delay associated with the time domain, and the low CPU load characteristic of

FFT-based frequency domain processing. To deal with the limitations inherent in

convolution, namely that to perform ideal convolution of two unending signals, an

infinite amount of memory and processing power are eventually required, transient

detection has been applied to segment the signals in a musically relevant way. The

transient-assisted segmentation also makes the effect more intuitive for users, as

it increases the users ability to interact rhythmically.

A GUI was developed, and the effect was implemented as a VST plug-in, to allow

users to easily apply the effect in DAWs.

The effect was prototyped in Matlab, and later implemented in Csound and C,

using the Cabbage framework for the VST.

NORGES TEKNISK-NATURVITENSKAPELIGE UNIVERSITET

SammendragFakultet for informasjonsteknologi, matematikk og elektronikk

Institutt for elektronikk og telekommunikasjon

Master i elektronikk

Krysskonvolusjon av sanntidslydsignaler til musikalske anvendelser

by Antoine Henning Bardoz

Lars Eri Myhre

I denne oppgaven foreslas en fremgangsmate for konvolusjon av to sanntids lydsig-

naler, til bruk i live-opptredener eller post-produksjon. I motsetning til tradis-

jonelle konvolusjonsteknikker, som krever en forhandsdefinert impulsrespons som

ett av inngangssignalene, tillater var metode konvolusjon av to kontinuerlig opp-

daterte, og ukjente, signaler, slik at to musikere kan forme hverandres klanglige

og tidsmessige bidrag.

Malet var a skape en effekt som hres ut som konvolusjon, tilbyr lav utgangs-

forsinkelse, og gir tilfredsstillende tilbakemelding til musikere. For a oppna dette

har en kombinasjon av tids- og frekvensdomeneteknikker blitt brukt. Dette kom-

binerer lav CPU-belastning, takket vre FFT-basert frekvensplanprosessering,

med den lave forsinkelsen assosiert med tidsdomenet. For a handtere begren-

sningene forbundet med konvolusjon, nemlig at for a utfre ideell konvolusjon av

to uendelige signaler, kreves det etter hvert uendelig minne og prosessorkraft, har

transientdeteksjon blitt brukt til a segmentere signalene pa en musikalsk relevant

mate. Segmentering ved hjelp av transienter gjr ogsa effekten mer intuitiv for

brukerne ved a ke deres evne til a samhandle rytmisk.

Et grafisk brukergrensesnitt ble utviklet, og effekten ble implementer som en VST

plug-in, slik at brukere enkelt kan benytte effekten i DAWer.

Effekten ble prototypet i Matlab, og senere implementert i Csound og C. Cabbage-

rammeverket ble benyttet for VST-implementasjonen.

Acknowledgements

We would like to extend a special thanks Sigurd Saue for giving us valuable sug-

gestions and technical insight, without which we would truly have been lost.

In addition we would like to thank Jan Tro for keeping music alive at Glshaugen,

and making this all possible; yvind Brandtsegg for technical and artistic insight,

as well as tips from a users perspective; and Tor A. Ramstad for signal processing

guidance.

A special thank you goes to Rory Walsh for developing Cabbage and for being ex-

tremely helpful through the forum at www.thecabbagefoundation.com. We would

also like to thank the Csound community for developing Csound and for quick and

crucial help through the Csound developers list.

For wasting our time with mindless babble and keeping us sane, we thank our

study hall companions Thomas Christiansen, Niklas Skyberg, Bendik Paulsrud,

Jrund Kaarstad Dahl and Rune Svensrud.

For their musical contributions, we thank Thomas Etholm-Kjeldsen, Jakob Eri

Myhre and Olaf Mundal.

Antoine would like to thank Lars for truly giving his all during this semester,

contributing heavily every step of the way, coming up with important ideas, and

remaining motivated, as well as motivating, until the very last minute.

Lars would like to thank Antoine for a partnership which will not be forgotten.

His knowledge in signal processing, programming and music has been infectious.

His effort has been remarkable.

iii

Contents

Abstract i

Sammendrag ii

Acknowledgements iii

List of Figures vii

Abbreviations x

Symbols xi

1 Introduction 1

1.1 Problem Description . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 How to Read This Thesis . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Theory 5

2.1 Convolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.1 Time Domain . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.1.2 The (Circular) Convolution Theorem . . . . . . . . . . . . . 6

2.2 The Fast Fourier Transform and Frequency Domain Multiplication . 7

2.3 Theoretical Foundation for Real-Time Blockwise Convolution . . . . 8

2.4 Transients and Transient Detection . . . . . . . . . . . . . . . . . . 12

2.5 Latency Tolerance for Musicans . . . . . . . . . . . . . . . . . . . . 13

3 Development Tools 14

3.1 Matlab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2 Csound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.3 Cabbage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

4 Algorithm 17

4.1 Preliminary Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 17

4.1.1 Short Description . . . . . . . . . . . . . . . . . . . . . . . . 18

4.1.2 Buffer Up Signals . . . . . . . . . . . . . . . . . . . . . . . . 18

iv

Contents v

4.1.3 Convolution Computation . . . . . . . . . . . . . . . . . . . 20

4.1.4 Put Convolution Result on Output . . . . . . . . . . . . . . 20

4.2 Algorithm Version 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4.2.1 Short Description . . . . . . . . . . . . . . . . . . . . . . . . 22

4.2.2 Buffer Partitioning . . . . . . . . . . . . . . . . . . . . . . . 23

4.2.3 Cross Convolution of a Segment . . . . . . . . . . . . . . . . 24

4.2.4 Output Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4.3 Algorithm Version 2 (Transient Detection) . . . . . . . . . . . . . . 25

4.4 Algorithm Version 3 (Parallel Processes) . . . . . . . . . . . . . . . 27

4.4.1 Alternative 1: ThrowAll (Used in Final Implementation) . . 28

4.4.2 Alternative 2: ThrowLast . . . . . . . . . . . . . . . . . . . 29

4.4.3 Alternative 3: TwoProc . . . . . . . . . . . . . . . . . . . . . 30

4.4.4 Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5 Results 34

5.1 Preliminary Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.2 Algorithm Version 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 38

5.3 Algorithm Version 2 . . . . . . . . . . . . . . . . . . . . . . . . . . 42

5.4 Algorithm Version 3 . . . . . . . . . . . . . . . . . . . . . . . . . . 44

5.4.1 ThrowAll (Final Algorithm) . . . . . . . . . . . . . . . . . . 44

5.4.2 ThrowLast . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

5.4.3 TwoProc . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

5.5 Graphical User Interface . . . . . . . . . . . . . . . . . . . . . . . . 50

5.5.1 Gain Knobs . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

5.5.2 Transient Detection Section . . . . . . . . . . . . . . . . . . 51

5.5.3 Convolution Section . . . . . . . . . . . . . . . . . . . . . . 52

6 Discussion 53

6.1 Preliminary Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 54

6.1.1 Why the Preliminary Algorithm Fails . . . . . . . . . . . . . 54

6.1.2 Independent Buffer Sizes, Overlap on Output and Fading ofOverlap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

6.2 Algorithm Version 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 56

6.2.1 Delayed Change . . . . . . . . . . . . . . . . . . . . . . . . . 57

6.2.2 Indistinct Transients . . . . . . . . . . . . . . . . . . . . . . 58

6.2.3 Destructive Interference . . . . . . . . . . . . . . . . . . . . 58