Musical Interactivity Through Alternate Controllers

Below is a research paper exploring some of my research from Spring 2014.  The first half starts with my research and is followed by implementations.  To view and listen to just the major project that was developed (an xbox controller instrument) go to – – or the same information is contained at the end of this post.  


This quarter (Spring, 2014) I explored alternative controllers and interfaces, machine cognition and musicianship, and human-computer musical interactivity.

In particular I researched the performance practice associated with alternate controllers and ways these controllers could be used to interact with improvising programs.

I applied this research through multiple projects, but primarily in using an Xbox controller as an instrument for improvisation.


Alternate Musical Interfaces

In Electronic and Computer Music by Peter Manning there is an extended discussion on the history of performance controllers.  Manning describes the prevalence of the keyboard as the primary electronic musical interface acting as a key shaping factor in the development of musical controllers.  While the keyboard vastly outnumbers other interfaces, during the 1990s and 2000s there was an extensive amount of research exploring alternate controllers, although few of these have ended up commercially produced (Manning, 461).   Among these alternate controllers some take the form of traditional instruments, such as the vBow, (2002) or the Overtone Violin, (2005), or the EWI.  There has also been extended work utilising other technology, such as the WiiMote to perform musically.  Finally, other alternate controllers are built off custom designs allowing completely new ways of interacting with musical material.


Part of understanding the creation and use of musical interfaces is to understand what defines a musical instrument itself.

“Musical instruments are devices designed to transform the actions of performers into acoustical energy” (Dodge and Jerse, 404).

Dodge and Jerse further describe musical instruments as tools that give the musician certain parameters to explore.  They elaborate that the clarinet can be broken done into a two dimensional control surface combining actions of the fingers with wind pressure to create two control parameters.

MIDI drum pads demonstrate that musical control interfaces design and application is significant to many musicians; a drum pad essentially replicates an output that can be performed with one key on a keyboard.   Manning counters this assertion with: “the use of a MIDI keyboard to trigger such events was a poor substitute for the art and craft of the performing percussionist” (Manning, 450).  I believe understanding the significance of MIDI drum pads provides essential indications to the utilisation and design of alternate musical interfaces.

A MIDI drum pad allows the performer to retain the original performative feel associated with performing on percussion.  This performative concept will work to shape different outward musical results than could be achieved performing the same music on a keyboard.  The ability to reuse skills developed in acoustic traditional percussion directly to the electronic domain is also significant in it’s ability to allow musicians to utilise years of training.   This knowledge on the relevance of the MIDI drum can be incorporated into design; that there is a unique energy and level of precision derived from different forms of gestural control.

It should be noted there may also be a level of subjectivity that encourages the idea that a different action from a performer is less effective or less musical.  A percussionist is likely to feel much more musically inclined perfuming on a drum pad, as opposed to pressing a key on a keyboard.  This subjectivity should be taken as part of the musical performance, as ultimately the context and history of the performer will impact the use and effectiveness of an alternate musical interface.


Performance Practice
Laetitia Sonami is a sound artist primarily known for her work with “The Lady’s Glove”, a glove containing a variety of sensory outputs.  “The Lady’s Glove” is an instrument developed over twenty years and Sonami demonstrates extreme virtuosity and control over this instrument.  Her instrument by nature involves a distinct performance style that coexists with the sound creation process.  Sonami regularly discusses the idea of the sound being ‘embodied’ within the interface.  Part of this concept involves the idea of ‘embodiment of a performative action’ and the relationship between the instrument and the sound itself.

It becomes an instrument when the software starts reflecting and adapting the limitations and possibilities of the controller. (Rodgers, 229)

She elaborates that the instrument and sounds it creates should be reflective of it’s own control scheme and not just a system to ‘push buttons and trigger things” (although this could be compared to the operation of a piano).
While Sonami’s work involves a distinct style of performance that creates a distinct sonic world, laptops and Xbox controls often create a distinctly un-performance like experience that doesn’t seem to represent any specific sonic world.  Xbox controllers and laptops are designed for ergonomic and efficient use of small movements.  This design practice creates a powerful control system, although one that outwardly shows no strong reaction, and there often isn’t a real sense of change coming from the instrument.  Movements made in each interface are on a small scale, laptop’s in particular can draw the performers’ attention away from the audience and towards the screen.  However, many of these features are common to traditional musical instruments, a pianist often makes smaller movements across an interface where the audience only sees small changes.

Sonami also explores the idea that originally she found laptop performers uninteresting but over time came to realise that what she was really interested in was the intimacy and “the focus the performer radiates” (Rodgers, 230).  For whatever reason she found this lacking in many laptop performers.  To my mind a laptop can very easily act as an extra wall between the performer and the audience, especially when the audience has no concept of what is actually being changed within the laptop.   Perhaps many of the flaws associated with laptop performance are due to the performers themselves and as further performance experience grows as a whole, a more established tradition will take place.  From my own observations most instruments seem to invoke a certain performance style, many clarinetists move and perform in a style unique to the clarinet.

Another key barrier in electronic musical interfaces is the disassociation of the physical sound being created with the sound source.  This is a barrier that has been associated with computer music from the very beginning.

Pierre Boulez: “We are here faced with definite limitations; the psychological reactions of an audience to which music is fed by loudspeakers can hardly be avoided where the audience is deprived of the possibility of associating a sound with a gesture.”  (Manning, 61)

Even in these early discussions by Boulez in the 1960’s, he describes the idea of gesture and sound association and the problem’s these pose towards electronic music.  The sound of any instrument has only become associated with the instrument itself over time.  Alternate interfaces allow this barrier to extend even further by the ability to create such diverse sound worlds, that no specific sounds (or even general musical concepts) can be associated with instrument.  I would argue that in our current time it is this association of a sound idea with a gesture or control that is important alone, an audience is now very accustomed to an instrumental gesture represented through a speaker (in particular through DJ’s and electric guitars).


Cross Domain Mapping

Understanding ways of mapping information from one domain to another is crucially important to alternate controllers.  Well-applied mapping will make virtuosity easier to attain while potentially generating gestural material.   In Conceptualizing Music: Cognitive Structure, Theory, and Analysis, Lawrence M. Zbikowski explores the area of cross domain mapping in terms of how we apply the concept of high and low to represent pitch.  He mentions that this is contextually derived, other cultures consider pitch variations as being young or old (Suya) or lower pitches as smaller and higher pitches as larger (Bali and Java) or even pitch relationships as waterfall characteristics (Kaluli of Papua New Guinea).  Each of this samples comes down to understanding one domain (sound) through the use of another domain.  The idea of mapping constraints in a way that is unexpected within the modes absorbed by culture could provide interesting gestural results for an alternate controller, although this would create an interface with a steeper learning curve.


The Xbox Controller

While the Xbox controller wasn’t designed to be used as a musical instrument it’s flexibility and control precision allows for effective musical control.  One of the benefits of using a predesigned control such as this is that there has already been extensive research and quality control placed in the design.  Microsoft official figures for the budget aren’t openly available however in interviews the research costs for the control reach hundreds of millions, with over two-hundred design prototypes.  They also mention (admittedly as part of the publicity) that the controller is refined “to the closest tenth of a millimeter to ensure the best fit and comfort in gamers’ hands”. The controller is inherently designed to be versatile to allow for a large variety of gameplay as well as being comfortable and ergonomic over extended use.

As has been previously discussed a different control scheme such as this also encourages new performative actions and musical choices and controls that wouldn’t necessarily occur on a keyboard.  For these reasons the Xbox controller is a reasonable choice for a musical interface providing high levels of quality control, within what is essentially a highly developed system to control parameters in a computer program.




Original Mouse Based Interface

The first patch I created explored using the mouse position on the screen to dictate the pitch and speed.  This information would then be performed on a virtual guitar, piano, drum and bass.   This piece was mainly used to explore self generating material that could be altered through a simple interface.  From this patch I learnt that even though the controller allowed significant changes it didn’t feel particularly unified.  Having so much free choice over the speed meant that outputs from the patch often sounded unintentional.

The example below shows the setup/controller as a working concept as opposed to being a final piece.



Irvine Jazz

This patch used the xbox controller to perform as a jazz small ensemble all within the one control setting.  I think the final output was workable as it did sound like a small jazz ensemble performing and generating material while being controlled by an Xbox controller.

As a use of the Xbox controller I think it was flawed, as the design process involved first knowing parameters and then applying these to the controller.  From my research I firmly came to decide that the controller (to act as an instrument) should dictate how it functions and not be used to just control parameters.

The two examples below show the setup/controller as a working concept as opposed to being a final piece.



Xbox Performer

Below are some recordings made in June 2014, of myself playing an xbox 360 controller and Anna Okunev.   As a working concept we explore the various roles of each instrument in the duo.  We plan to continue this concept much further and see this as the starting point for future works between the xbox controller and violin.

Richard Savery – Xbox 360 Controller (Left channel)
Anna Okunev – 5 String Electric Violin and Pedals (Right channel)



This concept was an attempt to utilise the Xbox controller in a musical way; that allows significant interaction with other performers.  Its design was geared towards high levels of flexibility with the intention to be used in particular within improvisations.  I also attempted to use the controller in a somewhat standard way, there are no odd combinations of buttons, it’s use fits patterns often used by the Xbox.


To perform with this setup a certain level of knowledge is required, buttons function differently at various times and the interface doesn’t really provide user feedback to help navigate through the options.  I intentionally included this idea, while ease of use is important, I felt forcing a level of virtuosity onto the performer was worthwhile.  A level of virtuosity is needed to understand how each section operates, as well the general principles of synthesis that are utilised.  In particular an understanding of frequency modulation is required to avoid having every sample sound the same; it is very easy to get an array of crazy electronic sounds that aren’t necessarily musical.


The synthesis modes were chosen to create a diverse range of varied effects. I use frequency modulation to create shorter notes (up to one second) and a combination of noise and sine tone’s for the other methods (explained in more detail below).

Aside from using the project as a method to develop an alternate interface I also felt as  though the interface was an effective method to apply realtime frequency modulation.  In addition to three synthesis modes – each one allows rhythmic patterns to be looped – the performer is given control of the global tempo.  This can be used to speed up or slow down the rate of each rhythmic pattern.


It is possible to have an unlimited number of different sound creations controlled by the Xbox.  I choose to limit it to three different synthesis options as within these choices there already is the ability to create a diverse amount of sounds.   Further methods could create an overwhelming array of sounds that will overcomplicate the sound base of the controller.


Visual Aesthetic
I decided to add in a display to create gestural associations with sounds and to display part of the inner workings of the control scheme.  I kept the display limited to not distract the audience, but enough that certain changes in the display would come to be associated with controls during the performance.  I imagined the display acting as the second body of the instrument (such as the piano body, and the controller represents the keys).  In an ideal performance the performer would be part of the projection, so standing in the middle surrounded by the moving components and may have a enlarged display showing the movements of the controller itself.



Xbox Performer – Instructions

The xbox controller operates around four different modes of control, chosen by the D-pad; left, right, up and down.  Left, right and down each allow a different form of sound synthesis while up controls the global tempo.  The choices that are made are then  represented visually through the GUI.


Button layout of a wireless Xbox 360 controller


The first action (assuming the controller is connected and the Dac on) is to press start.  This will then allow the D-pad to choose which synthesis method (left, right, up and down).  Once this is chosen, up and down on the D-pad allow you to choose which voice within left and right you wish to target.


For example to choose the right method and target the second voice you would:
-press start
-choose right on the D-pad (chooses right voice)
-press up on the D-pad (moves to 2nd voice)

The first voice is automatically chosen if no preference is given.

At any time within the piece you can move between each different synthesis method by pressing the start button.


Control Scheme
X = Noise
Y = Sine
B = Turns on loop
A = Play Note

Right Trigger = Function shape
Right Stick (left and right) = moves filter gradually
Right Stick (up and down) = moves filter in larger increments (toggled on and off with right bumper)
Left Bumper with Right Joystick (up and down) = changes Q of filter
Left Stick (left and right) = moves pitch gradually
Left Stick (up and down) = moves pitch in larger increments

This synthesis combines white noise with a sine tone (either sound can also be used alone).  The function shape effects both the sine tone and noise and extends up to 10 seconds in length.  You have control of the pitch of the sine tone, and the filter settings on the noise (Q and frequency).   Each sample can be up to 10 seconds long.

This part can be looped, and has 6 different potential voices.  Each loop is 8 bars long, as based on the global tempo.


Control Scheme
B = Turns on loop
A = Play Note

Left Trigger = Function shape of Modulation
Right Trigger = Function shape of Amplitude
Right Stick (left and right) = moves harmonicity gradually
Right Stick (up and down) = moves harmonicity in larger increments
Left Stick (left and right) = moves carrier frequency gradually
Left Stick (up and down) = moves carrier frequency in larger increments

This section of the controller uses Frequency Modulation and allows control over modulation, amplitude, carrier frequency and harmonicity.  Each sample can be up to 2 seconds long.

This part can be looped, and has 6 different potential voices.  Each loop is 8 bars long, as based on the global tempo.


X = Noise
Y = Sound
A = turns section on or off

Left Joystick (all directions) = controls noise filter
Top Right Bumper = turns on/off joystick (allows you to save a position)
D-pad (left/right) = chooses speed, ranging from a whole note to 16th note.

This synthesis acts differently from right and left, and while allowing a strong element of control, also leaves some sections unchangeable by the user.  It uses a phasor connected to a sine tone to create a sequence of pitches that sound somewhat randomly generated.  The pulse is linked to the global tempo and the speed can be changed; in combination these two controls allow for a lot of variety.  The noise and sine tone can be used individually or in unison.


Control Scheme
D-pad (left and right) – Set global tempo

The global tempo alters the speed at which loops will be played back and the speed of Down.  Depending on what is happening this may or may not cause a significant impact on the sound.


Common Information
Inlet 1 a is global on/off
Outlet 2 sends current voice number and a 1 or 0 for its status (for example 6 0)



The Display


click image to enlarge

Artwork by Mads Peitersen

The display is based around the same principle, of left, right, down and up.

The upper left hand side shows the filter and amplitude function of left, the number represents the frequency of the sine tone, and the images below represent which voices are currently being looped (in this case 2, 4 and 5).

The upper right hand side shows the modulation (left) and amplitude (right) function of right, the left number represents the frequency, the right number the harmonicity and the images below represent which voices are currently being looped (in this case 2, 4 and 5).

The two numbers at the bottom represent the tempo and the speed of down.


Possible Future Additions

I theoretically could add a large number of future synthesis options. I think a couple more could be added in the future without clouding the output.

I’d like to add further computer input, so some sections will work as small algorithmic composers.

Overall I’m happy with this as a basis for a control scheme, although I could add pitch memory for the loops.  I purposely chose to exclude this to avoid adding too many distracting textures, but it may be worthwhile in the future.







Cope, David Virtual Music: computer synthesis of musical style Massachusetts Institute of Technology, 2001

Dodge, Charles and Jerse, Thomas A. Computer Music : Synthesis, Composition, and Performance, 2nd ed. New York: Schirmer Books, 1997

Manning, Peter Electronic and Computer Music, 4th ed, Oxford University Press, 2013

Rodgers, Tara Pink Noises: Women on Electronic Music and Sound Duke University Press, 2013

Rowe, Robert Interactive Music Systems, MIT Press, 1993

Rowe, Robert Machine Musicianship, MIT Press, 2001

Temperley, David Music and Probability MIT Press, 2007

Zbikowski, Lawrence M. Conceptualizing Music: Cognitive Structure, Theory, and Analysis Oxford University Press, 2005


Sonami, Laetitia, Lady’s Glove (Retrieved June 2010),

Xbox Wire Staff, The New Generation Xbox Controller (posted June 6, 2013)

Crossley, Rob The $100M Story Behind The Xbox One Controller, (posted on Aug 12th, 2013)


Leave a comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>