The Conductor Program Página 1 de 33

Home | Site Map | Watch | FAQ | History | Store | Contact

The Conductor Program — computer-mediated musical

performance
by Stephen Malinowski

Everybody is an electronic musician. Some of us push only one button in a performance, and some of us push many buttons.
— Vladimir Ussachevsky

Introduction
There are many ways people share the responsibility of making music: they can collaborate in composing and performing music, a
composer can leave performing to other musicians, a group of musicians can hire a conductor, a performer can rely on an instrument
maker to build a machine capable of turning musical gestures into sound. A less commonly used way to share musical responsibility is
for a machine to be responsible for storing the notes of the score and playing back those notes in the proper order, leaving a human
performer with only the responsibilities of the conductor — rhythm, tempo, dynamics, expression, interpretation. With the advent of
computers and computer music, development of such an approach (which, when implemented with computers, is sometimes called
the conductor program) has taken off, with several people experimenting with its possibilities. In this article, I describe some of this

work, including my own implementation of the conductor program, Tapper (which is available as freeware ). My goal is for
the article to provide an introduction to the idea of the conductor program and for Tapper to provide an introduction to the experience of
doing computer-mediated musical performance. Neither is intended to be comprehensive, but both are works in progress, and I'm open to
suggestions for simplifications, corrections and additions.

Concepts
First let's go over some of the concepts. What is going on when a performer is using the conductor program? How is using the conductor
program different from other things musicians commonly do? How is one implementation of the conductor program different from
another? There are a lots of ways to think about these questions. You can ask: What kinds of choices are musicians making? What roles
do the computer and human performers play? What aspects of musical expression are performers responsible for? Where does the
acoustic energy come from? Where does the performance energy come from? How tightly is the performance coupled to the performer's
gestures? What is the relationship between pitch choice, pitch memory, and pitch responsibility? Is a person using the conductor
program really performing? ...

Choices

Considered abstractly, the responsibility of a musician is to make (and act upon) three types of choice about the allocation of acoustic
energy

z where in the frequency spectrum does the energy go (frequency )?

z when in time does the energy go there (time )?
z and how much of it goes there (dynamics)?

These three types of choice are seldom considered separately, and they can interact in lots of ways:

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 2 de 33

All musicians make choices about time, frequency and dynamics, but musicians in different roles are responsible for different kinds of
choices; these fall somewhat loosely into four groups:

z compositional choices — choices typically made by a composer, improviser or arranger, less often by a performer, and seldom or
never by a conductor (these decisions are typically not made by the user of the conductor program, either)
z choices of how notes are distributed (distribution) among performers, choices which are typically not made in real-time, and so are
mostly made by composers and arrangers
z broad expressive (or interpretive) choices — choices that all musicians make frequently; however, performing musicians make
these choices in a much more specific way: a composer can indicate accelerando in the score, but it is the performer who has the
final say on exactly how fast the music will go
z fine expressive choices, usually only made by live performers: subtle changes in pitch, timbre, timing, and dynamics (which are
typically not notated, and therefore are not communicated from composer to performer through the score, but are instead passed
among performers as part of their style of playing)

Which choices are made by musicians in which roles? This is summarized in the following table.

For each group, a given type of choice is colored according to when whether the musician in that role is responsible for that decision
always, usually, sometimes, rarely , or never:

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 3 de 33

As you can see, choices made by the user of the conductor program fall somewhere in between those made by a conductor and those
made by a performer. A live performer can make real-time choices about pitches; the user of the conductor program cannot, and is in that
way more like a conductor. A conductor cannot control the fine details of dynamics, rhythmic microstructure and articulation in real
time; the user of the conductor program can, and is in that way more like a live performer. Of course, the extent of the effect of these
choices varies from one implementation of the conductor program to another. For example, if the conductor program is controlling a pipe
organ synthesizer, the performer will not have any control of expressive timbre. Or, if the implementation of the conductor program is
only tracking the user's beats, then the user will have little or no control over the music's rhythmic microstructure.

Energy

In addition to the responsibility for making musical choices, there's responsibility for (or, at least, the labor of) doing the work to make
the music sounds. All musical performers make use of "sound reinforcement technology" in one form or another (if only in that their
instrument has a resonator), but there's always the question: who's doing the work of creating the sound, who is supplying the acoustic
energy? A singer, or the player of any non-amplified instrument is doing all the work (and, in addition to supplying the acoustic
energy, brass players and singers are also responsible for supplying the primary vibrating object — lips and larynx — singers also
supply a primary resonator). An organist isn't doing any of the work of creating the sound (except perhaps for the sound of feet hitting
the pedals, or the sound of keywork on a tracker organ), nor are conductors.

Apart from creating the sound energy, there's the energy required to control it. An organist may be creating none of the acoustic energy,
but the physical demands of playing the organ (requiring activity of all four limbs) are certainly as great as those of any other performer.
A musical performer is, in some sense, an athlete, and a conductor is perhaps the most athletic of musical performers — there's no
question that a conductor is doing physical work. We might call this performance energy — energy required to control and modulate the
sounds.

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 4 de 33

Like organists or conductors, users of the conductor program are supplying none of the acoustic energy (at least, not in any modern
implementations), but they are supplying a certain amount of performance energy. Would it be advantageous for them to supply more?
If you've done the work, have you performed the piece? If you haven't done any work, have you not performed the piece? I don't think
there's a single answer to this ...

Coupling

For any choice a musician makes, there's the question: how tightly is the performer's choice coupled to the sound that results? A singer's
interpretive ideas can expressed immediately as sound, with a minimum of intermediate steps (just neurons and muscles); a singer is
tightly coupled to the sound. A pianist's ideas take longer to effect the sound, because motions of the pianist's fingers must move the keys
of the piano, which move the action, which moves the hammers, which move the strings. A conductor's interpretive ideas must
additionally go through another performer, and a composer's ideas ... well, you get the idea. One way to quantify the tightness of the
coupling is by the minimum time delay that's introduced; these are something like this (just my back-of-the-envelope approximations for
now):

I've put in two extreme examples of the conductor program: my Tapper and Max Mathews' Radio Baton. Tapper can have a minimum
coupling time that's faster than a piano because of two factors: a synthetic piano can respond faster than a real piano, and Tapper doesn't
require the performer to move her hands as far. The coupling delay I've given for a composer is somewhat arbitrary, based only on
personal experience (it takes a couple of minutes to make a truly compositional change to a piece, write it down, get the parts in front of
the musicians, and have them play it). There are, of course, contexts in which a composer is working closely with the performers, and
can say "make that an e-flat" and have a coupling delay that's only a few seconds ... but it's hard to say that they're composing, exactly ...

You can see that, in terms of coupling delay, using the conductor program is similar to other kinds of music performance.

Pitch

All musicians need to know about pitch, but musicians in different roles have different areas of pitch-related responsibility. A composer
typically decides what the pitches (at least, some of the pitches) will be. A performer needs to know what the pitches are, either by sight-
reading during the performance or by memory. Does a conductor need to know what all the pitches are? The answer to that depends on
the breadth of the conductor's role. If the conductor is merely beating time to keep the performers, a detailed knowledge of the score is
not necessary, but to the extent that the conductor is responsible for interpreting the music, she should be familiar with the score; after all,
how can you know what the notes mean if you don't know what they are?

Beyond knowing what the pitches are, there's the matter of keeping track of which ones are about to occur and causing the corresponding
notes to come into being. Learning how to do this is a big part of what musicians work on when they are developing technique and
practicing — and with good reason: if you can't produce the right note at the right time, the other estimable features of your performance
will be hard to appreciate.

A conductor or the user of the conductor program is typically not responsible for causing pitches to come into being. There are a lot of
ways to look at this. If there is virtue in being responsible for something, then conductors are less virtuous than other performers, but if
effort spent being responsible (and learning how to be responsible) for producing notes of the correct pitches is unnecessary (or better
spent on some other aspect of musical performance), then conductors are spending their time better, which is perhaps also a kind of
virtue. If you're responsible for producing the notes, then you are certain to know what the pitches are, likely to know the piece better,
and thus likely to be in a better position to understand it and perform it expressively. This is not guaranteed, however (as audiences
who've listened to renditions performed by memory that were nonetheless dull will attest).

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 5 de 33

Technique

Related to the skill required to remember a sequence of pitches and render them as notes is the ability to subtly modulate the dynamic
level, pitch, timbre, and articulation of those notes. This kind of practice is intimately tied to musical interpretation; as a performer is
learning a piece, he will be having ideas about how the notes should sound, and will be developing and refining the techniques that allow
them to sound that way. To the extent that an implementation of the conductor program allows the user to control these aspects of the
performance, the expressive technical side of the conductor program user's practice will be similar to that of a "normal" performer.

Implementations
Next, let's examine how these considerations come into play in various implementations of the conductor program (and its precursors) ...

ca. 1800 Music boxes

Machines capable of controlling the playback of music date back to the middle ages, when pinned barrel mechanisms driven by pre-
wound clockwork were used to control pipe organs (and later, harmoniums and the plucked-comb instruments we now think of as "music
boxes"). One might dismiss such devices as being outside the realm of “artificially assisted performance,” but I think that’s a mistake: the
decision that "now, there will be music" is a musical decision; causing musical sounds to come into existence by force of will is
inherently gratifying.

The 1870s saw the introduction of hand-cranked, paper-roll-controlled "organettes." These offered the player control of tempo and, in
some cases, volume. The device shown above is a pin-barrel organette.

ca. 1900 Player piano

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 6 de 33

The player piano took the idea of expressive control of a mechanically-reproduced performance much further than the music box. It's
been long enough since player pianos were replaced by phonographs that few people remember the details of how a player piano was
used. It wasn't just a matter of pumping the pedals (though this was demanding enough to give you the feeling that you'd "done the
work") — there were also buttons for applying accents and levers for controlling tempo and dynamics. These are described in the 1925
"On Playing the 'Pianola'", by Reginald Reynolds, and demonstrated by Julian Dyer here. Some rolls had indications of when an accent
should be applied (in addition to lyrics for sing-along). These controls worked pneumatically, so the coupling wasn't as tight as it would
be if you were playing the piano directly, but it was still tight enough that you felt you were controlling the instrument, and responsible
for what it was doing.

1973 Strangio

The expressive capabilities of the music box and player piano were not extended until more than a half-century later, when Christopher
Strangio invented and patented his "COMPUTER-AIDED MUSICAL APPARATUS AND METHOD" (U.S. Patent 4,022,097), which
described a method of electronically storing data corresponding to notes of a musical composition so that a human performer could
subsequently trigger them and thus cause them to sound. His implementation (described in detail in his own words here) was hand-built
from discrete logic (remember, no PCs yet!) ...

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 7 de 33

... using a custom-modified reel-to-reel tape recorder to store a digital representation of the notes. The keys of a 1960's electronic organ
were the input device (five keys for each hand), and the organ's oscillators provided the output sound. Two separate paths of storage,
control and playback were provided, which allowed the score to be divided into to two parts, each of which could be independently
controlled by the performer.

In Strangio's implementation, individual key-presses on the organ keyboard were translated with a very short delay (less than 30
milliseconds) into individual notes, thus the coupling was very tight.

ca. 1975 Mathews

It's commonly believed that Max Mathews invented everything about electronic music, single-handedly, in the 1950s. While this is not
strictly true, it's not too much of an overstatement to say that he has been everywhere (Bell Labs, IRCAM, CCRMA...) and done a little
bit of everything (music synthesis, speech synthesis, human learning, acoustics, robotics, drawing, typography, conducting ...) related to
electronic music. Mathew's work on the conductor program began in response to Pierre Boulez's request for a method for controlling the
speed of playback of tape recordings. Mathews tried several kinds of interfaces, settling finally on a radio-frequency-based design (the
Radio Drum and Radio Baton) first developed by Bob Boie at Bell Labs (a version of which Mathews describes in U.S. patent
4,980,519).

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 8 de 33

In Mathews' implementations, the absolute position of the batons could be sensed in a 3D space extending a few feet above the receiver
(the rectangular object in these two photos). The positions of the batons could be interpreted in two different ways: their absolute
position could be mapped to various parameters, and information could be derived from their relative motion. So, for example, close and
far positions could correspond to loud and soft, while left and right would correspond to staccato and legato, with up and down motion
indicating the beats. Since there were two batons, each one could control a separate instrument or group of instruments, for example, the
string instruments and wind instruments of an orchestra.

Where Strangio followed a "performance" paradigm (one keypress equals one chord), Mathews' implementation was really a "conductor"
program; the performer was not responsible for individual notes per se; instead, there was a "conductor track," separate from the notes of
the pitch, which specified the positions of the beats the conductor was expected to indicate with gestures. Notes which fell between these
beats would sound according to the tempo established by the preceding beats. Thus, the coupling was fairly loose — the performer's
intention to change tempo had to wait until the next beat in the conductor track.

ca. 1985 Casio

Casio re-invented the conductor program idea and licensed its use from Strangio, including it in such devices as the PT-1 (which allowed
you to record a piece and then play it back by pressing either of the two blue "one button play" buttons at the right). Because it was
monophonic (with instruments that make early cell phone ring tones sound good), had a limited melodic range, and had a memory of only
100 notes, it was hard to take the PT-1 seriously as a musical instrument, and it and the other Casio devices incorporating the one button
play feature never made much of an impact. Still, I was intrigued when I saw one; I programmed the first 100 notes of Flight of the
Bumblebee into it then played it back as fast as I could wiggle two fingers ...

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 9 de 33

You can download this clip from Vimeo.

1987 Subotnick / Coniglio

Morton Subotnick began using computers for music during his 1985 residency at MIT. There, using David Levitt’s software program
Hookup (and with the assistance of Miller Puckette), he began developing algorithms that could follow the rhythmic gestures of a
conductor. He continued this work the following year at Cal Arts, where Mark Coniglio joined him, helping first with programming
(writing Macintosh versions of his Hookup algorithms) and later, with software design. The result of their collaboration eventually
became the Mac program Interactor.

In Subotnick’s Hunger (which premiered at the 1987 Los Angeles Festival), the conductor program participated in control of both the
musical composition itself and audio-visual effects synchronized with the music: camera zooms, lighting, equalization adjustments,
etc. In And the Butterflies Begin to Sing (premiered the Santa Fe Chamber Music Festival in 1988), the computer’s interaction with the
performers was extended to include modifications to pitches in response to the details of the pianist’s performance).

Subotnick’s went the furthest with the conductor program idea with a piece for orchestra and computer sound called A Desert Flowers
(premiered by the Cleveland Chamber Orchestra in 1989). In it, the conductor uses a MIDI baton built by Pat Downs called the Air
Drum. It measured movement in several directions; there were six movements of the wrist it could recognize. Interactor knew the overall
metric organization of the piece as well as specific notes in the piano and percussion parts and therefore could, like an orchestral player,
base its timing on both the conductor’s beats and the notes it was seeing in the MIDI from the piano and percussion players.

ca. 1988 Schloss

Andrew Schloss started using the Radio Drum at IRCAM, and has subsequently developed software to use with it and performed with it
a lot. His use of the Radio Drum extends beyond using it to control the expression of existing compositions into the realm of
improvisation. For example, a gesture with a baton could control the release of a certain set of pitches

1988 Malinowski

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 10 de 33

In the late 1980s, when I was working on the performance editor for the Music Animation Machine software, I was frustrated with the
amount of time it took to edit a passage to make it more expressive. Some pieces were easy enough that a recording of my own
performance needed only minor touch-ups, but others (especially, pieces for large numbers of instruments or ones with fast passagework)
needed to be recorded a part at a time or at a substantially reduced tempo; this resulted in passages in which the notes were right but the
fine structure of the dynamics, timing and articulation was not. Since the editor operated on one attribute (time, pitch, dynamic level) of
one note at a time, this editing was very laborious. It occurred to me that while I couldn't necessarily play a passage perfectly, I could
possibly play the rhythm more expressively, or do the accents more expressively — and that since the computer already knew the pitches,
I could add an editing mode in which a single attribute, that I would perform in real-time, would be applied to the notes already in the
score. I started experimenting with this by writing the first version of Tapper, a very simple DOS program with no user interface to
speak of. It was incredibly fun to play! — and since I was interested in patents at the time, I did a patent search for related ideas, which
led me to Christopher Strangio's work. We corresponded, and he became one of the first (and remains one of the most enthusiastic)
Tapper users.

A few years later, I incorporated the Tapper functionality into the Music Animation Machine software. Functionally, the current
version is the M.A.M. version (minus the rest of the M.A.M.), with modernized GUI. In it, the score is presented in bar-graph form, with
vertical lines indicating the positions of the beginnings of notes (and thus serving as a rudimentary type of rhythmic notation):

In Tapper, the gestures that control playback are single keystrokes on either a MIDI keyboard or the ASCII (typewriter) keyboard on the
computer. Notes are grouped into alternating sets of play groups (which are explicit and indicated in the display) and release groups
(which are implicit and not shown). Play groups are groups of note starts (MIDI NOTE ON events) that happen at the roughly same
time, for example, notes in a chord that is played on a certain beat. Release groups are all the notes ends (MIDI NOTE OFF events) that
happen between one play group and the following play group. When you press a key on the MIDI keyboard, Tapper plays any note ends
in the upcoming release group (there may be none) and all the note starts in the next play group. When you release a key and there are no
more keys pressed, Tapper ends all the notes in the next release group. The dynamic level of the notes being played depends on both the
score and the velocity of the MIDI keystroke keypress that triggers them: for each group of notes started by a given keypress, the loudest
note in that group will be played at the level dictated by the velocity of the keypress; the other notes in that group will have dynamic
levels that maintain their dynamic proportion to the loudest note. If you use the ASCII (computer) keyboard instead of a MIDI keyboard,
only the press actions happen (you have no explicit control of release groups except those between play groups or at the end of the piece),

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 11 de 33

and the dynamic levels are those in the score.

Notes and chords that come out of Tapper correspond one-to-one to MIDI events produced by the performer's actions and happen with
minimal delay; the coupling of Tapper is therefore very tight. This is a sword that cuts both ways: it offers the performer good control of
rhythmic, dynamic and articulatory nuance, but it requires that the performer know the rhythmic structure of the piece perfectly (or be
able to infer it from the display of the score); a rhythmic mistake often results in a musical catastrophe.

Here are two demonstration movies that show the kind of control Tapper gives the performer. First, a passage from Brahms' Capriccio,
opus 76 #2, played on Tapper without much expression:

Then, the same passage played with expression:

The first public Tapper performance was by Liza Dalby in October, 1994, at an all-Brahms concert. She played the G-minor Rhapsody,
Opus 79, #2 (there were no catastrophes).

Some thoughts on the possible future development of Tapper were stimulated by an email from David Christian.

1989 Friedman

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 12 de 33

Mathematician and pianist Harvey Friedman independently invented the conductor program in the late 1980s, and released his Macintosh
implementation of it, Instant Pleasure, under license from Strangio, in 1992. Herewith, a brief tour of the user interface of the software:

Clicking on the Instant Pleasure icon

takes you to the copyright screen,

and thence to the file selection screen

Instant Pleasure supported both standard MIDI files and files in its own proprietary format. When files in the proprietary format were
used, some extended features of the software were available; for example chords could be controlled several ways:

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 13 de 33

Controlled meant that the performer had complete control of the chords, and had to be responsible for triggering every note in every
chord. Divided was similar to that, except that the chords and melodies had to be triggered by the hand to which they were assigned in
the score (allowing the player to do a rubato performance). Automatic meant that the chords were played automatically, in sync with the
performer (so, for example, if the performer stopped, the chords would stop too), and Independent meant that the performer was only
responsible for the melody, and the chords would be played independently (like an accompaniment, not necessarily in sync with the
performer if the performer deviated too far from the correct performance).

The control of legato could work two ways:

Controlled meant that the articulation was completely controlled by the performer; Automatic meant that the articulation from the score
would be used (that is, the performer would control the times the notes started, but the software would control when they ended).

The GUI (ah, Macintosh ...) could show either the rhythm ...

or the lyrics.

Instant Pleasure supported touch sensitive MIDI keyboards,

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 14 de 33

which meant that the user could control timing, phrasing and dynamics. The controllable "thresholds" were ...

Volume (this prevented keys which you brushed against accidentally from causing notes to be triggered), Time (notes played within this
time window would be considered "part of the same trigger" and wouldn't causes an extra note to play), and Chord (same as Time,
except for chords).

Instant Pleasure was briefly marketed commercially, and was reviewed in some music journals. Interestingly, various people have used it
in psychological studies (if you're curious about this, Google for variations on these search terms: "instant pleasure" "software program"
music study).

1990 Kawai

Around 1990, Friedman licensed his work to Kawai, who included it (still under the name "Instant Pleasure") in their GMega synthesizer
(shown here). Later, similar functionality was also included in some of their pianos, under the name Concert Magic. In these
implementations, the scores were limited to those supplied by Kawai, stored permanently in ROM (read-only memory).

1993 Zimmerman/Wantman

Thomas Zimmerman and Samuel Wantman invented and patented a system for applying performance gestures to a pre-stored score.
Zimmerman and Wantman dug into many of the juicy issues, and their U.S. patent 5,488,196 is a great read. Let me touch on a few of
the high points.

An important advance of this work beyond Strangio's is that of making a distinction between two aspects of the performer's
control: finger control (the type of control typified by wiggling your fingers, characterized by speed of change and precise timing) and
energy control (the type of control typified by gestures of the arms, or by breathing, characterized by slower, continuous change). In the
implementation of this that I saw a video of Zimmerman using, the interface was in the shape of a violin; the finger control was
composed of two push-buttons on the fingerboard, and the energy control was a bow, able to sense pressure, position, and velocity.

The place where Zimmerman and Wantman did what I consider the nuts and bolts of the conceptual work was in figuring out exactly how

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 15 de 33

to use these two types of data. Their approach goes like this: look at the two data strings in combination, and calculate a gesture output
and an expression output. The gesture output corresponds roughly to "when do the notes happen" and the expression output describes the
dynamic changes (that are free to be asynchronous to the gestures). For example, a change of finger would typically result in a gesture
output, but so could a change of bow direction. Then, convert the gesture output into note timings, and send them plus the expression
data to the synthesizer.

A second important piece of work by Z&W was to develop the method by which the gesture output gets converted into note timings. In
Mathews' implementation, this task was simplified by the fact that there was a conductor track for the software to follow, and the
conductor's gestures were expected to be well-defined and not too closely space. But if the performer's gestures can cover the gamut
from quick finger wiggles to large, less distinct motions of the player's limbs, the task of mapping gestures to notes is harder. Z&W's
design not only addresses this, but also allows for variability in the performer's level of expertise, rubato, multiple performers,
synchronizing an accompaniment, etc. Here's a block diagram from the patent:

One distinctive feature of this implementation is that allowed a degree of expressive control not found in any other implementation: if the
score contained a trill, the performer could trill as fast and for as long as desired, meaning that there wasn't a one-to-one correspondence
between notes in the score and notes that were produced. In this sense, the implementation allowed a small degree of freedom that might
be called improvisatory.

It was with a mixture of pleasure and dismay that I learned that the patent covers both the use of this technology for music "re-
performance" (the same application as Strangio, Mathews, et al.) and for editing (the idea which was the source of my interest in the
invention) — pleased because it meant that the idea is "out there" in the world; dismayed because I'd been planning to use it myself. Ah,
well — that's patents for you.

1996 Marrin-Nakra

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 16 de 33

Teresa Anne Marrin-Nakra's work focuses on the activities of a conductor, and her implementations of the conductor program follow the
conducting paradigm the most closely.

In her 1996 Master's thesis, Marrin-Nakra describes the Digital Baton (a device designed and implemented by her, Joseph Paradiso, Tod
Machover, Christopher Verplaestse and Margaret Orth), a device which records conductors' gestures (see U.S. Patent 5,875,257) for
details). The first version of this device contained accelerometers to detect the orientation, position and motion of the device, plus five
finger pressure sensors. A second version added optical position sensing (infrared light emitted by the baton was received by a detector
positioned a few meters away from the performer). The software that used the information from these sensors was fairly simple,
translating measurements (and triggers and threshold crossings based on those measurements) directly into control signals used in
interactive musical compositions; gestures per se were not extracted from the sensor data.

In the concluding sections of her master's thesis, Marrin-Nakra discusses possible directions for future research, many of which she
pursued subsequently. Extending the capabilities of the Digital Baton was her Conductor's Jacket, a device that measured a conductor's
muscle tension, breathing, heart rate, skin conductance, and temperature. The analysis of the data collected from the jacket was followed
by the development of a system which could extract expressive features (e.g. beats, tempi, pitch choices, accents) from the real-time
output of the jacket. This work was the subject of her PhD thesis.

These gestural interfaces have been used to control various types of performance, including the conductor program (seeYou're the
Conductor: A Realistic Interactive Conducting System for Children and YoureTheConductor.com for a description of one installation)
and other, more improvisatory compositions.

Marrin-Nakra went on to found Immersion Music, a nonprofit organization in the Boston area, to develop and present this kind of thing.

1999 Plank

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 17 de 33

James Plank's piano-oriented implementation of the conductor program, MusPlay (described in this article), is notable in that it allows a
broad range of levels of pianistic skill on the part of the performer. If you play the piece as written, MusPlay will give you almost as
much control as if you were playing it for real. If you play wrong notes, MusPlay will correct them. If you wiggle your fingers in
approximately the right places at the right times, the piece will play. If you do something completely unrelated to the piece, MusPlay will
make its best guess about how to map what you're doing onto the stored composition. To handle this range of styles of control, MusPlay
has elements of both tight and loose coupling. A keypress of the correct pitch at close to the time the program expects it will result in the
corresponding note sounding immediately, just as if it were being played. However, if the keypress doesn't correspond to a note in the
score, MusPlay will guess which note you meant, and if your timing varies a lot, it will guess about where you intend the beats to be. So,
the tightness of the coupling varies, depending on how close you are to MusPlay's prediction about what you're doing. Also notable is
that the encoding of the score includes special consideration for trills, allowing the player a degree of improvisatory control.

2003 Fallgatter

In 2003, James Fallgatter assembled a team (including James Plank and others) called Aqwertian Music to develop and market a
QWERTY keyboard-based version of the conductor program based on Plank's work. This implementation, called Music At Your
Fingertips, relies on the performer's ability to touch-type; its graphical user interface presents a piano-roll style display in which the notes
are assigned letters, which are typed to play the music. A cursor moves through the bars as they are being played, and waits when the
program is waiting for you to press the next key; therefore, you can infer the rhythm of the piece by how long it takes the cursor to move
through the note; if you're playing it at the correct tempo, the cursor will move smoothly. Here is the display for the first few measures of
Beethoven's Für Elise:

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 18 de 33

Plank's ideas (as used in his MusPlay) for how to convert keypresses to notes when the performer doesn't get it exactly right have been
incorporated into Aqwertian's software, and thus the coupling spans a range of tight to moderately loose.

2004 Chew

photo by brian morri

In the implementation of the conductor program conceived by pianist, engineer and professor Elaine Chew, the performer controls the
tempo and dynamic level of the music by driving through a virtual environment (here, with Chew at the wheel):

This implementation was developed at USC's Integrated Media Systems Center (IMSC) in collaboration with professor Alexandre
François (Chew's husband) and students Jie Liu and Aaron Yang as part of the Expression Synthesis Project (ESP), and is described in
more detail in this article from the Proceedings of the 2005 International Conference on New Interfaces for Musical Expression.
François was the designer of IMSC's Software Architecture for Immersipresence (SAI), which formed the basis for ESP's interactive
software, which Liu and Yang helped realize.

There are several notable innovations in this system. The interface takes advantage of the user's experience in driving a car in several
ways. First, the function of the accelerator (gas) and brake pedals is familiar. Second, the structure of the musical composition being
performed is reflected in the geometry of the road. So, for example, a passage where the music slowed down might correspond to a
stretch of road with sharper curves; the performer wouldn't be forced to slow down, but would be encouraged to. In this way, the terrain
is a kind of graphical notation for tempo. There's a third, more subtle concept involved here: tempo in music is related (in a way which
musicians understand intuitively but which has not been well described theoretically) with physical motion, especially motions involving
the whole body, such as gait and dance. When musicians make tempo changes, the rates of change are not arbitrary; experimenters' first
attempts to create accelerandos and ritardandos algorithmically resulted in tempo changes that did not sound musical or natural. So, by

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 19 de 33

explicitly mapping tempo to physical motion, Chew's implementation takes advantage of the performer's experience of moving physically
in the world.

Another innovation in Chew's implementation is that tempo and dynamic level are controlled by the speed and acceleration of the virtual
car. This works because changes in dynamic level are well-enough correlated with changes in tempo that one can be used to estimate the
other.

Chart of Comparison

There are almost as many types of interface to the conductor program as there are implementations. Strangio's version used Push-
Buttons, as did Casio's (though Strangio's controlled two independent streams of music and Casio's only controlled one). Max Mathews'
Radio Drum and Radio Baton measure position in space, as does Teresa Marrin-Nakra's; hers also measured orientation; she also
developed a Jacket which measures many aspects of the performer's position and muscle tension. Plank, Friedman and I all developed
versions using a MIDI Piano Keyboard, and Fallgatter and I also developed versions using the ASCII Typewriter Keyboard. My
Lizard Trampoline explored the feel of a Flexible Surface (and I'm hoping that somebody will develop a version using an interface like
this one). Zimmerman and Wantman developed a Violin-Style interface which generated very different kinds of control from the fingers
and the bow. A while back, there was a device called the Video Harp which showed promise (and I believe they did a conductor-
program-like thing with it, but I've yet to dig up the details), and there have been various approaches to Sensing Hand and Body
Position, including computer image tracking, radio, and capacitance. Just as a violin is a very different instrument than a piano, each
implementation of the conductor program is its own thing. A few important differences between implementations are summarized in this
table:

implementation coupling note tempo dynamics articulation pedal tracks interface improv score
music box loose no yes no no n/a 1 crank no graph
pump,
buttons
player piano loose no yes yes no yes 1 no graph
and lever
(s)
Strangio
U.S.Pat. tight yes yes no yes? n/a 2 buttons no no
4,022,097
Mathews
loose no yes yes no n/a 1 baton(s) no no
Radio Baton
Casio
tight yes yes no ? n/a 1 buttons no no
PT-1
Schloss ? ? yes ? ? n/a 1 ? yes no
Malinowski
medium no yes no no no 1 crank no graph
Cranker
MIDI
Malinowski (keyboard)
tight yes yes yes yes yes 1 or 2 no graph
Tapper or ASCII
keyboard
MIDI
Friedman (keyboard)
tight yes yes yes yes ? 1? no symbol
Instant Pleasure or ASCII
keyboards
Zimmerman /
variable yes yes yes yes n/a 1 violin-like trills no
Wantman
Marrin-Nakra
baton or
Conductor's loose no yes yes no n/a 1 no no
jacket
Jacket
Plank MIDI
medium yes yes yes yes yes? multi? trills graph
MusPlay (keyboard)
Fallgatter
ASCII or graph/
Music At Your tight yes yes no yes yes 1 no
MIDI symbol
Fingertips
Chew steering,
loose no yes* yes* no no :-) 1 no road
ESP gas, brake

*In Chew's implementation, tempo and dynamics are controlled in tandem (not independently).

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 20 de 33

The only feature that all implementations share is that they allow you to control the tempo of the music.

Applications
"Okay, so it's an interesting idea. But what's it good for?" A fair question, and one for which I have only a partial answer.

Personally, the main way I've used Tapper is the same way I use a piano: to play music for my own enjoyment. I can play the piano
pretty well without computer assistance, and Tapper has many limitations, so I tend to use it for pieces that run afoul of my limitations
more than they do Tapper's. At the top of the list are pieces that are not written for the piano. I've gotten a great amount of satisfaction
playing movements from Bach's Brandenburg Concerti and organ music. For pure thrill, the two pieces tied for first place are the first
movement of Mozart's sonata for two pianos (K. 448) and Sousa's The Stars and Stripes Forever.

Then, there are pieces which are written for piano, but which present technical challenges that I'd rather avoid. I played Grieg's Butterfly
before I ever used Tapper, and thought I did a pretty decent job with it, but with Tapper, my interpretation changed completely —
especially the rubato. When I play it "for real" now, it is much closer to how I play it with Tapper, but never quite as fluid. I could play
some of Chopin's etudes, but they are a lot of work, and when I want to play them at the tempo I want (rather than what I can manage),
out comes Tapper. Occasionally, a piano piece presents a combination of articulatory and technical problems, such that I can get the
notes right, or the articulation, but not always both. One of these is Brahms' opus 76 #2 Capriccio which, with Tapper, I can play with
the right kind of playful delicacy (as shown in the movie above).

Harvey Friedman named his implementation of the conductor program Instant Pleasure. I wouldn't say that my pleasure with Tapper has
been instant — I still have to practice pieces to play them well with Tapper. But it is certainly pleasurable. How does it compare to
playing music on the piano "for real"? All things being equal, and with the current implementation of Tapper, I'd prefer to play a piece
on the piano. But all things are seldom equal, and there's a wide range of overlap between the amount of pleasure I get from playing the
piano and from playing Tapper. At its best, playing a piece on Tapper is completely engaging, thrilling, expressive — everything you'd
want, and is certainly preferable to playing something badly on the piano. However, a piano sound coming out of a loudspeaker is not the
same as a piano sound coming out of a piano, tapping keys on a MIDI piano keyboard is not as gratifying as playing a real piano, and
Tapper doesn't give you as much control as a real piano does, so with the present implementation, it's at a disadvantage.

Besides pleasure and gaining new insights into pieces, there is another use I have for Tapper: when I want to make renditions of pieces to
use as a basis for music visualization, I sometimes use a combination of Tapper performing and manual editing. This was how I made the
rendition of Bach's Toccata and Fugue in D-minor that's on the Second Demonstration Reel: I started with a version of the score that had
the trills removed, which I performed with Tapper; then I added the trills and cleaned up a few places where I'd slipped up. You can
listen to (and watch) the result here. If the conductor program were incorporated more fully into a performance editing tool, this
approach could be extended by alternating between "by-hand" editing (e.g. changing a single note timing/dynamic/articulation by
dragging it with a mouse) and "re-perform" editing in which some attributes of a new performance through the conductor program would
be applied to the pitches in the score (this has been done to some extent in the "NTEMPO" feature of Virtuoso Works' NOTION
software). Another piece I used the conductor program with is Debussy's Clair de lune (listen/watch here ); this piece fit the Tapper
paradigm well, so there was very little editing after the fact.

Other people have used the conductor program for pleasure in the way I have. Some of these people have enough technical skill on "real"
instruments that the conductor program is a means to extend their musical experience into areas that ability or practice time would
otherwise keep them from entering. For some, Tapper is their preferred way of playing music. Age is not a critical factor — James
Plank's 4-year-old daughter learned to play his MusPlay software before learning a "real" instrument. Some have used it in public
performances.

There are some uses of the conductor program which seem likely to be effective, but which I don't have any experience, either direct or
indirect. For example, it seems likely that a singer might accompany herself by playing the piano part to a song with the conductor
program. Song accompaniments often fall in the category of "music I'd rather not have to spend time practicing," so pianists too might
want to use the conductor program (especially in rehearsals). And I'd certainly like to try playing the piano part to Schubert's Der
Erlkönig with the conductor program — my right hand always runs out of steam.

I showed Tapper to guitarist James Edwards . He told me that because it's so hard to learn guitar music, guitar students have heard
themselves play a piece badly, haltingly, and with poor interpretation for a long time before they have the control to play it expressively
— by which time they are no longer able to imagine the piece sounding any way other than how they've heard themselves play it. He
said he'd like his students, before learning a piece on the guitar, to first learn it with the conductor program, so that they could develop
their expressive concept of the piece and have that firmly in mind while they were practicing.

There are also some more prosaic uses for the conductor program. For example, if you're studying a piece, it can be useful to step
through it at a reduced tempo to hear what's going on more clearly. The step-through feature can also be used when proofreading
(proofhearing?) a score.

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 21 de 33

Discussion
The following section is an unsorted catch-all for subjects worth mentioning but for which my thoughts are too under-developed to fit
into this article in a more structured way.

Incidental or Essential

Musicians have to learn lots of things. Some of these things are directly related to musicianship: being able to do rhythmic gestures, to
recognize what's going on in music, to perform while simultaneously listening closely to other musicians performing, etc. Some,
though, are incidental: the player of a modern flute has to learn very different fingerings than the player of a baroque flute, yet a singer
doesn't have to learn fingerings at all; a singer has to learn how to produce a pleasing tone but an organist does not, etc. The user of the
conductor program is relieved of some responsibilities that other musicians must bear. To the extent that these responsibilities are
incidental or unnecessary, their reduction is a good thing, but if they are things that put the performer into more direct contact with the
essential aspects of music, then something is being lost. Which responsibilities are essential and which are incidental? I'm sure there is a
lot of difference of opinion about this; personally, I believe that ...

... it is incidental that musicians have

... but that it is essential that they learn ...
to learn ...
...scales. Every pitch is unique; every note is
...fingerings. Because there are seven
different from every other note. The quality of
pitches in diatonic scale but only five fingers
a particular pitch, the distance between one
on the human hand, most instrumentalists are
scale degree and another — these things are as
saddled with the chore of learning how to
basic to a musician as knowing what blue
associate finger motions with notes in a scale.
looks like is to a painter. You can't know
This task is multiplied by twelve because a
about music and not know where the notes are,
scale can start on any of the available
how they relate to one another, and what they
chromatic pitches, and again some other
sound like. Also, it's essential for musicians to
number to include all the scale variants
be able to articulate fast notes with ease and
(major, various forms of minor, modal, blues,
control It's reasonable to try to "design out"
whole-tone, etc.). Keyboard players must
some of the more onerous manual demands of
additionally learn how to play with both
musical instruments, but it is not possible to
hands, how to alternate fingers in such a way
eliminate the requirement that a musician be
that there is always a finger available at the
familiar with the elements of musical
right time, and how to play multiple melodies
composition, or keep track of notes happening
with one hand.
too fast to conceive of individually.
...how to play chords and arpeggios. As
with scales, there are lots of different types of
chords, and each type of chord can start on
any pitch, so there are lots of them to learn.
...harmony. Just as every pitch is unique, so is
Moreover, chords and arpeggios are more
every chord and every chord progression.
difficult than scales because there are more
varieties of distance between chord notes, and
on most instruments, moving by a scale step is
easier than moving by a larger interval.
...physical motion that corresponds to
...physical motions peculiar to a particular musical motion. Music is all about motion:
instrument. Have you ever noticed what a melodic motion, harmonic motion,
bassoonist has to do with her thumb? It's rhythmic motion. We are sensitive to motion
pretty unnatural. Organ players need to learn in music precisely because we are gestural
to play melodies with their feet (and with one creatures; we understand musical gestures
foot while controlling a swell pedal with the innately. A musician's body knows all about
other foot) and to push and pull stop pistons gesture; if the musician can learn to transfer
without interrupting their player. Even that knowledge into sound, all will be well. A
something as simple as turning pages (which musical instrument that requires a musician to
singers can do without much special training) do a smooth gesture to produce a smooth
can be elevated to an acrobatic feat when sound is requiring that the musician have the
combined with the requirement that you not feeling of a smooth gesture in her body; this
interrupt the music and have enough of the feeling, and the resulting gesture, reify the
score memorized that you don't have to be musical idea. A brain without a body would
looking at it while the page is turned. Not to make a very inhuman music! Dancers, singers,
mention the demands of embouchure, holding and conductors have a certain advantage over
an instrument under your chin, etc... instrumentalist; their physical/musical
demands are inherently more natural.

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 22 de 33

...ultra-precise finger positions. On a violin,

a difference of a few hundredths of an inch
can be the difference between beauty and
ugliness, between pleasure and pain, between
Heifetz and ... well, me. Keyboard playing is
less demanding, but accuracy is still an issue:
you can have your hand in the right position,
have the right finger ready at the right time,
and bring it down with the right velocity and ...intonational nuance.
the right amount of muscular support behind
it, but if it falls quarter of an inch too far to the
left or right, the result can be anything from
the note being too loud or soft, to the wrong
note being played; and even if the note still
happens to sound correctly, the sensation of
contacting a key in a manner other than what
you expected is a distraction.
...awkward breathing. The oboe requires ...breathing. Pianists and players of string
such a small volume of air that the air in an instruments don't have to breathe in any
oboist's lungs goes stale before it is used up; particular way to make sounds on their
the performer must therefore breathe out at the instruments, but musical phrasing is so closely
end of a phrase before breathing in for the next related to breathing that most musicians
phrase. This is an extreme example, but few breathe musically as part of their playing, and
wind instruments place no unusual demands would find it harder to express the music
on the player's lung capacity. properly if they were prevented from doing it.
...practicing to modulate the tone. What's
...practicing to produce a good tone. What the right way to move from one note to
determines whether the notes a performer another? On most instruments, you can tell
produces are, in themselves (that is, apart from whether a skilled musician or a beginner is
the musical context in which they appear), playing by listening to how they play a single
pleasing? For instrumentalists, the quality of note; and on most of the rest, you can tell from
the instrument plays a part. Practice of course two notes. A single note can be played in a
makes a big difference. For singers, it is wide variety of ways, as can the connection
something like physical beauty: there are between two notes. How a note should sound
things one can do, but some people are simply depends on context, and musicians must learn
more gifted. what those contexts are, and how to make the
notes sound the right way at the right time.
...physical endurance. Perhaps the most
...mental endurance. Even if a musician's
treacherous example of this is the brass
skills are so well-developed that playing music
player's embouchure. It takes years to develop
is physically effortless, keeping one's focus
a good embouchure, but a few days without
through a long piece of music, and keeping it
practice is enough for it to deteriorate, and a
well enough that the piece makes sense as a
few hours of playing can be enough to use it
whole, is a feat.
up for the day.
...finger patterns associated with playing ...controlling multiple parts in contrapuntal
contrapuntal keyboard music. I can play music. Attention is unary, but counterpoint
most of Bach's Well-Tempered Clavier involves multiple objects of attention. How
tolerably well, and I have to say that it is one can a performer do it? Playing contrapuntal
of the hardest things I've ever learned to do. music is something like juggling; though there
That would be okay, I supposed, except that are several balls, you're usually only throwing
learning it has very little to do with learning one at a time; the skill is to shift your attention
how to conceive of how to perform from one ball to another. And with
contrapuntal music. Rather, it is a collection contrapuntal music, you need to pay attention
of a thousand and one tricks, stratagems, to a musical line at the point where it is
heuristics and an equal number of only changing from doing one thing to doing
partially successful compromises. If I had six another, and switch your attention rapidly
hands, none of this would be necessary. enough to never "drop the ball."

How does the conductor program fare in this light? This varies significantly from one implementation to another. In general, compared
to real performing, the incidental responsibilities are fewer and some of the essential responsibilities are still present, but some are not.
It's not too difficult to imagine a way to make the conductor program more like real performing in any given aspect, but as
responsibilities are added, the difficulty increases, and the advantage over real performing decreases. A possible way out of this corner is
suggested by the way a keyboard player manages to do several things at once. Human attention is limited, and the number of things
going on when playing multi-part music exceeds this limit, so keyboard players learn to perform various actions without having to attend
to them continuously. This approach could be extended to the conductor program: just as a real conductor can reduce her effort when the
performers are doing the right thing without attention, a user of the conductor program could, for example, let the tempo go on "auto-

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 23 de 33

pilot" while attending to articulation, then control the balance, then punch up a few accents, then take control of the tempo, etc. In such a
scheme, the performer can chew as much as he cares to bite off.

Musical gesture

Musical gesture is related to physical gestures. With conventional musical instruments, the structure of music is related to the structure of
the gestures that are required to produce it — a big gestures make a bigger sound than a smaller gesture, a fast physical gesture is required
to make a fast musical gesture. Every instrument has its own mapping from gesture to sound; a quick motion with your right hand has a
very different effect depending on whether you're playing a violin, a trombone, a gong, or the Theremin. One reason to learn to play
more than one instrument is that each has its own lessons about gesture to teach. Singing involves gestures with the lungs that are not
integral to piano playing, but a pianist can learn a lot about phrasing by learning to sing expressively.

For conventional instruments, the gestures required to produce sounds are a compromise between what feels right to a performer and
what's necessary from an acoustical or mechanical standpoint. Presumably this compromise means that these gestures are not optimal for
the performer. When you put a computer between the performer and the sound-production apparatus, the instrument's acoustical and
mechanical constraints are removed, and any mapping between the performer's gestures and the musical result is, theoretically, possible
(though some mappings, like wiggling your tongue, might require an interface that has not yet been developed).

What do we want to do with this freedom? What are the optimal gestures for producing music? There are various ways to approach this
question. One way is very pragmatic and direct: ask people. What gesture would you like to do to make such-and-such a sound?
Another way is to study the gestural aspect of music, and develop a theory for how this relates to the human system of nerves, muscles,
bones, and masses. I believe that both approaches have merit.

Imperfect gestures

With a real instrument, an imperfect performance gesture can lead to an imperfect sound. How imperfect the sound is varies from
instrument to instrument — pressing a key too hard on an organ has relatively little effect, but having your finger position wrong by an
eighth of an inch when you're playing a high note on the violin can make the difference between bliss and blech. In general, though, the
greater the imperfection of the gestures, the greater the imperfection of the result. This is a good thing, because one performer's mistake
is another's ornament; letting imperfection through is a natural side-effect of letting expression through.

This issue comes up when designing an implementation of the conductor program: what kind of freedom does the performer have to
deviate from the most-likely gestures, and what effect do such deviations produce? You don't want a little mistake to turn into a big
disaster (like it does in Tapper). The art here is to figure out a good mapping between the range of possible gestures and the range of
possible outcomes. Making it impossible to make a sound you'd never want to produce might seem like one reasonable goal, but it begs
the question: what are those sounds? Where is the borderline between "I might sometimes want that sound" and "I will never want that
sound"? There probably isn't an answer to that.

A useful principle is: the more unlikely a sound is, the more effort it requires to produce it. Since what is likely and what is unlikely will
change from one context to another, a mapping that could change is probably a good idea ...

Ease vs. control

I began by discussing some of the ways musicians divide up responsibility for making and enjoying music. There are lots of different
ways, and they vary a lot in terms of ease of performance, ease of learning, quality of the result, pleasure in performance, pleasure in
listening, pleasure in learning, extra-musical pleasures, etc. Musical activities could be placed on a "spectrum of control" like this

z improvise in your own style (complete control)

z improvise in somebody else's style
z improvise on somebody else's compositions
z play somebody else's composition without improvising
z play a part of somebody else's composition (for example, with other performers, music minus one, karaoke)
z control some aspects of the performance of the whole or part but not others (as with the conductor program)
z control only your extra-musical response to hearing the music (dancing with it)
z just listen (no control)

In this spectrum, there are various "sweet spots" — mixes of freedom and involvement that are in some way optimal. Listening to music
and dancing to music are very popular, and certainly qualify as sweet spots. As you increase control and responsibility, the level of
gratification goes up, but so does the degree of (for want of a better term) work. The fact that the conductor program lies in the region
between a very popular type of involvement (dancing) and a more gratifying but more demanding type of involvement (performing for

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 24 de 33

real) suggests that it might have the potential for becoming a popular activity.

Keyboard as bow

As a pianist, my first approach to using a piano keyboard to control the conductor program (with Tapper) was to follow the melodies, to
do finger patterns that reminded me of the finger patterns I'd use if I were playing some part of the piece on the piano. Sometimes,
though, I would encounter a passage for which the first fingering that came to mind was awkward (or downright difficult), and I started
trying other finger patterns (the more effective of which are described here). What I found is that Tapper is not just a new use for a piano
keyboard; it's actually a new kind of instrument. The finger patterns, divorced from the constraint that they relate to pitches, become like
the finger patterns of a guitarist's right hand, or the violinist's bow. In fact, the feeling of playing Tapper reminds me, in a certain way, of
bowing. The most effective finger pattern for a passage is one which follows the phrasing. I've found that this way of approaching the
keyboard has transferred to my "normal" playing.

Improvisation

If the score is in the computer, what about improvisation? Is there any room for this? The answer is "yes, but with freedom comes
responsibility." And complexity. Of those described here, only the Zimmerman/Wantman implementation allows flexibility in which
pitches are played: if the score indicates a trill, the player controls how fast it goes and how long it lasts, and thus how many notes the trill
contains. Four things are necessary for this to work: the score must be encoded in such a way that the trill is represented as a trill (as
opposed to being individual notes), the software must be written to respond to a trill in the score differently than for untrilled notes, the
performer needs to know where in the score the trills are, and the performer needs to know what to do to make a trill happen. There are
other ways this could happen; for example, there could be a special gesture a performer could do that would cause a trill to happen
regardless of what was in the score. Or, the notes of the trill could be controlled by some other aspect of the performer's gestures, like
how fast they're playing or how loudly (though this takes it a little out of the realm of improvisation).

There are other simple types of improvisation that performers of notated music might want: the ability to roll chords at will, add extra
notes to chords, play a note or chord more or less often than written, play a chord with an unusual voicing (bringing out one note,
for example). The design issue is: how would these things be controlled? If you develop a system of special gestures that have special
effects, you reduce the amount of "gesture space" that's available for non-special effects.

A possible way around this would be for a performer to play some parts of the score with the conductor program (say, the
accompaniment) and other parts "for real." Such a feature could be selectable in real-time during the performance: for example, pushing
a pedal could put the right hand part into "for real" mode to allow the performer to ornament or improvise.

Another possibility is to divide the controller into separate sections: the "conducting" section and the "improvising" section.

Where does the conductor program end and some other type of application, a more improvisation-oriented one, begin? There's no clear
dividing line. A score could be represented not by pitches, but by some description of the piece that was looser (e.g. chords changes, or
tonalities, or melodic motives), with the performer's gestures mapped to pitches conforming to that description. At some point, we're not
really talking about "conducting" anymore ...

What musical notation to use?

An implementation of the conductor program may present information about what's in the score to the performer. Some of the
implementations described here do not present it at all (Strangio, Mathews, Casio, Schloss, Zimmerman/Wantman, Marrin-Nakra).
Friedman's implementation uses a symbolic notation for rhythms; mine, Plank's and Fallgatter's use variations on bar-graph (piano-roll)
notation. Conventional notation is an option, but some people want to use the conductor program because they don't know how to read
conventional notation, so it is perhaps not ideal.

No musical notation specifies everything about a performance; any notation is only an approximate indication of what is to be played, a
reminder. What is the most appropriate, most effective reminder for a user of the conductor program? There are a lot of ways to go with
this question. Could there be a notation that expressed elements of rhythmic microstructure ? Or the kinds of timbral, amplitude and
pitch modulations that are possible on string and wind instruments?

Sadly, the part of conventional notation that's most obscure to non-musicians, rhythm, is the part that would be most valuable to a user of
the conductor program. This suggests that a new, more graphical, more intuitive rhythmic notation might be appropriate. Conventional
notation evolved to satisfy many constraints, many of which are relaxed or removed in the context of the conductor program implemented
on a personal computer. For example:

z the conductor program is often responsible for the pitch, so the notation needn't be as pitch-centric

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 25 de 33

z the notation will be drawn by a computer (instead of a person), and therefore doesn't need to be easy to draw
z the notation can scroll by, so it can be much bigger
z color can be used more easily

One possibility is to make a rhythmic notation in which the rhythmic hierarchies are made more explicit, perhaps with a graphical
notation; for example, this rhythm

might be indicated something like this

to show how the measure is subdivided.

Master-class mode

Depending on the implementation, various amount of information about a composition's performance is stored in the score used by the
conductor program. If the performer is responsible for everything except pitches, only the pitches need to be stored; if the responsibility
for the performance is more equally shared between the performer and the computer, the stored score many also contain information
about dynamics, tempo changes, timbre, etc. This raises the question: how do these details get into the score? An approach which has
been discussed a fair amount (but not, as far as I know, implemented in any commercial software) is for the computer to learn these
details from the performer.

A possible scenario would go like this: the stored score would start out very straight, containing more or less what's in a conventional
score. Each time the performer conducted the piece, information about note timings, dynamics, articulation, etc. would be extracted from
the performer's gestures and added to the score as probabilities. These probabilities would shape the computer's playback during
subsequent performances. The degree to which this information is used would depend on the implementation: if the computer were only
responsible for playing notes falling between beats indicated by the performer, it would only adjust those notes, but if the paradigm were
more like real conducting, the computer could anticipate ritards (and not play notes too early), diminuendos, etc.

This approach could also be used to create renditions of pieces to be played back by the computer alone, in which case the performer
assumes the role of the director of a master-class. In this scenario, the performer might choose to make "suggestions" about only a certain
aspect of the performance. For example, if the tempo in a ritard needed work, the performer could tell the computer to incorporate only
timing information into its stored score, then conduct a performance of the coda, paying special attention to the tempo of the ritard, and
ignoring dynamics.

One interesting design challenge here would be to create a "coaching language" for a performer to communicate the nature of the changes
to be incorporated into the score; for example, if they performer wanted the computer to play a passage "with more delicacy," what
features of a sample rendition by the performer should the computer incorporate into its stored score and which should it ignore?

Motivation

It's been suggested that removing too many obstacles from the path to musical performance would make people less interested in it, that
the challenges of musical performance provide an essential motivation. Since it was never part of my motivation, I was surprised when I
first heard this idea, but I know that I'm not any more "average" than anybody else, so I made an informal survey of the musicians I knew
personally. Since the responses were in the form of long essays and often did not answer the question of motivation directly, the results
presented here involve a certain amount of my judgment. However, I tried to err in the direction opposite my preconception. If it seemed
like the person might be saying that the challenges of playing a musical instrument provided some motivation, I counted it as "moderate
effect." If they spoke a lot about challenges, I counted it as "essential" even if they didn't say that the challenges were essential. Only in
the cases where people explicitly said it had no effect (or an anti-effect) did I score them that way (by "anti-effect" I mean that the person

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 26 de 33

said that if there had been less challenge, they would have been more motivated). Here are the results:

I'm not sure how to interpret these results. Since it is a cross-section of active musicians (at various levels from skilled amateur to
professional), I would expect it to be skewed in favor of people for whom musical challenges were welcomed or at least not a big
negative factor. Does the fact that the majority of responders said it had a small effect, no effect, or a negative effect mean that, on
balance, musical challenges are not important?

Another unanswered question is how people who are not musicians would respond. I've never known anybody who said "oh, I tried
learning a musical instrument once, but it was too easy, not enough of a challenge, so I quit" and I've met people who'd studied music as a
child and regretted having given it up, presumably because it would be too much work to learn it as an adult.

So, it's still an open issue. All I can say for sure is: it's not yet clear to me that musical challenges are an important part of what motivates
most musicians.

How many performers?

It's certainly satisfying for a single performer to use the conductor program. Thomas Zimmerman's patent describes a system in which
several performers play computer-assisted instruments together. What is the point of diminishing returns? Would you want to have a
hundred musicians all playing computer-assisted instruments together in an orchestra? I think not ... The optimal number depends on the
music. A piece of solo piano music probably only requires the attention of a single musician to perform it adequately, but a complicated
piece could certainly take more; after all, there are orchestral pieces which require two conductors. There is a practical upper bound,
though, because the complexity of music is limited by how much a person can hear. If a hundred people are all playing different things, a
single listener can't hear them all separately — and this applies to both the performers and the audience. We can follow a single musical
idea, or two, or three ... but beyond three or four distinct musical things, our attention breaks down and we hear groups of things, not
individual elements. How many distinct musical ideas can a person control expressively? The limit imposed by human attention is
further limited by the nature of the interface. A flute player seldom plays more than one melody at a time, a guitarist more than two, a
pianist more than four, an organist more than five. One of the reasons that organs have pedalboards but pianos do not is that human
attention limits a keyboard player's ability to play multiple lines expressively, but the expressive demands of an organ, which does not
include control of dynamic level by key velocity, are somewhat less than those of the piano. None of the current implementations of the
conductor program explore the question of higher-bandwidth user interfaces, but are instead based on existing models of music
instruments and conducting. Could a person with the right user interface control the conductor program well enough to play a late
Beethoven string quartet as expressively as four string players could? I believe the answer is either "yes" or "pretty darn close," but that it
would require a user interface very different from what's been developed so far ...

What a conductor does

It's called the "conductor program" but that doesn't mean that a person using it is doing everything a conductor does — or vice versa.
There are some parallels, though. For example:

as a conductor, you... as a user of conductor program, you...

...choose the repertoire (sometimes). ...choose the repertoire (usually).
...study the score. ...study the score.
...develop an idea of how the music should be performed. ...develop an idea of how the music should be performed.
...practice. ...practice.

...find gestures that best communicate your intent to the ...find gestures that best communicate your intent to the

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 27 de 33

performers. software.
...decide how you look to the audience. ...decide how you look to the audience (if any).
...decide how you look to the performers. N/A
...direct rehearsals, talk to the performers. N/A
...interact with all the people who make an orchestra happen. N/A

Conducting as a social, interpersonal activity is quite different than conducting as an abstracted, music- and information-
centric activity ...

Other interfaces

There are lots of interfaces that might be good for use with the conductor program. Here are a few I know about

name (and
picture description, discussion
link)

Collects signals from up to 12 EMG

(electromyographic) sensors and
BodySynth
sends them wirelessly to a receiver,
where they are mapped to MIDI.

If the conductor is going to control

lots of musical effects, one approach
Chord Board
is for each effect to have its own
button.

A continuous controller with three

dimensions (left/right, forward/back,
Continuum and pressure) presents a useful
gesture space; this one can sense
multiple fingers in that space.

The hope for this was that a rotating

gesture would map well onto the
repeating nature of musical beats,
phrases, etc. It turned out not to
work very well, because circular
motion is hard to do smoothly, and
Cranker
the interface provided no tactile
feedback about where the beats
and/or phrases were, so it was hard
to make them happen at the right
time. Haptic feedback might
improve this.

Digital agility a plus; absence of

Data Glove
haptic feedback is a minus.

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 28 de 33

This system includes 14 sensors to

Digital Dance detect the angles of the performer's
limbs.

Einway Large-scale interaction a plus.

Immersion focuses on haptic

Immersion
feedback, a plus for music interfaces.

Senses the vertical and horizontal

Lightning positions of two hands; performer is
free to move around (no wires).

The Lizard Trampoline provides

multi-finger control of both multi-
finger timing (from contact of
Lizard
conductive glove fingertips with
Trampoline
screen) and pressure
(displacement/tension of screen).
Too bad it's only a concept so far.

Marimba Like a normal marimba, but with

Lumina more axes of control.

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 29 de 33

Your basic MIDI wind controller;

MIDI Sax good for continuous control of
dynamics.

This interface (invented by industrial

design student Stephen O'Hearn),
was described in the "Ask Mr.
Moog" column of Keyboard
Magazine, January 1989, page 106.
O'Hearn It has many axes of large-gesture
control (pressure of thumbs, rotation
of hands, bending of main staff) plus
a two-dimensional finger controller
(each black/white dot is a trackball),
and a "function-key" pedalboard.

Nice large surface for expansive,

Sensor Carpet
full-body gesture.

Sensor Chair

Senses position of hands, butt. Good

for audience; lack of haptic feedback

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 30 de 33

a minus.

The cross-bars of the Tone Ladder

Tone Ladder
are mapped to notes of a scale.

Includes both sensing and haptic

VBow feedback for the four degrees of
freedom of the performer's bow.

The VideoHarp had some nice

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 31 de 33

features: a large area for gestures,

sensing of multiple fingers, and easy
for the audience to watch.

Video Harp

The zipper on the front of a sweater

or jacket is in the perfect position for
a hand-operated musical interface,
and allows many of the kinds of
gestures that bowing affords string
Zipper players: two directions of stroke that
can be alternated, three dimensions
of control (stroke, horizontal,
tension); a zipper controller could
also have a pressure sensor in the
handle.

Computer accompaniment

In the conductor program, the computer plays music described in a score in response to a performer's gestures. In computer
accompaniment, the performer performs "for real" and the computer provides the rest of the parts of the piece. Computer accompaniment
is similar to the conductor program in many ways (the software collects gestures from the performer, analyzes them, and plays back notes
from a score, etc.) and is typically considered a more "practical" (and hence marketable) application, since musicians need and use
accompanists already.

The future of the conductor program

How could the conductor program be improved?

Zimmerman, Wantman and Marrin-Nakra have begun several avenues of exploration of the physical part of the interface that seem ripe
for further development: differentiating different kinds of control (e.g. trigger versus modulation), capturing more information from the
performer's gestures and physical state, analyzing this information to extracting "gestural intent," and mapping this intent to the control of
the performance. The visual part of the user interface needs work too: for the conductor program to be most effective, music notation
needs to be redesigned from the ground up so that the information most important to a conductor/interpreter is presented naturally and
efficiently.

Ideally, the interface to the conductor program would eliminate the distinction between the performer's physical control of the music, the
gestures necessary to perform the music, and the visual score that represented the music. The score would be represented as an
evolving collection of dynamic sound objects, with which the performer would interact directly. The intent of the composer, the
performer in rehearsal, and the performer in real-time would all be made manifest both visually and aurally.

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 32 de 33

But what about non-technical considerations? Is there a place in our society for the conductor program? Will people want to use the
conductor program? Will something else take the place of the conductor program? I have high hopes for the future of the conductor
program, but I'm faced with this piece of evidence: the idea has been around for more than thirty years, and there has been readily-
available hardware, adequate to show the idea's potential, for more than half that time, and yet it's still failed to get past the "interesting
idea" point for all but a handful of practitioners (and some lucky children in Boston). Hopefully this article will help more people know
about its potential.

Acknowledgments
I extend my thanks to all the implementers and users of the conductor program mentioned in this article and to David Pihl, George
Pontis, and Ed Severinghaus. A special thanks to Mervin Lane, who provided the motivation for pushing this project to the top of my to-
do list.

Unclear?
If any part of this article is incorrect, confusing, or unclear, please let me know.

References and links

An inadequate list of references.

Study using Friedman's Instant Pleasure to examine the musical responses of children with disabilities.

"Technology-Assisted Research in Music Cognition: Enhancing Instruction in Expressive Music Performance," Robert H. Woody

Richard Boulanger, user of the Radio Baton

"Abstract Cinema and Light-Music," Bulat Makhmudovich Galeev, who is affiliated with the Prometheus Institute

Bert Bongers, "Physical Interfaces in the Electronic Arts"

"Music-Driven Motion Editing: Local Motion Transformations Guided By Music Analysis"

"Considering Human Memory in Designing User Interfaces for Computer Music," Otto Laske, Computer Music Journal, Volume II,
Number 4, pages 39-45

"Electronic Music Interfaces ," Joseph A. Paradiso, IEEE Spectrum, December 1997, pages 18-30

"The Double Revolution of the Theremin," Patrick Rashleigh

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011
The Conductor Program Página 33 de 33

http://www.musanim.com/tapper/ConductorProgram.html 01/09/2011