Adrian Freed, Ahm Lee, John Schott, Frances Marie Uitti, Matt Wright, Michael Zbyszynski
CNMAT UC Berkeley, adrian@cnmat.berkeley.edu, ahmsolo@yahoo.com, john@johnschott.com, francesmarieuitti@yahoo.com
ABSTRACT
In this paper we examine the hardware (sensors and connectivity), physical performance (placement, sensor type, tactile feedback),
and musical performance (software development and control) concerns inherent to extending control to six-string chordophones in general, and specifically to the electric guitar (plucked) and electric cello (bowed).
1. INTRODUCTION
This paper examines the fruits of collaborations between CNMAT researchers and eminent guitar players (principally John Schott) and recently with the renowned cellist Frances-Marie Uitti in the fall of 2005 (sponsored by a UC Regents lectureship program). The augmented cello completed during that collaboration was used in performance at the end of her residence on November 4th 2005.
The starting point for the cello project was a 6-string cello built by Eric Jensen [4]. The main, unique feature of this electric cello is a deep notch in front of the bridge co-designed by Ms. Uitti and Mr. Jensen. This allows Ms. Uitti to play using two bows simultaneously – one above and one below the strings – for chordal and other polyphonic textures [18] [12]. We were curious how much of our previous work on polyphonic signal processing for guitars could be leveraged for a bowed instrument in the hands of player who has already vigorously pursued the polyphonic potentiality of the instrument. We will describe the concepts developed for chordophone augmentation and compare and contrast the application of these ideas to guitar and cello.
1.1 Tuning Augmentation
Ms. Uitti uses a variety of non-traditional tunings to take advantage of the possibilities afforded by multiple stops and two bows.
One approach to supporting these different tunings is to use independent pitch shifting DSP algorithms on the signals captured by piezoelectric pickups under each string at the bridge. This method is used commercially for guitars and used notably by musicians who adopt many unusual tunings, Joni Mitchell, for example, who composes using scordatura tunings as a starting point [15]. In our hex guitar signal processing work we identified several important challenges with electronic pitch shifting:
1) Numerous noticeable artifacts in the shifted sound.
2) Conflict between the acoustic sound and electronic sound in live performance
3) Unacceptably long latencies especially for low-pitched strings.
For the guitar we handled the first problem by careful choice of parameters in the pitch shifting algorithm and careful equalization and processing of the sound before and after pitch shifting. The second problem was mitigated by use of an electric guitar in a large (800-seat) performance space. Careful choice of musical material avoided the latency problem. A typical example of this is John Schott’s Aggregate Delay” patch where feedback delay loops were timed to create a rhythmic and spatial counterpoint between 6-tone chord cluster “calls” and complementary 6-tone “responses”.
With the cello we pursued another avenue by adding a mechanical tension-modulating device at the heel of the instrument.
Figure 1. Cello heel with string tuning device
This device was originally developed for guitars by Hipshot Inc [10]. We adapted it primarily by accommodating the larger cello string ball-ends. The device is normally floated from the heel of an instrument but we instead added it to an extension of the heel to maintain the existing “short string” length. This affords bowing below the bridge on the short strings.
Set screws allow three possible pitch settings for each string enabling microtonal, 1/4 tone, 1/2 and whole tone tunings.
This arrangement works well and it is an example of the benefits of exploring non-electronic solutions to instrument augmentation challenges.
2. Gesture Sensing Augmentations
2.1 Foot Control
Foot control is standard practice for electric guitarists and MIDI-based switch and pedal controls are readily available.
For John Schott’s guitar performances we used a footswitch array for preset selection and a dedicated laptop display to visually confirm the preset selection and affirm correct functioning of the software and interface hardware. One disadvantage of this approach is that it constrains the performer to stay in visual range of a static display. The popularity of clip-on guitar tuners with graphical displays suggests that guitarists will appreciate the advantages of integrating the display with the instrument and we anticipate new affordable, flexible panel [9] displays will be commonplace in future musical instruments.
Commercially available and custom foot control options proved to be hard for cellists to use because cellists use their legs to counteract the considerable torque generated by bowing. Their feet have to be firmly planted on the floor to comfortably do this for long periods with the necessary stability to support solid performances. Alternatives have been explored to this seated playing position including stands and harnesses[8] but these are not widely accepted for ergonomic reasons. We therefore decided to focus our efforts on new interaction opportunities for the fretting and stopping hands – the core of the cellist’s technique. A similar approach has been described for the violin [5].
2.2 The stopping/fretting hand
For the stopping hand we provided a row of FSR’s (Force Sensing Resistors) on the edge of the neck closest to the low-pitched strings. FSR’s have the advantage over switches of having a low profile and providing an extra control dimension (pressure). They also cost no more because in this situation installation cost eclipsed the parts cost.
The FSR’s were centered at the semitone positions of the string. This provides both a natural location (already thoroughly part of the cellist’s technique) and no part of the hand inadvertently touches this part of the instrument. The semitone positioning also suggests a convenient labeling of each control in a musical score.
On the other edge of the neck we installed a continuous pressure-sensing strip accessed typically with the thumb.
FSR strips are cheap and convenient but unlike rotary potentiometers and linear sliders they don’t provide any tactile memory of a parameter setting. We addressed this by adding a slider on the side of the lower part of the instrument body. This most commonly was used to adjust the sound balance between processed and direct cello sound.
Figure 2. Cello Body showing neck and body FSR
We also installed a switch array directly below the bridge and an arrow of circular FSR’s at the top of the body of the instrument. The switch array is used to make major “preset” changes during performance where the tactile feedback of the switches was important to confirm the change. Installing a small touch screen here would have allowed us to label the presets but we learned that some performers prefer instrument interfaces where there is no dependence on visual feedback.
Figure 3. Cello Heel with switch array, hex pickups and slider
Fret scanning has been explored for guitar controllers as exemplified in the Zeta Mirror 6 Guitar controller. We tried an analogous approach on the cello by measuring the electrical resistance of the string from a conductive fingerboard to the nut but found that the distance/resistance function was highly non-linear and varied from string to string, presumably because of the exotic alloys and varying solid and stranded winding construction techniques used in cello strings. We also tried to sense string stop position using a resistive strip designed as a “ribbon” controller but found readily available strips too wide and short for this application.
The primary application of sensing fretting or stopping position is to enhance pitch detection quality. We have found pitch detection of bowed string signals to be easier than plucked guitar strings because of the higher energy partials, harmonicity and better pickup performance.
2.3 The bowing/picking/plucking hand
Even when guitarists use a pick enough fingers are free to turn knobs and operate switches built into the instrument. The bowed string player has less freedom of movement and fewer free fingers with the bowing hand. We opted to exploit the player’s bow control and developed a custom “bowable” knob using a rotary absolute position encoder-a device that outputs a voltage corresponding to the angle of rotation of a shaft from a reference position. We attached a wheel to the shaft of a commercially available encoder with a surface preparation that the bow could easily grip. We installed the wheel behind the heel of the instrument where it can be thought of as an extension of the “short string” bowing technique.
Figure 4. Sensor Wheel
The encoder shaft itself has low mass and little friction so we were able to create a variety of different masses, gear ratios and detents by modifying different wheels.
3. Sensor and Sound Data Capture
For our earlier guitar work we used commercially available MIDI and USB devices for sensor data acquisition as illustrate in Figure 5.
Figure 5. MIDI foot pedals and USB touch input
For the cello we used the two sensor ports of CNMAT’s Connectivity processor [2].
This connectivity processor has a motherboard with 8-channels of balanced audio D/A conversion, 2-channel headphone output, AES-3 I/O, ADAT optical I/O, multi-channel sync. I/O, MIDI I/O, a high speed (GIG) and 100BaseT Ethernet [1].
Figure 6. Connectivity Processor motherboard
This motherboard supports two daughter cards: one is used usually for audio input, e.g., 8-channels of balanced audio; the other is a gestural input card.
The basic gestural input card supports two DB25 connectors carrying power and 16 channels of 0-5V inputs on each with sample rates up to 8kHz. By adapting the switch matrix to output analog voltages and building resistive dividers for the FSR’s all the cello sensors we rendered as 0-5V signals for input card.
Figure 7. Gesture inputs and Digital I/O
The piezo sensors for each string and two additional piezo pickups near the tail of the short strings were converted by custom-built charge amplifiers built into a special daughter card for the Connectivity processor.
Figure 8. Short String Piezo Pickups
These analog signals are conditioned, converted into digital signals, serialized and aggregated into an Ethernet stream that was processed by custom software in Max/MSP. Sound output was also routed through Ethernet packets to the connectivity processor and demultiplexed into 8 balanced analog audio outputs. The advantage of the Connectivity processor over commercially available audio and sensor IO devices is that timing relationships between all the data are accurately established by the hardware with a single stable clock source allowing for precise control of latency and high reliability of the overall system. Traveling performers also appreciate the benefits of having to carry only their instrument, a laptop and a single unified interface box.
4. Performance Software
The general development strategies for the musical performance software for cello and guitar were the same. A series of ideas were prototyped during rehearsals in a rapid exploratory way. The best of these were culled and recoded into a flexible, modular and documented patch. As focus shifted to the final performance these pitches were assembled into a main supervisory patch that managed all the signal and gesture routing and also switched active patches according to
selections by the performer.
One programming challenge is to give the performer as much meaningful control as possible without overwhelming them with parameters that they will find useless or, worse yet, distracting. It is important to work in a style that allows the programmer to quickly remap controllers and values to any location in the patch, and empowers the performer to feel that the software is actually responding to her actions.
To that end, overall control of the performance subpatches was managed using a combination of OSC (Open Sound Control)[17] and the pattr family ofobjects. Each of the hardware sensors was given a unique address in an OSC namespace, allowing individual subpatches to tap into the appropriate control data. Configurations that activated one or more subpatches were stored as presets in the pattrstorage object and triggered via the switch array (below the bridge). Smooth
crossfades between successive configurations were achieved with pattr‘s built-in interpolation features.
Figure 9. Performance patch
These features allowed the cellist
to dynamically remap the meaning of her performance gestures
according to the needs of the musical situation, quickly and
smoothly moving between one set of patches and the next.
No matter what patches are in
effect, the cellist always has control of her throughput gain,
and the overall gain of the effects. Single controllers
are mapped to each these gains, and remain fixed throughout the
performance. This was important to allow the performer to
react instantly to the musical situation, especially if the
processing does not fit the character of the musical
moment.
5. Signal Processing Patches
Dozens of signal processing
patches were developed for both instruments. We will focus here
mainly on those for which interesting differences were observed
in their application to guitar and cello.
5.1 Vocal Effect
For this effect we used a separate
bank of five resonant formant filters for each string. These
were tuned dynamically by interpolating between vowel pairs
stored from a data set that included a, e , i, o, u for
soprano, alto, bass, contrabass, and tenor voices. The
appropriate vocal data set was matched to the tessitura of each
string. Vibrato was created artificially by interpolated delay
line modulation and modulated by pressure of the fingerboard
FSR strip. This was used as a micro-rhythmic contrast against
Uitti’s normally fluctuating vibrato, creating changing beating
patterns and synchronizations. Vowel pairs were chosen using
the fingerboard FSR’s and interpolations were controlled within
the patch.
This patch was derived from
earlier guitar work starting as the idea of a phonetic
generalization of the “wah wah” pedal. We added distortion to
each string to create high frequency energy to make the
spectral peaks of the vowel formants more obvious. The
artificial delay-line modulation vibrato was more critical in
the guitar case because it is difficult on many guitars to
achieve wide controlled “vocal” vibratos which are of course a
staple of cello technique. Vowel choices were made using pedal
board switches. Vowel interpolation was controlled using a
spring-loaded pedal. Vibrato depth was controlled by a regular
pedal with the other foot.
5.2 Double-stop Convolution and Pair-wise Distortion
The key idea of this patch is to
use a separate convolution for all the double stop combinations
and to process and spatialize the output of the convolved pairs
independently. Since the convolution was performed by FFT’s we
were able to save computation by sharing the forward transform
of each string signal.
Convolution works well in this
situation because sound is only output if there is a signal in
both inputs of the convolution. This is a fruitful area of
exploration because double stops are a reliable musical gesture
and the performer has immediate access to many independent
streams of processing without having to choose them ahead of
time with other gestures.
Figure 10. Double Stop Convolution
We explored pair-wise string
processing in the guitar applications for distortion (“fuzz”)
effects. For complex chord clusters distortion generates
numerous sum and difference partials that tend to muddy the
identity of chords. One solution to this is to simply apply
distortion (we used a hyperbolic tangent function)
independently on each string. This had the desired effect of
brightening the timbre of the instrument but we found in many
situations a modest amount of cross modulation was desirable so
we employed distortion on mixes of signals from adjacent string
pairs and the outer pair.
5.3 Sustained notes
Guitarists employ different
techniques to increase the perceived lengths of notes: fast
tremolos, electromagnetic “bows”, and feedback oscillation.
Cellists can simply use the bow. This difference explains why
granular synthesis was used differently in the two instruments.
For the guitar John Schott employed grains for an “Infinite
sustain” effect used with multiple grain buffers recorded in
real-time for his performance of “Acrobatic Fly” on Oct 27
2001. A TacTex MTC Express multipoint pressure pad was use to
control grain choice size and processing [13, 14, 16].
Figure 11. “Acrobatic Fly” Real-time Granular display,
5.4 Quad Granular and Circular Panning
Two patches were combined in this
effect with the intent of surrounding the direct sound of the
cello with a diffused aura of related fragments. The
fragmentation was achieved with a pair of stereo granulators,
specifically munger~ (from the PerColate [11]
collection). These were set to create relatively long
(2000ms ±200ms), widely spaced (500 ms ±250ms),
irregular grains. Grains were generated from a 3000ms buffer,
and could play back either forwards or backwards at the speed
of the original performance. Each granulator is
independent, and their outputs were interlaced and sent to the
circular panner.
The panning patch diffused the
sound in a circular array, maintaining a 180-degree separation
between each channel of each granulator. That is, if left and
right for the first granulator appeared at 45 and 225 degrees
from the listener, the second granulator would appear at 135
and 315 degrees. Each granulator generated grains at
random locations in their stereo field, so the result was a
complex constellation of sounds. The entire sound field
was rotated by the performer using the rotary encoder behind
the heel of the cello. This gave the performer sensitive
and expressive control of the direction and rate of the
perceived motion. The angular displacement of the sounds
was generated by Ville Pullki’s VBAP objects[6],
allowing the angle to be specified independently of the
specific number and location of loudspeakers.
6. Future Work and Conclusion
We conclude that much of the core
hardware interface technology and signal processing concepts
and implementations can be shared between augmented plucked and
bowed chordophones. Furthermor a toolset developed for 6
stringed instruments is readily transferred to related
chordophones with fewer strings.
Most of the differences in
approach between bowed and plucked instruments stem from
ergonomic considerations and the brighter timbre and loudness
of bowed strings.
Although the placement and use
will vary from instrument to instrument we have confirmed that
both kind of chordophone can be enhanced with integrated 2-D
multipoint touch surfaces and displays. Recent developments
will help make these affordable [3].
The position encoding wheel/bow
sensor interaction shows a lot of promise. We are exploring use
of programmable detents and weights to see how much tactile
feedback can be exploited by the musician. We are also
exploring new instrument interfaces built around this sensor.
We will explore the addition of a servo motor to the drive of
the encoder, a strategy that has been used to research violin
bowing [7].
7. Acknowledgements
This paper discusses work and tools that were generously supported over a long period by many organizations and individuals including: Gibson Guitar, Zeta Music. TacTex, David Wessel and the Chambers fund, UC Regents, and Waves Audio Ltd.
8. References
1. Avizienis, R. and Freed, A., OSC and Gesture features of CNMAT’s Connectivity Processor. in Open Sound Control, (Berkeley, California, 2004).
2. Avizienis, R., Freed, A., Suzuki, T. and Wessel, D., Scalable Connectivity Processor for Computer Music Performance Systems. in International Computer Music Conference, (Berlin, Germany, 2000), ICMA, 523-526, http://cnmat.CNMAT.Berkeley.EDU/ICMC2000/pdf/connectivity-processor.pdf.
3. Han, J.Y., Low-cost multi-touch sensing through frustrated total internal reflection. in Symposium on User Interface Software and Technology, (Seattle, WA, USA, 2005), ACM.
4. Jensen, E. Jensen Electric Cellos, 2006, http://www.halcyon.com/jensmus/cello.htm.
5. Overholt, D., The Overtone Violin. in NIME 2005, (Vancouver BC, 2005).
6. Pulkki, V. Virtual sound source positioning using vector base amplitude panning. Journal of the Audio Engineering Society, 45 (6). 456-466.
7. S. Serafin, M.B., S. O’Modhrain, C. Nichols. , Expressive controllers for bowed string physical models. in DAFX, (Limerick, Ireland, 2001).
8. Steinberger, n. Cello Stand Instructions, 2006, http://www.nedsteinberger.com/instruments/cello/cr/celloinstr.html.
9. Thompson, M.E. Flexible Displays, 2006, http://www.universaldisplay.com/.
10. Trilogy. Hipshot Trilogy Multiple Tuning Bridge, 2006, http://www.hipshotproducts.com/trilogy.htm.
11. Trueman, D. and DuBois, R.L. PeRColate, 2001, http://music.columbia.edu/PeRColate/
12. UItti, F.-M. Two Bows, 2006.
13. Wessel, D. and Wright, M. Problems and Prospects for Intimate Musical Control of Computers. Computer Music Journal, 26 (3). 11-22, http://mitpress.mit.edu/catalog/item/default.asp?sid=A6D3F1DE-E4D0-47CD-A282-ED08B47DA533&ttype=6&tid=8962.
14. Wessel, D., Wright, M. and Schott, J., Intimate Musical Control of Computers with a Variety of Controllers and Gesture Mapping Metaphors. in International Conference on New Interfaces for Musical Expression (NIME), (Dublin, Ireland, 2002), 171-173, http://cnmat.CNMAT.Berkeley.EDU/Research/NIME2002/NIME02WesselWrightSchottDm.html.
15. Whitesell, l. Harmonic palette in early Joni Mitchell. Popular Music, 21 (2). 179-193, http://www.journals.cambridge.org/action/displayAbstract?fromPage=online&aid=107747.
16. Wright, M., Problems and Prospects for intimate and satisfying sensor-based control of computer sound. in Symposium on Sensing and Input for Media-Centric Systems (SIMS), (Santa Barbara, CA, 2002), 1-6, http://cnmat.CNMAT.Berkeley.EDU/Research/SIMS2002/Wright-SIMS2002.pdf.
17. Wright, M., Freed, A. and Momeni, A., Open Sound Control: State of the Art 2003. in International Conference on New Interfaces for Musical Expression, (Montreal, 2003), 153-159, http://www.music.mcgill.ca/musictech/nime/onlineproceedings/Papers/NIME03_Wright.pdf.
18. Zorn, J.E. Arcana. Granary Books, 2002.
