Small Fish - Essay

Cybernetic Music Spaces

Small Fish, Chamber Music with Images for Computer and Player, started as a very unambitious endeavour in 1999 but would evolve into one of the most successful artistic CD-ROM productions of the ZKM. It presents a perfect example of the opportunities that the creative environment of the Centre for Art and Media provided at the crossroads of art, technology and science (Furukawa, Fujihata, & Muench, 1999). Small Fish consists of 15 interactive audio-visual games in which configuration, movements, and collisions of symbols, objects and images are used for controlling a flow of melody, harmony, and rhythm generated in real-time through a MIDI sound synthesiser. Users can interact with this software as a music instrument. 

The custom-developed graphic user interface for Small Fish references the metaphor of visual music scores: Two-dimensional spaces designed for the computer monitor by Masaki Fujihata, comprising of an individual background image for each score and a variety of different graphic objects placed in each of them. These graphical objects do not only carry visual qualities and connotations but also, first and foremost, serve as visual representations of the defining elements for the generation of sound according to the musical structure conceived by Kiyoshi Furukawa. Users can manipulate the position and movement of these objects through standard functionalities of the computer mouse, and this user interaction influences and alters the creation of sound within the boundaries of the musical structure.

The primary theme of Small Fish is ‘interaction’. It started in 1999 as an interactive music CD-ROM, and over two years evolved into an installation art, a concert, and a workshop version. In 2012, the development of the work made a full circle when the CD-ROM version was reintroduced as a software application for mobile communication devices such as Apple’s iPhone. Throughout all versions, however, the fundamental underlying artistic concept remained the same. Small Fish presents an electronic canvas for the establishment of highly interactive and playful user involvement in the generation of sound sequences through a manipulation of virtual or embodied objects that form part of the symbolic code. These symbols are simultaneously representations of the musical structure as part of the visual score and central components in the interactive creation of sound as they are defined as instruments with distinct musical qualities or as elements of music notation. The artwork introduces this conceptual amalgamation of the objects’ functionality of both representation and creation within an interactive visual user interface as ‘Active Score Music’. 

At an artistic-aesthetic level, Small Fish references concepts of abstract expression and synaesthesia, pioneered by artists of classic modernism such as Kandinsky and Schoenberg. However, digital computing at the end of the 20th century provided opportunities to augment and extend these concepts significantly. Small Fish does not advocate the detachment of technology from content but introduces a distinctly unique quality to the production and reception of music.  It defines a virtual environment that promotes the mutual interaction of unique signifying codes in the creation of artistic content. The artwork was designed as an electronic space in which meaning is created through a semiotic interplay of semantic encodings of visual components such as dots, lines, and colours, with those of musical elements such as tone and pitch of various instruments. In Small Fish, those visual and musical elements are inseparable parts of an intuitive process of creating a tangible experience of, and appreciation for, the space of abstract connotations. It presents a model for user engagement that does not demand cerebral interaction during the semantic decoding of audio-visual elements but allows drawing meaning straight out of direct interaction with images and objects, as Fujihata emphasised: “Thought does not follow experience; it is itself experience” (Fujihata, 2000). 

For a successful adaptation of an artistic concept into the mathematical-logical realm of computational machines, it is imperative that the software design corresponds well to the artistic idea. In the development of interactive artworks, it is often overlooked that technical concepts are also bearers of meaning. They potentially influence the artistic outcome to a significant degree, and the underestimation of the importance of the technological tools and processes often leads to aesthetic and artistic difficulties, as Norman et al. noted in a review of Furukawa’s virtual reality opera (Norman, et al., 1998, p. 26). For Small Fish, the underlying technological concept was designed to not only facilitate the realisation of the initial artistic concept but to provide new avenues for the artistic idea by intimately coupling the development of technical and artistic issues in the process of constant communication between the three authors. Through this communicative process, it was possible to avoid focusing on technological aspects of human-computer interaction as a quality in itself. In Small Fish, technology is both an integral as well as a concealed part of the artwork that could only be realised within the framework of a computer. However, it did not demand any prior knowledge of working with computers. While interacting with the software, users were hardly aware that they were indeed involved in a complex human-computer interaction.

Small Fish was developed on Macromedia Director, a standard multimedia platform in the 1990s, not least because all artists involved were familiar with this software environment. Furthermore, its scripting language was capable of supporting the envisioned modular structure of a network of interrelated and interdependent objects that was at the heart of the artistic concept (1).  These small algorithmic units operate according to simple behaviour descriptors to perform a precisely defined task in controlling a corresponding image displayed on the screen. They can communicate and swap information with each other, but they are oblivious to the overall context of the program. Such an approach signifies a highly decentralised approach, and forms a small complex system of interconnected objects quite similar to what Valentin Braitenberg described in his 1984 publication concerned with questions of cybernetics, artificial intelligence, and artificial life (Braitenberg, 1984). Such a system quickly develops a level of complexity that makes it virtually impossible to understand the underlying rules for the interaction of its components just by observing the behaviour from the outside. Modelling Small Fish as a complex system of objects provided the opportunity to define an abundance of different combinations within strictly determined boundaries without deploying a random generator to produce variation. The conscious decision to completely exclude chance is essential to the musical concept since only a predetermined set of rules assures that a sequence of tones created during the game is principally repeatable. Bizarre and unpredictable happenings manufactured purely at random would have been fundamentally antagonistic to Small Fish’s idea of establishing a semiotic space for the generation of artistic meaning. Only by limiting the possible behaviour of each element to a set of rules enabled the user to comprehend the musical structure proposed by composer Furukawa. 

However, particular events can no longer be forecast accurately in a complex system. The artist finds himself confronted with the paradox of never being able to hear or see his work in its totality. A model of interaction based on a stimulation-and-response approach is comparatively easy to define but imposes significant limitations to the audience’s aesthetic experiences. An environment composed of independent objects that are communicating and mutually influencing each other’s behaviour even without direct user input substantially enriches this aesthetic experience but makes it virtually impossible to imagine all theoretically conceivable configurations. The artist only defines the aesthetic boundaries for the artwork, while each user creates a unique art experience with every new interaction with the system. Such an interaction model that does not solely rely on constant user input also implies that Small Fish can be confidently left to its own devices for a while without resulting in the program being engaged in an endless loop of eternal sameness. If in doubt, the system interacts with itself. 

The success of Small Fish led to a substantial number of exhibitions in museums, which required the development of a presentation set-up that could accommodate the involvement of a bigger audience in a public space. A large-scale projection replaced the computer monitor, and a table was placed in the middle of the exhibition area, resulting in a not entirely satisfying solution conceptually located somewhere between the cultural interfaces of cinema and an open theatrical stage. The software was still running as a single-user application, and the user needed to assume the prominent role of main ‘actor’ in front of the entire audience, an exposure that was rather frightening for many visitors. 

The development of an interactive art environment based on the original CD-ROM addressed the issue of an appropriate representation of Small Fish’s artistic concept in the context of the White Cube of a museum. This version consisted of a ceiling-mounted projector, a ceiling-mounted infrared camera as part of the machine-vision system, real objects placed in the projector area on the floor, and two interconnected computers for evaluating the video signals and for generating the audio-visual content in response to user activities. 

Several conceptual adjustments to the artistic concept were required to convert a single-user desktop computer application to a multi-user spatial interactive art environment. These changes inevitably resulted in a notable shift towards the facilitation of embodied interaction in physical space that came at the expense of the complexity of a virtual musical structure. Firstly, it was nonsensical to provide the possibility to switch between several audio-visual environments, since such a feature would require a diverse audience to make a joint decision as to which of these environments they want to select. Furthermore, the provision of any interactive mechanism for changing such settings would have been difficult to comprehend by the audience if it involved a specific machine-recognisable pattern in the active area. Moreover, it would destroy the integrity of the artwork if such a selection device was positioned outside the artwork, e.g. in the form of a physical control switch.

Secondly, the complexity of the musical structure, as well as the aesthetics of the projected visual objects that served as the interface to the interactive art environment, needed modification to meet the reality of a noisy public exhibition space with ambient light. With these challenges in mind, Furukawa used the CD-ROM game template of Garden for the development of a new sound configuration that emphasised audibility over complexity and presented a significantly simpler musical structure compared to the CD-ROM. Equally, the visuals were altered to display a larger size and more vivid colours that could compete with the ambient light. Finally, the system-generated movements of objects were now limited to the new virtual actor on the floor. A little small fish ‘swam’ on the blue surface and created sound structures according to its position, and in response to any intersection with one of the objects. The metaphor of a small fish in a virtual underwater world certainly referenced the context of a multimedia exhibition dedicated predominantly to children that first presented Small Fish Floor. However, it proved immensely successful not only for children, so that this semiotic reference remained the same in subsequent exhibitions and versions. A few months later, the 2000 Ars Electronica Festival exhibited Small Fish Floor in essentially the same way. Although, due to real space constraints that did not allow for a large floor projection, we traded the floor with a table, and reduced the objects in size, so that users could sit at a table and move objects quite similar to the setting of playing a cardboard game.

The technical setup was far more advanced and demanding when compared to the original CD-ROM. The real-world objects with which the audience physically interacted carried a unique pattern of infrared reflection material on the upper side to facilitate proper identification of each object. The machine vision system analysed the position and rotation angle of these objects on the projection area through the camera. The system was connected via a serial port to the Small Fish interactive music application on a separate computer, which processed the spatial information to generate movement and musical sequences according to the location of objects. Changing the position of real objects altered the position of the associated virtual image projected at its position, and also resulted in a change in the musical structure. The images generated by the Small Fish interactive music software were projected onto the objects on the floor through the ceiling-mounted projector, while the generative sound was processed in real-time according to the user interaction through a MIDI synthesiser and an audio system.

While the installation was working very efficiently in technical terms, the computer system was unfortunately rather difficult to set up. Especially the serial communication between the Windows PC that ran the machine vision system and the Apple G3 Computer that hosted the Small Fish application was troublesome at times. To get rid of one of the two computers, we started to work on the simplification of the overall system in 2001 for an exhibition in the Tokyo Miraikan National Museum of Emerging Science and Innovation. This new version implemented a different technical-conceptual approach to the machine-vision system. This attempt turned out to have rather interesting conceptual implications in the larger context of questions related to the physicality of computer environments. 

We replaced the infrared camera system with an ordinary consumer video camera that was connected directly to the computer on which the Small Fish application was running, and modified the Small Fish software to facilitate the evaluation of the video signal inside this software without external communication devices. This set-up required a change in the physical characteristics of the real-life objects. For the previous machine-vision system, the signifying property of the objects was until then the transparent infrared foil. In the new version based on visible light, the signifying property was the specific colour of the object. While this approach had the disadvantage that the precise detection of the object was more reliant on the particular lighting setting in the exhibition hall, it had the great advantage that these objects could now be shaped as smooth cushion-like textiles instead of thin wooden boxes on wheels. The new conceptual approach also allowed dismissing all visual representations of virtual objects that were formerly projected onto the wooden boxes, since now the colourful cushion objects took their part in the interaction framework. 

Apart from simplifying the technological aspects, this version constituted the embodiment of virtual objects in real life. In the first version, the projection of virtual images onto real objects denoted a clear conceptual demarcation line between virtual and real space. These objects drew their proof of existence in the interactive art environment only by virtue of a transparent infrared foil that belonged neither to the semantic realm of the physical objects nor the virtual computer environment. The foil was nothing more than a conceptual crutch included for the sole purpose of getting hold of information about the spatial positioning of these objects. In the second version, however, the embodied objects were part of the virtual computer environment by virtue of the physical property of their colour. Here, virtual and real spaces were intrinsically connected through a machine-vision system that recognised a quality of the object itself, not a mere signifier lacking semantic connotations with the object. The new Small Fish Floor version introduced a playful elucidation to a pertinent issue in the conceptual framework of computer-based interactive art environments. It dismissed the semiotic split of embodied and virtual objects and their semantic detachment from each other in such environments (2). In Small Fish Floor v.2, physical objects are properties of both worlds, there is no difference in semantic meaning between real objects and their virtual attributes, apart from their ability to trigger sound as part of the artistic concept. They signified the same in both realms: Flexible barriers to the movement of a small fish.

Small Fish Floor deliberately restricts the realm of virtual reality to the ambiguous role of its coloured objects and does not attempt to represent an embodiment of the user within the virtual environment. The machine-vision system does not recognise the user itself, only his or her actions with objects are relevant as the input interface to the system. Small Fish Floor’s spatial set-up resembles a stage for the interactive formation of a nonlinear audio-visual narrative in hybrid space. It signifies an open theatrical space, composed of real objects in an illuminated area, combined with a computer application that serves as virtual meta-actor, a function within the virtual scenography that is conceptually positioned somewhere between a director, producer, conductor, actor and deus-ex-machina. In the context of the ‘White Cube’ of an exhibition hall, the scenography of Small Fish Floor, which places the audience around a central stage, has links to Kiesler’s 1924 utopian design for the Raumbuehne. However, different from performances of trained actors that included digital technology, Small Fish Floor obliterates the separation between audience and actors entirely. The overall conceptual approach of Small Fish to provide a user interface that does not require any prior knowledge of digital technology, accentuated even more in its spatial version, supports a fluid transition of visitors from the passive role of spectator to involvement as an interacting performer. Access to the ‘active’ area of the performative space is unlimited, and it presents no barrier other than the criterion of the visitor’s general disposition to participate in a playful activity. 

As mentioned above, Small Fish Floor’s celebration of user-interaction with a digital artwork came at the expense of the complexity of musical structures. Different from the original CD-ROM version, it was not intended as a musical instrument, but as a platform for playful spatial engagement with a virtual audio-visual environment. It is characteristic of a rarely articulated phenomenon of interactive art related to the interdependencies between the level of user interactivity and the amount of enunciated and conveyed artistic content. An increasing number of opportunities by users to interactively influence the process of creating the gestalt manifestation of the artwork is prone to emphasise the concept of technology-based interactivity itself and almost inevitably result in the impossibility of formulating explicit artistic messages in a traditional sense. With some sarcasm, one could fantasise about the ultimate interactive digital artwork as a computer, equipped with all sorts of programmes, and connected to all types of peripheral devices: An empty digital environment with which any user could create any artistic statement. The definition of artistic content in a digital environment does not add ‘interactivity’ to the system but introduces an increasing number of limitations to the potential of the machine in proportion to the artistic content that is to be communicated by the artwork. In the context of featuring technological aspects of interactivity as part of the artistic statement in an interactive artwork, the element of self-referentiality in interactive art that Medosch noted (Medosch, 2005, p. 25) is indeed related to the phenomenon that the creation of explicit artistic content conflicts with the concept of user-interactivity.

It is, therefore, interesting to note that synchronously with the development of Small Fish Floor, we also developed a concert version of Small Fish that brought back prominently the emphasis on the musical content. This version facilitated the interplay between a trumpet player and interactive music software. Both players were able to alter the positions and the behaviour of virtual objects on the screen, the trumpet player by using specific notes and pitch, and the player at the computer through a mouse and keyboard interface. The significantly extended options for interacting with the application to create musical structures in real-time made this version a complex instrument for musicians with an expert background and knowledge. Consequently, it was first performed in a concert hall, the Brucknerhaus, during the Ars Electronica Festival 2000. 

For reasons of completeness, it should be mentioned that Small Fish’s synesthetic property of visual representation of sounds also led to a customised concert version for children with hearing disabilities. Notably, while Small Fish in all versions was impressively successful, only this version was honoured with an award: It won the Interaction Design Prize at the 2001 Good Design Award ceremony in Tokyo (Japan Institute of Design Promotion, 2001).

© Wolfgang Muench . This essay forms part of the Author's PhD Thesis (Muench, 2017) , with minor edits to adjust to the online environment. 

Endnotes

1. The following description of Small Fish draws on the author’s 1999 essay Behind the Interface, published in the CD-ROM booklet of Digital Arts Edition #3 (Muench, 1999). The booklet provides more detailed information on various issues and techno-artistic approaches. The thesis’ text reuses expressions and phrases of the original essay.

2. This semantic detachment is a result of the disparate connotations they carry in physical life as opposed to the function that is ascribed to them in the user interface to a virtual environment. As an example might serve Jill Scott’s installation A Figurative History, in which diverse objects such as an antique Greek vase or helmet are used as interface to initiate the display of animated films related to one of the main historical figures presented in the artwork (Hahne, 2003, p. 171). Although these objects are contextually linked to the narrative background of the figures, there remains a semantic barrier between the real world meaning of a vase and its functionality in the virtual environment as a switch to trigger a short narrative film sequence

Bibliography

• Braitenberg, V. (1984). Vehicles: Experiments in Synthetic Psychology. Cambridge, Mass.: MIT Press.

• Fujihata, M. (2000). Notes on Small Fish.  Retrieved 14.08.2016, from ZKM Center for Art and Media Online Archive, http://hosting.zkm.de/wmuench/sf_about

• Furukawa, K., Fujihata, M., & Muench, W. (1999). Small Fish: Digital Arts Edition #3. Karlsruhe, Stuttgart: ZKM / Hatje Cantz Verlag.

• Hahne, M. (Ed.). (2003). Coded Characters. Media Art by Jill Scott. Ostfildern: Hatje Cantz Verlag.

• Japan Institute of Design Promotion. (2001). Good Design Award: Interaction Design Prize.  Retrieved 17.08.2016, from Japan Institute of Design Promotion, http://www.g-mark.org/award/describe/2243?token=I3jvZqLovt

• Medosch, A. (2005). Technological Determinism in Media Art. MA Interactive Digital Media, Sussex University. Retrieved from http://theoriebild.ung.at//pub/Main/TechnologicalDeterminismInMediaArt/TechnoDeterminismAM.pdf

• Muench, W. (1999). Behind the Interface. In A. Sommer & ZKM (Eds.), Digital Arts Edition #3. Karlsruhe, Stuttgart: ZKM / Hatje Cantz Verlag, Stuttgart.

• Norman, S. J., Bowers, J., Staff, H., Schwabe, D., Wallen, L., Fleischmann, M., & Sundblad, Y. (1998). Extended Performances: Evaluation and Comparison. Esprit Project 25379, eRena Project Deliverable 2.3 Stockholm: KHT.

Illustration Credits

1. Small Fish: Parrot (1999). Screenshot from the CD-ROM produced by the ZKM Center for Art and Media (Coproduction of Institute for Music and Acoustics and Institute for Image Media). © Masaki Fujihata, Kiyoshi Furukawa & Wolfgang Muench. Courtesy the Artists & ZKM.

2. Small Fish Floor v1 (2000). Photo of the artwork taken at the exhibition D/J Brand at University Art Museum, Tokyo University of the Arts, Tokyo, Japan, 2005. © Masaki Fujihata. Courtesy Masaki Fujihata

3. Small Fish Floor v2 (2001). Photo of the artwork taken at the exhibition science + fiction, C3 Foundation, Budapest, Hungary, 2001. © Wolfgang Muench.

4. Small Fish Tale / Concert (2000). Photo of the artwork taken at the festival ZKMusik, Center for Art and Media, Karlsruhe, Germany, 2002. © Wolfgang Muench.