“drop”: An Interactive Art Installation with Waterdrop

Projection-Mapping
SADAM FUJIOKA, President, anno lab inc., Part-time instructor, Fukuoka Jogakuin University, Japan
This paper describes an interactive art installation titled “drop.” It is the first artwork using the Waterdrop
Projection-Mapping (WPM) system, which animates levitating waterdrops. With this artwork, the anno lab
team infuses physical characteristics into computer graphics and materializes them as tangible pixels. WPM
consists of a waterdrop generator and an ultra high-speed projector. The team uses an ultra high-speed
projector to cast stroboscopic spotlights mapping on waterdrops to create an optical illusion of animating
each waterdrop individually. This is a new technique to show computer animation by animating levitating
waterdrops. This technique explores a new horizon to create animations with tangible pixels that the viewer
can touch physically. 27
CCS Concepts: • Applied computing → Media arts; • Computing methodologies → Animation; • Human-
centered computing → Human computer interaction (HCI).
Additional Key Words and Phrases: Projection mapping, Physical computer graphics, Media art installation
ACM Reference Format:
Sadam Fujioka. 2021. “drop”: An Interactive Art Installation with Waterdrop Projection-Mapping. Proc. ACM
Comput. Graph. Interact. Tech. 4, 2, Article 27 (August 2021), 8 pages. https://doi.org/10.1145/3465613

1 PURPOSE
“drop” is an interactive art installation created to be a part of the permanent artwork for the Art
Museum of Nature and Human Non-Homogeneity (Oita/Japan) [Art Museum of Nature, n.d.]. For
the installation, my team at anno lab invented the Waterdrop Projection-Mapping (WPM) system,
which animates levitating waterdrops. We infuse physical characteristics into computer graphics
and materialize them as tangible pixels. This is a new technique to show computer animation by
animating levitating waterdrops.

2 SYSTEM
Waterdrop Projection-Mapping consists of a waterdrop generator and an ultra high-speed projector
(Figure 1) [WaterPearl, n.d.; Tokyo Electron, n.d.]. A waterdrop generator has 40 nozzles in line
and each nozzle ejects waterdrops about 60 times per second. Therefore, it creates a screen of
waterdrops. It has a water circulator system so it doesn’t need sustainable water supply. An ultra
high-speed projector casts stroboscopic spotlights composed carefully at 960 frames per second (fps)
on periodic falling waterdrops at precise times and positions, resulting in hundreds of waterdrops
that start animating with color. WPM creates a drop pixel: We call it dixel, with an overlaying digital
image. A waterdrop generator is installed above on the ceiling, and lights are rear-projected onto
waterdrops falling from above. Scientifically, the system employs stroboscopic effect to create an
Author’s address: Sadam Fujioka, President, anno lab inc., Part-time instructor, Fukuoka Jogakuin University, Japan,
sadam@annolab.com.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee
provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and
the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored.
Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires
prior specific permission and/or a fee. Request permissions from permissions@acm.org.
© 2021 Association for Computing Machinery.
2577-6193/2021/8-ART27 $15.00
https://doi.org/10.1145/3465613

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.
27:2 Sadam Fujioka

optical illusion, also applied to 3D Zoetrope [Stroboscopic, 2021]. Conventional applications for this
effect with waterdrops, so-called levitating water, are widely known. However, those applications
can’t control each waterdrop individually.
“drop” also uses an Azure Kinect sensor to detect a visitor’s interaction with waterdrops. With
this interaction, we could add reactions to each waterdrop when a visitor touches the installation.
There are four software applications for the “drop” installation: the WPM viewer application
for “drop,” the Kinect sensor application, the weather data analysis application, and the “drop”
sound application. The WPM viewer application generates image data in 960fps to cast stroboscopic
spotlights mapping on waterdrops from an ultra high-speed projector. The Kinect sensor application
detects visitor action toward waterdrops for interactions. The weather data analysis application
accesses the web page of the Japan Meteorological Agency and analyzes weather data. The “drop”
sound application plays background music and sound effects from 2.1ch speakers.
So far the projector is not so bright that we need a completely dark room for WPM. In addition,
each stroboscopic spotlight rear-projected on a waterdrop is cast only once in 60 times at 960fps so
it is not too strong for a visitor’s eyes. Although it uses water with some electric devices, a water
generator is very stable and we use one of the commercial machines that are installed in many
other places. In addition, we carefully installed the other devices to be separate from wet areas
and/or waterproofing devices. Shock prevention is perfect.

3 ARTISTIC CONTRIBUTION
There are several conventional artworks using waterdrops such as Rain Room by rAndom Interna-
tional [rAndom, 2012], 3D Water Matrix by EPIDEMIC [EPIDEMIC, n.d.], and G Active: Water Made
Active by Cole Paviour and Unit9 [Paviour, 2017]. Those works control opening and closing valves
mechanically to make some shapes with waterdrops. In particular, G Active: Water Made Active
uses stroboscopes to create stop-motion animation. It shows 3D animation but it is not possible to
show it in real time. On the contrary, WPM shows real-time animation of levitating waterdrops
without controlling opening and closing valves but by composing images of an ultra high-speed
projector to cast projection-mapping on periodically falling waterdrops.
In the context of tangible pixels, there are some conventional artworks such as Kinetic Sculpture
by ART+COM [ART+COM, 2008] and the Mechanical Mirror series by Daniel Rozin [Rozin, n.d.].
Both have succeeded in giving physical characteristics to digital animation with tangible objects.
WPM also has tangible objects as waterdrops but its biggest advantage over these conventional
artworks is that it is possible to show animation without physical constraints in 60fps because it
employs stroboscopic effect to create an optical illusion.

4 PRINCIPLE
WPM appears to levitate waterdrops by making stroboscopic spotlights with images for an ultra
high-speed projector and casting them at precise positions at precise timings on falling waterdrops.
When you make a sequence of images with a pitch-white screen image once in every 16 frames
and the rest are pitch-black screen images in 960fps without making spotlights, you will be able
to illuminate all of the waterdrops under the same conditions as casting a stroboscope light at 60
times per second (Hz). Because the frequency of generating waterdrops is also 60Hz, it seems as if
all of the waterdrops are levitating and still in the air (Figure 2). When you shift the phase of the
projection cycle of a pitch-white screen image forward by one, the positions of all the evenly spaced
waterdrops are also shifted upward by a distance of about 16 divisions of the interval. Conversely,
when you shift the phase backward by one, the waterdrops will be shifted downward (Figure 3). By
applying this technique, we can show smooth animation of each waterdrop by making a spotlight
with images instead of pitch-white screen images and casting it on at a precise position at a precise

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.
“drop”: An Interactive Art Installation with Waterdrop Projection-Mapping 27:3

Fig. 1. Waterdrop Projection-Mapping system and a drawing of the “drop” installation. (©anno lab)

timing. In “drop,” there are 23 waterdrops at a certain moment for each nozzle 2.7m high. We can
show a waterdrop animation with 16 segments of their intervals, so there are 368 dixels in height.
There are 40 nozzles, so the resolution is 40 × 368 dixels.
An Azure Kinect sensor is mounted on the ceiling capturing downward, and it detects visitor’s
action toward waterdrops. The Kinect Sensor application detects this interaction and sends its
information to the WPM viewer application when a visitor’s hand touches a waterdrop (Figure 4).
In addition, when he/she touches a waterdrop, the WPM viewer application sends a command to
the “drop” sound application to play a sound effect for the interaction.

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.
27:4 Sadam Fujioka

Fig. 2. Projection on a waterdrop once in 16 frames, and a simulation image of the result. (©anno lab)

The weather data analysis application analyzes weather, precipitation, and tide level data from
the Japan Meteorological Agency and sends them to the WPM viewer application for changing
visuals.

5 SCENES OF “DROP”
The work consists of three scenes where we practiced three different types of animation: “Wave”
(line animation), “Tide” (face animation), and “Rain” (point animation). “Wave” is a scene of 40
waterdrops forming a sine wave (Figure 5). When touched by hand, each waterdrop flies upward,
reappears from below, and returns to its original position. It represents the concept of tangible dixel
effectively. In “Tide,” each waterdrop rises and falls inside of the diamond shape (Figure 6). When a
visitor holds up his/her hand toward a waterdrop, neighboring waterdrops around it stop moving
and stand still in the air. The speed of rising and falling of waterdrops that are slowly changing
reacts to the tide level data obtained from the Japan Meteorological Agency; the diamond shape
also changes. “Rain” is a unique scene presented only on a rainy day. It includes waterdrops falling
from above like rain (Figure 7). When a visitor holds up his/her hand toward the waterdrops screen,
all of the waterdrops freeze, and the visitor can control them by moving his/her hand up, down,
left, or right. The weather forecast for the area is obtained from the meteorological API. Also, the
amount of the falling waterdrops depends on precipitation in the area.
In all of these scenes, the animation is given the substance of waterdrops, but the visual effect is
heterogeneously affected by the natural water environment. Each scene is 2–3 minutes in length
and there are transition animations between scenes. The sound for each interaction consists of
background music, transition sounds, and sound effects [Nakamura, n.d.].

6 MOTIVATION
The Art Museum of Nature and Human Non-Homogeneity was planned and directed by us in a
collaboration with architects [Kocochi, n.d.; Fuu-Architect, n.d.] and has the theme of expanding
interaction with elements of nature by technology. We assigned a natural element to each room.
“drop” was installed in the Sea Room, where the natural element is water. We also designed the
concept of the museum of artworks reacting to the natural environment, so we added the weather
analysis application for “drop.” We are still in the trial-and-error stage regarding the potential of
WPM, and we want to keep updating and adding scenes or making other works using WPM.

7 FUTURE PERSPECTIVE
My team at anno lab and I are creating audiovisual artwork using the WPM system. This is a
benchmark test of dixel animation. Although you can play existing animation with WPM, it is
not so effective. It is more effective to design the movement of each waterdrop. This is also a
trial to develop a creative environment for animation artists. We are planning to collaborate with

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.
“drop”: An Interactive Art Installation with Waterdrop Projection-Mapping 27:5

Fig. 3. A principle of dixel of WPM, and a simulation image of a dixel animation of WPM. (©anno lab)

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.
27:6 Sadam Fujioka

Fig. 4. A Kinect sensor application for user interaction. (©anno lab)

Fig. 5. “Wave,” a scene of 40 waterdrops forming a sine wave. (©anno lab)

audiovisual animation artists and are designing some tools that they can use to create animations
with Adobe After Effects. Through collaborations we hope to spread WPM not only to art scenes
but also to commercial and entertainment scenes. We are also thinking of expanding the system to
include more layers or non-planar projection-mapping by devising the placement of the nozzles of
a waterdrop generator and an ultra high-speed projector.

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.
“drop”: An Interactive Art Installation with Waterdrop Projection-Mapping 27:7

Fig. 6. “Tide,” a scene of waterdrops rising and falling inside of the diamond shape. (©anno lab)

Fig. 7. “Rain,” a scene of waterdrops falling from above like rain. (©anno lab)

8 GLOSSARY
• stroboscopic effect: The stroboscopic effect is a visual phenomenon caused by aliasing that
occurs when continuous rotational or other cyclic motion is represented by a series of short

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.
27:8 Sadam Fujioka

or instantaneous samples (as opposed to a continuous view) at a sampling rate close to the
period of the motion.
• rear-projection: A projection from the back of the screen.
• projection mapping: A technique that uses a projector or several projectors to perform
projection on a real object to create an illusion.
• Azure Kinect sensor: A camera sensor device manufactured by Microsoft to detect the
movement of a person.

REFERENCES
Art Museum of Nature and Human Non-Homogeneity. n.d. Web page retrieved April 5, 2021 from https://nature-and-
human.art/
ART+COM Studios. 2008. Kinetic Sculpture—The Shapes of Things to Come. Web page retrieved April 5, 2021 from https:
//artcom.de/en/?project=kinetic-sculpture
EPIDEMIC. n.d. 3D Water Matrix. Web page retrieved April 5, 2021 from http://www.epidemic.net/en/art/3D-water-matrix/
index.html
Fuu-Architect Design Office. n.d. Web page retrieved April 5, 2021 from https://freedomlab.jp/companies/view/3024
Kocochi Architect NISHIOKA Miki Registered Architect Office. n.d. Kocochi Architect web page retrieved April 5, 2021
from https://kocochi.com/
Yuichi Nakamura (KAMRA). n.d. Web page retrieved April 5, 2021 from https://kamra.invisi-dir.com/
Cole Paviour. 2017. G Active: Water Made Active. Unit9 web page retrieved April 5, 2021 from https://www.unit9.com/
project/gactive-gatorade/
rAndom International. 2012. Rain Room. Barbican. Web page retrieved April 5, 2021 from https://www.barbican.org.uk/read-
watch-listen/random-international-rain-room
Daniel Rozin. n.d. Mechanical Mirror series. Web page retrieved April 5, 2021 from http://www.smoothware.com/danny/
Stroboscopic effect. 2021. Wikipedia: the Free Encyclopedia. Retrieved April 5, 2021 from https://en.wikipedia.org/wiki/
Stroboscopic_effect
Tokyo Electron Device, LTD. n.d. DynaFlash web page retrieved April 5, 2021 from https://www.inrevium.com/product/tb-
6v-dynaflash/
WaterPearl, co, ltd. n.d. WaterPearl web page retrieved April 5, 2021 from https://www.waterpearl.co.jp/waterpearl.html

Proc. ACM Comput. Graph. Interact. Tech., Vol. 4, No. 2, Article 27. Publication date: August 2021.