SCHEDULE

Physical Interaction Design (DOM-E5043) by Juan Carlos Vasquez / Niklas Pöllönen

1st Week

 Tue 29.11.2016 9:00 – 12:00

 Introduction to Physical Interaction Design

 Getting started with Arduino

 Alternatives to Arduino – Similar devices

 Wed 3oth Nov 2016 9:00 – 12:00

 Arduino + Pure Data

 Sensors

 Thu 1.12.2016 9:00 – 12:00

 Natural user interfaces

 Microsoft Kinect
 Leap Motion

 Fri 1.12.2016 9:00 – 12:00



 An introduction to Human Computer Interaction applied to creative projects –

Types of algorithms and Wekinator. Examples.

 Discussion about projects

 Assignment: Return a project proposal on Tuesday 6.12.2016

 Guidelines for the final project and proposal

2nd Week

 Tue 6.12.2016 – No Session

 Return a Project Proposal (1 A4). Send by the end of day to juan.vasquezgomez@aalto.fi

 Wed 8.12.2016 9:00 – 12:00

 Motors & Actuators

 Capacitive sensing – Capacitive Sensors + Introduction to Velostat and examples

 Thu 9.12.2016 9:00 – 12:00
 

 Raspberry Pi II
 Raspberry Pi and Arduino

 Standalone operation
 Assignment: Prepare a short presentation about your project for Friday.

 Fri 10.12.2016 9:00 – 12:00

 Strategies for Music Composition for Interactive Interfaces

 Project presentations and starting of project work.

3rd Week – Project work and Tutoring

Project Week (FabLab).
 12th, 13th, 14th, (No Session on 15th due to Demo Day). 9:00 – 12, at FabLab (Hämeentie 135 –
Arabia Campus, Second Floor)
 Final Presentation: 16.12.2016

DESCRIPTION
Physical Interaction Design course aims to explore and investigate the tools, concepts and practices for
planing and building new interactions with digital environments.The course focuses on music and sound
as lenses to discuss approaches and methodologies for the creation of interactive installations, physical
media/image/sound projects and artistic strategies for the creation of experimental musical
compositions. With a hands on approach this course invites students to experiment with sensor
technologies, acoustic transducers, micro controllers and thus confront fundamental concepts and
technical issues faced during the design of digital music environments.

The course explores multimodal interaction practices for adapting physical interaction to daily life
applications and contemporary art works. The course will be taught using the Pure Data and Arduino
open-source environments, accompanied by short readings. Physical Interaction Design course is a
project-based course. At the end of the course, students will submit and present their projects.

GRADING
 20% class participation

 10% assignments
 70% final project

LOCATION:
Paja (Media Lab 4rd Floor)

SUGGESTED READINGS:
 Designing for Interaction: Creating Innovative applications and Devices by Dan Saffer
  The Resonant Interface: HCI Foundations for Interaction Design by Steven Heim

 Handmade Electronic Music: The art of Hardware Hacking by Nicolas Collins

 Where the Action Is: The Foundations of Embodied Interaction by Paul Dourish

Lesson 1

Introduction to Physical Computing and Interactivity

 A Brief Rant on the Future of Interaction Design  by Bret Victor
 Amplify what we can do, instead of trying mimicking un natural gestures that we have
never done historicly on glass
 Every object in the world offer feedback we are not aware of
 Wintergatan – marble machine
 Programming Interactivity - O'Reilly Media
 Mixing Audio: Concepts, Practices, and Tools (Roey Izhaki)– Good book for audio
Processing

Interaction vs. Transmission

 Transmission doesn’t make a dialog, just send info

 Interactivity
 Reactive Interaction – system response for user input
 System makes a task and user irregulates it, modifying the outcome
 Full-fledged conversation – words, tones, facial expressions, AI, deep learning, neural
networks

Examples:

Sound Chaser – Yuri Suzuki

Analog Tape Playing Glove by Signal-to-Noise

Beacon by Cinimod Studio

Wonder Mirror – Daniel Rozin

Organelle by Critter and Guitari (Arduino n raspberry, puredata)

Equilibrium Variant by Roberto Pugliese (robots interact without interference of a human, mic and
speaker trying to avoid the Larsen effect, reaching perfect equilibrium)
Complexity of Interaction

 The richer the interaction, the more can go wrong

 Is surprise something desirable

 What does it mean if I’m the performer?

 Until what point design interferes with artistic expression

 What kind of skills are required?

 Is it better to be versatile or expert in one function?

Arduino Basics
Audio boards:
Bela: an Embedded Platform for Low-Latency Audio

Teensy

Introduction to Electronics
 Understanding electricity by Tom Igoe
 All about circuits – online textbook about electronics
 Look at the Sensor wiki to get an idea of sensors and actuators that are available.

Examples:

 SNOO Smart Sleeper – Mic +threshold > motor motion and white noise

 AXoloti XY – Axoloti + MIDI > OCSIloscope (inspired by: ryoji ikeda – datamatics)

 Janken Robot

First Arduino Code for use with Sound Touch in Pd:

void setup() {

pinMode(A0, INPUT);
pinMode(A1, INPUT);

Serial.begin(9600);
}

void loop() {
 // put your main code here, to run repeatedly:

int readValue = analogRead(A0);

int readValue2 = analogRead(A1);

Serial.print(readValue);
Serial.print("\t");

Serial.print(readValue2);
Serial.println();

delay(100);

 Soundtouch files got to be in the same folder of the project

Patch:
Second Practice – Usage with ALLINPUTSFirmata

Connected components to Arduino:

Button, potentio-meter
Computer Vision

Examples

 Dynamic Projection Mapping onto deforming non-rigid Objects

 Myo (very closed SDK), Easy to extract Accelerometer data, position, very fast gesture
recognition, puredata patches available

FaceOSC - Facetracker.net
 Uses OSC protocol

 OSC data

 By default, sends to localhost, port 8338. If you want to change this, see the below section
about editing the settings.xml file.

 Pose

 center position: /pose/position

 scale: /pose/scale

 orientation (which direction you're facing): /pose/orientation

 Gestures

o mouth width: /gesture/mouth/width

o mouth height: /gesture/mouth/height

o left eyebrow height: /gesture/eyebrow/left

o right eyebrow height: /gesture/eyebrow/right

o left eye openness: /gesture/eye/left

o right eye openness: /gesture/eye/right

o jaw openness: /gesture/jaw

o nostril flate: /gesture/nostrils

 Raw

o raw points (66 xy-pairs): /raw

 Keyboard controls

o r - reset the face tracker

 o m - toggle face mesh drawing

o g - toggle gui's visibility

o p - pause/unpause (only works with movie source)

o up/down - increase/decrease movie playback speed (only works with movie source)

 Gui controls

o "pose" and "gesture" toggle whether to send the pose and gestural stats over OSC,
respectively. "raw" toggles whether to send the raw points of the face mesh (66 XY-
points = 132 values).

 Settings

o To change the default settings of FaceOSC, including the OSC receiving IP and port, right-
click FaceOSC and select "Show Package Contents", then navigate to the settings.xml
under /Resources/data and edit accordingly.

Patch Example:

Introduction to Kinect

 Tracking users (no skeleton)

 Combination between Synapse and a puredata patch (through OSC)

 Kinectar – Establish invisible points for performance. Easy to map MIDI points and send to any
DAW

Leap-Motion

 IR-Hand tracking

 Issues with tracking and latency

Mapping and Machine Learning with Wekinator

 Relationship between actions and reactions

 Operate with data from sensors, buttons

Types of mapping:

 One to one - Direct mapping input to output

 One to many

 Many to one

 Many to many – complex math operations (arithmetic, trigonometric, etc)

 All to all

Wekinator receives input, takes a decision, sends to OSC

Modes:
Classification

 computes a class level – category for inputs

Regression

1. Computes numerical values from inputs

2. Output changes smoothly as the input changes

3. Can be used to control continuous parameters like volume, position

Dynamic Time Warping

Computes similarity between two sequences

Communicate with processing using OSCP5

Examples available in Wekinator Starter Packs

Good Course:

https://www.kadenze.com/courses/machine-learning-for-musicians-and-artists/info
Motors Capacitive Sensors

Examples
Motorized Strand beast
Makey Makey – For children and plug-n-play approach

Types

DC-Motor – Operates by voltage (current amount), opposite connection changes directions

 Needs its own power supply

 More powerful

Servo – DC Motor + positioning aids – Simpler connection

 Moves by a symbolic pattern to affirm connection

 Simple Arduino program with potentiometer:

#include <servo.h>

Servo myservo; // servo object

int potpin = 0; //pot to reviece ptentiometer

int val; // pot value

void setup() {
// put your setup code here, to run once:

myservo.attach(9);
}

void loop() {

// put your main code here, to run repeatedly:

 val = analogRead(potpin);

val = map (val,0,1023,0,180);

myservo.write(val); // sends pot value to servo

delay(15); //time to servo to reach position - delays the whole program

}

 ServoFirmata – Used to control ONLY servos through firmata

o pdServo Object found in pduino help file

o Radical changes overloads information on the servo

o combine simpleanalogFirmata and servoFirmata to deal with both in pd

 high value capacitor in between poles stabilizes current into the motor

 better to connect Arduino to external power supply

 DC Motors

o https://itp.nyu.edu/physcomp/labs/motors-and-transistors/dc-motor-control-using-an-
h-bridge/

 QT113 Touch Sensor

o

o Using AllInputsFirmata – connected to digital input

o Algorithm to stabilize flickering

 [speedlim] pd object limits in miliseconds the input rate

 [change] object bangs only when there’s a change in data

 o Simpler method produces a lot of noise – only for sensing if someone’s nearby

 Velostat

o Ingredients:

o Copper, tape, wires, inbetween 2 layers of velostat

o - A +
0
Raspberrypi

Examples:

 Barcode music

 Zimoun

Installation:

 Download X11 (mac) or xming

 FTP client

 Patch:

Important to auto-functionality of pd patch:

Auto run Arduino connection:

Auto-start dsp

Initial connection:

open terminal
ssh -x pi@(IPaddress)
user:pi password: “raspberry” - if not “test”
  In putty enable x11 forwarding

 Link local address acquired from pi running standalone using ifconfig:

 Current ip: pi@fe80::d37f:4c3e:9fa5:df9%3

Use a program from the boot:

Modify buffer file:

Sudo nano /etc/rc.local

/usr/bin/pd-extended -nogui /home/pi/potsynth/simplest.pd &

(& makes it run in th ebackground)

Ctrl+X Saves the buffer

Shutdown

Sudo shutdown now // shuts off raspberry

Sudo shutdown //shuts off only the connection

Audio adapter is a must in order to make sound work without noises.

Composition as definition of a Process, in a dialog

 Piano Phase (1967)

o Small idea develops into a massive discourse

o Same phrase with different instruments/conditions

o Balance between variety and cohesion

o Composition technics refer meanings

o Example from NOISA:

FABLAB