Developing Interactive Systems with OpenFrameworks

1. Interactive multimedia systems. Introduction to openFrameworks

Denis Perevalov perevalovds@gmail.com

See in-depth details in my book Mastering openFrameworks Books examples are free, see masteringof.wordpress.com

Preface
What is interactive multimedia system?

Interactive multimedia system

Examples
FunkyForest

Emily Gobeille and Theodore Watson (OpenFrameworks creator) for Festival 2007 CineKid in the Netherlands http://zanyparade.com/v8/projects.php?id=12

Examples
Hand from above

by Chris O'Shea

Examples
Body Paint

by Mehmet Akten

Definition
Interactive multimedia system is hardware and software multimedia system which 1) Is the real-time system. 2) Can input data by using various sensors, cameras and other sources of signals. 3) Can output data by the graphics, sound, haptics, robotics and other devices.

Interactive Multimedia System

The idea Concept
Development Computer vision Computer graphics Computer sound / music

Design

Implementation Equipment and installation

"Physical Computing

Ways of developing interactive systems

- Low-Level Libraries - Middle-Level Platforms - High-level Environments

Low-level libraries

Processing, analysis and image recognition

(Open Graphics Library) High-speed graphics

(Open Computing Language) Parallelization and speed up the calculations, in particular, means GPU.

(Open Audio Library) Sound

Box2D - 2D physics engine

Bullet - 3D physics engine

Web server

Video 1 Video 2

and so on ...

Sredneurovnevye Platform
This is a platform for "Creative coding", includes a large set of functions and libraries that are integrated for convenient Programming.

Processing
Language: Java For computer vision Java is slow.

openFrameworks
Language: C / C + +

Cinder
Language: C / C + + Recently appeared, gaining popularity.

Video 1

Video 2

Video 3

High-level protection
Visual programming environments, which allows to implement projects without actual programming. It is important that they can be expanded by the plugins made with low-level libraries. Also they can work together with middle-level platforms.

Max / MSP / Jitter

Focused on audio and VJ-ing.

VVVV
Focused on visual effects.

Unity3D
Focused on highquality 3D.

Example: Sniff - Interactive Dog, Unity3D + OpenFrameworks http://www.vimeo.com/6400266

Fields of application
Using only computer vision and computer graphics (and sound) currently produces a wide range - Advertising, - Entertainment, - Training, - Scientific, - Health, - Art interactive systems.

Course Description

What we will do
(1) The main interest
- Creation of interactive multimedia-based systems Recognition of video and audio signals. (2) Moderate interest

- 3D graphics, physics simulation, sound generation.

(3) Basis for application development - OpenFrameworks

Course Content
1. Introduction to OpenFrameworks
- Basics OpenFrameworks. General principles of real-time systems, vision systems and interactive systems. - 2D-graphics. - Receiving and processing images from the camera, the basics OpenCV. - Receiving and processing of sound. - Generation of sound, the playback of audio samples. - 3D-graphics. - Basic mapping. - Meet with high-level programs Unity3D, MultitouchDesigner, QuartzComposer. Installing Openframeworks connection with them. - Connecting external devices via the Arduino.

Course Content
2. Lecture "The Strategy of interactive art." by Ksenia Fedorova
(Curator of Yekaterinburg r la National Centre for Contemporary Art, Ven. factor is the art and culture, Postgraduate Department. aesthetics, ethics, theory and cultural history Philosophical Faculty USU)

Course Content
3. Working on your projects Listeners will be asked to perform under our supervision a number of projects related to video analysis and generation of graphics / sound.

Recommended Reading
OpenFrameworks
Joshua Noble "Programming Interactivity: A designer's Guide to Processing, Arduino, and openFrameworks "

For the translation of the book to your language use http://www.onlinedoctranslator.com/translator.html

Links
OpenFrameworks homepage www.openframeworks.cc

OpenFrameworks applications list www.creativeapplications.net/category/openframeworks/

Introduction in OpenFrameworks

What is OpenFrameworks
OpenFrameworks open library (framework) for C++, designed to work with multimedia.
Developed as a tool for designers and artists working with interactive design and media art.
One of the main advantages OpenFrameworks - extreme ease of use.

It is very popular tool for creating interactive systems, image processing, installations and various projects working with graphics, sound and input / output to external devices.

Where does OpenFrameworks

Windows Mac OS X Linux

iPhone OS

History of creation
OpenFrameworks was developed by: Zach Lieberman,Theo Watson, Arturo Castro, Chris O'Shea, together with other members of the Parsons School of Design, MediaLabMadrid, Hangar Center for the Arts, etc. Start of development - in the Parsons School of Design (New York) where Lieberman was a student.

http://www.flong.com/projects/tables/

When to use OpenFrameworks

- Multimedia capabilities are needed (input-output video input / output audio connection of external devices) - High-speed analysis of input data is needed (which is usually can be made only on the C/C + +) - A rather simple logic of data processing calculate->draw->calculate->draw->..." - No or little amount of GUI is needed - A small amount of time for developing is desirable

When not use OpenFrameworks

- Requires a lot of GUI (text editor) instead, use development tools, GUI - QT, Cocoa, ... - Require complex logic control rendering (3d game) instead, use engines like Unity3D, ... - Multimedia capabilities are not needed (web server) -You have money & time & desire for industrial application development so you can create your project from a number of low-level libraries.

Application structure
Architecture design of openFrameworks aimed to handling multimedia information in real time.
This results in a - Application appearance - Application structure.

Application appearance

Normally openFrameworks application has two windows - a graphics window and console window for the logs.

Application structure

Application structure

Getting Started with OpenFrameworks

1. Install VisualStudio C++ Express Edition 2. Download and install OpenFrameworks http://www.openframeworks.cc/download - There select "visual studio 2010 FAT" "FAT" means that many addons and examples are included. 3. Test installation. Go to the folder openFrameworks/app/examples, then compile and run one example.

Create a new project "Pendulum" with OpenFrameworks

1. In the folder openFrameworks/app/examples take an example emptyProject and copy it to /App/myApps/Mayatnik 2. Enter the code in the function of your class that derives from ofBaseApp

setup (); //set parameters at startup update (); //computation, analysis of input data draw (); //draw the current state

An example of a pendulum with an elastic band

The text of the "pendulum" testApp.h

// You dont need to change this file
# Ifndef _TEST_APP # Define _TEST_APP # Include "ofMain.h" class testApp: public ofBaseApp { public: void setup (); void update (); void draw (); void keyPressed (int key); void keyReleased (int key); void mouseMoved (int x, int y); void mouseDragged (int x, int y, int button); void mousePressed (int x, int y, int button); void mouseReleased (int x, int y, int button); void windowResized (int w, int h); }; # Endif

The text of the "pendulum" testApp.cpp, p. 1 / 3

# Include "testApp.h" // this class is already defined // Variables for ofPoint pos0; // Point of suspension ofPoint pos; // Current position ofPoint velocity; // Speed // Set void testApp:: setup () { pos0 = ofPoint (500.0, 200.0); pos = ofPoint (700.0, 200.0); velocity = ofPoint (0.0, 0.0); // OpenFrameworks it will redraw background each frame //with this color: red=255, green=255, blue=255, i.e. white ofBackground (255,255,255); }

The text of the "pendulum" testApp.cpp, p. 2 / 3

// Update the state void testApp:: update () { // Calculate the Hookes force ofPoint delta = pos - pos0; float len = sqrt (delta.x * delta.x + delta.y * delta.y); float guk = 0.01 * (len - 100.0); // 100.0 - length gum ofPoint gukForce = delta * (-guk / len); ofPoint g (0.0, 9.8); // Gravity // 2-th Newton's law float dt = 0.2; // TODO there must be reliance to real-time between frames float mass = 0.1; ofPoint force = gukForce + g * mass; ofPoint a = force / mass; velocity + = a * dt; pos + = velocity * dt; }

The text of the "pendulum" testApp.cpp, p. 2 / 3

// Draw void testApp:: draw () { // Gum ofSetColor (0, 0, 255);// Color blue ofLine (pos0.x, pos0.y, pos.x, pos.y); // Draw a line // Circle ofSetColor (255, 0, 0); // Color red ofFill ();// Insert a fill ofCircle (pos.x, pos.y, 20); // Draw a circle } ... Next comes the rest of the code empty functions for processing the mouse and keyboard. We do not need to change it now.

Homework
On the basis of "pendulum" do "Branching pendulum, with pendulums of different weights. It will have a much more interesting dynamics.

Projects to do
Choose one of the projects for the independent or collective development (or suggest your own idea): 1. 3D Sculpture Creating 2. Flying flowers 3. The dynamic mapping on the cube

Creating 3-D Sculpture

Script The sheet of paper with a marker out is dropped on the table. The system draws some 3-D body (cube or sphere). User takes in hand the second marker starts knocking them in 3D-body, beating of his minor particles. By rotate the sheet of paper and working with second marker,, you can make 3D sculpture. Technology To recognize the markers use Augmented Reality technology (example ARToolKit library). The sculpture can be a set of small cubes (50x50x50).

Flying flowers
Script Classical interactive installation. where the audience waved his hands in front of the camera, and drawn on the screen, such as flower petals. On the screen in a place where waved the audience - the petals scatter in different directions, there appears a picture. After some time the petals fly back into place. The viewer must actively waving his arms to clear the whole picture. Technology 1) Use optical flow and background analysis for analysis of users' movements 2) Do rendering - on openFrameworks, either on Unity3D or TouchDesigner (transfer data from openFrameworks through OSC).

The dynamic mapping on the cube

Script Project images onto cunes faces. Specificity in that cube can be moved, and the camera (or Kinekt) should track it and synchronize projector images accordingly. Thus, no matter how we moved the cube - it will be illuminated by the projector correctly. Technology 1) Tracking the cube faces using pattern recognition technology. Camera and projector calibration is needed also. 2) Do rendering on openFrameworks, either on Unity3D or TouchDesigner (transfer data from openFrameworks through OSC).