CS 352: Computer Graphics: Input Interaction

CS 352: Computer Graphics
Input
and
Interaction
Chapter 3 - 2
Interactive Computer Graphics
What good is Computer Graphics?

JS1k canvas examples: 1 2 3 4 5 Games, visual demosof what value? Is there a Christian perspective?
Communication of information to the user

Data visualization, simulation, GUI Even a word processor is an "interactive graphics program"
Communication between users

"Collaborative environments" are hot (multi-player games?)
Social networking is transforming the world
Interaction is an essential component
Chapter 3 - 3
Interaction
Much of the fun and utility of graphics is in interaction: accepting user input that affects the display Paint programs, modeling, games, word processors, spreadsheets, powerpoint
User initiates input events such as clicking on a menu or drawing a circle Computer response by changing graphical display
Chapter 3 - 4
Chapter 3 - 5
Projects 3
Projects 3 will be a paint program
we'll learn 2-D graphics, interaction, event-loop programming, double-buffering simple animation, basic image processing Pen, line, and rectangle tools Color, pen size selectors Image processing (sharpen, blur, detect edges) Open, save images Toolbar and menu for controlling application
Features:

Chapter 3 - 6
How to set up an application UI?

How How How How
to to to to
make make make make
a menu? a color picker? a toolbar? toolbar buttons pop in and out?
Chapter 3 - 7
How to paint?
How do I make a colored line follow the mouse or fingertip?
Interactive prog:

Input devices Event handlers Event loop
Chapter 3 - 8
Input devices

Interaction requires handling input devices
Physical: mouse, fingertip, keyboard, joystick,

digitizer, accelerometer, head tracker
Logical:

Text Locator Valuator (slider) Stroke Color picker
How to read?
Chapter 3 - 9
Input devices
Sample mode
Program reads the current state of input device (frequently) Each click or motion generates an event that goes on a queue Program can process events from queue
Event mode
HTML?
Chapter 3 - 10
Event-loop programming
Events: button press, slider value change,

menu selection, mouse move, keypress
Event loop:

Poll (or wait) for events Process events, update state Update display (ideally: wait for rest of frame time to elapse)
Chapter 3 - 11
State-driven
Typically, the event loop is driven by a big state table
e.g. depressing the paintbrush tool releases other tools, puts me in "paint mode"
Good libraries will handle some of the bookkeeping for you You still typically have to handle some state
Chapter 3 - 12
State diagram
Chapter 3 - 13
Event loop in HTML/JS
HTML/JS provides event queue, support for many basic events (mousedown, mouseup
mouseover, mousemove, keypress, key release, value change, button click, etc.)
You are on your own for higher-level events, e.g. clicking on a toolbar tool It's also possible to set a function to run every 15 ms, sample input devices
Chapter 3 - 14
Events for painting?
mousedown:

go into paint mode store mouse position draw initial dot exit paint mode if in paint mode

mouseup:
mousemove:
draw a line from old mouse position to current set old mouse position to current position
Chapter 3 - 15
Event handling with jQuery
Binding events to functions

$(cpaint.canvas).bind('mousedown', cpaint.drawStart)
Processing events
cpaint.drawStart = function(ev) { var x, y; x = ev.pageX - $(cpaint.canvas).offset().left; y = ev.pageY - $(cpaint.canvas).offset().top; ev.preventDefault(); cpaint.paintmode = true; cpaint.oldX = x; cpaint.oldY = y;
Chapter 3 - 16
Looking neat and spiffy
How to avoid crinkles in your paint strokes?
Chapter 3 - 17
Looking neat and spiffy
How to avoid crinkles in your paint strokes?
Draw connected paths
Or just use line caps
Chapter 3 - 18
Menus
There are many jQuery menu plug-ins
<ul id="mainmenu"> <li>File <ul> <li id="menuNew">New</li> <li id="menuOpen">Open</li> <li id="menuSave">Save</li> ------------------------------------$('#menuNew').bind('click', cpaint.clear); $('#menuOpen').bind('click',cpaint.open); $('#menuSave').bind('click',cpaint.save);
Chapter 3 - 19
Toolbar
How to make a toolbar?
How should buttons behave?

State diagram?
Chapter 3 - 20
Buttons

Widgets may have several states State should be evident in visual feedback E.g. how exactly does a button work? States?
State transition diagram
Most buttons: six states, with six different appearances

neutral neutral-highlighted neutral-depressed selected selected-highlighted selected-depressed
Events: mousedown, mouseup, enter, exit Transitions: what happens in each state under each event?
Chapter 3 - 21
Button state diagram

(buttons selectable, not unselectable)
N S
NH
SH N: neutral H: highlighted NHD SHD

(usually mouse over)
Press Move
S: selected D: mouse down
Chapter 3 - 22
Other button considerations
Could also consider

Tooltips Whether button merely clicks and releases or can be selected Whether button can be unselected (e.g. B/I/U vs. Left, Center, Right)
Want consistent appearance, behavior over whole program or whole computer Really need a library implementation and a strict set of UI guidelines
Chapter 3 - 23
4-State toolbar buttons: CSS

.toolbarCell { background-color:#ddd; width:20pt; height:20pt; border: solid #eee 2px; -webkit-box-shadow: 2px 2px 2px #666; }
#markerButton { background: url(img/paintbrush.png) no-repeat center center; }

.selected { -webkit-box-shadow: inset 2px 2px 2px #666; }
.toolbarCell:hover { border:solid #555 2px; }
Chapter 3 - 24
4-State toolbar buttons: JS

$('#markerButton').bind('click', {tool:"marker"}, cpaint.selectTool); -------------------
cpaint.selectTool = function(ev) { cpaint.tool = ev.data.tool;
// get tool name
$('.toolbarCell').each(function(index) { // unselect $(this).removeClass('selected'); // others }); var tool = '#' + cpaint.tool + 'Button'; // get ID $(tool).addClass('selected'); // select
Chapter 3 - 25
Paintbrush size selector
How?
Chapter 3 - 26
Color picker
Google "jQuery color picker"
Chapter 3 - 27
How draw a line?
What kind of feedback is normal?
Rubber Banding
Events and actions?
Chapter 3 - 28
Save & restore

Create your own off-screen canvas Copy it back on each mouse movement Events and actions?
Chapter 3 - 29
Save & Restore event handling
Mousedown

enter line mode remember mouse position as startx, starty save screen draw initial dot paste saved screen draw line from startx, starty to current mouse pos exit line mode
Mousemove
Mouseup
Chapter 3 - 30
Analysis
Drawbacks?
slow eats memory bandwidth for breakfast
copy smallest possible rectangle? Only points from line?
possible flickering
use double buffering?
Chapter 3 - 31
Double buffering

Have two frame buffers, "front" and "back" Draw into back buffer At vertical retrace time, swap buffers
This is a fundamental graphics technique not built into canvas though, some drawing aggregation seems to happen automatically, behind the scenes; not usually necessary in canvas, but browser dependent
Chapter 3 - 32
Fake Double Buffering in Canvas
Create off-screen canvas

canvasBuffer = document.createElement('canvas'); canvasBuffer.width = canvas.width; canvasBuffer.height= canvas.height; canvasBufferContext= canvasBuffer.getContext('2d');
Draw into off-screen canvas

canvasBufferContext.[drawSomeStuff]
Copy onto display canvas

context.drawImage(canvasBuffer, 0, 0);
Chapter 3 - 33
How to erase?
Draw background color
Chapter 3 - 34
Save, Load?
Save
Copy pixels to an image so users can right-click, save-as
Load

What are the security risks? Can only load files from the same server Can use a file chooser if it's a local app Security policies are not entirely in place
Chapter 3 - 35
Resizing

How would you allow the user to shrink, expand image? What would happen to image resolution?
Chapter 3 - 36
Store drawing commands
Realistically, to redraw effectively, you have to save all drawing commands

pen color red fill color blue line width 5 circle 5 10 12 textfont Times 12 text 10 10 "hello world"
Replay commands to redraw scene Could store commands in a file (e.g. Mac PICT file) For some kinds of drawings, files are smaller and more accurate than bitmaps
Chapter 3 - 37
Other toolkits

HTML offers basic UI capabilities in a crossplatform context Writing your own UI extensions is tedious jQuery plugins extend capabilities Ideally, all UI elements ought to be built in In the real world, they come in platformspecific 'toolkits' or 'windowing libraries' E.g. MFC, QT, OpenLook, Cocoa
Chapter 3 - 38
Image processing
Image processing examples:

blur, sharpen detect edges enhance contrast noise reduction posterize fisheye redeye reduction find faces
Chapter 3 - 39
How to fade an image?

Could average pixel colors with white Or just decrease alpha
Chapter 3 - 40
How to blur an image?

Blurring is an averaging process Convolution: apply a convolution kernel, e.g.

1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9 1/9
Center on pixel of interest Multiply each value by color val underneath Add results Gaussian smoothing
Chapter 3 - 41
Gaussian smoothing
Chapter 3 - 42
Chapter 3 - 43
How to sharpen an image?

It's a differentiation process What's Unsharp Masking?
Chapter 3 - 44
How to sharpen an image?

It's a differentiation process What's Unsharp Masking?
Subtract blurred version from original Add back to original
Chapter 3 - 45
Sharpening

Subtract neighbors Like subtracting a blurred version of the image Unsharp Masking E.g. convolve with a kernel like one of these:
Chapter 3 - 46
Edge detection?
Wikipedia
Chapter 3 - 47
Sobel Edge Operators
Look for horizontal and vertical variation

1 0 -1 2 0 -2 1 0 -1 -1
-2 -1
0
0 0
1
2 1
Could do this at different sales or resolutions Note: results maybe positive or negative numbers; must normalize
Chapter 3 - 48
Image Compression
What if you created say six different lowerresolution images and only stored the difference at each resolution?
Note: most of the data is in the highestfrequency component An early image compression technique
Chapter 3 - 49
The alg that changed imagery
Discrete cosine transform (DCT)
(Wikipedia)
Chapter 3 - 50
JPEG image compression

DCT to convert to frequency domain Perceptual modeling Coefficient quantization Entropy coding
Chapter 3 - 51
Research in image processing and computer vision
How would you clear up a blurry photograph?

How would you recognize a face in a photograph?
Chapter 3 - 52
Summary

Event loop programming Interaction and event handling in HTML State diagrams Painting Rubber banding Double buffering Basic image processing and convolutions "Photoshop Nano"

CS 352: Computer Graphics: Input Interaction

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CS 352: Computer Graphics: Input Interaction

Uploaded by

Copyright:

Available Formats

CS 352: Computer Graphics

Interactive Computer Graphics

What good is Computer Graphics?

Communication of information to the user

Communication between users

Social networking is transforming the world

Interaction is an essential component

Interactive Computer Graphics

Interactive Computer Graphics

Interactive Computer Graphics

Projects 3 will be a paint program

Interactive Computer Graphics

How to set up an application UI?

How How How How

make make make make

a menu? a color picker? a toolbar? toolbar buttons pop in and out?

Interactive Computer Graphics

How do I make a colored line follow the mouse or fingertip?

Input devices Event handlers Event loop

Interactive Computer Graphics

Interaction requires handling input devices

Physical: mouse, fingertip, keyboard, joystick,

Text Locator Valuator (slider) Stroke Color picker

Interactive Computer Graphics

Interactive Computer Graphics

Events: button press, slider value change,

Interactive Computer Graphics

Typically, the event loop is driven by a big state table

Interactive Computer Graphics

Interactive Computer Graphics

Event loop in HTML/JS

Interactive Computer Graphics

Events for painting?

Interactive Computer Graphics

Event handling with jQuery

Binding events to functions

Interactive Computer Graphics

Looking neat and spiffy

How to avoid crinkles in your paint strokes?

Interactive Computer Graphics

Looking neat and spiffy

How to avoid crinkles in your paint strokes?

Draw connected paths

Or just use line caps

Interactive Computer Graphics

There are many jQuery menu plug-ins

Interactive Computer Graphics

How to make a toolbar?

How should buttons behave?

Interactive Computer Graphics

State transition diagram

Most buttons: six states, with six different appearances

neutral neutral-highlighted neutral-depressed selected selected-highlighted selected-depressed

Interactive Computer Graphics

Button state diagram

SH N: neutral H: highlighted NHD SHD

S: selected D: mouse down

Interactive Computer Graphics

Other button considerations

Could also consider

Interactive Computer Graphics

4-State toolbar buttons: CSS