PhonePoint Pen:

Using Mobile Phones to Write in Air

Noting small pieces of information,
quickly and effortlessly
can be useful

Buy
Milk

So, what are the options ...

 State of the Art

▪ Sticky notes …
▪ organization is a nightmare
▪ not handy

▪ Typing on keyboard
▪ tiny keyboard sizes
▪ small inter-key spacing

▪ Audio recording
▪ cannot sketch information
▪ time consuming to browse through voice messages
So, need a solution that is

▪ Easy to use

▪ Always-with-me

▪ Allows sketching

▪ Searchable
 Proposed Approach: PhonePoint Pen

▪ User can write messages in air

▪ holding the phone like a pen

▪ Use built-in accelerometer

▪ capture hand movement

▪ Decode text and image

▪ sent to user’s email address for future use
 Some Motivations
• Make “post-it” notes on the fly
• Assistive communications for disabled persons and/or the elderly
• Sketching complex ideas, equations and non-standard information
• Emergency operations and first responders
 Primary Goals
• To explore the viability of using mobile phone accelerometers to
write in the air.
• To understand and overcome limitations with character recognition.
• To develop a prototype on the Nokia N95 platform and perform a
test study.
 Design Challenges (1)

▪ Lack of a Gyroscope
▪ Accelerometers only measure linear acceleration
▪ Linear Movements – Rotation Ambiguity

Proposed Approach:
• Hold Phone in Non-rotating Grip
• Determine Angular Orientation during the pause
 Design Challenges (2)

▪ Background Vibration (Noise)

▪ Sensitive Accelerometers
▪ Significant Jitter by hand vibrations

Proposed Approach:
• Smooth the accelerometer readings with moving
average over last n (=7) readings
• Suppress acceleration values < 0.5
 Design Challenges (3)

▪ Computing Displacement
▪ The physical position in the air is necessary to know the
size and relative position of the characters
▪ Erroneous Acceleration Reading
▪ Ambiguity when acceleration is zero

Proposed Approach:
• Detect Pause Using Moving Window
• Reset Velocity to Zero in Between Strokes
• If n readings were measured as all background noise,
it is likely a pause and the velocity is reset to zero
 Design Challenges (4)

Differentiating “A” from a Triangle

▪ Lifting Pen from the Paper

Proposed Approach:
• Impulse on the Z axis during the lift
• Pause and change in the direction as indication

P
 Design Challenges (5)

Identifying character transitions

▪ Certain ambiguities even if phone lifts are recognized
▪ Example, B and 13

Proposed Approach:
• Special gesture between characters, like a “dot”
• Delimiting characters: estimating next stroke based on
what is written till now
 Drawing a Rectangle

Raw Accelerometer Reading

 Drawing a Rectangle

Noise Smoothing
using Moving
Average

Background Noise
Suppression
 Drawing a Rectangle

Velocity Plot after

Avoiding Velocity
Drifts
Final Rectangle
Drawing a Rectangle

(From source article)

 System Design

Stroke Detection
▪ Identified a set of 6 strokes for English characters
▪ Compute a running variance of the accelerometer
readings
▪ If variance falls below a threshold, detected as pause
 System Design

Character Recognition
•Uses a stroke grammar
•The grammar is essentially a tree and expresses the
valid sequence of strokes to form an alphabet
•Confusion between N, D, and P
•Grammar ambiguity:
•O and S => direction of movement during second
stroke, if upwards +ve Y acc then O else vice versa
•D and P =>relative sizes of Y axis movements
•X and Y
•O and 0
Stroke Grammar

(From source article)

 System Design

Word Recognition: multiple heuristics

• Transition from one alphabet to next, longer pause
• If hand moves in the leftward direction, start of a new
character
• Asking the users to give a special gesture like “dot”
between characters
 System Design

P3- aware Spelling Correction

• Spell correction tools computes a list of valid English
words based on edit distance
• E.g., MQM => MOM, MAM, MUM (1 edit distance)
• P3 aware tool will suggest MOM with higher
confidence
• E.g., NIET => NET (1) and MET(2)
• P3 confuses M and NI with more probability than E
with IE
• P3 aware tool will suggest MET with higher
confidence
 System Design

Control gestures
• Separation between words=> long horizontal
movement or two dots
• Deletion => user shakes her hand at least 4 times
briskly
• To email => draw a check mark
 Implementation
• Prototyped on Nokia N95 phone platform.
• A server-side implementation was developed in MATLAB, which they
were able to use basic libraries for the signal processing.
• The on-phone processing version was implemented in Python, but
stripped-down to only one character at a time and simpler signal
processing methods.
 User Evaluation
•Tests were conducted with 10 average users to write the
English alphabet: mainly CS and Engineering students.
•4 subjects were trained on the system, while the other 6
“novice” subjects had writing less than ten characters.
•Another 5 individuals took part in 8-character tests at Duke
University Hospital under supervision. These patients either
suffered from cognitive disorders or motion impairments.
 Evaluation metrics
• Character recognition accuracy (CRA): fraction of successful typed
text recognitions
• Human readability Accuracy (HRA): to evaluate the quality of
geometric representation: display the geometrics characters to a
human, ask her to recognize them,
• Word recognition accuracy (WRA): randomly picked words from a
dictionary and users were requested to write them, WRA for basic
P3, WRA for English spelling correction, WRA with
P3-aware-correction, and WRA with human readability
 Performance Evaluation
•Human readability accuracy (HRA) of the characters was
around 83-85% for the student test subjects. Character
recognition accuracy (CRA) was 91.9% for trained users and
78.2% for novices
•Most novice users achieve similar CRA to trained ones
except 2 weak users
•However, the weak users achieve comparable accuracy by
writing on a table top
•Average character writing time is between 3.02-4.3 sec
•Word recognition accuracy degrades with increasing word
length, with spell correction WRA is 80% for 5 letter words,
HRA is 70%
(From source article)
 Experiences with Hospital Patients
•The hospital patient 8-character HRA test results were:
User 1 – 1/8
User 2 – 1/8
User 3 – 1/8
User 4 – 5/8
User 5 – 0/8 (could not operate button)
 Patient Barriers
• The PhonePoint Pen required a button to be pressed to begin and
end the writing.
• Shoulder, elbow, and wrist coordination for large 12in characters can
be difficult.
• Familiarity with mobile devices.
• IRB restrictions make it difficult to exhaustively test patient use.
• Emulating patients’ experience by having right-handed users write
using left hands
• CRA is 81.73%
 Improvements To Be Made
• Faster writing (only 3.02s per letter on average)
• Writing longer words/drawings
• Cursive handwriting
• Writing while moving
• More diverse test subjects (CS/Enginering majors likely more
technologically-inclined.)
• More advanced algorithms (Bayesian Networks, Hidden Markov
Models, etc.)
Conclusions and Related Research

• PhonePoint Pen compares well to other research in this area

considering its limited processing power and sensor hardware (just
the accelerometer)
• Air-gestures with 3D accelerometers (uWave)
• Vision-based gestures (Microsoft write-in-the-air)
• Stylus-based sketch recognition (SketchREAD)
• Wiimote, Logitech Air-Mouse, Nokia NiiMe
• Smart Pen and SmartQuill
 Strengths
• The idea of using phone like a pen to write on characters/words on
air is novel
• The selection of the set of strokes and coming up with a simple
stroke grammar is a significant contribution
• The paper is easy to read, pictures of air-written characters seem
interesting
 Weaknesses

• The processing is done at the server, the results shown are from the
server, how would the accuracy be with processing done at the
phone
• No discussion about ground truth collection, how are the images of
the words are shown
• The evaluation is limited w.r.t. number of users, scenario etc, the
novice users also had a small training session
• The design of the system limits writing speed, that further limits
applicability of the system in real world use cases.