You are on page 1of 60

# Closing the Gaps in Inertial Motion Tracking

## Sheng Shen, Mahanth Gowda, Romit Roy Choudhury

Inertial Measurement Unit (IMU)

Accelerometer

Gyroscope

Magnetometer
Inertial Measurement Unit (IMU)

Accelerometer

Gyroscope

Magnetometer
Inertial Measurement Unit (IMU)

Accelerometer

Gyroscope

Magnetometer

## Wide applications in motion tracking

Lot of work in inertial motion tracking

IMU 3D
Data Motion Trajectory
Tracking
Algorithm

## Open problem in mobile computing

Lot of work in inertial motion tracking

IMU 3D
Data Motion Trajectory
Tracking
Algorithm

## Open problem in mobile computing

This paper doesn’t solve it either … however, makes progress
Lot of work in inertial motion tracking

IMU 3D
Data Motion Trajectory
Tracking
Algorithm

## Open problem in mobile computing

This paper doesn’t solve it either … however, makes progress
Am struggling to explain our algorithm … so will start from scratch
One Prerequisite Slide: Rotation Matrices

• Rotation is a function
0 90° 0
Y: X:
1 −1 1
𝑅𝑜𝑡 90° = = Y:
0
1
X:
0

## • Mathematically, rotation is a matrix

cos 90° − sin 90° 1 0 0 −1
=
sin 90° cos 90° 0 1 1 0

## • Same for 3D Rotation

3×3
Rotation =
Matrix
MUSE: Our Goal is 3D Localization

Map
Initial Location
3D
Algorithm
IMU = Accel + Gyro + Mag Trajectory
Let’s Understand the Inputs

𝑥0 𝑦0 𝑧0 𝑇

Initial Location
Algorithm
IMU = Accel + Gyro + Mag

## Zoom into IMU data:

𝑎𝑥 𝑔𝑥 𝑚𝑥
Accel. = 𝑎𝑦 Gyro. = 𝑔𝑦 Mag. = 𝑚𝑦
𝑎𝑧 𝑔𝑧 𝑚𝑧
(Linear Acceleration) (Angular Velocity) (Earth’s Magnetic North)
Delta rotation in unit time
Can we solve localization with these inputs?

𝑥0 𝑦0 𝑧0 𝑇

Initial Location ? 𝑥𝑡
Algorithm 𝑦𝑡
IMU = Accel + Gyro + Mag
𝑧𝑡
One possibility is:

𝑥0 𝑦0 𝑧0 𝑇

Initial Location 𝑥0 𝑡 𝑥𝑡
𝑦0 + ඵ 𝐴𝑐𝑐𝑒𝑙. 𝑑𝑡 2 𝑦𝑡
Accel., Gyro., Mag.
IMU 𝑧0 0 𝑧𝑡

𝑎𝑥
But there is one BIG problem: Accel. = 𝑎𝑦 is in local reference frame
𝑎𝑧
𝑎𝑥
But there is one BIG problem: Accel. = 𝑎𝑦 is in local reference frame
𝑎𝑧
𝑎𝑥
But there is one BIG problem: Accel. = 𝑎𝑦 is in local reference frame
𝑎𝑧

As an analogy
New Delhi Airport What is Map
- Frontwards
3D
- Rightwards
Accelerating Trajectory
- Upwards
𝑎𝑥 : Frontwards w.r.t. the earth?
𝑎𝑦 : Rightwards
𝑎𝑧 : Upwards
This makes orientation necessary
This makes orientation necessary

## • What is 3D orientation? Orientation is the 3D rotation needed to make:

Frontwards Northwards
Plane’s Own Rightwards Eastwards
Upwards Vertical
Front

Up

Right
This makes orientation necessary

## • What is 3D orientation? Orientation is the 3D rotation needed to make:

Frontwards Northwards
Plane’s Own Rightwards Eastwards
Upwards Vertical

Up: Vertical

Front: North

Right: East
So, how will orientation solve the problem?
What we need to do is:
Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

## So 3D orientation is the key.

So how to get 3D orientation?

## Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction
So how to get 3D orientation?

## Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction
So how to get 3D orientation?

## Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction

Rotation 𝑎𝑦 = 0
Matrix 𝑎𝑧 −9.8

## Gravity says a lot about orientation,

but not sufficient

Gravity Downward
So how to get 3D orientation?

## Key idea: What rotation is needed such that

1) Gravity is exactly in my downward direction
2) North is exactly in my frontward direction

## 3×3 𝑎𝑥 0 Tilt + Heading is determined (all 3 DoFs)

Rotation 𝑎𝑦 = 0
Matrix 𝑎𝑧 −9.8
3×3 𝑎𝑥 𝑚𝑥 0 0
Rotation 𝑎𝑦 𝑚𝑦 = 0 100
Matrix 𝑎𝑧 𝑚𝑧 −9.8 0

My orientation
North Frontward
So is Gravity + North enough to get 3D Orientation?

## Only when object is static … but not otherwise. Why?

So is Gravity + North enough to get 3D Orientation?

## Only when object is static … but not otherwise. Why?

Because any motion of the object will reflect
in the accelerometer … thereby
polluting the gravity estimate
So how to get 3D orientation? (Another idea)

## Another Idea for Orientation: Integrate angular velocity from gyro

Initial 𝑡
New
Orientation + න 𝐺𝑦𝑟𝑜. 𝑑𝑡 = Orientation
(at time t)
0

## But gyro drifts, so only useful in short time scales

State of the art today: Sensor Fusion
Static Moving Static Moving Static Moving

Time
Rely mostly on Rely mostly on ……
gravity + North gyro integration

## Always know 3D orientation

Getting back to our goal
Initial 𝑥𝑡
Location
+ 𝑦𝑡
𝑧𝑡
Accel. Location
Static?
Y 3D Accel.
IMU Mag. Orient. IMU ඵ
(Local) 𝑡
N Gyro. 3D (Global)
Orient.
Getting back to our goal
Initial 𝑥𝑡
Location
+ 𝑦𝑡
𝑧𝑡
Accel. Location
Static?
Y 3D Accel.
IMU Mag. Orient. IMU ඵ
(Local) 𝑡
N Gyro. 3D (Global)
Orient.

## Main take away: Gravity is the main anchor for 3D orientation

But what if object is not often static

## This paper: MUSE

What does MUSE do?
MUSE = Estimate 3D Orientation even for moving object

## Static Moving Static Moving

Time
• Use magnetometer as our main anchor
• Joint estimation of orientation and location
What does MUSE do?
MUSE = Estimate 3D Orientation even for moving object

## Static Moving Static Moving

Time
• Use magnetometer as our main anchor
• Joint estimation of orientation and location
Earth’s North Pole

## Earth’s South Pole

From: Wikipedia
Earth’s North Pole Magnetic Field Direction
on Earth Surface

## Earth’s South Pole

From: Wikipedia
Earth’s Magnetic Field as Main Anchor

Z: Up

X: East

## Magnetic Field Direction

Earth’s Magnetic Field as Main Anchor

## This vector is not influenced by motion!

Let’s make it our main anchor

North

Horizontal Plane

## Magnetic Field Direction

Earth’s Magnetic Field as Main Anchor

## 3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Main Anchor
Magnetic Field Direction
Earth’s Magnetic Field as Main Anchor

## 3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Three questions:

## 1) What does it say about my orientation?

2) Magnetometer’s resolution is low …
3) How to know x, y, z of earth magnetic field?
Earth’s Magnetic Field as Main Anchor

## 3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Three questions:

## 1) What does it say about my orientation?

2) Magnetometer’s resolution is low …
3) How to know x, y, z of earth magnetic field?
Actual vs. Inferred Measurement

## Z: Outward Y: Front Z: Outward Y: Front

[Local] [Local]

X: Right X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒 𝒗𝑖𝑛𝑓𝑒𝑟

## Actual Measurement Inferred Measurement

(from magnetometer) (inferred from orientation estimation)
Actual vs. Inferred Measurement

## Z: Outward Y: Front Z: Outward Y: Front

[Local] [Local]

X: Right X: Right
𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒 and 𝒗𝑖𝑛𝑓𝑒𝑟 are equal, if our orientation is estimated perfectly …

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒 𝒗𝑖𝑛𝑓𝑒𝑟

## Actual Measurement Inferred Measurement

(from magnetometer) (inferred from orientation estimation)
Actual vs. Inferred Measurement

Z: Outward Y: Front

[Local]

X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒
𝒗𝑖𝑛𝑓𝑒𝑟

## 2-DoF Orientation Drift == How off 𝒗𝑖𝑛𝑓𝑒𝑟 is from 𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒

Actual vs. Inferred Measurement

Z: Outward Y: Front
∆𝑜𝑟𝑖𝑒𝑛𝑡

[Local]

X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒
𝒗𝑖𝑛𝑓𝑒𝑟

## 2-DoF Orientation Drift == How off 𝒗𝑖𝑛𝑓𝑒𝑟 is from 𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒

Actual vs. Inferred Measurement

Z: Outward Y: Front
∆𝑜𝑟𝑖𝑒𝑛𝑡

[Local]

X: Right

𝒗𝑚𝑒𝑎𝑠𝑢𝑟𝑒
𝒗𝑖𝑛𝑓𝑒𝑟

## New Orientation = Old Orientation + ∆𝑜𝑟𝑖𝑒𝑛𝑡

Actual vs. Inferred Measurement

## Magnetometer: Orientation(t) = Orientation(t) + 0.01 ∗ ∆𝑜𝑟𝑖𝑒𝑛𝑡

Earth’s Magnetic Field as Main Anchor

## 3×3 𝑚𝑥 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥

Rotation 𝑚𝑦 = 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦
Matrix 𝑚𝑧 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧

Three questions:

## 1) What does it say about my orientation?

2) Magnetometer’s resolution is low …
3) How to know x, y, z of earth magnetic field?
Earth’s Magnetic Field as Main Anchor

## How to know x, y, z of earth magnetic field?

• Obtain GPS address and check the database is one option …
• A easier way is to utilize pause opportunities

## 𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑥 3×3 𝑚𝑥

𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑦 = Rotation 𝑚𝑦
𝐸𝑎𝑟𝑡ℎ 𝑀𝑎𝑔.𝑧 Matrix 𝑚𝑧

## Known when object is static

Evaluation
Comparison Baselines

## MUSE Our algorithm

Ground Truth: Periodically bringing the phone to the same orientation, and excluding
those moments from calibration opportunities
Natural Activities
A Sample Trace
OK, we have 3D Orientation now… how to estimate location?
OK, we have 3D Orientation now… how to estimate location?

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)
OK, we have 3D Orientation now… how to estimate location?

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

Localization impossible if
accelerometer has any noise … why?
Motion models can bound location error

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)
Motion models can bound location error

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

## Motion model as priors

MUSE uses this motion model to
jointly estimate both { orientation + location }

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

## Motion model as priors

MUSE uses this motion model (in a particle filter) to
jointly estimate both { orientation + location }

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

## Motion model as priors

OK, we have 3D Orientation now… how to estimate location?

Initial Location 𝑥𝑡
Accel. Accel. + 𝑦𝑡
Gyro. 3D IMU ඵ 𝑧𝑡
IMU (Local) 𝑡
Mag. Orient. (Global)

## Motion model as priors

MUSE Localization Error

Fits motion
model the best

(cm)
Motion model breaks
Conclusion
Conventional Systems This paper

Use magnetometer as
Orientation
a new “gravity” sensor

Joint estimation of
orientation and location
Location