USMAN RAIS 781-2020 COMPUTER VISION 16/05/23

ASSIGNMENT 04

LUCAS-KANADE (LK) ALIGNMENT AND BAKER-MATTHEWS (BM)

ALIGNMENT

The Lucas-Kanade (LK) alignment and Baker-Matthews (BM) alignment

are two popular methods in computer vision for estimating the optical flow
or motion between consecutive frames of an image sequence. Both
methods have applications in various fields such as object tracking, video
stabilization, and motion analysis. Let's explore each alignment method and
its applications in detail:
Lucas-Kanade (LK) Alignment:
The Lucas-Kanade algorithm is a widely used technique for estimating the
optical flow between two frames. It assumes that the displacement of pixels
between frames is small and tries to minimize the difference between the
intensities of corresponding pixels in two frames.
The basic steps of the Lucas-Kanade algorithm are as follows:
a. Compute the gradient of the first frame.
b. Select a region around a pixel in the first frame.
c. Estimate the displacement by finding the best match between the pixel
intensities in the selected region of the first frame and a corresponding
region in the second frame.
d. Repeat the previous steps for all pixels in the first frame.
Applications of Lucas-Kanade alignment:
USMAN RAIS 781-2020 COMPUTER VISION 16/05/23

a. Object tracking: Lucas-Kanade is often used for tracking objects in video

sequences. By estimating the motion vectors of key points in the first
frame, it can track those points in subsequent frames, enabling the tracking
of objects in real-time scenarios.
b. Optical flow estimation: The LK algorithm is useful for estimating the
dense optical flow, i.e., the motion vectors of all pixels between two frames.
This information is valuable for tasks like video compression, action
recognition, and motion analysis.
1. c. Structure from motion: Lucas-Kanade can be used to estimate the
3D structure and camera motion from a sequence of 2D images. By
tracking feature points across frames, it can recover the 3D motion
and shape of the scene.
Here are some additional details about the Lucas-Kanade (LK) alignment
1. Intensity-based approach: The LK algorithm is based on the
assumption that the intensity values of pixels within a small
neighborhood remain constant between consecutive frames, under
small motion. It uses this assumption to estimate the displacement of
pixels.
2. Spatial and temporal coherence: The LK algorithm exploits spatial
and temporal coherence by considering the local neighborhood of
each pixel and assuming that nearby pixels undergo similar motion. It
formulates an equation system to minimize the difference between
the pixel intensities in the first frame and their corresponding
positions in the second frame.
3. Gaussian derivatives: To compute the gradient of the first frame, the
LK algorithm often uses Gaussian derivatives for better robustness
against noise and to capture both local and global image structures.
USMAN RAIS 781-2020 COMPUTER VISION 16/05/23

Baker-Matthews (BM) Alignment:

The Baker-Matthews algorithm is an extension of the Lucas-Kanade
algorithm that addresses limitations when dealing with large displacements
and occlusions. It uses an iterative approach to handle these issues and
improve the accuracy of motion estimation.
The key idea in the Baker-Matthews algorithm is to progressively refine the
estimation by considering multiple scales of image pyramids. It starts with a
coarse-to-fine alignment, where the initial estimation is done on
downsampled versions of the images. Then, the alignment is refined at
each scale to handle large displacements and occlusions.
Applications of Baker-Matthews alignment:
a. Robust object tracking: The Baker-Matthews algorithm is particularly
useful when tracking objects with large displacements or occlusions. By
incorporating iterative refinement and scale-space representation, it can
handle challenging tracking scenarios more robustly.
b. Augmented reality: In AR applications, accurate alignment of virtual
objects with real-world scenes is crucial. The Baker-Matthews algorithm
can be used to estimate the motion of the camera and align virtual objects
to the real environment, enhancing the visual coherence and realism of
augmented scenes.
2. c. Video stabilization: Camera shake can introduce unwanted motion
in videos. The Baker-Matthews algorithm can be employed to
estimate the motion caused by camera shake and compensate for it,
resulting in stabilized videos with reduced jitter.
In summary, both the Lucas-Kanade and Baker-Matthews alignment
techniques have applications in object tracking, optical flow estimation, and
USMAN RAIS 781-2020 COMPUTER VISION 16/05/23

motion analysis. While Lucas-Kanade is a fundamental method for

estimating optical flow, Baker-Matthews extends it by addressing
challenges like large displacements and occlusions, making it more robust
for various computer vision tasks.

Here are some additional details about the Baker-Matthews alignment:

1. Iterative refinement: The BM algorithm improves upon the LK
algorithm by iteratively refining the motion estimation. It starts with an
initial estimation, usually obtained using LK, and then progressively
refines it using a multi-scale approach.
2. Scale-space representation: BM uses a scale-space representation of
the image pyramid, where each level represents a different scale of
the image. By performing alignment at different scales, BM can
handle large displacements and occlusions that may occur in a single
scale.
3. Feature-based approach: While LK works with pixel intensities, BM
often employs feature-based techniques. It extracts robust features,
such as corners or scale-invariant keypoints, and estimates the
motion based on matching and tracking these features across
frames.
4. Robustness to occlusions: BM addresses occlusions by introducing a
motion confidence measure. It analyzes the consistency of motion
estimates across different scales and regions to detect occlusions
and handle them more robustly during the alignment process.
USMAN RAIS 781-2020 COMPUTER VISION 16/05/23

Applications:
1. Object tracking: Both LK and BM alignments are widely used for
object-tracking tasks in computer vision. By estimating the motion
between frames, they can track objects in videos or live camera
feeds, enabling applications such as surveillance, activity recognition,
and augmented reality.
2. Motion analysis: Optical flow estimation, which involves estimating
the motion vectors of pixels between frames, is a key application of
LK and BM. Optical flow information can be used for motion analysis,
such as understanding human movements, analyzing vehicle
behavior, or detecting anomalies in video sequences.
3. Video stabilization: Both LK and BM alignments have applications in
video stabilization. They can estimate the camera motion caused by
shake or jitter and compensate for it by applying the inverse motion to
the frames. This results in smoother and more stable videos.
4. Structure from motion: LK and BM alignments are used in structure
from motion (SfM) algorithms, which aim to reconstruct the 3D
structure and camera motion from a sequence of 2D images. By
tracking feature points across frames, these alignments contribute to
estimating the camera motion and 3D structure.
It's worth noting that the Lucas-Kanade and Baker-Matthews alignments
are part of a broader field of techniques for motion estimation and tracking.
These methods have been influential in computer vision research and have
inspired many subsequent developments in the field.