What happens during bundle adjustment?

Bundle adjustment is a critical step in the process of 3D reconstruction from images

using structure-from-motion (SfM) and multi-view stereo (MVS) techniques. It
involves refining the camera poses and 3D point positions simultaneously to
minimize the difference between the observed image features and the predicted
ones based on the estimated camera and scene geometry.

During bundle adjustment, the algorithm adjusts the estimated camera parameters
(position, orientation, and internal calibration parameters such as focal length and
principal point) and the 3D positions of the reconstructed points to minimize the
geometric error between the observed image features and the projected ones based
on the current estimates. This involves solving a non-linear optimization problem
with a large number of variables, where the goal is to find the parameters that
minimize the sum of the squared differences between the observed and predicted
image features.

Bundle adjustment typically involves minimizing a cost function that captures the
difference between the observed and predicted image features, as well as any
additional constraints such as smoothness or sparsity of the 3D structure. The
optimization is usually performed using iterative methods such as Levenberg-
Marquardt or Gauss-Newton, and can be computationally expensive, especially for
large datasets.

The output of bundle adjustment is an optimized set of camera poses and 3D point
positions that provide a more accurate representation of the scene geometry than
the initial estimates. This refined structure can be used for further processing such as
dense point cloud generation, texture mapping, or 3D model reconstruction.