See

discussions, stats, and author profiles for this publication at: http://www.researchgate.net/publication/265294266

Automatic stereo model reconstruction using

directly measure orientation parameters

CONFERENCE PAPER · NOVEMBER 2004

READS

30

4 AUTHORS, INCLUDING:

Antonio Maria Garcia Tommaselli Mauricio Galo

São Paulo State University São Paulo State University
127 PUBLICATIONS 134 CITATIONS 108 PUBLICATIONS 78 CITATIONS

SEE PROFILE SEE PROFILE

All in-text references underlined in blue are linked to publications on ResearchGate, Available from: Mauricio Galo
letting you access and read them immediately. Retrieved on: 10 November 2015
 AUTOMATIC STEREO MODEL RECONSTRUCTION USING
DIRECTLY MEASURED ORIENTATION PARAMETERS

Roberto da Silva Ruy

Antonio Maria Garcia Tommaselli
Elaine Reis Costa
Mauricio Galo
Dept. of Cartography, São Paulo State University at Presidente Prudente
(rruy, tomaseli, elaine, galo)@prudente.unesp.br

KEY WORDS: Photogrammetry, Acquisition, Correlation, Orientation, Reconstruction,

Georeferencing, Automation.

ABSTRACT

With the availability of digital cameras, satellite positioning systems (GPS) and inertial navigation
systems (INS), new applications of fast cartography became feasible, as disasters mitigation and
environment monitoring. Such applications require georeferenced information with characteristics of
speed and reliability that can be obtained by digital photogrammetry, using additional data from sensors
providing camera attitude and position. Generally, the parameters supplied by the sensors are under the
required accuracy to generate a stereo model without vertical parallax (py). To solve this problem, a
methodology is proposed in this paper, based on the reprocessing of the exterior orientation parameters
(EO) supplied by the sensors, using a modified coplanarity model. In order to refine these directly
measured orientation parameters, some tie points are automatically measured in each image using an area-
based correspondence method. The correspondence process is preceded by an analysis of the matching
areas, to increase the robustness of the process, avoiding suspect correspondences. After the EO
parameters refinement, the images are normalized through an epipolar resampling, in order to provide a
confortable stereoscopic visualization of the model and to facilitate the process of automatic Digital
Terrain Model generation. Some computational programs were developed and experiments were
performed, in order to verify the performance of the method with respect to vertical parallax elimination
and accuracy of reconstructed object space coordinates. These experiments will be presented and
discussed showing the effectiveness of the proposed approach for automatic generation of stereo models.

1. INTRODUCTION

With the evolution of digital image acquisition devices, sensors for direct positioning
and attitude determination, new applications of fast cartography became feasible, as disasters
mitigation and environment monitoring. These systems are based on digital cameras and sensors
for direct georeferencing (GPS+INS) and are used in the fast acquisition of georeferenced data.
A key problem with digital cameras is the sensor resolution and the sensitive area of the
sensor, when compared to convencional metric cameras in Photogrammetry. In this situation,
many frames are necessary to exposure a reference area on terrain. Thus, the complexity of EO
determination by aerotriangulation increases, because there are many control points and tie
points in the process, resulting in one expensive ground control surveying.
One of the alternatives to solve this problem is the integration of GPS and INS for direct
determination of the EO parameters. However, the parameters supplied by these sensors are
under the required accuracy to generate a stereo model without vertical parallax (Mostafa e

XI Simposio Latinoamericano Sobre Percepção Remota Y Sistemas de Informacion

Espacial - SELPER, 22-26 de Noviembre, Santiago, Chile, 2004.
Schwarz, 2001; Colomina, 1999 e Brzezinska, 1999), difficulting to the stereoscopic
visualization of the scene and affecting the 3D reconstruction.
In this paper, a methodology to remove the residual vertical parallax in the model is
presented. This approach is based in reprocessing of the orientation parameters supplied by the
sensors for direct georeferencing, using a modified coplanarity model, by a Least Square Method
(LSM). The observations are some tie points coordinates automatically measured in the model.
Finally, the images are normalized by epipolar resampling, in order to remove the vertical
parallax, providing a suitable stereoscopic visualization of the model, and improving the
correspondence process.

2. DIRECT GEOREFERENCING

Direct georeferencing consists of the determination of the camera exterior orientation

parameters during exposure, using the integrated GPS and INS. The satellite positioning system
(GPS) is used for camera perspective center location. In the inertial navigation system (INS), a
Inertial Measured Unit (IMU), composed by accelerometers and gyroscopes, determines the
camera orientation during the flight.
In this context, the integrated GPS and INS can be a powerful system. The inertial
system can help GPS by providing accurate initial position and velocity information after signal
loss. Even during satellite outages where the number of visible satellites drops below four, INS
will provide continuous trajectory information. On the other hand, the high absolute performance
obtained by GPS can help the inertial navigation system with accurate estimates on the current
behaviour of its error statistics. If Kalman filtering is used in traditional navigation approaches,
the internal INS errors are modelled as gyroscope drifts and accelerometer offsets, using GPS
informations (Cramer & Stallmann, 2002).
One point that must be considered in direct georeferencing is the reference system. The
orientation angles from GPS/INS are not comparable to the photogrammetric angles (ω, ϕ, κ).
Because the integrated GPS and INS solution was originally designed for navigation purposes,
the computed attitudes are interpreted as navigation angles (roll – r, pich – p, yaw - y). These
angles are obtained in a local navigation system, whose origin is located in the center of the
inertial sensors axes triad. Since the INS is moving relatively to the earth’s surface, this local
frame is not constant but moving with time. Consequently, it is necessary to transform the
navigation angles to photogrammetric attitudes, via the Cartesian earth-centered earth-fixed
coordinate system (Cramer & Stallmann, 2002).
In image orientation there is an important problem: the overall system calibration. Any
discrepancies between the assumed mathematical model used and the true physical reality during
image exposure will cause erros in object point determination. In direct georeferencing the
interior geometry of the images is of crucial important, because GPS/INS provides direct
measurements of the true physical camera position and orientation during exposure, whereas in
the bundle adjustment the values of the exterior parameters are estimated indirectly.
A great advantage of direct georeferencing is to minimize or even eliminate the need for
control points in exterior orientation process, mainly in areas with difficult access.
Although GPS and INS systems have high cost, the direct sensor orientation during the
flight can be advantageous, because:
• Fast determination of the EO parameters, because the aerotriangulation can be eliminated or
totally automatized;
• It allows to observe variant phenomenons in the time, that need fast cartography (flood, sign of
petroleum, etc);
• To limit the flight direction, avoiding to make a block of images;
• It allows the automatic control of the exposures of the camera;

The acurracy of the EO parameters provided by the integrated GPS and INS depends on
the system. Systems used for mapping present an accuracy around 0,1-0,3m for camera
perspective center location (X0, Y0, Z0) and 1-3’ for attitude angles (ω, ϕ, κ).

3. METHODOLOGY

The methodology proposed in this paper is based on the reprocessing (adjustment) of

the exterior orientation parameters provided by the integrated GPS and INS, for removing the
vertical parallax in the stereo model.
The steps involved in this methodology can be seem in Figure 1.

EO parameters
(GPS+INS) and
camera calibration
Stereopair
data

Automatic tie points measurement

Reprocessing of the EO parameters

Epipolar resampling

Analysis of results

Figure 1. Steps of the automatic stereo model reconstruction.

3.1 Automatic Tie Points Measurement

The reprocessing of the EO parameters using an adjustment by the Least-Squares

Method (LSM), requires some tie points that can be automatically measured in the model. In this
approach one simple tie point is enough to refine these parameters, since all parameters are
considerated as weighted constraints. Nevertheless, the adjustment process will be more reliable
if more tie points are included in this process, since points measured erroneously can be removed
by the residual analysis.
An approach for automatic tie points measurement was implemented, using an area-
based correspondence method, employing a correlation function and LSM matching. To avoid
false correspondences and to limit the search area, an a priori analysis, based on the gray values
of the template window was developed.
First, the area of the model needs to be determined. This is performed by projecting the
limits of the right image on to the left image by the collinearity equations, using the EO
parameters provided by GPS/INS. Epipolar lines are used to reduce the search space for the area-
based correspondence methods. For details, see Schenk (1999).
In the model, the templates (reference window) can be established. The template size
and the number of tie points are previously defined in a configuration file. The correspondence
process is preceded by an analysis of the matching areas, to avoid unsuitable (homogeneous)
areas. This a priori analysis is based on the computation of the covariance matrix (∑) of the
radiometric translation components of the matching areas in the template. This matrix is a
function of the noise variance and the gray level gradients on the template. Thus, the area-based
matching (cross-correlation technique and LSM) is applied only for those areas with acceptable
translation covariance matrix. This a priori analysis is performed in three steps, based in
Haralick and Shapiro (1993):

1) Computation of the noise variance ( σ n2 ) based on the template variance ( σ r2 ) and the
maximum correlation coefficient (ρ12):
σ r2 (1 − ρ12 ) (1)
σ n2 =
ρ12
The maximum correlation coefficient (ρ12) is a matching value theoretically correct for
the images. Since this approach is totally automatic, this coefficient was defined as 0,8, based in
empirical evaluations.

2) Computation of the normal equations coefficient matrix (N) based on the gradients of the
template:

Considering only radiometric translations between the images, the linearized model for
the correspondence process is given by equation 2, where the translation parameters (∆x, ∆y) can
be estimated in an adjustment by the Least Squares Method:

∆g = − f l (∆x) − f c (∆y ) + n (2)

where n = noise;
f r e f c = row (r) and column (c) gradients.

The normal equation coefficient matrix (N) is given by:

 m m

 ∑ f ri
2
∑ f ri f c  (3)
=  
i
i=1 i=1
N
 m m



∑i=1
f ri f c i ∑i=1
f c
2
i 


3) Computation of the accuracy of the translation parameters in function of the noise

variance ( σ n2 ) and the normal equations coefficient matrix (N):

The accuracy of the translation parameters (∆x, ∆y) is given by:

1  n22 −n12 
Σ∆x,∆y = σˆ n2 ⋅ N −1 = σˆ n2 ⋅ 
2 − n
 (4)
n11n22 − n12  21 n11 
where σ̂ 2n = the estimated noise variance considerated equivalent to the computed noise
variance ( σ 2n );
n11 ,..., n 22 = elements of the matrix N.

Then:
 n22 n11
σˆ ∆x = σˆ n ⋅ 2
, σˆ ∆y = σˆ n ⋅ 2 (5)
n11n22 − n12 n11n22 − n12

Based on the previous equations it is possible to note that the covariance matrix of the
radiometric translation components, and consequently the standart deviation of the parameters,
depends on:
• The noise variance;
• The number of pixels of the template, since the standart deviation decreases linearly when the
template size increases;
• The gray level gradients in the template that indicates the presence of borders.

Considering the noise variance as a constant in overall image, it is possible to determine

the locations which most probably will provide high accuracy in the correspondence process,
based only on the template data.
In this work the thresholds for unsuitable areas were empirically established after some
tests with real data. These tests were performed in various conditions, from homogeneous areas
to areas with high variance. Thus, based on these experiments, the followings thresholds were
considered:
Template variance ( σ r2 ) < 400 or ∑ trace > 0,05
If the template area is considerated problematic, the template is displaced for three
horizontal pixels. This approach is repeated until the template will be considerated suitable for
the correspondence process.

3.2 Reprocessing of the EO parameters

For reprocessing of the EO parameters, an adjustment by LSM based on the following

modified coplanarity model (eq. 6) was performed.

(X L − X L )* (v1 ⋅ w2 − v2 ⋅ w1 ) +
(YL − YL )* (u 2 ⋅ w1 − u1 ⋅ w2 ) +
2 1

(6)
(Z L − Z L )* (u1 ⋅ v2 − u 2 ⋅ v1 ) = 0
2 1

2 1

with:
 u1   x  u2   x 
  T    T 
 v1  = M 1  y  ,  v2  = M 2  y  (7)
 w1  − f   w2  − f 
1 2
where:
(X L , YL , Z L ) : camera perspective center location for the left image in object space coordinate
1 1 1

system;
(X L , YL , Z L ) : camera perspective center location for the right image in object space coordinate
2 2 2

system;
M1 : rotation matrix for the left image;
M2 : rotation matrix for the right image;
(x, y,− f )1 : image coordinates of a point in the left image;
(x, y ,− f )2 : image coordinates of a point in the right image.
 Since this model is represented by one equation for each measured point in the images,
then:
• Number of equations (r) = number of measured points in the images (np);
• Number of observations (n) = 4 * number of measured points in the images (np);
• Number of constraints (s) = number of parameters (u) = 12.

With S = r + s − u degrees of freedom it is necessary only one equation for the

reprocessing by the Least Squares Method, since all the exterior orientation parameters are
considered as weighted constraints. For details of this approach, see Ruy (2004).

3.3 Epipolar Resampling

This process is carried out for removing the vertical parallax and to provide a suitable
stereoscopic visualization of the model, with the resulting improvement also of the
correspondence process, because the conjugate entities are confined to the same row.
For transforming the original images in their normalized positions, two stes are required
(Mikhail, 2001):
• The images are transformed for their vertical positions, by computing the rotation matrixes;
• Starting from vertical images, the images are transformed for their normalized positions, by
computing the base rotation matrix.

4. EXPERIMENTS

Based in the methodology presented in section 3 some computational programs in

C/C++ language were developed. Some experiments were performed to access the vertical
parallax effect and the accuracy of check points coordinates on terrain, before and after the
reprocessing of the EO parameters. Two experiments with real data will be presented in this
paper.

First Experiment

In this experiment a pair of images taken by the KODAK PRO 14n digital camera was
used, over the Unesp campus in Presidente Prudente, SP, Brazil. The images were obtained from
a flight height of 1550m, with a scale of 1:30000, image resolution of 4500 x 3000 pixels and
pixel size of 8µm. The nominal camera focal length was 50mm.
Since this flight was performed without direct orientation sensors, the EO parameters
were estimated in an aerotriangulation process, using a digital photogrammetric workstation
(Socet set - LHSystem). In order to simulate direct orientation using GPS and INS, some random
perturbations were introduced in the position and attitude parameters. Standart deviations of
0,5m and 10’ were considered for the perspective center location and the attitude angles,
respectively. These values were used for simulating low cost direct orientation sensors.
The observations (tie points coordinates) were automatically measured in the model,
using the approach presented in section 3.1. For these experiments nine points were
automatically measured, but two of these points were eliminated after the residual analysis
(residuals higher than 0,5 pixel).
A set of experiments with this level of accuracy was performed and the results are
presented in Table 1. The Root Mean Square (RMS) values for five check points, and the vertical
parallax (py) in the model are presented considering three situations: 1) after the
aerotriangulation; 2) with the EO parameters degraded (before reprocessing) and; 3) after
reprocessing of the EO parameters.

Situations RMS X(m) RMS Y(m) RMS Z(m) py (mm)

1 0,68 0,63 1,67 0,0223
2 1,32 1,55 9,47 0,0834
3 1,60 3,30 4,27 0,0026
Table 1. RMS values of check points and vertical parallax in the model.

It was verified that the vertical parallax was reduced from 0,0834mm (10,43 pixels) to a
minimum value of 0,0026mm (0,33 pixel), which is smaller than the parallax after the
aerotriangulation.
The planimetric components showed higher errors after the reprocessing of the EO
parameters (See Table 1), probably due to the lack of accurate interior orientation parameters.
The RMS error of the computed elevations was reduced, but it still can be considered a high
value, due the lack of knowledge of the focal length.
In these experiments, systematic discrepancies on the check-points coordinates were
verified. The resulting model was translated and rotated in the space, with respect to the
reference system defined by the control points (Figure 2). In this context, if some control points
(minimum of three) are available, a seven-parameter transformation can be applied to correct the
ground coordinates, although this is out of the objectives of the direct georeferencing
methodology.

N
1m

1m (b)
E (a)

Figure 2. Discrepancies on the five check points coordinates in the model: (a) planimetric
components; (b) elevation component.

The visual effect of the vertical parallax in the model can be seen in Figure 3, that
shows a part of the model before and after the reprocessing of the EO parameters, with the
normalized images.
The influence of the vertical parallax on stereoscopic visualization of the model can be
seen in Figure 3. In Figure 3(a) stereoscopic visualization is difficult because of the high vertical
parallax value (about ten pixels). After the reprocessing of the EO parameters and epipolar
resampling however, the vertical parallax was significantly reduced (0,33 pixel), providing a
suitable stereoscopic visualization.
 (a) (b)

Figure 3. Vertical parallax in the model. (a) before and (b) after the reprocessing of the EO
parameters and epipolar resampling.

Second Experiment

In this experiment a stereo pair of aerial digital images with direct orientation data
(GPS+INS) was used. These images were provided by the Center for Mapping (Ohio State
University). The flight was performed with a digital camera produced by Lockheed Martin
Fairchild Semiconductors and the EO parameters were estimated by the integrated GPS Trimble
4000SSI and INS Litton LN-100, providing an accuracy about 0,3m (X0, Y0, Z0) and 1’ (ω, ϕ, κ).
The images were captured with a flight height around 650m, with an image scale of
1:12000, resolution of 4096 x 4096 pixels and pixel size of 15µm. The calibrated camera focal
length was 51,569mm.
The EO parameters were reprocessed after the automatic measurement of six tie points.
In Table 2 the original EO parameters, provided by the integrated GPS and INS, adjusted EO
parameters, and the vertical parallax values (py) are presented.

EO parameters Original parameters (GPS+INS) Adjusted parameters

X 0L ( m ) 138670,655 138670,649
Y0L (m) 137211,441 137211,369
L
Z (m)
0 1355,515 1355,297
ω ()
L o
2,93061190 2,96582343
ϕ ()
L o
-1,27860300 -1,30260697
κ ()
L o
177,45700910 177,48079091
R
X (m)
0 138472,650 138472,656
R
Y (m)
0 137216,778 137216,850
R
Z (m)
0 1358,557 1358,774
ω ()
R o
5,60728800 5,57025040
ϕ ()
R o
-3,93530190 -3,97882988
κ ()
R o
177,48427930 177,49709949

initial py (mm) 0,0583

py after the reprocessing (mm) 0,00022
L: left; R: right.
Table 2. EO parameters and vertical parallax, before and after the adjustment.
 In Table 2 it can be verified that the adjusted EO parameters are almost equivalent to the
original EO parameters (RMSCP = 0,1323m and RMSAttitude = 0,0321°), because of the good
quality of the original parameters. However, the vertical parallax was reduced from 0,0583mm
(about four pixels) to 0,00022mm (0,015 pixel).
Its is important to mention that the ground accuracy assessment was not performed in
this experiment due to the lack of ground control points.
After the reprocessing of the EO parameters the epipolar resampling was performed. In
Figure 4(a) it is presented a part of the stereo model using the original parameters (before the
adjustment), and in Figure 4(b) the same part of the model after the adjustment and with
normalized images.

(a) (b)

Figure 4. Vertical parallax in the model. (a) before and (b) after the reprocessing of the EO
parameters and epipolar resampling.

The stereoscopic visualization using the original stereopair was perturbed by the
parallax of about four pixels while the reprocessed images just a residual of 0,015 pixel was
verified.

5. CONCLUSIONS

This paper presented an approach for removing the vertical parallax in the model, when
considering EO parameters from integrated GPS and INS.
In all studied cases the proposed approach was efficient in removing the existing
vertical parallax, due to the level of accuracy of the directly measured EO parameters.
The experiments with the KODAK PRO 14n camera were limited because of the
absence of calibration data. Even using the nominal focal length (50mm) good results were
obtained. Some tie points were eliminated, probably due to the effect of non compensated lens
distortion.
The experiments performed with the direct georeferencing data, evidenced that even
with accurate EO parameters provided by GPS/INS, the vertical parallax is considerable (about
four pixels). After the adjustment the vertical parallax was reduced to 0,015 pixel, showing the
efficiency of this methodology.
It is important to say that this approach does not correct the ground points coordinates,
because the coplanarity model is independent of control points. It is rather an attempt to provide
a better solution concerning the stereo visualization. If low cost sensors were used for direct
georeferencing it will be necessary a geometric transformation to reduce the discrepances,
according to the experiments performed with the PRO 14n camera.
6. ACKNOWLEDGEMENTS

The authos wish to thank to:

• CAPES, by supporting this work and FAPESP that provided equipments and softwares;
• Dr. Charles Toth of Center for Mapping (DoT – Department of Transportantion), at the Ohio State University, for
providing real data.
• PROPP, that provided support for presenting this paper.

7. REFERENCES

Brzezinska, D. A. G., 1999. Direct Exterior Orientation Of Airborne Imagery with GPS/INS System: Performance
Analysis. Navigation, Vol. 46, No. 4, pp. 261-270.

Colomina, I., 1999. GPS, INS and Aerial Triangulation: What is the Best Way for Operation Determination of
Photogrammetric Image Orientation. IAPRS. München.v. 32, Part. 3 – 2W5. September 8-10. pp.121-130.

Cramer, M.; Stallmann, D., 2002. On the use of GPS/inertial Exterior Orientation Parameters in Airborne
Photogrammetry. OEEPE – Integrated Sensor Orientation: Test Report and Workshop Proceedings. Frankfurt,
Kartographie und Geodasie.

Haralick, R; Shapiro, L.G., 1993. Computer and robot vision. Addison-Wesley Publishing Company. v.2.

Mikhail, E. M.; Bethel, J. S. Mcglone, J. C., 2001. Introduction to Modern Photogrammetry. Inc. New York : John
Wiley & Sons, 479p.

Mostafa, M. M. R.; Schwarz, K. P., 2001. Digital Image Georeferencing from a Multiple Camera System by
GPS/INS. ISPRS Journal of Photogrammetry & Remote Sensing, 56, pp. 1-12.

Ruy, R. S., 2004. Geração Automática de Estereomodelos a partir de Imagens Digitais Georreferenciadas.
Dissertação (Mestrado em Ciências Cartográficas) – Universidade Estadual Paulista, Presidente Prudente.

Schenk, T., 1999. Digital Photogrammetry. The Ohio State University: TerraScience, v. 1.