Scanned with CamScanner316 the ain task of photogrammetry—analog, analytical and digital alike. js to reconstruct the object space from images, You may consider the re. construction as tne inverse Process of image formation, The latter pro- ceeds from the scene to the image while reconstruction begins with images and ends with a suitable description and representation of the sean One task of reconstruction deals with determining positions of features in object space from known quantities in image space. Before computing positions in object space, two major problems must be solved—the cam. era's interior and exterior orientation. The interior, relative and absolute orientation are the fundamental ori- entation procedures in analog and analytical Photogrammetry. Are these procedures sti!l necessary in digital photogrammetry—and if so—are there perhaps substantial differences in how to perform them in a digital envi- ronment? Pundits of platform orientation systems Ive long predicted the end of traditional orientation, including aerial triangulation. Recent advances expedite the process but do not supersede it. For one, the relative orien- tation and acrial triangulation also render information about the surface. Moreover, the orientation parameters compensate model deficiencies toa certain degree, that is, small differences between the mathematical model and the physical reali: Can we simply duplicate the ‘classical orientation procedures or does digital photogrammetry require new approaches? If you work in an in- teractive setting. for example on digital photogrammetric workstations, then the orientation tasks are eésentially performed in the same fashion as on an analytical plotier. We focus, in this part, on automatic orientation Procedures, point out important differences, and modify existing methods accordingly, or derive new solutions. In traditional photogrammetry, a human operator performs the orienta- tion. The crucial task for the operator is to identify and to measure suitable “points”. Economic reasons put a limit on the number of measured points, however. Simple but effective adjustment procedures have been developed to determine the orientation parameters and to assess their quality. The «lgorithms ae point-based. Jn contrast to humans, computers are far inferior in terms of selecting (extracting) suitable points and measuring (matching) them. It is easier ‘0 extract features instead. Strictly speaking, a point in a mathematical Sense does not exist in an image. What the human operator measures as @ “Point” is in fact the result of an elaborate image interpretation process— on abtlity that is far beyond the reach of a computer operator, Oe operant there is virtually no limit to the number of feature ; Paine tor can extract and process. Consider linear features, such as cdg Scanned with CamScannerChapter 13 Automatic Interior Orientation The main task of photogrammetry—analog and digital alike—is to recon- struct the object space from images. You may consider the reconstruction as the inverse process of image formation. The latter proceeds from the scene to the image while reconstruction begins with images and ends with a suitable description and representation of the scene. ‘One task of reconstruction deals with determining positions of features in object space from known quantities irt image space. Before computing positions in object space, two major problems must be solved, however. For one, we need to determine the exterior orientation of the camera—its position and attitude referenced in the object space. The other prerequisite is the interior orientation, the subject of this chapter. 13-1 ‘Purpose of Interior Orientation ‘The purpose of interior orientation is to establish a transformation from. measured or computed features in the image tc a 3-D Cartesian coordi: nate system that has its origin in the camera's perspective center, We call this system the image coordinate system while the reference system for °e digital image is the pixel coordinate system, or simply the pixel syste”. "it, 13.1 depicts the two systems. The digi diapositive, ; The imay ing 5 al image shown is a digitized coordinat fate ‘stem serves as a suitable reference for express: center. The facia in the image space. Its origin is in the perspec fiducial center te They 8 of the diapositive shown in Fig. 13.1 define u : Polat ofan ees offset between the fiducial center and the principal dure, the pace imation, PP, is known from the camera calibration proce Dest symmetry, PS, is the origin of the radial distorvo" Scanned with CamScanner13 Automatic tn "Or Orientation tic N terior i Figure 13.1: The 3-D Cartesian image coordinate system has iis origin in the per- spective center. The intersection of lines between opposite fiducial marks defines the fiducial center FC. The principal point PP is the vertical projection of the perspective center onto the pixel system. ‘Tne point of best symmetry, PS, is the origin of the radial distortion. The x,y’ coordinate plane is parallel to the pixel image and the positive *-avis points in the flight direction. Positions in the image space are expressed by point vectors. For exam- le, point vector p defines the position of point P in the pixel system. We have for point Pp a a3.) p=| Yn : Note that the third component is negative. This changes to a posit ® digi Me Fare case when the film negative instead of the diaposiie ized, "I ‘een the image The geometric reconstruction is a transformation between the Image Scanned with CamScannerentation ining Interior Orient 32 13,2 Deter {the object space coordinate system, To keep this and | jmate system ary in analytical photogramme coord omar y ch try to simple, it is CU qllganel oh thecal ff a model; perspective center, image point, and objec ¢ collinearity i employ the col point are ona straight IN only approximately describes the physical None, the collinear’ Deviations between mathematical model and Cees en aera The standard way to accomplish this rea re rections to the collinearity model— a process known as Se a analogy to approximating a function by expanding it 4 areal about initial values, you may consider the collinearity mode Eee al approximation of the image formation process and the mo one lied a it as the partial derivatives. To continue with this pain wall be discussing only “first order corrections" in this chapter. se arcequence, we ought to extend the definition of interior orientation to include at least the first order corrections, including refraction, radial distortion, film shrinkage, and scanner errors. 13.2. Determining Interior Orientation 13.2.1 Transformation from Pixel to {mage Coordinate System The transformation of a point P in the pixel system (rp. Cp) to image coor Ginates is rather simple as the following equations demonstrate (see also Tig. 1.3 on page 6. Xp = (y= co)pe | Yo = =p = re), 03.23 erlamsec | e are the pixel coordinates ze In metric units, At this point you may wonder where the pixel coordinates of the prin “pal point come from, If the image is obtained by a digital camera, then “evte are available from the camera calibration, Perhaps the more \ypical case is that of digitized diapositives. Here, the Principal point is not directly available, First, the fiducial center MUS’ sitet Sasuring the fiducial marks, followed by applying He The fiduci Set of mea where r9,¢ the pixel 5 of the principal point and py X Pe fs ef, microns, center is de sured fiducial m; jon. The termined by Way of a 2-D transformation. the arks in the pixel system is transformed to Scanned with CamScanner13 Autom, atic Inte HOt Orien nation yz? responding set in the fiducial system that is @Wailable froin 44 om he camers on. calidratto . qhe general form for transforming pixel Coordinates to the fic ‘ ° ONducial 65 «. en xh" is ial foe % Xp = QM + Ay2Cp +43 /f A Yr = AN p + Arey + ary 5 fora similarity transformation the coefficients a; ij are a =S-COSK = ay = -5 + sina 21 = —a12 a2 = ay where the scale factor s is related to the pixel size. Note that in the case of rectangularly shaped pixels (p, = pr), two scale factors must be intro. cuced. Hence, an affine transformation should be chosen, leading to the following coefficients for Eq. 13.3 Qi = Sx (Cos(w—eEsine)) — ay2 = ~sy sinew a Sx (sin(a+Ecosa)} — a22 angle ¢ denotes the non-orthogonality of the coordinate axes. The last step translates the fiducial system into the image coo: System. The shift parameters Xo,’ are known frem the camera calibrati Procedure, azar he factors that oy some of " fess Jormation PL aus the collinearity model, let us cons peace ae to deviate from the physical reality © a gata maseeS The 8: 13.2 shows the major steps involved in obtainIng TT acquis ton left side of the figure depicts the traditional . aa re right side ith’? an aerial camera, followed by scanning HE HM iy aye scene (ote data acquisition by digital cameras: POC ccod in St Orde all image, the following factors oURNt Te "Image refinement. Scanned with CamScanneri entation 132 determining Interior OFFeNtaLle 323 scene i i ee Tam based camera {expat camera 7 i th, —_1—_. i = ‘im evelopment | ieee aces apostive 1 a —E 2 igh image ioral image ! : ‘meer ereraion Seer onentaton Figure 13.2: Major steps involved in image formation. The left side sketches data acquisition using a film-based camera, fol- lowed by digitizing the diapositives with a scanner. The right side of the figure shows image formation with dig- ital cameras. Refraction Fig. 13.3 shows how an oblique light ray ig refracted by the atmosphere. According to Snell's law, alight ray is refracted at the interface of two differ ent media, The density differences in the atmosphere are in fact different media. The refraction causes the image to be displaced outward. The radial displacement caused by refraction can Le computed based on a model atmosphere. 3.5) dy = Kir+"5) u3 K «= ( 2410 2410? 107° H?=GH +250 ~ T= 6h — 250017 with y : pate SINT the distance from the principal point PP to the imaged POE! — aA, Scanned with CamScannernegative : pp perspective center datum Figure 13.3: Oblique light rays are refracted by the atmosphere. The effect is quite similar to that of radial distortion. ‘ecl length, #7 the flying height, and h the ground elevation. With dr computed from Eq. 13.5 we have for the displacement of an Smege point x, 97 ar. = Xar 3.6) dry = —dr Ca mera Imperfections The late i 204 sou a8 formed during the time of exposure pone Sohne gag ZEEE IC vith the collinearity model. FHS iat distortion. displaces on Cd from the path through the lens by radial dis 8 Off ax sitive distortion increa MMe lateral po! Points in a radial fashion, A positive dis not exactly where are al may, reduces it. Ostortgn eMfication while a negative distortion reduces I an f N Values ess of camera : on, values are determined during the proce function of the . c er as a *adlus or gg Usually listed in tabular form, either as @ * > radial distor the 7 ot dd be the ra “Cangle at the perspective center. Letdd Scanned with CamScannerion jor Orlentat ns mining Inter 5 13.2 peter" points on a cirele (centered on the point of best 5 " sthmetry ° (v2 v2, T hon the displacement of the im: . yn value rae tio ARE Coord. ps), radius Y= nates are x ddy = pad 37 ddy = sad . (13.8) For acrial cameras, the distortion values are very small. Hence, jt suf. fices to linearly interpolate the values from the table Provided in the cali. bration protocol. For cameras with significant radial distortion, it may be necessary to interpolate dd from an odd-power polynomial of the form dd = por + pire +perrt--- (13.9) The coefficients p; are found by fitting the polynomial curve to the dis- tortion values. Other camera imperfections include tangential distortion, non-flatness of image plane, non-perpendicularity of image plane and optical axis. These are examples of second order image refinement. Film Shrinkage : 5 During the development process, the film undergoes consideratle mechan- al stress. This, together with bathing and drying, causes deformations. The developed film, say a diapositive, is not an exact geometric copy of the latent image. It is difficult te model these exrors. Luckily enough, the base material of aerial films is extremely stable, and under normal circum stances, the film shrinkage is neglected. oe pares mechanical stress along the film roll than across. More- ference berween ae be uniform, Therefore, one may model the dif Such a transformation image and diapositive by an affine transformation. ould only take global shrinkage into account. TO consider local fil im shrinkage (sec rder image re’ rn eseau camera! fs needed, ge (second order image refinement) at Scanner Errors H¢ from ae TAs not perf ‘imaged get Plane plage the instance ; eitized bY@ Tial photography, the film is digitized bY ec TI the Ccl and introduces new errors. ThaUls: of vreseau ete. This 84 oF exposure, 1 alle Fa, an accurately ruled grid is ¢ ‘ Haccurately ruled grid “a erate PWS A More rigorous elimination O Scanned with CamScanner13 Automatic inte " Oriontatio OMALE Interior Orientar ion ed image fs NOt geometrically identical with the di itz anne apdsitive eaten e. In Chapter 8 we discussed some of the nwt: MOF wlth me question Is, Now, how fo model them, Potarttal errors ane otogrammetric scanners are accurate and reliable Systems th sofied carrier Stages of analyteal plotters for the photo carrier he tse pica geometric ACCUTACY of less than 5 pm. A hallmark of these nas : Maat they closely fellow Abe's principle®, This important prince jolated by desktop scanners. Here, the plane defined by the esa ing gystem (cross-slide system carrying the optics and sensor elements) ae ae diapositive (glass plate on top of measuring system) are not arate allel. Consequently, a projective transformation should be caning to describe the relationship between the diapositive and the digital image jrturns out that an affine transformation approximates a projective trans formation quite well if the two planes are nearly parallel. Unless data is available from a scanner calibration, an affine transfor- mation may sufficiently describe the relationship between diapositive and digital image. rhe Image Refinement Using Digital Cameras The previous sections described some of the differences between the math- ematical model of central projection (collinearity mode!) and the physical model in the case of obtaining imagery by film-based cameras. Let us now briefly analyze the situation for the case of digital cameras. Assuming the 2erial case, refraction affects digital cameras as much as aerial cameras and ‘we ought to consider it. Digital cameras su‘fer the same shortcomings as aerial cameras as far 2s geometric distortions are concerned. In fact, many digital cameras have considerably larger radial and tangential distortions. Moreover, the ngid: ity of the camera body is inferior to aerial cameras. As a consequence, the metric properties are more likely to change between calibrations. It ap- Pears that digital cameras should be calibrated more frequently than aerial cameras, : Fig. 13.2 on page 323 indicates a difference between the sensor image and the digital image. The sensor image is analogous to the latent MARES @film-based camera. It is the image captured by the detector clo the case of a CCD array, this image consists of accumulated charge a i Not directly accessible as such, The processes of transferring the NNT” *mplifying and measuring it, and converting It (0 & digital ere : Aifferences between the sensor image and the digital MAKE si by ae sa ents. a and ts the abject to be measured she te ayes, fOr ent y phabbe Put forward the principle tha 48 the measuring device, represented by coordina Scanned with CamScannerentation autonomous Interior Orie 13.4 Au Aarne Sone en ee : A Tt jelot sisielevisl aia sainl sans eo Ange “Sm . ageStation Z. The Mgure 13.5: Interior orientation performed on Intergraph’ 5 imagstatin 2 a * system offers an option to measure the fidi eatonaualy: ie operator can accept or reject the result, The ee three parts. The left display contains an overvie mage, The lighted fiducial mark (left lower corner) is displayed i he igh 308 the wel cee stray Sooner f the screen shows the fiducial center automatically, The lower third oft 7 {wo windows with measurement and computation results. 13.4 Autonomous Interior Orientation In this section We contrast int approach and motiv: Problems to be solv Sections et ; an auton cractive interior orientation at a malate the ate such a development. Moreover, We rurthet nade. Fu ¢d and analyze assumptions that can be M laborate on possible solutions, Scanned with CamScanner mous —13 Automatic Interior Otientatio n packground and Motivation ) oMOUs interior orientation, subsequently led Alo, that performs the entire orientation process with aoe ggyste! fa “black box” process, if yi out human int- jon sort ol a“bla pI SS, I you wish. The term “auto e ot orientation” is often used to express various degrees of co on Laas in measuring fiducial marks on digital photogrammetric ae ons. Eve if the human operator's involvement fs simply to accept o pect the result, there is a substantial difference between the “100% auto- rac system” and the autonomous system; the latter cannot Tely on the fety net of ahuman operator who may come to the rescue should some- ing go wrong. Hence, autonomous systems need to be foolproof. As a result, the development effort is much higher compared to an “automatic” a that runs in a (protected) interactive environment. You are about roask the right question: the interactive interior orientation requires only 2 few minutes—so why bother with an autonomous system? The motivation for developing an AIO system is the desire to build an au- tomation chain consisting of various photogrammetric processes. Fig. 13.6 illustrates this scenario. Tto Tonmanzes mages ona {~ ver ]_ ese] isnguiaton [—"]_generaton i et | J Figure 13.6; Automation chain, After scanning, the interior orientation executes autonomously, followed by the relative and absolute orientation of aerial triangulation. With known exterior orientation data, DEMs and orthophotos can be generated. These processes do not require an 7 teractive environment, such as digital photogrammetric work taons: automation chain. should the ssstem : a a traditional Dot 4 problem, then the image can be digitized again. nat ions in metry environment, all subsequent PrOCESSES WATT aaive orient etehtation parameters are known. Such 5 the COST oper hhingg LOM After relative orientation, normalized IMABES CET hrs (See Chapter 12), Now you can display the 1m starting point of the i executes parallel to the scanning proce: Scanned with CamScannerLad Autonorsas {tortor Orientation wy dd vers She mode! store’ copleally Alternatively, yOU tay be jy terested! a 20> scmuring Ue ¢ terlor orlentation of all Anages Invelved fy erested I 4 : project lie sriangulation) whieh Iya provequisite for generating DiMy from more sa one model, time an 13.4.2 Objective of Autonomous Intertor Orientation The followiss ust contains the major objectives of an autonomous Interior orientation rstem, identificatios and subpixel localization of fiducial marks ts the main ob- jective. “4entification includes the task of determining which of the fiducie. mark Is detected. Subpixe} localization Is required because the pms. size is most likely Jarger than the expected precision of the fiducie, “enters. autonomous process requires a general and robust olution, accommo: datiny eifferent types of fiducial marks, A system is robust If it can cope wth different problems as they may occur Ina production en- Mironment, For example; the film may be placed upside cown In the scanner, some fiducials may be only partly digitized, In addition to opera's mistakes, there are film imperfections to consider, such as noist, blemishes, low contrast due to the cosine to the fourth power Grop-o'f lave, vignetting, and over exposure when the fiducial mark is projected on the film, Fig. 13.7 on the following page shows some fiducial marks with different degrees of complexity for an AIO sy to identity and locate precisely and reliably. Giapositive: . . positives or negatives inay be present and must be handled. + color o Sah F black and white film no assumptions can be made, both types may weur, é ssoluth ure esoltlan imagery the system should cope with image resolutions aan hea center may have been lost, As long as the fiductl from the feature uely {dentifable, the location may be dete ned! atures that describe a fiducial, 13.4.3 Assumptions me of the ay he assumptions that one can reasonably make whol the ener Tat st aualty of the solution, tn fact, we describe the downy be brought to hear on the solution, Hone or seer Festricting knowted Scanned with CamScanner13 Autom: atic Interior Orientation Figure 13.7: Different types of fiducial marks under various condi- tions are shown. Examples (a)-(c) show the typical strc: ture consisting of a small disc marking the center. sur rounded by an annulus and cross for easier recognition. The example in (c) shows a typical real world problem \sitha film scratch running across the fiducial mark. The unusual fiducial mark in (a) is from a reseau camera. : . e “these assumptions are violated, then the system should recognize this ee nment accordingly. If it must terminate the process it should do so fully—that is, with proper explanations. ystem is known together with rel- on is assumed that the scanning catibration canner calibra ew dan Parameters, such as maximum range and a, 1s assumed that the camera type is known, including ca raen ‘ta and suitable fiducial mark descriptions Scanned with CamScanner3.4 Autonomolls Interior Orientation 333 13. ration about the scanner we can deduce a maximum Fone and maa enews which the film may be placed, as illus. rotation ang! » in predicting the search space where fiducial hel aved in PIB. 13d. woth tae me Rapected. no scanner calibration data fs avatlabe, then marks a . there is still a chance to consider some scanner errors by using an affine transformation (sce Sec, 13.2.2). From the inform Figure 13.8: Search space for the left upper fiducial mark. The film size is smaller than the net scanning area. Since it is not known exactiy where the operator piaces the film, extreme positions and a possible rotation angle are assumed for predicting the search space. A traditional camera calibration repert contains information about the position of the fiducial center, the véfset to the principal point, the radial distortion, and the calibrated focal length. However, additional informa: tion is required for AIO. For example, each fiducial mark should have & unique identifier, such as a number or a signature that can be ex racted from the imagery. Moreover, an ideal image (template) of the fiducial mark must be available, If relational matching methods are used, th2n We also need a structural descrintion, It is clear that a great deal of additional information is required. The standard calibration report does not sufles 13.4.4 Approaches to AIO The chief proble peo i al marks 2d their precise and be solved is the identification of the fiducial ® hese To tasks. A ¢ and robust localization. It is reasonable to separate PProaches may pursue one of the Following two strates Scanned with CamScanner13 Automatic Interio, terior Orientai ti ra ion exbased approach: The subimage that contains a fiducial mark jg pinarized. This approach resembles the Gestalt law of figure/ground separation (see 4.4.1). The precise localization is performed by cross- correlating an exact copy of the fiducial mar\ image: feature-based approach: Here, extracted features are matched with the features of the ideal fiducial mark. This approach casts AIO as an object recognition problem. Features to be extracted consists of ele- ments of a fiducial rnark, for example straight lines, circles, crosses, and squares. Different matching methcds may be employed. Exam- ples include shape matching and relational matching. k with the foreground The following sections elaborate on the details of the two approaches and highlight advantages as well as drawbacks. 134.5 The Anatomy of a Fiducial Mark The fiducial marks are located on the upper surface of the inner cone of enaerial camera. They are located either at the four corners of the format opening and/or in the center of each of the four sides. Fiducial marks are ‘egistered on the film by projecting an image of the mark through a smal! ‘ens housed inside the cone. Since each fiducial mark is built and projected Separately, small variations may occur. Fiducial marks have typical patterns, made up of geometric structures including straight line segments, crosses, solid squares and circles, and ‘sauli. Most of the structures serve the identification process. The actual * fenter of a fiducial mark, whose coordinates are known from camera cali- ation, is a small disc, just slightly bigger than the measuring mark of an “shtical plotter?, The size of the center ranges from 50 tc 100 microns. sin ite Straightforward to model fiducial marks geometrically. A “scription of the mark shown in Fig. 13.9 is a list of parts: ((circle1; center=(0,0); radius=rt)} (circle2; center=(0,0); radius=r2)} ((disc; center=(0,0); radius=rd)} (ine); length=1);...(line4; length=l4)} A For exam: Die Ne claborate description would continue with relations. For exan our line h 7 rigin, and hey inte ments are located symmetrically about the orig! ~ Tot the annulus halfway. ic : is post ing mark Is Po Tithing cnc@suring accuracy is obtained when the white meas! slightly Scanned with CamScanner13.5 AreaBased Approach Figure 13.9: Design drawing of a fiducial mark. The design data depict the corner fiducials of the RC 30 camera by LH Systems, Heerbrugg, Switzerland Dimensions are in units of microns. The data is useful for generating a geometric model that will be used for matching extracted features from the image. Elementary structures are line segments. annulus, and disc. There are relations among these structures that are useful ina rclational matching scheme. The four rectangles on the right side - indicate the position of the fiducial mark with respect to the image co- ordinate system. The example shown indicates the right upper corner fiducial. The radiometric model of a fiducial mark is trivial. The elements making up the mark are evenly bright against a dark background. The first strat egy for locating the fiducial marks is based on the radiometric model and Proceeds as foreground/hackground separation, for example, The second strategy builds on the geometric model of the fiducial m rk. 13.5 Area-Based Approach Suppose we have extracted subimages such that each contains & fd mark. Now he simple radiometric model of a fiducial mark—bright ™" forward Stk background—supgests binarizing the subimages. A ae Proach is to define ranges of brightness values, for example BSI res} ve have rable threshold values from the gray level histogram. We’ pray 35.1. In tach introduced this met method in the image processing chapter, SCC: 3 Scanned with CamScanner: 13 Autom: 336 atic Intertor oO . lentatio, ances show FB. 3.14 on page 68 is a fiductal mark, <<. 4 ot 2} (b) 10} Figure 13.10: The histogram of the fiducial mark in (a) has two widely’ separated clusters (b). Binarizing the subimage with the threshold value ¢ renders the expected result (c). is mn involved? The histogram of the subimagce with the fiducial mark Of fray vad bimodal. As Fig. 13.100) demonstrates, the dark cluster hoes es 5 Really separated from the bright cluster. Choosing the Dace wath we £8 the valley between the two clusters, and assigning al than ¢ yy te EY value smaller than ¢ to binary’ O and gray values greater inary 1, delivers the expected result. as Fig 1g ne Procedure, applied to another subimage, is far less successful Fated, choo, reveals. Although the two clusters are reasonably feller evere nanine the threshold ¢ at the lowest point does not deliver what binary jpg Pine for. Selecting different thresholds leads to quite differ Mages as Figs, 13.11(¢)(f) convince you. The problem is rooted in Scanned with CamScanner13.5 Area Based Approach rhe fact that the gray Jevel distribution of the background and foreground overlap. tn the transition zone between the two peaks we have mixed Dixels that may belong to either of the two groups, ‘One approach to determining a suitable threshold value, suggeste by Otsu (1979), is to minimize the weighted sum of group variances, Let 1 iu the threshold value, dividing the histogram into two groups. Let o?(t) he the variance of the first group and ott) the variance of the second group. Then of, the weighted sum of the group variances, is defined as ORtt) = si(thoztty + sx(toz(t) (13.10) achere s)(f), S2(t) are the probabil spectively. : Now'we find the best threshold value ¢ where o2.(t) reaches a minimum. To determine the group variances of (t1 and o3(t), we first need the group means p(t) and po(t). The following procedure, together with Eq. 13.10, must be repeated for different threshold values ‘ies for group one and group two, re- sith = Spin (13.1) ai sat) = > pu) (13.12) jet : tpi) i ee try = See PDT (13.13) a att) os : : potty 2 Sse PU (3.14) = S2tt) ofuy = Liei= men? (3a) att) oly «Sete e to? (3.10) Sit) Nip. 13.1100 shows the binariz Principle of minimizing the group describe more sophisticated thre: 4 comprehensiv Whatever ap; 1 image of the fiducial mark, using f ariances, Haratick and Shapiro (190 olding methods. Saltoo (1988) ProN'® © review of binarization methods. aia ieee the fiducial mark from the background. Sor veh SOVETAL piste a nny do not belong to the fiducial mark. Non! Ne fiducial mark are classified as background (bins ing the ) Je that Scanned with CamScanner3 Automatic inter OF Orientation he example shown in Fig. 3.14 on page 68 is a fiducial mark, a (a) () 40 207 : : (b) Figure 13.10: The histogram of the fiducial mark in (a) has two widely separated clusters (b). Binarizing the subimage with the threshold value ¢ renders the expected result (0). What is involved? The histogram of thie subimage with the fiducial mark ot gray ely bimodal. As Fig, 13.1000) demonstrates, the dark cluster threshong > is neatly separated from the bright cluster. Choosing t i Diels with alue ¢ in the valley: between the two clusters, and assigning 4 than ¢ ie Bray value smaller than ¢ to binary 0 and gray values greate The Mary 1, delivers the expected result. as fig. 1g pe PFOcedure, applied to another subimagests far ess sucths led, chee reveals: Although the two clusters are reason by ieee ve wong POMBE the threshold £ at the lowest point does not deliver WN Vinary 4 PINE for, Selecting different thresholds leads to quite GUCTNT Mages as Figs. 13.11(¢)-(f) convince you. The problem Is rete’ is jess successful, Scanned with CamScanner13.5 Area:Based Approach) 337 fact that the gray lev el distribution of the backs round and foregroung ee tn the transition zone between the two peaks we have inixed Pixels overlap. r that may belong to cither of the wo groups. One approach to determining @ suitable threshold value, suggested by Otsu (1979), is to minimize the weighted sum of group variances. Let ¢ be the threshold value, dividing the histogram into two groups. Let OF (t) be the variance of the first group and 0310) the variance of the second group, Then of, the weighted sum of the group variances, is defined as Rit) = si(thoz ity + s2(taz(t) (13.10) where 51 (1), $2(t) are the probabilities for group one and group two, re- spectively. Now we find the best threshold value ¢ where o3.(t) reaches a minimum, To determine the group variances of (t1 and o7(t), we first need the group means #2) (t) and pi2(t). The following procedure, together with Eq. 13.10, must be repeated for different threshold values sutt) = 3.11) st) = (13.12) wylt) = zt (33.13) batt) = Bier asa oft) = Batiste (13.15) oot) = Baa = nto? (13.16) Tig. 13.11(f) shows the binarized image of the fiducial mark, using the pence of minimtaing the group variances. Haralick and Shapiro (1992) sese re sophisticated thresholding methods, Sahoo (1988) provides mprehensive review of binarization methods, Whatev a we c; ‘cr approach we may choose, it is very unrealistic to assume that 4n neatly separate the h pisets omen fiducial mark from the background. Some of the several pixety “ a binary 1 do not belong to the fiducial mark. Moreoves Sof the fiducial mark are classified as background (binary 0 Scanned with CamScanner13 Autom: 338 [eet Fete as 13.11: The histogram of the fiducial mark depicted 0 (2) has a uanstion Zone containing pixels that may belong to either of the Wwe C TT Threshold values at 55, 61, 69, and 74 have been selene Sulting binary images, shown in (c) through (D are quite di ™ay lead to different fiducial center locations. Scanned with CamScanner138 Areastiased APPFOAE h Bay ye only way Wo sort things oul is to follow the binarization withay ? guided by the knowledge about the shape and size of i ra Here, the first step Is to form binary 0 and binary ie a connected component algorithm, for example. Once regions explicit, their area, shape, moments and boundaries can be de Thi necessary for identifying the regions that belong to th vyek, Single, isolated pivels, oF small isolated regions that are | foreground probably’ do not belong to the fiducial mark. Hence be eliminated. Ping fide lons ty are tag, termineg fiducia, abled aq + they can 1 re 13.5.1 Precise Localization We now turn to the problem of determining the fiducial center as precise}, as possible. To achieve results comparable to analytical photogrammer cubpixel accuracy is necessary because the pixel size most likely exceeds the expected accuracy by an order of magnitude. Assuine a typical pixel size of 15 pm to 30 pm. Then, the fiducial center should be determined to one tenth of a pixel. The larger the pixel size, the higher the subpixe! accuracy requirement. But with a larger pixel size, fewer pixels make up the fiducial center, and its position is determined less accurately! Let us anaiyze the problem by way of an example. Fig. 13.12 ou the fol lowing page illustrates the process of sampling the fiducial center. Suppose a square-shapéd fiducial mark, size 60 ym, is ideally imaged on the fm Now we scan the film with 20 pm resolution. As indicated in Fig. 13.12 the fiducial mark does not align exactly with the pixel tessellation. As2 consequence, pisels along the boundary’ also receive energy’ and are turned on, The numbers in Fig. 13.12(b) express the area that the fiducial mark cavers for in each pixel. The four pixels in the center are fully covered and assigned a value of ten. The pixel at the lower Jeft corner receives only 8% of the energy of the central pixels. Let us turn these numbers, a gray values. Then, Fig. 13.” 2(b) reflects the gray level image of the fiducis! mark. After thresholding, Fig. 13.12(c) is obtained, Note that regardless how the treshold value is chosen, the resulting binary image of the A cial mark differs from the original in various ways, for example 18 sizes" shape. Worse, it also differs in location. We conctucte that Samet thresholding cause dislocalization of the fiducial mark. Now, the question is how to determine the center of the ae ae pes fst method that comes to mind is to consider ; . Although simple, this approach is affected by §° 4An ideal image meanea penne aid use th : catty we shou spread tae iHaHe Meas a perfect copy’ of the object, Thearetieally “6 on to obtain a more realistic imap tise Scanned with CamScanner"13 Automatic Interior Orientation Faure 13.12: 4 square-shaped fiducial center, size 60 jm, is ideally imaged on the film (a). The fitm is digituzed with 20 yim pinei size. The resultirg fs Sray level image is shown in-(b). The numbers indicate the fractional area of each pixel that is “covered” by the fiducial (ten Is full cov erage). Thresholding the gray level image leads to the binary im=ge in (©. Note that the binary image is disferent from the original in shape, size and even f. location, Sampling and thresholding cause dislocalization. threshotay : cna ets Different thresholds may lead to different shapes and othe to different Incations. een cross! (Ortelation (ge PTOACh for defining the subpixel location is bases OF Ay atk tem Se SEC. 10.4.1), Here, we generate an ideal image of the FANE ‘relation Ste) and correlate it with the thresholded image: pepe] Chtaing (, Position, the correlation factor is determined. 3% 34 TN Madrayie gf, Ximum correlation coefficient in its center Is USEE TY S*face whose coefficients can be conveniently deter ™ Scanned with CamScanner13.5 AreaBased Approach bas a linear observation equation, Jeastesquares, We HAVES Peet areata ay ith pial 9 the correlation coefficients in the 3% 3 window The with Pi Aas ! ! , a of the surface are found by solving the following two equations for x andy Qax+cved = 0 ti3.t%, To assess the accuracy of the subpixel location, we analyze the quadrayc function. For example, a significant change in the tangent around the may; mum signals higher accuracy. This change is manifest in the second deriva. tives. Hence, coefficients a and b render a measure for the accuracy of the subpixel location. By now you may have concluded that binarizing the image is probably not the best approach. Regardless of how sophisticated the threshold- ing procedure, binarization is somewhat arbitrary. A better approach is to leave the gray level image of the fiducial mark as it is and correlate i: with a template that is computed at various subpixel locations. Fig. 13.13 illustrates the procedure. First, the image of an ideal fiducial mark is computed by taking into account the line spread function. Now, this image is sampled at different subpixel iocations, for example in intervals cf one fifth of the pixel size Sampling effects cause all of these templates to de slightly different. They are correlated with the image ef the fiducial mark. The template that yields the maximum correlation coefficient is considered to be most similar to the image. The subpixel accuracy before (Eq. 13.17) is now determined in the same fashion as sample 13.1: An ideal fiducial mark, size 3x3 pivels, is imaged at the subpixel location 4r = 0.2 and Ac = 0.6, The first entry in Table 13 T contains the rest Of computing the centroid of the binarized imax The centroid is off bY 0.09 and 0.29 pixels. The next entry refers to cross-correlating the ideal template with the binarized image, Entr obtained by subsampling the temyplate at p 3—7 represent the results el locations indicated in the comments column, The last line represents entry 4, except that the right unrer piel ofthe fiducial mark image was sett zor for siting hlemish, if Scanned with CamScanner13 Autom: we Figure 13.13: The template (ideal fiducial mark) is sampled at different subpixel locations 4r,Ac. The gray values are functions of the fractional part of the pixel that is “covered” by the mark. For exam: ple, the left upper pixel is covered by {1 Ar)(1 dc). These experimental results con ot bebinarized. In Trinder et a j CMe o similar conclusions. firm that the fiducial mark images should , 136 Feature-Based Approach | Fiducial marks ar posure time, is i ¢ artificial objects that are projected onto the film during Almost every fiducial mark has a simple, regular shape. MViles Us to Tepre.-cnt the fiducials as structural descriptions and to a Structu.al elements in the image. Fa infig. lay ollowing discussion we use the sample fiducial marl 3 on Diemish 2" "is representative of most fiducial marks. Moreover, c pra Scratch right through the fiducial mark—presents one 0 the man n ny sake 4 a tony lems that a robust system must cope with in an operational ent, film ‘ tele three structural and gee present in MSc, . Fro 2 ns the cam clements, or shape primitives if you wish, that any fiducial mark: straight line segments, circles, cra manufacturer's design data, exact dimensions length of lines, and their spatial relationships Scanned with CamScanner atic Interior Orientation I. (1995) and related Papers, the authors.13.6 Feature-Based Approach My ‘Fable 13.1: Results of subpixel target detection, ———————————e | corr. | Location Error ; : covtt. | row | col | row col Comments 0.111 | 0,889 | -0.089 binary image: cenisony 0.9258 | 0.078 | 0.921 | -0.122 binary image: een 0.9776 | 0.221 | 0.600 | 0.021 | 0.000 | sampled ato.4/og 0.200 | 0.608 | 0.000 | 0.008 | sampled at 0.2/0.4 0.200 | 0.621 | 0.000 | 0.021 | sampled at 0.2/0.8 0218 | 0.618 | 0.018 | 0.018 | sampled at 0.4/08 0.210 | 0.474 | 0.010 | -0.126 | blemish sounune © g 3 & With the shape primitives and the relationships we build higher struc- tures. For example, two concentric circles of different radii constitute an annulus, two parallel line segments build a line pair, and line pairs with dif- ferent orientations make up a cross. Continuing with this line of thought we realize that with a few simple shape primitives and the many relation- ships among the primitives we can build useful structural descriptions of fiducial marks. Such descriptions are quite general and can be used tor many different fiducial mark types, as long as they are made up of circles, straight lines, and perhaps discs. Fig. 13.14 shows the shape primitives and derived structures, incluc- ing spatial relationships. With these simple constructs one can devise 2 powerful strategy for detecting and localizing fiducial marks. 1. Detect edge pixels as the atomic part of shape primitives. 2. Group edge pixels together that belong to one shape prirnitives [OF examle straight line segment or circle. - Check the relationships among the shape primitives and build higher order structures, for example annuli, line pairs, and crosses. 4. ; a Compute the fiducial mark center from all shape primitives: trough the jem ate You may consider this matching of shape primitiy NOt performed simultaneo) ships are checked, If sa Increases. One of the annulus and the approach as structural matching: al es and the spatial relationships amons ion usly. After each grouping process, the Flt" tisfactory, the confidence related to the Fe Most effective checks concerns the cente! Cross for they must be identical. cognition rs of th Scanned with CamScannerrelationships structure . segment graphic primitive —_ line cncies graphic primitive Oo oO i line pair 2 parallel line segments = tine pair with gap 2 parallel tine pairs =n 2 parallel line pairs with gap cross symmetric; perpendiculas annulus, 2 concentric Circles rimitives. nuli and crosses, can be construct Paris of the structural descripti Based on these, an- ied that are important ion. Trefotlowing secti-s tives (Circles and Ns address the p: "Sy loca Straight lines), of che ting the fiducial mark center, Toblem of detecting the shape prim- cking the relationships, and of pre- 13.6, Detecting Circles We Use el Ureleg pe Hough transfo: 5, to find rm technique, introduced in Chapter 5, to rele can by © represented in the Spatial domain by (x ~ x0)? + (y = yg)? =? ‘th %, — 1a 38 the center,» the radius and x,y the variable the repr - Vibe es. Then, + be variables, SeMtayj Pont On the circle and let Xo Yor? nn ation in the parameter space is again ac Scanned with CamScannerroach 136 Feature-Based App! ba oxi)? + (Yo - Hi (03.2) we have established a simple relationship between spatial dom; narameter space. A point ona circle in the spatial domain transfor 5 circle in the parameter space, where the center is given by the coo, of the point. ; ; ; : Fig. 13.15 illustrates the relationship. Points 1 to 5 lie ona circle in the spatial domain. They all generate circles in the Hough space that intersect in a single point—the center of the circle in the spatial domain. ain ang ms into dinates parameter 5 2 30 2% 90 40 50 70 x-axis parameter a a (b) Figure 13.15: ilustration of the relationship between points on a circle in the sP* tial domain (a1 and their representation in the parameter space () Points become circles, c.g. point 1 transforms into circle 1. ee on a circle in the spatial domain, e.g. points 1 to 5, generate Cet in the parameter space that intersect in a single point—the ace center in the spatial donuain. _tet us apply this procedure to find the circles that are part of 2 Glal mark. ‘The radii of these circles are accurately known from do Ga. Hence we can reduce the 3-D parameter space t0 0 ae at. this tan is a subimage, size 512 x 512 pixels, pixel size aes the fide cial mark, pee subimage is selected to make sure that it oo roi) For eerreaet he first step entails detecting edges, shown in Fis eter space * Ske Pixel the respective circles are generated in the parame Scanned with CamScanner13 Automatic Interior Orientation y r Table 13.2: Results of circle detection, Circum | Peels | Radius | ference | detected | Row | Cot (93 90 80 343 | 64 24 92 62 343 | 65 25 96 44 343 | 63 17 74 41 242 18 76 58 343 | 65 9 80 38 347 | 6 1 7 | 62 Consider the parameter space as being represented by a two-dimensional arg clled the accumulator array. The two dimensions of the array cor: respond to Xp and 39, respectively. All cells in the accumulator array are intdized to zero. As each edge pixel in the spatial domain is evaluated, the corresponding circle in the parameter space is generated, and repre- sented by incrementing the affected cells of the accumulator array. After froctssing all edge pixels we search for peaks in the accumulator array. Thecell value corresponding to the peak indicates how many pixels lie on adircle, and the peak's row and column indicate the circle's center in the image, Results of finding circles in the image of Fig. 13.16(a) are listed in Ta- 4132. The aominal radius of the larger circle is 24 pixels, and the smaller has radius of 18 pixels. We find the fact that the two circles are con- anak Confirmed in Table 13.2, The ratio of detected pixels to maximum tent Of pies per circle (circumference) is another criterion to assess aE wommance of the arcle detection. These ratios, together with the aes of concentricity, justifies the hypothesis that we have found mag, TK This hypothesis is further tested by finding other fd tha arg 5 UCTUFRS, such as straight lines. Fig. 13.16(c) shows the phe Table 13 LF the annulus of the fiducial mark. Note the fourth entry in Touts yg Me TOW and column values are not consistent with the Ee a accent the circle detector found 41 edge pixels that happen Tadius 17, far from the fiducial mark! R62 62 De *tecting Straight Lines . . f the "cl may the problem of detecting straight fines that re et straight a tk. From the design data we expect two sets of parallel strais Scanned with CamScanner136 Feature-Based Approach (c} iB Figure 13.16: Mustration of detecting circles and crosses. The subimage in (a) has 312 x 312 pixels with a pixel size of 40 wm. A LoG operator wt w= 2 was used for detecting edges (b). The result of detecting cis {s shown in (c). Here, a new subimage, of size 128 x 128 pixtsi* Chosen such that the hypothesized fiducial mark is centered. Pit® that have been found by the #45° line detector are shown in(@ li 30 ae and -45° angles. Moreover, their length is known. ane spat domain and rey) £ BE the representation of a straight line tT Fesented by: i i+ ¥i be point P; on the line. The parame ansfore imoa strane 1OP¢M and by the intercept c. Then, POI Ny spt Benerate sere ne ¢ = =x, m +4. To generalize: pointsin Noyes © straight lines in the parameter space. At the interscc! Scanned with CamScanner13 Automatic interior Orientation ne parameters 11, C of the line in the Spatial domain, ve the ‘i a netic ci int jimtercept representation of 6 ahs 48 HOt without problems, ‘he ae parallel to the y-axis, for example. However, since we are a ted in lines with slopes of m = =1 we do not Need to consider st - resentation., wwe hav der Come grote ne mp rio with the fiducial mark (a) and hi ant 7 depicts the scenario wi a) and th fig, 13.17 mulator array (b). Its dimensions are chosen such responding Aa to ~2 can be accommodated. The interce; sep etctlines within the area marked by dashedinee ¢ cor: that a Pt range is de- in Fig. 13.17(a), signed t0 range intercept (c) g g s ° col tb) (a) a straight line in Figure 13.17: Mustration of the reiationship bewween points on a strais! the spatial domain (ai and their representation in the paren Space (b). Points transform into lines. The inter cruel ae termines the parameter of lines in. the spatial gama The ren pssible intercepts is shown in (a) by dashed lines. Riera oe dimension in the accumulator array (b), The cameron heres Parameter is application dependent. We need cf *1 for detecting the crosses of the fiducial mark. 13.164) g ss of the fiducial hows the pixels that are part of the cross of the tends itself to a waé that the analysis of the accumulator array crs amediat mttSt Of the detected straight lines. For ane. we cn Moreover OW Parallel the line pairs are and how mu -e deduce the angle. af 2M the difference between the tO somata, we can actermine Aeaige eS Quantities ave knowns from Gesisn aarsegntidence that 'e Menage * Af they agree with the expectations. the “4 fiducial Mark increases. Scanned with CamScanner136 Feature-Based Approach 4g 13.6.3 Precise Localization i «the problem of reliably identifying th fidy. ANY effort yoy The previous section addre: rob cial mark, Although we know where it is, we have not made to determine its position as accurately as possible. The success with identifying the major structures of the fiducia} suggests we take a different approach to determine its center, T] we compute the center from the structural elements rather than q from the center pixels. In our example two concentric circles an straight line pairs indicate the center. ‘Asa preliminary step we may consider determining the edge pixels mor accurately. Fig. 13.18 shows a gray level profile through the fiducial ie center. We clearly recognize the sharp rise in gray levels across the annulus The fiducial's center also stands out, but not as distinctly as the ring around it. What is the peak between the center and the right cross-section of the annulus? This is the scratch through the fiducial mark that gave us much trouble in the area-based approach. It is conceivable to remove it since we concluded during the recognition process that it is not a structural part of the fiducial. Hence, the edge pixels of the annulus and the cross could be determined to subpixel accuracy. Mark, hatis, rectly id four 255 gray level distance [pixels] Figure 13.18: This gray ievel profile cuts through the center of the fiducial mat shown in Fig. 13.16(a). The first and last peak reflect the crossing © the aiinuius (from left to right). The fiducial center stands OU but less distinctly than the annulus, The dominant peak is us by the scratch in the film. Let us First now de . =i determine the center by a least-squares adjustm he We fit a cir cle o o jrcle. same proces: known radius through the pixels of the outer circles Sis repeated for the inner circle. Next, we fit stralé | A Scanned with CamScanner; 3 Automatic Interior Orientation 3 the cross pixels and force them to pass through the center of th he ust cles: ane ‘t oc Ww yet the we » circles be represented by (xi - x0)? + (94 Yor -R? = (xj = Xo) + (9j-Yo = RF = yy " (13.22) (13.23) xj, yu the pixels on the outer circle with known radius Ry and x;, y; the eee the inner circle whose radius is R>. Assuming random errors for al positions, Eqs. 13.22 and 13.23 constitute observation. equations ae the right-hand side contains the residuals 1,1). Unearizing 13.22 and 13.23 with respect to the two Parameters x, V9 leads 10 ry & - 2X) — XoIANG — 2094 — MOAN — (XG - x, ? + (Yi = Yo)? = RF WF WRAY NoAX ~ 2(94 ~ Vola ~ (Xy ~ Xo)? + (34) = Io)? — RF Every pixel on a circle gives rise tu one observation equation. For exam- ple, the circle detection discussed in the previous section leads to approx- {mately 120 circle pixels (see Tab. 13.2 on page 346). We can also merge the cross pixels into the adjustment. Suppose the two narrow parallel lines canbe represented by their center line. From the design data of the fiducial mark we know whether or not this centerline passes exactly through the ‘iducial center, In that case, the centerline can be represented by one point and the slope, We have tn (13.24) Ym = Yo~ Mz(Xm— rod = Tm aoe Yn = Yo — M1 (X%n - Xo) pee *u. Yn are the pixels of the line with slope my and Xm. ¥m the pixels a line with Slope inz. As before, Xo,.¥o is the fiducial center. Se 7 the slopes and the center are unknown, £9s. 13.24 and 13.25 must "2ri2ed. We obtain =m 0326) MAX. ~ Ayo + (rn — Io) — MA(Xn— Xe) = 0327) MAX — Ave + (Vm — Yo) — M2(Xm ~ Xo) 5 Now 7 tions ag n PAVE a total of 4 parameters (Xo. Yo.71 te Due to Pee reduatt 28 detected pixels on the circles and the C085. Or iy Morcovge n@@RCy, blunders can be detected and elim! vn€ accuracy and reliability increase. d as many equa the Scanned with CamScannerREFERENCES Otsu, N. (1979). A threshold selection method trom gray-le¢ IEEE Transactions es ‘ and Cybernetics, 9, ( Sahoo, P. (1988). A survey’ of thresholding techniques. Com, puter \y Graphics, and Image Processing, 41, 233-260. PULEr Vision Schickler, W. (1995). Ein operationelles Verfahren zur neren Orienticrung von Luftbildern. Zeitschrift fiir Ph Fernerkundung, 63(3), 115-122. automatischen in togrammetrie un a Schickler, W. and Z. Poth (1996). The automatic interior orien its daily use. In International Archives of Photogrammetry Sensing, 31(B3), 746-751. tation ang and Remore Trinder, J.C. J. Jansa, and Y. Huang (1995). An assessment of the precision and accuracy of methods of digital target detection. IspRS Journal of Photogrammetry and Remote Sensing, 50(2), 12-20. Scanned with CamScanner