You are on page 1of 11

Imaging system fundamentals

Gerald C. Holst
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Optical Engineering 50(5), 052601 (May 2011)
Imaging system fundamentals
Gerald C. Holst
JCD Publishing Company
2932 Cove Trail
Winter Park, Florida 32789
Email: Jerry@JCDPublishing.com
Abstract. Point-and-shoot, TV studio broadcast, and thermal infrared
imaging cameras have signicantly different applications. A parameter
that applies to all imaging systems is F/d, where F is the focal ratio,
is the wavelength, and d is the detector size. F/d uniquely denes the
shape of the camera modulation transfer function. When F/d<2, aliased
signal corrupts the imagery. Mathematically, the worst case analysis as-
sumes that the scene contains all spatial frequencies with equal ampli-
tudes. This quanties the potential for aliasing and is called the spurious
response. Digital data cannot be seen; it resides in a computer. Cathode
ray tubes, at panel displays, and printers convert the data into an analog
format and are called reconstruction lters. The human visual system is
an additional reconstruction lter. Different displays and variable viewing
distance affect the perceived image quality. Simulated imagery illustrates
different F/d ratios, displays, and sampling artifacts. Since the human vi-
sual system is primarily sensitive to intensity variations, aliasing (a spatial
frequency phenomenon) is not considered bothersome in most situations.
C
2011 Society of Photo-Optical Instrumentation Engineers (SPIE). [DOI: 10.1117/1.3570681]
Subject terms: sampling; modulation transfer function; imaging system; aliasing;
reconstruction; spurious response.
Paper 110144TR received Feb. 14, 2011; revised manuscript received Mar. 5,
2011; accepted for publication Mar. 7, 2011; published online May 10, 2011.
1 Introduction
Point-and-shoot, TV studio broadcast, and thermal infrared
imaging cameras have signicantly different applications.
Common to all is an optical system and detector array which
are linked together by F/d, where F is the focal ratio, is
the wavelength of interest, and d is the detector size. In the
frequency domain, it is the ratio of the detector cutoff to the
optical cutoff. F/d uniquely denes the shape of the camera
modulation transfer function (MTF).
The MTFis the primary parameter used for systemdesign,
analysis, and specications. It describes how sinusoidal pat-
terns propagate through the system. Because any scene can
be decomposed into a Fourier series, the MTF approach in-
dicates how imagery will appear on the display. In general,
images with higher MTFs are judged as having better image
quality. However, there is no single ideal MTF shape that
provides the best image quality.
Sampling is an inherent feature of all electronic imag-
ing systems. The scene is spatially sampled in both direc-
tions by the discrete detector locations. It creates ambiguity
in target edges and produces moir e patterns when viewing
periodic targets. Aliasing becomes obvious when image fea-
tures approach the detector size. It distorts the image and the
amount of distortion is scene dependent. It is pronounced
when viewing periodic structures and these are rare in na-
ture. Aliasing is seldom reported when reproducing natural
scenery.
Mathematically, worst case analysis assumes that the
scene contains all spatial frequencies with equal amplitudes.
This quanties the potential for aliasing and is called the spu-
rious response. However, real scenes have a limited spectrum
and image quality is a subjective measure. This means there
is no method of validating the theory with imagery. MTF
0091-3286/2011/$25.00 C 2011 SPIE
theory and sampling issues are just two slices through the
multidimensional image quality space. They provide guid-
ance for camera design but do not uniquely quantify image
quality.
Assuming the detectors are in a rectangular lattice, the ll
factor (Fig. 1) is the ratio of areas
Fill factor = FF =
d
H
d
V
d
CCH
d
CCV
=
A
D
A
PIXEL
. (1)
The photosensitive area (A
D
) is d
H
d
V
. Larger detectors
collect more photons (higher sensitivity). Unfortunately, the
current trend is to make smaller detectors. The detector
center-to-center spacing (pitch) denes the sampling fre-
quency (u
S
=1/d
CCH
and v
S
=1/d
CCV
) and pixel area (A
PIXEL
= d
CCH
d
CCV
). Note that the detector size and pixel size can
be quite different and this leads to confusion when describing
system performance.
For convenience, a one-dimensional (horizontal) ap-
proach is used with FF = 1. Then d = d
H
= d
CC
= d
CCH
.
The equations and graphs are easily modied for nite ll
factors.
As the world becomes digital, we tend to ignore linear
system theory (developed for analog systems) and sampling
theory (analog-digital-analog conversion). The analog output
of each detector is immediately quantized. The digital data is
processed (image processing), digitally transmitted, and then
sent to a digital display. We cannot see digital data. It must be
transformed into analog data. Each display medium [cathode
ray tube (CRT), at-panel display, or printer] modies the
image in a different way.
2 Linear Shift Invariant Systems
Four conditions must be met to achieve a linear shift in-
variant (LSI) system: 1. the radiation is incoherent; 2.
signal processing is linear; 3. the system mapping is
Optical Engineering May 2011/Vol. 50(5) 052601-1
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
d
H
d
CCV
d
V
d
CCH
Detector
Pixel
Fig. 1 Detector and pixel relationship.
single-valued; and 4. the image is spatially invariant. An
LSI system only modies the amplitude and phase of the
target.
Nonlinear image processing, present in nearly every imag-
ing system, violates the one-to-one mapping requirement. For
convenience, image processing will be considered a linear
process. Single-valued mapping only occurs with non-noisy
and nonquantized systems. No system is truly noiseless, but
can be approximated as one when the signal-to-noise ratio
(SNR) is high.
A sampled-data system may be considered globally shift-
invariant on a macroscale. As a target moves from the top
to the bottom of the eld of view, the image also moves
from the top to the bottom. On a microscale, moving a point
source across a single detector does not change the detector
output. An imaging system system is not shift-invariant on a
microscale.
In spite of these disclaimers, an imaging system is treated
as quasilinear over a restricted operating region to take ad-
vantage of the wealth of mathematical tools available. For
mathematical convenience, an electronic imaging system is
characterized as a linear spatial-temporal systemwith respect
to both time and two spatial dimensions. Although space is
three-dimensional, an imaging system displays only two di-
mensions.
2.1 Linear System Theory
An object can be thought of as the sum of an innite array
of impulses located inside the object boundaries. Thus, an
object can be decomposed into a two-dimensional array of
weighted Dirac delta functions, (x x

), (y y

)
o(x, y) =

=
o(x

, y

) (x x

)(y y

)x

. (2)
An optical system produces an image and the process is
symbolically represented by the operator h
SPATIAL
{}
i (x, y) =

=
h
SPATIAL
{o(x

, y

) (x x

)(y y

)x

}.
(3)
For small increments, this becomes the convolution integral
i (x, y)
=

o(x

, y

)h
SPATIAL
{(x x

)(y y

)} dx

dy

, (4)
and is symbolically represented by the two-dimensional con-
volution operator **
i (x, y) = o(x, y) h
SPATIAL
(x, y). (5)
The function h
SPATIAL
(x, y) is the optical systems response
to an input impulse. The resulting image is the point spread
function (PSF). Equation (4) is simply the summation of all
the impulse responses. If i(x, y) passes through another LSI
system, i

(x, y) = i (x, y) h

SPATIAL
(x, y). As the number
of LSI systems increases, multiple convolution calculations
become tedious.
Since convolution and multiplication are Fourier trans-
form pairs, convolution in space becomes a multiplication in
the frequency domain
I (u, v) = O(u, v)H
SPATIAL
(u, v). (6)
H
SPATIAL
(u, v) is the optical transfer function and is usually
labeled as OTF(u, v). The MTFis the magnitude and the phase
transfer function (PTF) is the phase of the complex-valued
OTF. Symbolically
OTF
SPATIAL
(u, v) = MTF
SPATIAL
(u, v) e
j PTF(u,v)
. (7)
Spatial frequency can be dened in image-space (at the
focal plane) with units of cycles/mm or in object-space (cy-
cles/mrad). They are related by u = u
o
/f l. To maintain di-
mensionality, if u
o
is measured in cycles/mrad then the focal
length, f l, must be measured in meters to obtain u in cy-
cles/mm.
The MTF and PTF alter the image as it passes through
the system. For LSI systems, the PTF is of no special in-
terest since it only indicates a spatial or temporal shift with
respect to an arbitrarily selected origin. An image where the
MTF is drastically altered is still recognizable whereas large
nonlinearities in the PTF can destroy recognizability.
The point spread function is assumed to be separable
in Cartesian coordinates (taken as horizontal and vertical).
Separability
1
reduces the analysis so that complex calcula-
tions that include cross-terms are not required. Separability
in Cartesian coordinates requires that
h
SPATIAL
(x, y) = h
SPATIAL
(x)h
SPATIAL
(y). (8)
Separability in polar coordinates requires
h
SPATIAL
(r, ) = h
SPATIAL
(r)h
SPATIAL
(). (9)
The PSF of an aberration-free optical system can be char-
acterized by a function that is separable in polar coordinates.
The detector is assumed to be rectangular. Its PSFis separable
in Cartesian coordinates, but is not separable in polar coordi-
nates. The collective PSF of the detector and the optics is not
separable in either polar or Cartesian coordinates. The errors
Optical Engineering May 2011/Vol. 50(5) 052601-2
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
associated with separability are usually small
2
and therefore
most analyses use the Cartesian separability approximation.
The detector array is assumed to be composed of rectangu-
lar (or square) detectors spaced in a rectangular (or square)
grid (Fig. 1). Any other spacing (e.g., hexagonal) can only
be analyzed on a case-by-case basis.
3
The electronic imaging system response consists of both
an optical response and electronic response. Time and
spatial coordinates are treated separately: h
SYSTEM
(x, y, t)
= h
SPATIAL
(x, y)h
ELECTRICAL
(t). This is reasonable. Optical
elements do not generally change with time and therefore
are characterized only by spatial coordinates. Similarly, elec-
tronic circuitry exhibits only temporal responses. The detec-
tor provides the interface between the spatial and temporal
components, and its response depends on both temporal and
spatial quantities. The conversion of two-dimensional op-
tical information to a one-dimensional electrical response
assumes a linear photodetection process. Implicit in the de-
tector response is the conversion from input photon ux to
output voltage (or amps).
The electronic circuitry is assumed to modify the horizon-
tal signal only (although this depends on the system design).
With appropriate scaling, the electronic frequencies can be
converted into spatial frequencies. This is symbolically rep-
resented by f
e
u:
H
SYSTEM
(u) = H
SPATIAL
(u)H
ELECTRONICS
( f
e
u), (10)
and
H
SYSTEM
(v) = H
SPATIAL
(v). (11)
A system is composed of many components that respond
to spatial and temporal signals. Here lies the advantage of
working in the frequency domain. If multiple LSI com-
ponents exist in the spatial and/or electronic domains, the
individual MTFs can be multiplied together. Equivalently,
multiple convolutions in space or time are equivalent to mul-
tiplications (or cascading) in the frequency domain. For in-
dependent MTFs
MTF
SYSTEM
(u, v) =
n

i =1
m

j =1
MTF
SPATIAL
(i, u, v)
MTF
ELECTRONICS
( j, f
e
u). (12)
When coupled with the three-dimensional noise
parameters
4
the MTF uniquely denes system performance.
The MTF determines how the system responds to spa-
tial frequencies. It does not contain any signal intensity
information.
Image formation is straight forward. Over the region that
linear system theory is valid, the scene is transformed into
its frequency components O(u, v). Each frequency is then
multiplied by MTF
SYSTEM
(u, v) to provide I
SYSTEM
(u, v).
Then the inverse transform provides I
SYSTEM
(x, y).
While an imaging system is composed of many sub-
systems, generally the MTF is dominated by the op-
tics, detector, electronic lters, digital lters, and display
medium. Adding electronic and digital lters to the anal-
ysis obscures the fundamentals of image creation. Here,
the electronic and digital lter MTFs are assumed to be
unity over the spatial frequencies of interest. The basic
0
0.2
0.4
0.6
0.8
1
0 0.2 0.4 0.6 0.8 1
Spatial frequency (u/u
C
)
M
T
F
O
P
T
I
C
S
Fig. 2 Optics MTF for a clear (unobscured) circular aperture.
MTF is
MTF
SYSTEM
= MTF
OPTICS
MTF
DETECTOR
MTF
DISPLAY
.
(13)
Camera manufacturers have no control over how an ob-
server will process the imagery and therefore their analyses
generally omit the display MTF. The perceived MTF depends
upon display characteristics and the human visual system
(HVS) interpretation
MTF
PERCEIVED
= MTF
SYSTEM
MTF
HVS
, (14)
and is used for predicting image quality metrics. These met-
rics will not be discussed here. However, MTF
HVS
plays an
important role when viewing imagery (discussed in Sec. 5).
2.2 Optics MTF
A complex optical system is replaced with a simple lens
that has the equivalent focal length. For an aberration-free,
radially symmetric optical system, OTF
OPTICS
is the same
in the horizontal and vertical directions. Since the OTF is
positive, it is labeled as the MTF. In the horizontal direction,
the diffraction-limited MTF for circular aperture (Fig. 2) is
MTF
OPTICS
(u) =
2

cos
1

u
u
C

u
u
C

u
u
C

.
(15)
The image-space optics cutoff frequency is u
C
= D/(f l)
= 1/(F), where D is the aperture diameter and F
= f l/D. Because the cutoff frequency is wavelength depen-
dent, Eq. (15) and Fig. 2 are only valid for noncoherent
monochromatic light. The extension to polychromatic light
is lens-specic. Most lens systems are color corrected (achro-
matized) and therefore there is no simple way to apply this
simple formula to predict the MTF. As an approximation to
the polychromatic MTF, the average wavelength is used to
calculate the cutoff frequency:
AVE
= (
MAX
+
MIN
)/2.
2.3 Detector MTF
The detector OTF cannot exist by itself. Rather, the detector
OTF must also have the optical MTF to make a complete
imaging system. In the horizontal direction, the OTF of a
single rectangular detector is
OTF
DETECTOR
(u) =
sin(du)
du
. (16)
Optical Engineering May 2011/Vol. 50(5) 052601-3
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
0 1 2 3 4
Spatial frequency (u/u
D
)
O
T
F
D
E
T
E
C
T
O
R
Fig. 3 Detector OTF.
The OTF is equal to zero when u = k/d (Fig. 3). The rst
zero (k = 1) is considered the detector cutoff frequency, u
D
.
It is customary to plot the OTF up to u
D
. This is probably
done because it is unknown what effect the negative OTF has
on overall image quality. Since the OTF is positive up to u
D
,
it is called the MTF. Plotting the MTF up to the rst zero
erroneously suggests that the detector does not respond to
frequencies greater than u
D
. The optical system cutoff limits
the absolute highest spatial frequency that can be faithfully
imaged and not u
D
. Nevertheless most analyses (and that
considered here) consider the response up to u
D
only.
2.4 Display MTF
Display specications are a mix of CRT terminology, video
transmission standards, alphanumeric character legibility,
and graphics terminology. CRTs are low cost with high reso-
lution, wide color gamut, and high luminance. Flat panels do
not have all these attributes. Nevertheless, at panel displays
will probably replace all CRTs in the near future.
Flat panel displays are assumed to have rectangular pix-
els. Usually the number of pixels matches the number of
detectors. When referred to image space,
OTF
DISPLAY
(u) =
sin (d
CC
u)
d
CC
u
. (17)
As with the detector, the display response is (erroneously)
plotted up to the rst zero. When FF = 1 the display MTF is
identical to detector MTF.
3 Sampling
Sampling is an inherent feature of all electronic imaging
systems. The scene is spatially sampled in both directions
by the discrete detector locations. Sampling theory states
that the frequency can be unambiguously recovered for all
input frequencies below Nyquist frequency. After aliasing,
the original signal can never be recovered. The mathemat-
ics suggests that aliasing is an extremely serious problem.
Objection-ability depends upon the scene, F/d, d
CC
, dis-
play medium, and viewing distance.
3.1 Sampling Theorem
The sampling theorem as introduced by Shannon
5
was ap-
plied to information theory. He stated that if a time-varying
function v(t) contains no frequencies higher than f
MAX
(Hz), it
is completely determined by giving its ordinates at a series of
points spaced 1/(2f
MAX
) seconds apart. The original function
Fig. 4 Replicated spectra. The ideal reconstruction lter response is
unity up to f
N
and zero thereafter. It eliminates the replicated spectra
leaving only the analog base band.
can be reconstructed by an ideal low-pass lter. Shannons
work is an extension of others,
6
and the sampling theorem
is often called the ShannonWhittaker theorem. Reference 7
provides an in-depth discussion on sampling effects.
If sampling occurs every T seconds, the sampling fre-
quency is f
S
= 1/T. The resultant signal is
v
SAMPLE
(t ) = v(t )s(t ), (18)
where s(t) is the sampling function is equal to (t nT). Since
multiplication in one domain is represented as convolution
in the other, the sampled frequency spectrum is
V
SAMPLE
( f
e
) = V( f
e
) S( f
e
), (19)
where V( f
e
) is the amplitude spectrum of the band-limited
analog signal and S( f
e
) is the Fourier transform of the sam-
pler. The transform S( f
e
) is a series of impulses at nf
S
and
is called a comb function. When convolved with V( f
e
), the
resultant is a replication of V( f
e
) about nf
S
(n = to
+). Equivalently, the sampling frequency interacts with
the signal to create sum and difference frequencies. Any in-
put frequency, f
o
, will appear as nf
S
f
o
after sampling.
Figure 4 illustrates a band-limited system with frequency
components replicated by the sampling process. The base
band ( f
H
to f
H
) is replicated about nf
S
. To avoid distortion,
the lowest possible sampling frequency is that value where
the rst replicated spectrum just adjoins the base band. This
leads to the sampling theorem that a band-limited system
must be sampled at twice the highest frequency ( f
S
2f
H
).
Nyquist frequency is dened as f
N
= f
S
/2.
After digitization, the data reside in data arrays (e.g., a
computer memory location). The signal must be converted
(reconstructed) into an analog signal to be useful. If the orig-
inal signal was oversampled ( f
S
2f
H
) and if the reconstruc-
INPUT
ALIASED OUTPUT
0 T 2T 3T 4T 5T 6T
Fig. 5 An undersampled sinusoid will appear as a lower frequency
after ideal reconstruction.
Optical Engineering May 2011/Vol. 50(5) 052601-4
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
Relative spatial frequency
M
T
F
0 u
S
2u
S
3u
S
Replicated spectra
Fig. 6 Overlapping spectra. The amount of aliasing (and hence im-
age quality) is related to the amount of overlap.
tion lter limits frequencies to f
N
, then the reconstructed
image can be identical to the original image.
Within an overlapping band ( f
S
< 2 f
H
), there is an ambi-
guity in frequency. It is impossible to tell whether the recon-
structed frequency resulted from an input frequency of f
o
or
nf
S
f
o
(Fig. 5).
3.2 Aliasing
To apply the above sampling theory to imaging systems, let
f
e
u, f
S
u
S
, and T d
CC
. As the sampling frequency
decreases, the rst replicated spectrum starts to overlap the
base band (Fig. 6). It is the summation of these spectra that
distort the image.
A bar pattern consists of an innite number of frequen-
cies. While the fundamental may be less than the Nyquist
frequency, higher-order terms will not. These higher-order
terms are aliased and distort the signal. In Fig. 7 the input bar
pattern fundamental is 1.68 u
N
and the aliased fundamental
is 0.32 u
N
. Since higher-order frequencies are present, the
reconstructed bars appear more triangular than sinusoidal.
Since aliasing occurs at the detector, the signal must be
band-limited by the optical system to prevent it. This is
achieved by designing a system with F/d 2 or by us-
ing an optical low pass lter (OLPF).
9
While monochrome
aliasing is tolerable, color aliasing is bothersome. Single chip
color arrays always have a window over the array. The rst
impression is that the window is designed to protect the ar-
ray. This is an ancillary feature. The window is actually a
birefringent crystal that acts as an OPLF. The OPLF reduces
the MTF and reduces image contrast. Since color aliasing is
unacceptable, the reduced MTF is a small penalty to pay.
Fig. 7 Aliasing. Input (left) and aliased output (right). An ideal recon-
struction lter was used. Imagery created (Ref. 8) by MAVIISS.
Fig. 8 Reconstruction with a at panel display (left) and a CRT (right).
See footnote on page 8.
3.3 Reconstruction
Digital data cannot be seen because it resides in a com-
puter memory. Any attempt to view a digital image requires
a reconstruction lter.
10
Most imaging systems rely on the
display medium and HVS to produce a perceived continu-
ous image (discussed in Sec. 5). Display media include laser
printers, half-toning, fax machines, CRTs, and at panel dis-
plays. The display medium creates an image by painting a
series of light spots on a screen or ink spots on paper. The
spot acts as a low pass reconstruction lter. Each display
medium has a different spot size and shape resulting in dif-
ferent frequency responses. The perceived imagery will be
different on each display type.
A at panel display is not an ideal reconstruction lter.
It passes signicant frequency components above u
N
and
this makes the image blocky or pixelated. A CRT will re-
move the higher frequencies (above u
N
) but also attenuates
the in-band frequencies to create a somewhat blurry image
(Fig. 8). The ideal reconstruction lter abruptly drops to zero
at u
N
. As illustrated in Fig. 9, a sharp drop in one-domain
produces ringing in the other (Gibbs phenomenon). In these
two gures, the optics and detector MTFs are unity over the
spatial frequencies of interest. This emphasizes howdifferent
reconstruction lters affect image quality.
Moir e patterns occur when periodic targets are viewed.
Some published articles provide resolution charts. The sam-
pling artifacts become more noticeable when viewing targets
at an angle with respect to the array axis. Figure 10 illustrates
various artifacts when viewing a TV resolution chart. It com-
pares a Bayer pattern with the Kodak TrueSense color lter
array (CFA) and their respective demosaicking algorithms.
Fig. 9 Reconstruction with an ideal reconstruction lter. See footnote
on page 8.
Optical Engineering May 2011/Vol. 50(5) 052601-5
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
(a) (b) (c) (d) (e)
Fig. 10 Moir e patterns vary according to the CFA type and demo-
saicking algorithm. Details are provided in Ref. 11. The heavy hori-
zontal lines represent the estimated TV resolution afforded by each
camera. The on-line PDF version is in color where color aliasing can
be seen. This gure has been enlarged. At normal viewing dis-
tances, the individual pixels cannot be resolved. View this gure from
several feet.
This gure illustrates the difculty in quantifying resolution.
It illustrates distorted edges and periodic structures.
3.4 Resampling
Nearly all analyses focus on the spatial sampling created
by the detector array. If the camera output is analog, then
a frame capture board redigitizes the image for computer
processing. The board can digitize the analog signal at a rate
that is different than u
S
. Additional samplers in the overall
system can add new sampling artifacts.
Imagery imported into word documents or other programs
are automatically interpolated to provide a smooth continu-
ous image. As the image is enlarged interpolation smoothes
it. This is not so when the image is resampled without inter-
polation. In Fig. 11, an enlarged portion of a scene captured
by a 10 Mpixel point-and-shoot camera was inserted as a
desktop picture. The at panel display resampled the scene.
Since the at panel elements (12801024) did not align with
the camera pixels (36642748), resampling artifacts become
obvious: straight lines appear as a stair step (jaggies).
Fig. 11 Original image (left) and image seen on the WindowXP desk-
top background scene (right). Each picture is 210245 pixels (HV).
The arrows point to the most obvious sampling artifacts. Careful ex-
amination reveals numerous others. The on-line PDF version is in
color.
Relative spatial frequency
M
T
F
u
N
MTF
POST
(u)
MTF
PRE
(u)
MTF
PRE
(u
S
-u)
MTF
POST
(u)MTF
PRE
(u
S
-u) MTF
POST
(u)MTF
PRE
(u)
Fig. 12 Practical reconstruction lter. MTF
POST
(u)MTF
PRE
(u
Su
)
was created by the sampling process. It was not in the original scene.
3.5 Spurious Response
In one dimension, the reconstructed image
12
is
I (u) = MTF
POST
(u)

n=0
MTF
PRE
(nu
S
u)O(nu
S
u).
(20)
MTF
PRE
contains all the MTFs up to the sampler (the de-
tector). For this paper, MTF
PRE
=MTF
OPTICS
MTF
DETECTOR
.
MTF
POST
represents all the lters after the sampler. For this
paper MTF
POST
=MTF
DISPLAY
. Equation (20) can be written
as
I (u) = MTF
POST
(u)MTF
PRE
(u)O(u)
+MTF
POST
(u)

n=1
MTF
PRE
(nu
S
u)O(nu
S
u).
(21)
The rst term is the image spectrum when no sampling is
present and is called the direct response. Sampling created
the remaining terms and these may be considered an aliasing
metric. If u
S
2u
H
and the reconstruction lter response is
zero for all frequencies greater than u
S
u
H
(see Fig. 4), the
second term is zero.
Considering the rst fold back frequency (n = 1) and
assuming O(u) = 1, Schade
13
dened the spurious response
as
Spurious response=

0
MTF
POST
(u)MTF
PRE
(u
S
u) du

0
MTF
POST
(u)MTF
PRE
(u) du
.
(22)
The highest scene spatial frequency is limited by the op-
tical cutoff. The upper limit in the denominator is u
C
. The
highest spatial frequency in the aliased signal (numerator)
is limited by the reconstruction lter. The assumption that
the scene contains all frequencies [O(u) =1] with equal am-
plitude is for mathematical convenience. Perhaps Schades
spurious response should be called the potential for alias-
ing metric. Figure 12 illustrates the spurious response when
a practical post-reconstruction lter is used. The spurious
Optical Engineering May 2011/Vol. 50(5) 052601-6
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
0
10
20
30
40
50
0 10 20 30 40 50
d

(

m
)
F (m)
No aliasing
LWIR design
MWIR design
Visible
F/d = 2
Fig. 13 Design space for visible, MWIR, and LWIR cameras (FF =
1). There is no aliasing when F/d 2.
response value depends critically upon the reconstruction
lter used.
4 Optics-Detector Subsystem
Image metrics may be described in the spatial domain where
the optical blur diameter is compared to the detector size or in
the frequency domain where the detector cutoff is compared
to the optics cutoff. Either comparison provides an image
quality metric
14
that is a function of F/d. Table 1 summa-
rizes the two limiting cases. Since the transition from one
region to the other is gradual, it is difcult to select an F/d
value that separates the two regions. It is nominally set at F/d
= 1. From a sampling viewpoint, the important parameter
15
is F/d
CC
(also called F/ and Q). When FF = 1, F/d
= F/d
CC
. Figure 13 illustrates overall design space for
visible, mid-wave infrared (MWIR) and long-wave infrared
(LWIR) cameras. As listed in Table 2, the more popular de-
signs are detector-limited.
The MTF at Nyquist frequency is often used as a mea-
sure of performance (Fig. 14). As MTF(u
N
) increases, image
quality should increase. Unavoidably, as the MTF increases,
aliasing also increases and image quality suffers. Figure 15
provides 256256 pristine images. Figures 1618 illustrate
the imagery for three different F/d ratios with FF = 1.
Figure 8 illustrates F/d = 0.55. All images were recon-
Table 1 Optics-limited versus detector-limited performance.
F/d
System
performance Spatial domain Frequency domain
<1 Detector-limited Airy disk much
smaller than
detector
Optical cutoff much
greater than the de-
tector cutoff
>1 Optics-limited Airy disk much
larger than de-
tector
Optical cutoff much
less than the detec-
tor cutoff
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0 0.5 1 1.5 2
F /d
M
T
F

(
u
N
)

Fig. 14 MTF
OPTICS
(u
N
)MTF
DETECTOR
(u
N
) when FF = 1.
structed with a at-panel display. As evident in these gures,
image distortion is more obvious with periodic targets and
straight lines. Low F/d imagery exhibits the most aliasing.
In Fig. 16 the 3-bar targets appear as one, two, or distorted
3 bars. High F/d imagery (Fig. 18) has no aliasing or dis-
tortion. But the imagery is blurry because the MTF is so
low.
5 Viewing Distance
Whether the image is displayed on a CRT, at-panel dis-
play, or printed, the viewing distance signicantly affects
perceived quality. As illustrated in Fig. 19, as the target fre-
quency increases, the perceived modulation decreases. As the
distance increases a small object will eventually subtend an
angle smaller than that which can be resolved by the HVS.
If several objects are close together, they are perceived as
one where the saturation, hue, and brightness are an average
of all the objects. In this context, the HVS is an additional
reconstruction lter.
5.1 Flat Panel Displays
For monochrome systems, the minimum viewing distance
occurs when the individual display pixels are barely per-
ceptible. At closer distances, the pixels become visible and
this interferes with image interpretation. The at panel pixel
consists of three color elements (red, green, and blue). At nor-
mal viewing distances, the eyes MTF attenuates the spatial
frequencies associated with the display pixels thus produc-
ing a smooth continuous image. Assuming a display pixel
pitch of 0.26 mm and a typical viewing distance of 0.355 m
(14 in.), each pixel subtends 0.73 mrad and each color ele-
ment subtends 0.24 mrad. The HVS can resolve 0.29 mrad
(equivalent to 20/20 vision). Thus at 0.355 m, each pixel is
Table 2 F/d for F = 2.
Spectral Typical
AVE
response Detector d (m) (m) F F/d
Visible CMOS 2.2 0.55 1.1 0.5
MWIR InSb 18 4.0 8.0 0.44
LWIR Microbolometer 18 12 24 0.75
Optical Engineering May 2011/Vol. 50(5) 052601-7
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
Fig. 15 Pristine imagery.
Fig. 16 Imagery when F/d = 0.2. See footnote on this page.
Fig. 17 Imagery when F/d = 1.0. See footnote on this page.
Fig. 18 Imagery when F/d = 2.0. See footnote on this page.
Table 3 Digital still camera prints (assuming 300 pixels per inch is
acceptable).
Application Typical array size (HV) Print size (inches)
Cell phone 640480 2.11.6
Point-and-shoot 28972173 9.67.2
6.3 Mpixel
Point-and-shoot 36502738 12.19.1
10 Mpixel
Professional SLR 73025477 24.318.2
40 Mpixel
resolved and each color element is not (desired result). At
0.154 m (6 in.), each color element subtends 0.56 mrad and
can be just perceived. When too close, the individual color
spots become visible and the impression of full color is lost.
For all the imagery in this paper, it is suggested that you
move several feet away to achieve the same visual angle
that would normally exist.

At normal viewing distance,


the HVS attenuates the amplitudes of the spatial frequen-
cies associated with the blockiness and the imagery appears
continuous.
5.2 Prints
Color printing sufces with four colors (red, green, blue, and
black). There are no grays. It is either ink or no ink. For
lighter colors, a small ink spot is surrounded by a white area.
To increase saturation, the ink spot is larger. For xed pixel
size the white area is smaller. The HVS blends the ink spot
and white area to have a perceived saturation.
Printed imagery is considered excellent when there are
more than 300 dots/in (dpi). Photos look good when there are
about 300 pixels/in. To allow for color printing, the printer
should provide about 3 times more dots, or at least 900 dpi.
The values in Table 3 assume 300 pixels/in creates an ex-
cellent image. Larger images can be created. If pixel repli-
cation is used, at some point they will start to be blocky
(you can see the individual pixels). However, blocky im-
ages are rarely seen because software always interpolates
the data to create a smooth image. At small viewing dis-
tances this smoothed image may appear blurry. In contrast to
wet-lm cameras, image enlargement does not provide more
resolution. Resolution is xed by the pixel size and focal
length.
6 Summary
Performance for all cameras (point-and-shoot, TV studio
broadcast, and thermal infrared imaging) can be described
by an MTF with F/d being an important design metric.
The difference between detector-limited and optics-limited

The 256256 image (Fig. 15) was downsampled to 3232 detectors. The
imagery is enlarged so that your eye MTF and the printing do not signi-
cantly affect the image quality. This allows you to see the distortion created
by sampling and the system MTF degradation. Image quality depends upon
viewing distance. Viewthe images at several feet to viewto simulate normal
distance (same visual angle).
Optical Engineering May 2011/Vol. 50(5) 052601-8
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
Distance (x) Distance (x)
I
n
t
e
n
s
i
t
y
P
e
r
c
e
i
v
e
d

I
n
t
e
n
s
i
t
y
Fig. 19 Perceived modulation (right) when viewing a sweep frequency target (left). Very high spatial frequencies are perceived as a uniform
gray.
is important. When detector-limited (F/d < 1), changes in
the optical system will have little effect on image quality.
Likewise, in the optics-limited region (F/d > 1), changing
the detector size will have minimal effect on image quality.
While Fig. 13 illustrates overall design space, most imaging
systems today are detector-limited (Table 2).
The MTF will always be used for lens design. It will be
used for scientic and military system optimization. It is
of lesser concern for commercial applications. The MTF at
Nyquist frequency is often used as a measure of performance
(Fig. 14). As MTF(u
N
) increases, image quality should in-
crease. Unavoidably, as the MTF increases, aliasing also in-
creases and image quality suffers. This suggests there may be
an optimum MTF, or, equivalently an optimum F/d. This is
not so. Cameras are designed for a specic application with
F/d being a secondary consideration.
There is no aliasing when F/d 2. This limiting condi-
tion is known as critical sampling or Nyquist sampling of
the optical blur in the astronomy community.
16
The latter
termis not good terminology since the blur diameter consists
of all frequencies up to the optical cutoff. When F/d 2
the imagery replicates the scene exactly. This may be an im-
portant consideration for medical imaging where a sampling
artifact could be construed as a medical abnormality or a
space probe where it is impossible to obtain ground truth.
Having F/d 2 may be overly restrictive. In-band MTFs
(frequencies less than the Nyquist frequency) are reduced in
amplitude (Fig. 18). If the SNR is sufciently high, a boost
circuit can increase the MTF and sharpen the image. Note
that if F/d <2, the boost circuit will also increase the aliased
signal [(MTF
POST
(u)MTF
PRE
(u
S
u) in Fig. 12]. If too much
boost is used, the amplied aliased signal can degrade image
quality.
Sampling theory suggests that an ideal reconstruction
lter could be used. Although unrealizable, it can be ap-
proximated by a high order Butterworth lter. This works
well if there is no aliasing. With aliasing, the sharp cut-
off lter creates undesirable ringing (Fig. 9). Flat panel dis-
plays are not ideal reconstruction lters and will not produce
ringing.
Aliasing was quantied by the spurious response. The
information in the higher frequencies has been aliased to
lower frequencies. But it is not known howto interpret this in-
formation. Mathematically, worst case analysis assumes that
the scene contains all spatial frequencies with equal ampli-
tude. Real scenes have a limited spectrum and image quality
is a subjective measure. This means there is no method of
validating the theory with imagery. MTFtheory and sampling
issues are just two slices through the multidimensional image
quality space. They provide guidance for camera design but
do not uniquely quantify image quality. The perceived image
quality depends upon F/d, d
CC
, display medium, and view-
ing distance. Changing any one or all of these parameters
affects the perceived image quality.
Nearly every visible digital camera aliases the scene
(Fig. 13). Is this really bad? Sampling artifacts are seen rou-
tinely. The amount of aliasing is scene specic and may or
may not be bothersome. It becomes apparent when viewing
test patterns (Figs. 10 and 16), picket fences, plowed elds,
railroad tracks, and Venetian blinds. In fact, while aliasing is
present (Fig. 11), the imagery may be considered excellent.
The color rendition on the at panel display is excellent. The
scene content is exciting and the few jagged lines are just
ignored. On the other hand, a printed image is enlarged to
1620 in. (see Table 2) and looks wonderful.
Why is aliasing acceptable? The eye is primarily sensi-
tive to intensity variations and less so to frequencies. There-
fore, sampling artifacts in imagery are usually tolerable.
We have become accustomed to the aliasing. Simply watch
television. Pay attention to the folks wearing clothing with
narrow stripes or small patterns. Note the changing patterns
as they move. Stripes will change in shape, width, and color.
In contrast, the ear is a frequency detector and any dis-
tortion is immediately obvious. The sampling theorem must
be strictly followed for auditory-based processes but not for
imagery.
Different display media create different images. Image
quality is scene dependent. Go to any store selling TVs and
observe the difference between identical TVs. Note the
differences at different viewing distances. As the distance
decreases, the differences become more apparent. Simply
stated, all displays are designed for an assumed viewing dis-
tance. The printed image will never be the same as that seen
on the at panel display.
Finally, wet-lm based cameras did not alias the scene.
With technological advancements in digital cameras, image
quality has decreased. This is the trade-off between instant
imagery, ability to digitally transmit the imagery, and im-
age quality. Current trends are to build smaller detectors
because it leads to lower cost and lighter cameras. Staring
array detectors create photoelectrons that are stored in charge
wells. But the charge well capacity decreases as the pixel area
decreases. The next generation camera will have a smaller
dynamic range, smaller SNR, and poorer image quality.
Optical Engineering May 2011/Vol. 50(5) 052601-9
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms
Holst: Imaging system fundamentals
References
1. R. G. Driggers, P. Cox, and T. Edwards, Introduction to Infrared and
Electro-Optical Systems, pp. 2125, Artech House, Norwood, MA
(1999).
2. R. H. Vollmerhausen and R. G. Driggers, Analysis of Sampled Imaging
Systems, SPIE Tutorial Text TT39, pp. 2831, Bellingham, WA (2000).
3. O. Hadar, A. Dogariu, and G. D. Boreman, Angular dependence of
sampling modulation transfer function, Appl. Opt. 3628, 72107216
(1997).
4. G. C. Holst, Electro-optical Imaging System Performance, 5th ed.,
pp. 362374, JCD Publishing Company, Winter Park, FL (2008).
5. C. E. Shannon, Communication in the Presence of Noise, Proceedings
of the IRE Vol. 37, pp. 1021, Institute of Electrical and Electronics
Engineers, New York (1949).
6. A. B. Jerri, The Shannon Sampling Theorem Its Various Extensions
and Applications: A Review, Proceedings of the IEEE, Vol. 85(11),
pp. 15651595, Institute of Electrical and Electronics Engineers, New
York (1977).
7. G. C. Holst, Sampling, Aliasing, and Data Fidelity, JCD Publishing,
Winter Park, FL (1998).
8. MAVIISS (MTF based Visual and Infrared System Simulator) is
an interactive software program available from JCD Publishing at
www.JCDPublishing.com.
9. G. C. Holst and T. L. Lomheim, CMOS/CCD Sensors and Camera
Systems, 2nd ed., pp. 287289, JCD Publishing Company, Winter Park,
FL (2011).
10. G. C. Holst, Are reconstruction lters necessary?, In Infrared Imaging
Systems: Design, Analysis, Modeling, and Testing XVII, G. C. Holst,
Ed., Proc. SPIE 6207, 62070K (2006).
11. D. Tamburrino, J. M. Speigle, D. J. Tweet, and J.-Jan Lee, 2PFC (two
pixels, full color): Image sensor demosaicing and characterization,
J. Electron. Imaging 19(2), 021103 (2010).
12. G. C. Holst, Electro-Optical Imaging System Performance, 5th ed.,
pp. 197199, JCD Publishing Company, Winter Park, FL (2008).
13. O. H. Schade, Sr., Image reproduction by a line raster process,
in Perception of Displayed Information, L. C. Biberman, Ed., pp. 233
278, Plenum Press, New York (1973).
14. G. C. Holst, Imaging systemperformance based upon F/d, Opt. Eng.
46, 103204 (2007).
15. R. D. Fiete, Image quality and FN/ for remote sensing systems,
Opt. Eng. 38, 12291240 (1999).
16. M. Clampin, Ultraviolet-optical charge-coupled devices for space in-
strumentation, Opt. Eng. 41, 11851191 (2002).
Gerald C. Holst is an independent consul-
tant for imaging system analysis and test-
ing. His varied background includes serving
as a technical liaison to NATO, research sci-
entist for DoD, and a member of the Martin
Marietta (now Lockheed Martin) senior tech-
nical staff. He has planned, organized, and
directed the internationally acclaimed SPIE
conference Infrared Imaging Systems: De-
sign, Analysis, Modeling and Testing since
1990. He is author of over 30 journal articles
and 6 books. He is a SPIE fellow and a member of OSA.
Optical Engineering May 2011/Vol. 50(5) 052601-10
Downloaded From: http://spiedigitallibrary.org/ on 08/03/2014 Terms of Use: http://spiedl.org/terms