AG20103: Soil Technology

(l-t-p: 3-0-0 Credit: 3)

Instructor: Somsubhra Chakraborty

Room: R-207, PHTC
Soil Science
Office Phone: 83156
Email: somsubhra@iitkgp.ac.in

SOIL SENSORS
 Soil Sensors: the Need of the Hour
• Can transform global agriculture
• Right source of plant nutrient, in right rate, right time and right place
Digital
agriculture

• To build synergy among data analytics, new age sensors and

IIT Kgp’s geospatial technologies
vision

• A decade of research in
• Rapid natural resource management via generation of spectral libraries of soil, crop and water
• Development of microfluidics, MEMS and new-age sensors
• Digital mapping and geospatial modelling
Preparedness
 PXRF: Advanced Portable Sensor
• Energy released by replacing of
inner shell electrons by outer shell
electrons (termed fluorescence) is
quantified by a silicon drift detector
in the aperture of the instrument .
• Scanning (dwell) time for a given
sample is typically ∼60–90 s.
• Advantages:
• Field portability
• No consumables
• Non-destructive
• Multi-elemental analysis
(providing simultaneous analysis
of ~20-30 elements)
• Parameters:
• Soil pH, EC, CEC, P, K, Ca, Mg, S,
Micronutrients, Gypsum, %BS,
Heavy metals, permafrost pH,
LULC, Parent material, Profile
horizonation, Geochemistry,
Compost EC, Compost CEC, Water
heavy metals, Leaf elements
 Reference Methods

Soil Science
USDA NRCS: 7.3.1 X- Society of
US-EPA Method Ray Fluorescence, America
6200 Handheld
XRF Spectra

2 New approaches
1.Combining XRF beams as such
2.Selecting the peak and extracting the FWHM and Hmax
PXRF for LULC
Forests (147 samples), agricultural
lands (149 samples), and iii) converted
lands (converted from forest to
agriculture over the past 20 years) (127
samples).
Classification Results

CART
 Other PXRF Work Published

• Elemental analysis of Vegetation analysis in Sierra Leone;

Weindorf.
vegetation
Elements in Vegetation

Leaf Elemental Analysis PXRF for Tea Leaf Classification

SEM-EDAX
Diffuse Reflectance Spectroscopy
• Fast
• Cost-effective (almost zero recurring cost)
• Needs minimum or no sample pretreatments
• High Throughput and Non-invasive
• Portable

• Parameters:
• OC, Available N, P, K
• pH and clay
• Moisture and heavy metals

VisNIR DRS

Soil arsenic map at Kolkata landfill agricultural site using DRS

VisNIR-DRS Basics

• Visible and near infrared

diffuse reflectance
spectroscopy is a technology
that uses a mug light or
contact probe to produce
reflections at different
wavelengths (350-2,500 nm)
• Field portable
• Simple
• Reflectance patterns can be
correlated to a number of
soil properties
How Diffuse Reflectance Works

Vis: 350 to 700 nm

NIR: 700 to 2500 nm

Soil Material
 VisNIR-DRS Equipment

▪ AgriSpec DRS ▪ PSM-3500 DRS

(Analytical Spectral (Spectral Evolutions)
Devices, Inc.) ▪ Cost: ~$70,000
▪ Cost: ~$60,000
Types of Reflection: Specular vs. Diffuse
Basics of Infrared Spectroscopy

Scissoring
Asymmetrical stretching Symmetrical stretching

Rocking Twisting Wagging

Soil VisNIR Spectra
One can measure the light energy at
various wavelengths = a spectrum

The maxima and minima of spectral

reflectance curves – minima are
caused by absorption, and we call
these absorption features or
absorption bands.

Differences in absorption and

scattering for different wavelengths
can be used to identify the
compounds.
Components of Spectroradiometer
1. Light source
2. A dispersive unit (monochromator)
3. A detector
4. (Fibres)
5. Absorbance / reflectance standard
How Spectroradiometer Works?

Holographic grating

Total Internal Reflection

optic
Control SWIR2 (1800-2500 nm) SWIR1(1000-1800 nm) Vis-NIR (350-1000 nm)
software

Fiber
1.4-2 nm

Separation filter
512 Si photodiode array
Light
InGaAs detector
~600 channels
InGaAs detector
~600 channels
2 nm Source/probe
Reflection Light

Sample
Converts incident photons to
Digital Voltage electrons (photocurrent)

signal
16 bit A/D converter
Working Mechanism
Spectralon Panel
❑ 99% reflectance
❑ Optimize spectrometer
❑ White reference: simplifies reflectance
❑Removes ambient temperature effect
❑ Removes humidity effect
❑ Optimize : probes change

www.asdi.com
Spectral Resolution
Spectral Preprocessing
Raw scan Seamless spectra

Smoothed spectra Average spectra

Spectral Preprocessing

1St derivative 2nd derivative

 Why Derivative Spectra
Derivative: change in reflectance over a bandwidth Δ𝜆 = 𝜆𝑗 − 𝜆𝑖 , where 𝜆𝑗 >𝜆𝑖
𝑑 𝑑(𝑛−1)
nth derivative= 𝑠 (Tsai and Philipot, 1998).
𝑑𝜆 𝑑𝜆(𝑛−1)

❑ Increasing resolution
❑ Background elimination
❑ Noise filtration

http://www.youngin.com/application/AN-0608-0115EN.pdf
 Why Derivative Spectra

http://www.youngin.com/application/AN-0608-0115EN.pdf

❑ High resolution: separate 2 analytes ❑ Suppression of broad bands

with similar 𝜆𝑚𝑎𝑥 values
 Spectral Preprocessing
❑ 1st derivative of reflectance spectra
❑ Fourier Transform
❑ Discrete wavelet transform
❑ Classification : linear discriminant analysis, support vector machines, and
random forest
❑ Multivariate regression/ML:
✓ stepwise multiple linear regression
✓ partial least squares regression
✓ principal component regression
✓ penalized spline
✓ GAM
✓ RF
✓ SVM
✓ ANN
✓ CNN
✓ ACO
✓ …..and many more
Fourier Transform
Image Transforms
• Many times, image processing tasks are best
performed in a domain other than the spatial domain.
• Key steps
(1) Transform the image
(2) Carry the task(s) in the transformed domain.
(3) Apply inverse transform to return to the spatial domain.
Fourier Transform
Fourier Series Theorem
• Any periodic function f(t) can be expressed as a
weighted sum (infinite) of sine and cosine functions of
increasing frequency:

is called the “fundamental frequency”

t t
Fourier Transform
Fourier Series (cont’d)

α1

α2

α3
Fourier Transform
Continuous Fourier Transform (FT)

• Transforms a signal (i.e., function) from the spatial (x)

domain to the frequency (u) domain.

where
 Fourier Transform
❑ SEE VIDEO

https://www.youtube.com/watch?v=spUNpyF58BY
Wavelet Transformation
Motivation

Some signals obviously have spectral

characteristics that vary with time
Criticism of Fourier Spectrum
It’s giving you the spectrum of the
‘whole time-series’

Which is OK if the time-series is stationary

But what if its not?

We need a technique that can “march along” a

timeseries and that is capable of:

Analyzing spectral content in different places

Detecting sharp changes in spectral character
Fourier Analysis is based on an indefinitely long cosine
wave of a specific frequency

time, t
Wavelet Analysis is based on an short duration
wavelet of a specific center frequency

time, t
 Wavelet
A wavelet is a mathematical function
useful in digital signal processing
and image compression

In signal processing, wavelets make

it possible to recover weak signals
from noise .

Addison, 2002

Types of Wavelet
Wavelet Transformation
❑ Wavelet transform of CWT
1 +∞ 𝑡−𝑏
T(a,b)=√𝑎 ‫׬‬−∞ 𝑥(𝑡)ψ∗ ( 𝑎
)dt
❑ Discrete wavelet transform
1 𝑡−𝑛𝑏𝑜𝑎𝑜𝑚
ψm,n (t)= ψ( 𝑎 𝑚 )dt
√𝑎𝑜𝑚 0

❑ When a0=2, b0=1

ψ m,n (t)= 2-m/2 𝜓(2-mt-n): “dyadic grid”
❑ ‫׬‬−∞
+∞
ψ m,n (t)ψ m′,n′ (t)={ 1 if m=m’ & n=n’
{ 0 otherwise
❑ Orthogonality: useful for
complete regeneration of the
original signal without
redundancy

Addison, 2002
Wavelet Transform

Inverse Wavelet Transform

All wavelet derived from mother wavelet

Inverse Wavelet Transform

time-series wavelet with

scale, s and time, t

coefficients
of wavelets

build up a time-series as sum of wavelets of different

scales, s, and positions, t
Wavelet Transform We are going
to ignore the
complex
time-series
conjugate
from now
on, assuming
that we’re
using real
wavelets

coefficient of wavelet
complex conjugate of
with
wavelet with
scale, s and time, t
scale, s and time, t
Wavelet

normalization
shift in time

change in scale:
big s means long
wavelength
wavelet with
scale, s and time, t

Mother wavelet
Shannon Wavelet

Y(t) = 2 sinc(2t) – sinc(t)

mother wavelet

t=5, s=2

time
Fourier spectrum of Shannon Wavelet

frequency, w

Spectrum of higher scale wavelets

Thus determining the wavelet coefficients at a fixed scale, s

can be thought of as a filtering operation

g(s,t) =  f(t) Y[(t-t)/s] dt

= f(t) * Y(-t/s)

where the filter Y(-t/s) is has a band-limited spectrum, so the filtering

operation is a bandpass filter

A band-pass filter, also bandpass filter or BPF, is a device that passes

frequencies within a certain range and rejects (attenuates) frequencies
outside that range.
 Discrete wavelets: choice of scale and sampling in time

sj=2j

Scale changes
and
by factors of 2
tj,k = 2jkDt

Then g(sj,tj,k) = gjk Sampling widens by factor of 2

for each successive scale

where j = 1, 2, …
k = -… -2, -1, 0, 1, 2, …
dyadic grid
 The factor of two scaling means that the spectra
of the wavelets divide up the frequency scale into
octaves (frequency doubling intervals)

w
1/
8wny ¼wny ½wny wny
As we showed previously, the coefficients of Y1 is just the
band-passes filtered time-series, where Y1 is the wavelet, now
viewed as a bandpass filter.
This suggests a recursion. Replace:

w
1/
8wny ¼wny ½wny wny

with
low-pass filter w

½wny wny
And then repeat the processes, recursively …
 Chosing the low-pass filter

It turns out that its easy to pick the low-pass filter, flp(w). It must
match wavelet filter, Y(w). A reasonable requirement is:

|flp(w)|2 + |Y(w)|2 = 1

That is, the spectra of the two filters add up to unity. A pair of such
filters are called Quadature Mirror Filters. They are known to have
filter coefficients that satisfy the relationship:

YN-1-k = (-1)k flpk

Furthermore, it’s known that these filters allows perfect

reconstruction of a time-series by summing its low-pass and high-
pass versions
To implement the ever-widening time sampling

tj,k = 2jkDt
we merely subsample the time-series by a factor of
two after each filtering operation
time-series of length N

HP LP
Recursion for
wavelet
2 2
coefficients
g(s1,t)
HP LP g(s1,t): N/2 coefficients

g(s2,t): N/4 coefficients

2 2
g(s2,t): N/8 coefficients
g(s2,t)
HP LP Total: N coefficients

2 2
g(s3,t) …
 Wavelength ( N) Filter Bank
HP LP g(s11,t): N/2 coefficients
g(s10,t): N/4 coefficients
2 2 g(s9,t): N/8 coefficients
Total: N coefficients
g(s11,t) Wavelet coefficients at scale 7,6,5 were
HP LP extracted
Stepwise Multiple linear regression
2 2

g(s10,t)
HP LP

2 2
g(s9,t) …
Coiflet low pass filter

time, t
Coiflet high-pass filter

time, t

From http://en.wikipedia.org/wiki/Coiflet
Spectrum of low pass filter

frequency, w
Spectrum of wavelet

frequency, w
time-series

stage 1 - hi

stage 1 - lo
Stage 1 lo

stage 2 - hi

stage 2 - lo
Stage 2 lo

stage 3 - hi

stage 3 - lo
Stage 3 lo

stage 4 - hi

stage 4 - lo
Stage 4 lo

stage 5 - hi

stage 6 - lo
Stage 4 lo

stage 5 - hi

stage 6 - lo
Had enough?
 Putting it all together …

|g(sj,t)|2
short
wavelengths
scale

long
wavelengths
time, t
Wavelet Thresolding
Noise corrupted spectra Haar wavelet transform

Thresholding Inverse Haar reconstruction

Data Mining

Spectral
preprocessing
database select make model
selected transformed
data
data

Interpretation&
data prospecting
rule formulation
and surveying

Direct Kernel Methods

www.drugmining.com
Soil Clay

Waiser et al. (2007)

Soil Organic and Inorganic C

SOC SIC

Morgan et al. (2009)

Petroleum Contaminated Soils
0.5

0.4

Reflectance
0.3

0.2 Soil A No hydrocarbon

Soil A hydrocarbon
0.1 Soil B No hydrocarbon
Soil B hydrocarbon
0

500 1000 1500 2000 2500

wavelength, nm

Petroleum Contaminated Soils

Selection of Significant Wavelengths

Petroleum Contaminated Soils

PCA Screeplot

Separation of Contaminated and Non-contaminated Soils

PSR model
Petroleum Contaminated Soils
BP Deepwater Horizon Oil Spill
Calibration Models
On-site Prediction
Mapping Soil Properties via DRS

Variable Indicator Approach

Output Maps
Indian Applications of DRS: Soil As

Dhapa Landfill, Kolkata

Soil As Map
Soil As Pools by DRS

Significant Wavelengths

Modeling Soil As Pools

Removing the Impact of Soil Moisture
Portable X-ray Fluorescence • Elemental data (PXRF) is coupled
Spectrometry DRS
with spectral data (VisNIR) to
create predictive algorithms of soil
physical and chemical properties
• Allows for on-site analysis in
minutes
• Spatial variability applications for
variable rate agriculture (precision
farming) and environmental
assessment/cleanup

Provisional patent awarded – Spring, 2014

Indigenous Sensor Developed by my Students

Spectrum
Portable VisNIR Camera

Upgraded version
 Soil OC box + Smartphone

Soil Organic Carbon Prediction CNN for Image Processing

2.5
y = 0.52x + 0.32
2 R² = 0.84 The images acquired were of dimension
Measured OC %

1.5 4096 x 2304 (w x h). These were cropped

1 in the same manner to 2950 x 2304 (w x h)
0.5
for using in the Neural Network.
0
0 1 2 3 4 5
Predicted OC %
NixPro color sensor
• Nix Pro™ an
inexpensive color
sensor
• Features:
o Rechargeable and portable
o Has its own light source- 2x
High-CRI LEDs
o Connects to smartphones via
Bluetooth
o Has free android and iOS app
o Costs around $350 US
o 2.4 x 1.7 inches (6 x 4.2
centimeters)
o 1.5 ounces (43 grams)

NixPro
USB microscope image+ deep learning for soil OC classification

USB Microscope for Soil Image Processing by CNN

Before histogram equalization After histogram equalization

Portable Soil Moisture Sensor
▪ The prototype consists of a
transmitter(speaker) and a
receiver(condenser microphone) placed
at a known distance and the phase
difference between the transmitted and
received signals is accounted for the
time travel of the sound wave.
▪ The velocity, is defined as,
Acoustic soil moisture sensor

▪ We use phase locked loop(PLL)

technique to calculate the phase
difference between the two signals.It
uses the continuous wave methods to
determine the sound velocity.
Portable Soil Moisture Sensor

Acoustic soil moisture sensor

 Mobile Image Based Soil Recognition

Variables (axes F1 and F2: 99.22 %)

1
Smartphone for Soil Analysis
3 0.75 R value
0.5 G value
2
F2 (23.50 %)

0.25
B value
1 0
-0.25
PC2

0
-4 -2 0 2 4 -0.5

-1 -0.75
-1
-2 -1 -0.75 -0.5 -0.25 0 0.25 0.5 0.75 1
F1 (75.72 %)
-3 Active variables
PC1
 Conclusions

• Advanced sensors are the future of soil analysis

• Both PXRF, DRS and new-age sensors are capable of
rapid and cost-effective identification of soil clay and other
properties
• The premise is sound-we need more research
THANKS