Profiling Poverty

using Machine Learning

Task Link: https://datamasters.ru/en/task?id=480

Sarvar Abdullaev
 Motivation
• Nearly 800 million people in the world live below $1.90 per day
• Almost 32% of the population of developing countries live below
$3.10 per day
• Half of today's extreme poor are children who:
– mostly live in rural areas prone to natural disasters
– 65% work in agriculture
– almost 40% have no formal education
 Motivation
• The World Bank is aiming to end extreme poverty
by 2030 by adopting following agenda:
– Promote inclusive and sustainable growth
– Invest in people
– Promote resilience
• We will review following technical issues
– How to measure poverty?
– How to understand poverty?
– How to predict poverty?
 How to measure poverty?
• Three steps need to be taken in measuring poverty (see Ravallion, 1998):
1. defining an indicator of welfare
2. establishing a minimum acceptable standard of that indicator to separate
the poor and the non-poor (often known as a poverty line)
3. generating a summary statistic to aggregate the information from the
distribution of this welfare indicator and its position relative to minimum
acceptable standards
• Try to find answers to following questions too?
– Why is it important to measure poverty?
– What factors determine poverty?
– What data should be collected?
– How can we estimate missing data from other indicators?
– How shall we compute the level of poverty in a most appropriate way?
 Methods of Measuring Poverty
• Individual income? Below $1.9 per day?
• Household income? Below $3.1 per day?
• Global Multidimensional Poverty Index (MPI)

Source: https://ophi.org.uk/multidimensional-poverty-index/
 Methods of Measuring Poverty
• Headcount index
– percentage of poor population
• Poverty gap index
– how far from poverty line the poor population is
• Squared poverty gap index
• Sen index
• Sen-Shorrocks-Thon index
• Time taken to exit
 How to understand poverty?
• Poverty profiling examines the patterns of poverty, to
see how it varies by geography, by community
characteristics (e.g. access to school, healthcare, etc),
and by household characteristics (e.g. by education of
household head, by size of household).
• Answering following questions are critical to construct
a poverty profile:
– What are the underlying factors that result in poverty?
– What are the common or the contrasting features of
vulnerable groups?
– What other unknown patterns and structures can be
revealed from available surveys?
 How to understand poverty?
• More questions:
– Does poverty vary widely between different areas in the country?
– Are the most populated areas also the areas where most of the poor live?
– How is income poverty correlated with gender, age, urban and rural, racial, or
ethnic
characteristics?
– What are the main sources of income for the poor?
– What products or services—tradables and non-tradables—do the poor sell?
– To what extent are the rural poor engaged in agriculture? In off-farm
employment?
– How large a factor is unemployment? Underemployment?
– Which are the important goods in the consumption basket of the poor? How
high is the share
of tradables and non-tradables
– How is income poverty linked with malnutrition or educational outcomes?
 How to understand poverty?
• More questions:
– What are fertility characteristics of the poor?
– To what public services do the poor have access? What is the quality
of the service?
– How important are private costs of education and health for the poor?
– Can the poor access formal or informal credit markets?
– What assets—land, housing, and financial—do the poor own? Do
property rights over such
assets exist?
– How secure is their access to, and tenure over, natural resources?
– Is environmental degradation linked to poverty?
– How variable are the incomes of the poor? What risks do they face?
– Are certain population groups in society at a higher risk of being poor
than others are?
 Methods of Understanding Poverty:
Unsupervised Learning Algorithms
• K-means clustering is a method
for finding clusters and their
centers in a set of unlabelled
data.
– Choose randomly R cluster centers
– Iterate following two steps to
minimize cluster variance:
• identify subset of data points that is
closer to a given center than any
other center
• compute mean of data points within
each cluster and set them as new
centers of the cluster
– Stop when change in cluster
centers is negligible
Unsupervised Learning Algorithms
• Learning Vector Quantization (LVQ) - cluster
boundaries change with every new input which is
attracted to a correct prototype and repelled
from other prototypes

==>
Unsupervised Learning Algorithms
Unsupervised Learning Algorithms
• Association Rules can determine interesting
relations in large datasets
For example:
{butter, bread} => {milk}
How association rules are
selected? If their support and
confidence is above given
threshold.

Support Example:
supp({beer, diapers})=0.2

Confidence Example:
conf({butter,bread}=>{milk})=
0.2/0.2=1.0
 How to predict poverty?
• What are the key predictors of poverty?
• Can we build a classification model for
predicting if a given household is poor or not?
• Can we build a regression model for predicting
key indicators of poverty such as household
income or MPI?
• What other supervised learning methods can
be applied for predicting poverty?
 Methods of Predicting Poverty:
Binary Classification
• Household survey data of Malawi
published by World Bank contains:
– 347 features such as answers to
questions:
• What is your main source of cooking
fuel?
• Do you have electricity?
• How many cellphones the household
own?
• Rice consumption
• Bread consumption
• More here:
– https://microdata.worldbank.org/inde
x.php/catalog/3016/data-
dictionary/F3?file_name=MWI_2010_
household
– 12271 data points 𝑓 𝑥 = 𝛽0 + 𝛽1 𝑥1 + ⋯ + 𝛽𝑛 𝑥𝑛
– Every data point is labelled: "Poor" or
"Not Poor" 𝐿𝑜𝑠𝑠 𝑓, 𝑦 = −(𝑦 log 𝑓 + 1 − 𝑦 log(1 − 𝑓))
• How to use these features to 𝐶𝑜𝑠𝑡 𝑓, 𝑦 = 𝐴𝑣𝑔(𝐿𝑜𝑠𝑠 𝑓, 𝑦 )
building a prediction model?
 Logistic Regression Classifier
1
• 𝑙𝑜𝑔𝑖𝑠𝑡𝑖𝑐 𝑓 𝑖 =
1+exp(−𝑓 𝑖 )
• 𝑓 𝑖 = 𝛽0 + 𝛽1 𝑥1,𝑖 + ⋯ + 𝛽𝑚 𝑥𝑚,𝑖
• where regression coefficients
𝛽0 , 𝛽1 , … , 𝛽𝑚 should be trained
using gradient descent algorithm.
• 𝛽𝑗,𝑡+1 = 𝛽𝑗,𝑡 − 𝛾𝑓 ′ 𝑖
• Evaluation metrics:
– Accuracy = [TP + TN] / Total Population
– Precision = TP / [TP + FP]
– Recall = TP / [TP + FN]
– ROC curve relationship between TP and
FP - the higher the area under the curve
the better the model
 Other Classifiers
• Linear Discriminant Analysis (LDA)
• Naïve Bayes
• Support Vector Machines
• Decision Trees
• Random Forests
• XGBoost
• Multilayer perceptron
• Deep learning
• See for above examples:
– https://github.com/worldbank/ML-classification-
algorithms-poverty
 DrivenData's Pover-T Test Challenge
• In February 2018, DrivenData.org conducted a ML challenge on predicting poverty.
• World Bank provided detailed data on household consumption starting from food,
transportation, access to healthcare and even to sporting events.
• Task:
– Find the strongest predictors of poverty.
– Construct a classifier model which uses these predictors to accurately classify households.
• Benchmark used to evaluate solutions is Logistic Regression
• Winners:
– 1st Place (Ag100): Ensemble of gradient boosting (LightGBM) and neural network.
– 2nd Place (sagol): Tree-based algorithms along with CatBoost, XGBoost and LightGBM
– 3rd Place (LastRocky): Ensemble of gradient boosting, decision trees and neural network
• More about contest and proposed solutions:
– https://github.com/drivendataorg/pover-t-tests
 References
• Sample Projects:
– https://github.com/worldbank/ML-classification-algorithms-poverty
– https://github.com/drivendataorg/competition-winners
– https://github.com/worldbank/LearningPoverty
– https://github.com/drivendataorg/pover-t-tests
• Open Datasets:
– https://microdata.worldbank.org/
– https://dhsprogram.com/data/available-datasets.cfm
• Useful Tools and Dashboards:
– http://iresearch.worldbank.org/PovcalNet/home.aspx
– http://povertydata.worldbank.org/poverty/home
• Reports and White Papers:
– http://shorturl.at/kGMN8
– http://shorturl.at/agloR
– http://shorturl.at/dtPQ5
• Books:
– Introduction to poverty analysis. Washington, D.C. : World Bank Group.
http://documents.worldbank.org/curated/en/775871468331250546/Introduction-to-poverty-analysis
– Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1, No. 10). New York: Springer
series in statistics.
– James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New
York: springer.