Data Exploration with

Python
Andrew Michelson, MD
Pulmonary/Critical Care
Institute for Informatics
Washington University School of Medicine in St. Louis

February 17, 2020

Institute for Informatics (I 2)

Disclosures
No relevant financial disclosures.

Many topics could be their own courses, so this will be a brief overview

The best techniques to analyze and clean your data will depend on the question your
asking and data you have

Institute for Informatics (I 2)

Class Structure

Institute for Informatics (I 2)

Objectives
1. Learn how to import data into Python

2. Discuss variable identification

3. Explore missing data and discuss its management

4. Explore univariate & bivariate analyses

5. Discuss outlier assessment and management

6. Explore data transformation

Institute for Informatics (I 2)

The Data
Source: MIMIC-III Demo Data

Contents:
• Vital Signs: Blood pressure, heart rate, respiratory rate, etc…

• Laboratory Values: White Blood Cell Count, Potassium, etc…

• And more, but we won’t use any of that today

Institute for Informatics (I 2)

The Working Environment
1. Python

2. jupyter-notebook

3. Import libraries
A. Pandas
B. Numpy
C. Seaborn
D. Datetime
E. Matplotlib
F. Scipy.stats

Institute for Informatics (I 2)

Importing Data Into Python
1. Python is a versatile and powerful language that can accept data from
many formats

2. In this class we import CSV documents from the MIMIC-III demo data

3. Use: dfNAME = pd.read_csv(filepath/filename, sep = ’,’)

Institute for Informatics (I 2)

Importing Data Into Python

Jupyer-Notebook
• Open Jupyter-Notebook
• Run Section 2: Import Libraries for DataSet Exploration
• Fill in the blank to import the following files:
• ICUSTAYS.csv
• PATIENTS.csv
• D_ITEMS.csv
• D_LABITEMS.csv

Institute for Informatics (I 2)

Variable Identification
Variable Name: Variable name

Variable type:
• Continuous (ex, age)
• Categorical (ex, sex)

Data Type:
• String
• Category
• Integer
• Float
• ManyString

Independent vs Dependent:

Institute for Informatics (I 2)

 Variable Identification
Identify your variables:

>> DataFrame.head( )

Patients dataframe

Note: you can use >> DataFrame.tail( ) to view the tail rows of the data frame

By adding in a number within the parenthesis you can specify how many rows to view

Institute for Informatics (I 2)

Variable Identification
View your data frame

ICU Stays

Institute for Informatics (I 2)

Variable Identification
How do we know how many rows and columns we have in total?

>> DataFrame.shape

How do we know the type of the data type?

>> DataFrame.info()

Institute for Informatics (I 2)

Variable Identification
Remove Extraneous Information that takes up space (visible and memory)

>> DataFrame.drop(items, axis, inplace)

Institute for Informatics (I 2)

Variable identification in Python

Go to section 3.0.1 and fill in the *** to start identifying your

variables

Complete until section 3.2: Merge Patients & ICU Data to

Create a single DataFrame

Institute for Informatics (I 2)

Manipulating Data in Python
Often data is collected from different sources and then
merged together for analysis.

>> DataFrame1.merge(DataFrame2, how = “left/right”,

on=[‘’])

After a merge, double check the shape, to make sure you

merged correctly

Institute for Informatics (I 2)

Variable identification in Python

Go to section 3.2: Merge Patients & ICU Data to create a single

DataFrame

Check the size of the new DataFrame to confirm a successful

merge

Institute for Informatics (I 2)

Missing Data
Very Common in clinical data

Why is data missing?

• Data extraction
• Data collection

Institute for Informatics (I 2)

Missing Data Categorization
1. Missing completely at random:
• The propensity for a data point to be missing is completely
random and not dependent on observed or unobserved data

2. Missing at random:
• Systematic differences between the missing and observed values,
but these can be entirely explained by other observed variables

Institute for Informatics (I 2)

Missing Data Categorization
3. Missing not at random
• There is a relationship between the propensity of a value to be
missing and it’s values

Institute for Informatics (I 2)

Missing Data Treatment

Adapted from: https://medium.com/ibm-data-science-experience/missing-data-conundrum-exploration-and-imputation-techniques-9f40abe0fd87

Institute for Informatics (I 2)

Missing Data: Case Deletion

List Wise Pair Wise

Delete all data Analyze all cases

where any where data is
missing available
value is present

Institute for Informatics (I 2)

Missing Data: Imputation
Goal is to fill missing data with estimated values

Most common methods: mean/median/mode:

• Population-wide
• Cohort-wide

Institute for Informatics (I 2)

Missing Data: Statistical-Model Imputation
Linear Regression
• Limitations:
• Reduces variability
• Overestimates the model fit and correlation coefficient

K-nearest Neighbor Imputation

• Limitations:
• The choice of k critical in getting desired results
• Very slow

Institute for Informatics (I 2)

Missing Data: Statistical-Model Imputation
Multiple Imputation by Chained Equations (MICE)
• Assumes data is missing at random
• Runs multiple regression models
• Each value is modeled conditionally
• Multiple data sets are made (usually at least 10)

Institute for Informatics (I 2)

Assessing Missing data in Python
Look for null entries
>>DataFrame.isnull( ).sum

Look for non-null entries

>>DataFrame.notnull( ).sum

Institute for Informatics (I 2)

Assessing Missing Data

Go to section 3.3: Assess Missing Data in NEW Patients

DataFrame and complete UP TO, but not including Import Vital
Signs

Institute for Informatics (I 2)

Data Mapping
Process of extracting and unifying data for further analysis

Measurements of interest could be mixed with measurements

not of interest

The same value can have different names

• Sometimes the differences in names is important, other
times its not

Occurs in many data sets, including MIMIC-III

Institute for Informatics (I 2)

Data Mapping
Vital Signs:
• Blood Pressure (systolic/diastolic)
• Heart Rate
• Respiratory Rate
• Oxygen saturation (%)
• Temperature

In MIMIC-III vital signs are mixed with other measurements in

the CHARTEVENTS.CSV

Institute for Informatics (I 2)

Data Mapping with Vital Signs
Systolic Blood Pressure Synonyms in THIS dataset:
• Non Invasive Blood Pressure systolic',
• 'Arterial Blood Pressure systolic',
• 'Manual Blood Pressure Systolic Left',
• 'Manual Blood Pressure Systolic Right’,

Institute for Informatics (I 2)

Data Mapping with Vital Signs
Count variable frequency
>> DataFrame.series.value_counts( )

Institute for Informatics (I 2)

Data Mapping with Dictionaries

Dictionaries are data structures

that consist of an unordered
collections of key-value pairs
that can be changed

Dictionary = {
<key>: <value>
}

Institute for Informatics (I 2)

Data Mapping with Vital Signs
To accommodate synonyms, or extract items of interest from a
larger data set, you can use a dictionary

Institute for Informatics (I 2)

Import the remaining data and assess
missingness

Go to section 4.2 Import Vital Signs complete up to section 5:

Univariate & Bivariate Analysis

Institute for Informatics (I 2)

Univariate Analysis
Explore variables individually

Basic descriptive analysis

Central Tendency Measure Dispersion Visualization

Mean Interquartile Range Histogram
Median Standard Deviation/ Box plot
Variance
Mode Skewness
Min Kurtosis
Max

Institute for Informatics (I 2)

Univariate Analysis: Skewness
Measure of the asymmetry of the probability distribution of a variable
• Positive or Right
• Negative or Left

Grading Skewness Severity

• Minimal: -0.5 and 0.5
• Moderate: -1 and -0.5 or 0.5 and 1
• Severe: < -1 or >1
https://en.wikipedia.org/wiki/Skewness

Institute for Informatics (I 2)

Univariate Analysis: Kurtosis
“The kurtosis parameter is a measure of the combined weight of the tails relative to the rest
of the distribution.”

Kurtosis >3: Positive

No Kurtosis/Normal

Kurtosis <3: Negative

https://www.spcforexcel.com/knowledge/basic-statistics/are-skewness-and-kurtosis-useful-statistics#kurtosis
https://bishalbanksonfinance.wordpress.com/tag/probabality-distribution/

Institute for Informatics (I 2)

Bivariate Analysis
A method to determine the relationship between 2 variables

1. Visualization: Scatter plots

2. Regression analysis: Find the equation for the line or curve that best fits the data

3. Correlation coefficients: A measure of association between two data points

Institute for Informatics (I 2)

Outliers
What is an outlier?

• A data point that appears far away and diverges from the overall pattern in a sample
• Can be univariate or bivariate

Institute for Informatics (I 2)

Outliers
How do outliers occur?
• Natural
• Sampling error
• Data entry error
• Data processing error
• Measurement error
• Intentional outlier
• Experimental error

Institute for Informatics (I 2)

Outliers
Why are they important?

• Alters population variance, leading to non-normal data distributions

• Alters performance of downstream analyses
• Biases results

How do you detect outliers?

• Visualization
• Bar charts
• Box plots
• Scatter plots (looking for bivariate outliers)
• There are many, many ways, but we will focus on visualization today!

Institute for Informatics (I 2)

Outliers: Univariate

Institute for Informatics (I 2)

Outliers: Univariate

Institute for Informatics (I 2)

Outliers: Bivariate

Institute for Informatics (I 2)

Outliers
How do you treat outliers? (Subject for an entire course!)

• Delete observations:
• Data entry error
• Data processing error
• Very few (subjective)

• Transform values
• Log conversion
• Binning
• Differential observation weights

• Impute
• Would avoid with natural outliers

• Treat outliers as a separate category

Institute for Informatics (I 2)

Assessing Data in Python: Pivot Tables
DataFrames must be properly structured before they can be plotted

Patient Label Value

John Smith Heart Rate 75
John Smith Respiratory Rate 15

Patient Heart Rate Respiratory Rate

John Smith 75 15

DataFrame.pivot_table(values = 'value', index = [‘columns’], columns='label')

Institute for Informatics (I 2)

Visualize Data Within Python
Declare the graph properties
>> fig, ax = plt.subplots(rows,columns, figsize = (width,height))

Locate a subset of data from within the larger dataframe

>> DataFrame.loc[DataFrame.column == ‘columnname’, ‘return column name']

Use Seaborn to make distribution and boxplots

>> sns.distplot(data, ax=ax[ X ])

>> sns.boxplot(x = data, ax = ax[ X ])

Pivot your dfce

>>DataFrame.pivot_table(values = 'value', index = [‘columns’],
columns='label').reset_index()

Use Seaborn to plot bivariate data

>>sns.pairplot(pivoted table)

Institute for Informatics (I 2)

Visualize Data Within Python
Seaborn can make a heatmap to help you more rapidly identify correlations
>> sns.heatmap(dflabs.corr(), vmax = 1)

Institute for Informatics (I 2)

Univariate & Bivariate Visualization with
Vital Signs

Go to section 5: Univariate & Bivariate Analysis and complete

until section 6: Data Transformation

Institute for Informatics (I 2)

Data Transformation
Skewed data
• Skewed data can violate model assumptions (logistic regression)
• Amplify a class imbalance, degrading model performance towards the tail of the
distribution

Heteroskedasticity
• The relationship between two variables shows increasing scatter (non-constant standard
error) at extremes of measurement of the dependent variable
• Two forms:
• Conditional: Unpredictable volatility
• Unconditional: Predictable volatility

Institute for Informatics (I 2)

Data Transformation: Heteroskedasticity
Conditional

Institute for Informatics (I 2)

Data Transformation: Heteroskedasticity
Unconditional

Institute for Informatics (I 2)

Data Transformation
Way to improve skewness and heteroskedasticity is to normalize your data
• Remove/manage outliers
• Log
• Cube Root
• Binning
• Normalization
• Sigmoid
• Hyperbolic tangent
• Etc…

Again, there are many different ways to do this and the best way will depend on your
planned analyses and the question you are answering

Institute for Informatics (I 2)

Data Transformation
To perform the log function on data, you take a Pandas Series as such:
>> DataFrame.Column = np.log(DataFrame.column)

To raise a value to the cube root

>> DataFrame.Column = DataFrame.column**(1/3)

Institute for Informatics (I 2)

Data Transformation

Go to section 6: Data Transformation and go until the end!

Institute for Informatics (I 2)

Questions?
Thank you!

Institute for Informatics (I 2)

References:
1. Grus, Joel. Data Science from Scratch. O’Reilly Media;2015.
2. Marcellino, P. Comprehensive data exploration with python.
https://www.kaggle.com/pmarcelino/comprehensive-data-exploration-with-python. 2/2018. Accessed:
2/12/2020.
3. Sheridan, E. Un-bottling the data. 12/2/2019.
https://towardsdatascience.com/un-bottling-the-data-2da3187fb186. Accessed: 2/12/2020.
4. Ojeda, T. Data exploration with python, part 3.
https://www.districtdatalabs.com/data-exploration-with-python-3. Accessed: 2/12/20.
5. Sunil, R. A comprehensive guide to data exploration.
https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/#two. Accessed: 2/12/2020.
6. Bratkovics, C. Exploratory data analysis tutorial in Python.
https://towardsdatascience.com/exploratory-data-analysis-tutorial-in-python-15602b417445. 6/16/19.
Accessed: 2/12/20.
7. Sunil, R. Ultiamte guide for data exomploration in Python using Numpy, Matplotlib and Pandas.
https://www.analyticsvidhya.com/blog/2015/04/comprehensive-guide-data-exploration-sas-using-python-nump
y-scipy-matplotlib-pandas/
. 4/9/2015. Accessed: 2/12/2020.
8. Akinfaderin, W. Missing data conundrum: exploration and imputation techniques.
https://medium.com/ibm-data-science-experience/missing-data-conundrum-exploration-and-imputation-techni
ques-9f40abe0fd87
. 9/11/2017. Accessed: 2/12/20.
9. Wade, C. Transforming skewed data. https://towardsdatascience.com/transforming-skewed-data-73da4c2d0d16.
8/21/2019. Accessed: 2/20/20.
10. Chow, J. Log transformation base for data linearization does not matter.
https://towardsdatascience.com/log-transformation-base-for-data-linearization-does-not-matter-22eb3c1463d0.
6/27/2019. Accessed: 2/12/20. Institute for Informatics (I 2)
11. Azur MJ, Stuart EA, Franggakis C, Leaf PJ. Multiple imputation by chained equations: what is it and how does it
Thank you!

Institute for Informatics (I 2)

Institute for Informatics (I 2)