Module-10 - Python Basic-2

LOGO
UNIV/POLTEK
Python Basics
Function, Data Structures, Data Manipulation
digitalent.kominfo.go.id
LOGO
UNIV/POLTEK
Functional Programming tools
• Python allows for the use of a special kind of function, a lambda function.
• Lambda functions are small, anonymous functions based on the lambda abstractions that
appear in many functional languages.
• Python can support many different programming paradigms including functional
programming.
Right now, we’ll take a look at some of the handy functional tools provided by Python.
LOGO
UNIV/POLTEK
Lambda functions
• Lambda functions within Python.

• Use the keyword lambda instead of def. >>> def f(x):
• Can be used wherever function objects are used. ... return x**2
• Restricted to one expression. ...
• Typically used with functional programming tools. >>> f(8)
64
>>> g = lambda x: x**2
>>> g(8)
64
LOGO
UNIV/POLTEK
Functional programming tools

def even(x):
• Filter if x % 2 == 0:
• • filter(function, iterable). Filters method return True
constructs an iterator from elements of an else:
iterable for which a function returns true. return False
iterable which is to be filtered, could be
sets, lists, tuples, or containers of any list(filter(even, range(0,30)))
iterators
• • Returns an iterator that passed the [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28]
function check for each element in the
iterable.
LOGO
UNIV/POLTEK
• Map
def square(x):
• • map(function, sequence) applies
function to each item in sequence and return x**2
returns the results as a list.
list(map(square, range(0,11)))
• • Multiple arguments can be provided if
the function supports it.
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
LOGO
UNIV/POLTEK
• Map def expo(x, y):

• • map(function, sequence) return x**y
applies function to each item in
sequence and returns the results list(map(expo, range(0,5), range(0,5)))
as a list.
• • Multiple arguments can be
provided if the function supports [1, 1, 4, 27, 256]
it.
LOGO
UNIV/POLTEK
• Reduce
• • reduce(function, sequence) returns a import functools
single value computed as the result of
performing function on the first two def fact(x, y):
items, then on the result with the next return x*y
item, etc.
• • There’s an optional third argument print(functools.reduce(fact,
which is the starting value. range(1,5)))
24
LOGO
UNIV/POLTEK

• We can combine lambda abstractions with functional programming tools. This is especially
useful when our function is small – we can avoid the overhead of creating a function
definition for it by essentially defining it in-line.
>>> list(map(lambda x: x**2, range(0,11)))

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
LOGO
UNIV/POLTEK
More Data Structures

• Lists
• Slicing
• Stacks and Queues
• Tuples
• Sets and Frozensets
• Dictionaries
• How to choose a data structure.
• Collections
• Deques and OrderedDicts
LOGO
UNIV/POLTEK
When to use Lists

• When you need a non-homogeneous collection of elements.
• When you need the ability to order your elements.
• When you need the ability to modify or add to the collection.
• When you don't require elements to be indexed by a custom value.
• When you need a stack or a queue.
• When your elements are not necessarily unique.
LOGO
UNIV/POLTEK
Creating lists
• To create a list in Python, we can use bracket notation to either create an empty list or an
initialized list.
mylist1 = [] # Creates an empty list
mylist2 = [expression1, expression2, ...]
mylist3 = [expression for variable in sequence]
Example:
mylist2 = [0,1,4,9,16]
mylist3 = [i for i*i in range(5)]
• The first two are referred to as list displays, where the last example is a list comprehension.
LOGO
UNIV/POLTEK
Creating lists
• We can also use the built-in list constructor to create a new list.
mylist1 = list()
mylist2 = list(sequence)
mylist3 = list(expression for variable in sequence)
• The sequence argument in the second example can be any kind of sequence object or
iterable. If another list is passed in, this will create a copy of the argument list.
LOGO
UNIV/POLTEK
Creating lists
• Note that you cannot create a new different list without initialization.
# mylist1 and mylist2 point to the same list

mylist1 = mylist2 = []
# mylist3 and mylist4 point to the same list

mylist3 = []
mylist4 = mylist3
mylist5 = []; mylist6 = [] # different lists
LOGO
UNIV/POLTEK
Accessing list elements

• If the index of the desired element is known, you can simply use bracket notation to index
into the list.
>>> mylist = [34,67,45,29]
>>> mylist[2] 45
45
• If the index is not known, use the index() method to find the first index of an item. An
exception will be raised if the item cannot be found.
>>> mylist = [34,67,45,29]
>>> mylist.index(67) 1
1
LOGO
UNIV/POLTEK
Slicing and sliding

• The length of the list is accessible through len(mylist).
• Slicing is an extended version of the indexing operator and can be used to grab sublists.
mylist[start:end] # items start to end-1

mylist[start:] # items start to end of the array
mylist[:end] # items from beginning to end-1
mylist[:] # a copy of the whole array
• You may also provide a step argument with any of the slicing constructions above.
mylist[start:end:step] # start to end-1, by step
LOGO
UNIV/POLTEK
Slicing and sliding

• The start or end arguments may be negative numbers, indicating a count from the end of the
array rather than the beginning. This applies to the indexing operator.
mylist[-1] # last item in the array

mylist[-2:] # last two items in the array
mylist[:-2] # everything except the last two items
• Some examples:
mylist = [34, 56, 29, 73, 19, 62]
mylist[-2] # yields 19
mylist[-4::2] # yields [29, 19]
LOGO
UNIV/POLTEK
Inserting/removing elements
• To add an element to an existing list, use the append() method.
>>> mylist = [34, 56, 29, 73, 19, 62]

>>> mylist.append(47)
>>> mylist
[34, 56, 29, 73, 19, 62, 47]
• Use the extend() method to add all of the items from another list.
>>> mylist = [34, 56, 29, 73, 19, 62]
>>> mylist.extend([47,81])
>>> mylist
[34, 56, 29, 73, 19, 62, 47, 81]
LOGO
UNIV/POLTEK
Inserting/removing elements
• Use the insert(pos, item) method to insert an item at the given position. You may also use
negative indexing to indicate the position.
>>> mylist = [34, 56, 29, 73, 19, 62]
>>> mylist.insert(2,47)
>>> mylist
[34, 56, 47, 29, 73, 19, 62]
• Use the remove() method to remove the first occurrence of a given item. An exception will be
raised if there is no matching item in the list.
>>> mylist = [34, 56, 29, 73, 19, 62]
>>> mylist.remove(29)
>>> mylist
[34, 56, 73, 19, 62]
LOGO
UNIV/POLTEK
Lists as stacks
• You can use lists as a quick stack data structure.
• The append() and pop() methods implement a Last In, First Out (LIFO) structure.
• The pop(index) method will remove and return the item at the specified index. If no
index is specified, the last item is popped from the list.
>>> stack = [34, 56, 29, 73, 19, 62]
>>> stack.append(47)
>>> stack
[34, 56, 29, 73, 19, 62, 47]
>>> stack.pop()
47
>>> stack
[34, 56, 29, 73, 19, 62]
LOGO
UNIV/POLTEK
Lists as queues
• Lists can be used as queues natively >>> from collections import deque
since insert() and pop() both support >>> queue = deque([35, 19, 67])
indexing. However, while appending >>> queue.append(42)
and popping from a list are fast,
inserting and popping from the >>> queue.append(23)
beginning of the list are slow >>> queue.popleft()
(especially with large lists. Why is 35
this?). >>> queue.popleft()
• Use the special deque object from 19
the collections module.
>>> queue
deque([67, 42, 23])
LOGO
UNIV/POLTEK
Other operations
• The count(x) method will give you the number of occurrences of item x within the list.
>>> mylist = ['a', 'b', 'c', 'd', 'a', 'f', 'c']

>>> mylist.count('a')
2
• The sort() and reverse() methods sort and reverse >>> mylist = [5, 2, 3, 4, 1]
the list in place. The sorted(mylist) and
reversed(mylist) built-in functions will return a >>> mylist.sort()
sorted and reversed copy of the list, respectively. >>> mylist
[1, 2, 3, 4, 5]
>>> mylist.reverse()
>>> mylist
[5, 4, 3, 2, 1]
LOGO
UNIV/POLTEK
Custom sorting
we can sort the list as follow (default = ascending)
>>> mylist = ['b', 'A', 'D', 'c']
>>> mylist.sort(key = str.lower)
>>> mylist
['A', 'b', 'c', 'D']
For descending
>>> mylist = ['b', 'A', 'D', 'c']
>>> mylist.sort(key = str.lower, reverse = True)
>>> mylist
[‘D', ‘c', ‘b', ‘A']
str.lower() is a built-in string method.
LOGO
UNIV/POLTEK
When to use sets

• When the elements must be unique.
• When you need to be able to modify or add to the collection.
• When you need support for mathematical set operations.
• When you don't need to store nested lists, sets, or dictionaries as elements.
LOGO
UNIV/POLTEK
Creating sets
• Create an empty set with the set constructor.
myset = set()
myset2 = set([]) # both are empty sets
• Create an initialized set with the set constructor or the { } notation. Do not use empty curly
braces to create an empty set – you’ll get an empty dictionary instead.
myset = set(sequence)
myset2 = {expression for variable in sequence}
LOGO
UNIV/POLTEK
Hashable items
• The way a set detects non-unique elements is by indexing the data in memory, creating a
hash for each element. This means that all elements in a set must be hashable.
• All of Python’s immutable built-in objects are hashable, while no mutable containers (such as
lists or dictionaries) are. Objects which are instances of user-defined classes are also hashable
by default.
LOGO
UNIV/POLTEK
Mutable operations
>>> myset = {x for x in 'abracadabra'}
• The following operations are not >>> myset
available for frozensets. set(['a', 'b', 'r', 'c', 'd'])
• The add(x) method will add element x >>> myset.add('y')
to the set if it’s not already there. The >>> myset
remove(x) and discard(x) methods will set(['a', 'b', 'r', 'c', 'd', 'y'])
remove x from the set. >>> myset.remove('a')
• The pop() method will remove and >>> myset
return an arbitrary element from the set(['b', 'r', 'c', 'd', 'y'])
set. Raises an error if the set is empty. >>> myset.pop()
• The clear() method removes all 'b'
elements from the set. >>> myset
set(['r', 'c', 'd', 'y'])
LOGO
UNIV/POLTEK
Mutable operations continued
set |= other | ...

Update the set, adding elements from all others.
set &= other & ...

Update the set, keeping only elements found in it and all others.
set -= other | ...

Update the set, removing elements found in others.
set ^= other
Update the set, keeping only elements found in either set, but not in both.
LOGO
UNIV/POLTEK
Mutable operations continued

>>> s1 = set('abracadabra')
>>> s2 = set('alacazam')
>>> s1
set(['a', 'b', 'r', 'c', 'd'])
>>> s2
set(['a', 'l', 'c', 'z', 'm'])
>>> s1 |= s2
>>> s1
set(['a', 'b', 'r', 'c', 'd', 'l', 'z', 'm'])
>>> s1 &= s2
>>> s1
set(['a', 'c'])
LOGO
UNIV/POLTEK
Set operations
• The following operations are available for both set and frozenset types.
• Comparison operators >=, <= test whether a set is a superset or subset, respectively, of some
other set. The > and < operators check for proper supersets/subsets.
>>> s2 = set('bard')
>>> s1 >= s2
True
>>> s1 > s2
True
>>> s1 <= s2
False
LOGO
UNIV/POLTEK
Set operations
• Union: set | other | …
• Return a new set with elements from the set and all others.
• Intersection: set & other & …
• Return a new set with elements common to the set and all others.
• Difference: set – other – …
• Return a new set with elements in the set that are not in the others.
• Symmetric Difference: set ^ other
• Return a new set with elements in either the set or other but not both.
LOGO
UNIV/POLTEK
Set operations
>>> s1
set(['a', 'b', 'r', 'c', 'd'])
>>> s2 = set('alacazam')
>>> s2
set(['a', 'l', 'c', 'z', 'm'])
>>> s1 | s2
set(['a', 'b', 'r', 'c', 'd', 'l', 'z', 'm'])
>>> s1 & s2
set(['a', 'c'])
>>> s1 - s2
set(['b', 'r', 'd'])
>>> s1 ^ s2
set(['b', 'r', 'd', 'l', 'z', 'm'])
LOGO
UNIV/POLTEK
Other operations
• s.copy() returns a shallow copy of the set s.
• s.isdisjoint(other) returns True if set s has no elements in common with set other.
• s.issubset(other) returns True if set s is a subset of set other.
• len, in, and not in are also supported.
LOGO
UNIV/POLTEK
When to use tuples

• When storing elements that will not need to be changed.
• When performance is a concern.
• When you want to store your data in logical immutable pairs, triples, etc.
LOGO
UNIV/POLTEK
Constructing tuples
• An empty tuple can be created with an empty set of parentheses.
• Pass a sequence type object into the tuple() constructor.
• Tuples can be initialized by listing comma-separated values. These do not need to be in
parentheses but they can be.
• One quirk: to initialize a tuple with a single value, use a trailing comma.
>>> t1 = (1, 2, 3, 4)
>>> t2 = "a", "b", "c", "d"
>>> t3 = ()
>>> t4 = ("red", )
LOGO
UNIV/POLTEK
Tuple operations
• Tuples are very similar to lists and support a lot of the same operations.
• Accessing elements: use bracket notation (e.g. t1[2]) and slicing.
• Use len(t1) to obtain the length of a tuple.
• The universal immutable sequence type operations are all supported by tuples.
• +, *
• in, not in
• min(t), max(t), t.index(x), t.count(x)
LOGO
UNIV/POLTEK
Packing/unpacking
• Tuple packing is used to “pack” a collection of items into a tuple. We can unpack a tuple using
Python’s multiple assignment feature.
>>> s = "Susan", 19, "CS" # tuple packing

>>> name, age, major = s # tuple unpacking
>>> name
'Susan'
>>> age
19
>>> major
'CS'
LOGO
UNIV/POLTEK
When to use dictionaries

• When you need to create associations in the form of key:value pairs.
• When you need fast lookup for your data, based on a custom key.
• When you need to modify or add to your key:value pairs.
LOGO
UNIV/POLTEK
Constructing a dictionary
• Create an empty dictionary with empty curly braces or the dict() constructor.
• You can initialize a dictionary by specifying each key:value pair within the curly braces.
• Note that keys must be hashable objects.
>>> d1 = {}
>>> d2 = dict() # both empty
>>> d3 = {"Name": "Susan", "Age": 19, "Major": "CS"}
>>> d4 = dict(Name="Susan", Age=19, Major="CS")
>>> d5 = dict(zip(['Name', 'Age', 'Major'], ["Susan", 19, "CS"]))
>>> d6 = dict([('Age', 19), ('Name', "Susan"), ('Major', "CS")])
Note: zip takes two equal-length collections and merges their corresponding elements into tuples.
LOGO
UNIV/POLTEK
Accessing the dictionary

• To access a dictionary, simply index the dictionary by the key to obtain the value. An
exception will be raised if the key is not in the dictionary.
>>> d1 = {'Age':19, 'Name':"Susan", 'Major':"CS"}

>>> d1['Age']
19
>>> d1['Name']
'Susan'
LOGO
UNIV/POLTEK
Updating a dictionary
• Simply assign a key:value pair to modify it or add a new pair. The del keyword can be used to
delete a single key:value pair or the whole dictionary. The clear() method will clear the
contents of the dictionary.

>>> d1['Age'] = 21
>>> d1['Year'] = "Junior"
>>> d1
{'Age': 21, 'Name': 'Susan', 'Major': 'CS', 'Year': 'Junior'}
>>> del d1['Major']
>>> d1
{'Age': 21, 'Name': 'Susan', 'Year': 'Junior'}
>>> d1.clear()
>>> d1
{}
LOGO
UNIV/POLTEK
Built-in dictionary methods

>>> d1.has_key('Age') # True if key exists
True
>>> d1.has_key('Year') # False otherwise
False
>>> d1.keys() # Return a list of keys
['Age', 'Name', 'Major']
>>> d1.items() # Return a list of key:value pairs
[('Age', 19), ('Name', 'Susan'), ('Major', 'CS')]
>>> d1.values() # Returns a list of values
[19, 'Susan', 'CS']
Note: in, not in, pop(key),and popitem()are also supported.
LOGO
UNIV/POLTEK
Ordered dictionary
Dictionaries do not remember the order in which keys were inserted. An ordered dictionary
implementation is available in the collections module. The methods of a regular dictionary are
all supported by the OrderedDict class.
An additional method supported by OrderedDict is the following:
OrderedDict.popitem(last=True) # pops items in LIFO order
LOGO
UNIV/POLTEK
Ordered dictionary
>>> # regular unsorted dictionary
>>> d = {'banana': 3, 'apple': 4, 'pear': 1, 'orange': 2}
>>> # dictionary sorted by key

>>> from collections import OrderedDict
>>> OrderedDict(sorted(d.items(), key=lambda t: t[0]))
OrderedDict([('apple', 4), ('banana', 3), ('orange', 2), ('pear', 1)])
>>> # dictionary sorted by value

>>> OrderedDict(sorted(d.items(), key=lambda t: t[1]))
OrderedDict([('pear', 1), ('orange', 2), ('banana', 3), ('apple', 4)])
>>> # dictionary sorted by length of the key string

>>> OrderedDict(sorted(d.items(), key=lambda t: len(t[0])))
OrderedDict([('pear', 1), ('apple', 4), ('orange', 2), ('banana', 3)])
LOGO
UNIV/POLTEK
Python Libraries for Data Science
Many popular Python toolboxes/libraries:

• NumPy
• SciPy
• Pandas
• SciKit-Learn
Visualization libraries
• matplotlib
• Seaborn
and many more …
44
LOGO
UNIV/POLTEK

NumPy:
 introduces objects for multidimensional arrays and matrices, as well as functions that allow to easily
perform advanced mathematical and statistical operations on those objects
 provides vectorization of mathematical operations on arrays and matrices which significantly improves
the performance
 many other python libraries are built on NumPy
Link: http://www.numpy.org/
45
LOGO
UNIV/POLTEK
SciPy:
 collection of algorithms for linear algebra, differential equations, numerical integration, optimization,
statistics and more
 part of SciPy Stack
 built on NumPy
Link: https://www.scipy.org/scipylib/
46
LOGO
UNIV/POLTEK

Pandas:
 adds data structures and tools designed to work with table-like data (similar to Series and Data Frames in
R)
 provides tools for data manipulation: reshaping, merging, sorting, slicing, aggregation etc.
 allows handling missing data
Link: http://pandas.pydata.org/
47
LOGO
UNIV/POLTEK
SciKit-Learn:
 provides machine learning algorithms: classification, regression, clustering, model validation etc.
 built on NumPy, SciPy and matplotlib
Link: http://scikit-learn.org/
48
LOGO
UNIV/POLTEK
matplotlib:
 python 2D plotting library which produces publication quality figures in a variety of hardcopy formats
 a set of functionalities similar to those of MATLAB
 line plots, scatter plots, barcharts, histograms, pie charts etc.
 relatively low-level; some effort needed to create advanced visualization
Link: https://matplotlib.org/
49
LOGO
UNIV/POLTEK
Seaborn:
 based on matplotlib
 provides high level interface for drawing attractive statistical graphics
 Similar (in style) to the popular ggplot2 library in R
Link: https://seaborn.pydata.org/
50
LOGO
UNIV/POLTEK
Loading Python Libraries

In [ ]: #Import Python Libraries
import numpy as np
import scipy as sp
import pandas as pd
import matplotlib as mpl
import seaborn as sns
Press Shift+Enter to execute the jupyter cell
51
LOGO
UNIV/POLTEK
Reading data using pandas

In [ ]: #Read csv file
df = pd.read_csv("http://rcs.bu.edu/examples/python/data_analysis/Salaries.csv")
Note: The above command has many optional arguments to fine-tune the data import process.
There is a number of pandas commands to read other data formats:
pd.read_excel('myfile.xlsx',sheet_name='Sheet1', index_col=None, na_values=['NA'])

pd.read_stata('myfile.dta')
pd.read_sas('myfile.sas7bdat')
pd.read_hdf('myfile.h5','df')
52
LOGO
UNIV/POLTEK
Exploring data frames

In [3]: #List first 5 records
df.head()
Out[3]:
53
LOGO
UNIV/POLTEK
Hands-on exercises
 Try to read the first 10, 20, 50 records;
 Can you guess how to view the last few records; Hint:
54
LOGO
UNIV/POLTEK
Data Frame data types

Pandas Type Native Python Type Description
object string The most general dtype. Will be
assigned to your column if column
has mixed types (numbers and
strings).
int64 int Numeric characters. 64 refers to
the memory allocated to hold this
character.
float64 float Numeric characters with decimals.
If a column contains numbers and
NaNs(see below), pandas will
default to float64, in case your
missing value has a decimal.
datetime64, timedelta[ns] N/A (but see the datetime module Values meant to hold time data.
in Python’s standard library) Look into these for time series
experiments.
55
LOGO
UNIV/POLTEK
Data Frame data types

In [4]: #Check a particular column type
df['salary'].dtype
Out[4]: dtype('int64')
In [5]: #Check types for all the columns

df.dtypes
Out[4]: rank object

discipline object
phd int64
service int64
sex object
salary int64
dtype: object
56
LOGO
UNIV/POLTEK
Data Frames attributes

Python objects have attributes and methods.
df.attribute description
dtypes list the types of the columns
columns list the column names
axes list the row labels and column names
ndim number of dimensions
size number of elements
shape return a tuple representing the dimensionality
values numpy representation of the data
57
LOGO
UNIV/POLTEK
Hands-on exercises
 Find how many records this data frame has;
 How many elements are there?
 What are the column names?
 What types of columns we have in this data frame?
58
LOGO
UNIV/POLTEK
Data Frames methods

Unlike attributes, python methods have parenthesis.
All attributes and methods can be listed with a dir() function: dir(df)
df.method() description
head( [n] ), tail( [n] ) first/last n rows
describe() generate descriptive statistics (for numeric columns only)
max(), min() return max/min values for all numeric columns
mean(), median() return mean/median values for all numeric columns
std() standard deviation
sample([n]) returns a random sample of the data frame
dropna() drop all the records with missing values

59
LOGO
UNIV/POLTEK
Hands-on exercises
 Give the summary for the numeric columns in the dataset
 Calculate standard deviation for all numeric columns;
 What are the mean values of the first 50 records in the dataset? Hint: use
head() method to subset the first 50 records and then calculate the mean
60
LOGO
UNIV/POLTEK
Selecting a column in a Data Frame

Method 1: Subset the data frame using column name:
df['sex']
Method 2: Use the column name as an attribute:

df.sex
Note: there is an attribute rank for pandas data frames, so to select a column with a name
"rank" we should use method 1.
61
LOGO
UNIV/POLTEK
Hands-on exercises
 Calculate the basic statistics for the salary column;
 Find how many values in the salary column (use count method);
 Calculate the average salary;
62
LOGO
UNIV/POLTEK
Data Frames groupby method

Using "group by" method we can:
• Split the data into groups based on some criteria
• Calculate statistics (or apply a function) to each group
• Similar to dplyr() function in R
In [ ]: #Group data using rank
df_rank = df.groupby(['rank'])
In [ ]: #Calculate mean value for each numeric column per each group
df_rank.mean()
63
LOGO
UNIV/POLTEK
Once groupby object is create we can calculate various statistics for each group:
In [ ]: #Calculate mean salary for each professor rank:

df.groupby('rank')[['salary']].mean()
Note: If single brackets are used to specify the column (e.g. salary), then the output is Pandas Series object.
When double brackets are used the output is a Data Frame 64
LOGO
UNIV/POLTEK
groupby performance notes:

- no grouping/splitting occurs until it's needed. Creating the groupby object
only verifies that you have passed a valid mapping
- by default the group keys are sorted during the groupby operation. You may
want to pass sort=False for potential speedup:

df.groupby(['rank'], sort=False)[['salary']].mean()
65
LOGO
UNIV/POLTEK
Data Frame: filtering

To subset the data we can apply Boolean indexing. This indexing is commonly
known as a filter. For example if we want to subset the rows in which the salary
value is greater than $120K:
df_sub = df[ df['salary'] > 120000 ]
Any Boolean operator can be used to subset the data:

> greater; >= greater or equal;
< less; <= less or equal;
== equal; != not equal;
In [ ]: #Select only those rows that contain female professors:
df_f = df[ df['sex'] == 'Female' ]
66
LOGO
UNIV/POLTEK
Data Frames: Slicing

There are a number of ways to subset the Data Frame:
• one or more columns
• one or more rows
• a subset of rows and columns
Rows and columns can be selected by their position or label
67
LOGO
UNIV/POLTEK
Data Frames: Slicing

When selecting one column, it is possible to use single set of brackets, but the
resulting object will be a Series (not a DataFrame):
In [ ]: #Select column salary:
df['salary']
When we need to select more than one column and/or make the output to be a
DataFrame, we should use double brackets:
In [ ]: #Select column salary:
df[['rank','salary']]
68
LOGO
UNIV/POLTEK
Data Frames: Selecting rows

If we need to select a range of rows, we can specify the range using ":"
In [ ]: #Select rows by their position:

df[10:20]
Notice that the first row has a position 0, and the last value in the range is omitted:
So for 0:10 range the first 10 rows are returned with the positions starting with 0
and ending with 9
69
LOGO
UNIV/POLTEK
Data Frames: method loc

If we need to select a range of rows, using their labels we can use method loc:
In [ ]: #Select rows by their labels:

df_sub.loc[10:20,['rank','sex','salary']]
Out[ ]:
70
LOGO
UNIV/POLTEK
Data Frames: method iloc

If we need to select a range of rows and/or columns, using their positions we can
use method iloc:
In [ ]: #Select rows by their labels:
df_sub.iloc[10:20,[0, 3, 4, 5]]
Out[ ]:
71
LOGO
UNIV/POLTEK
Data Frames: method iloc (summary)

df.iloc[0] # First row of a data frame
df.iloc[i] #(i+1)th row
df.iloc[-1] # Last row
df.iloc[:, 0] # First column

df.iloc[:, -1] # Last column
df.iloc[0:7] #First 7 rows

df.iloc[:, 0:2] #First 2 columns
df.iloc[1:3, 0:2] #Second through third rows and first 2 columns
df.iloc[[0,5], [1,3]] #1st and 6th rows and 2nd and 4th columns
72
LOGO
UNIV/POLTEK
Data Frames: Sorting

We can sort the data by a value in the column. By default the sorting will occur in
ascending order and a new data frame is return.
In [ ]: # Create a new data frame from the original sorted by the column Salary
df_sorted = df.sort_values( by ='service')
df_sorted.head()
Out[ ]:
73
LOGO
UNIV/POLTEK
Data Frames: Sorting

We can sort the data using 2 or more columns:
In [ ]: df_sorted = df.sort_values( by =['service', 'salary'], ascending = [True, False])
df_sorted.head(10)
Out[ ]:
74
LOGO
UNIV/POLTEK
Missing Values
Missing values are marked as NaN
In [ ]: # Read a dataset with missing values
flights = pd.read_csv("http://rcs.bu.edu/examples/python/data_analysis/flights.csv")
In [ ]: # Select the rows that have at least one missing value

flights[flights.isnull().any(axis=1)].head()
Out[ ]:
75
LOGO
UNIV/POLTEK
Missing Values
There are a number of methods to deal with missing values in the data frame:
dropna() Drop missing observations
dropna(how='all') Drop observations where all cells is NA
dropna(axis=1, how='all') Drop column if all the values are missing
dropna(thresh = 5) Drop rows that contain less than 5 non-missing values
fillna(0) Replace missing values with zeros
isnull() returns True if the value is missing
notnull() Returns True for non-missing values

76
LOGO
UNIV/POLTEK
Missing Values
• When summing the data, missing values will be treated as zero
• If all values are missing, the sum will be equal to NaN
• cumsum() and cumprod() methods ignore missing values but preserve them in
the resulting arrays
• Missing values in GroupBy method are excluded (just like in R)
• Many descriptive statistics methods have skipna option to control if missing
data should be excluded . This value is set to True by default (unlike R)
77
LOGO
UNIV/POLTEK
Aggregation Functions in Pandas

Aggregation - computing a summary statistic about each group, i.e.
• compute group sums or means
• compute group sizes/counts
Common aggregation functions:
min, max
count, sum, prod
mean, median, mode, mad
std, var
78
LOGO
UNIV/POLTEK
Aggregation Functions in Pandas

agg() method are useful when multiple statistics are computed per column:
In [ ]: flights[['dep_delay','arr_delay']].agg(['min','mean','max'])
Out[ ]:
79
LOGO
UNIV/POLTEK
Basic Descriptive Statistics

describe Basic statistics (count, mean, std, min, quantiles, max)
min, max Minimum and maximum values
mean, median, mode Arithmetic average, median and mode
var, std Variance and standard deviation
sem Standard error of mean
skew Sample skewness
kurt kurtosis
80
LOGO
UNIV/POLTEK
Graphics to explore the data

Seaborn package is built on matplotlib but provides high level
interface for drawing attractive statistical graphics, similar to ggplot2
library in R. It specifically targets statistical data visualization
To show graphs within Python notebook include inline directive:
In [ ]: %matplotlib inline
81
LOGO
UNIV/POLTEK
Graphics
description
distplot histogram
barplot estimate of central tendency for a numeric variable

violinplot similar to boxplot, also shows the probability density of the
data
jointplot Scatterplot
regplot Regression plot
pairplot Pairplot
boxplot boxplot
swarmplot categorical scatterplot
factorplot General categorical plot
82
LOGO
UNIV/POLTEK
Basic statistical Analysis

statsmodel and scikit-learn - both have a number of function for statistical analysis
The first one is mostly used for regular analysis using R style formulas, while scikit-learn is
more tailored for Machine Learning.
statsmodels:
• linear regressions
• ANOVA tests
• hypothesis testings
• many more ...
scikit-learn:
• kmeans
• support vector machines
• random forests
• many more ...
See examples in the Tutorial Notebook

83
LOGO
UNIV/POLTEK
Follow our social

media!
digitalent.kominfo
digitalent.kominfo
DTS_kominfo
Digital Talent Scholarship 2019
Pusat Pengembangan Profesi dan Sertifikasi

Badan Penelitian dan Pengembangan SDM
Kementerian Komunikasi dan Informatika
Jl. Medan Merdeka Barat No. 9
(Gd. Belakang Lt. 4 - 5)
Jakarta Pusat, 10110
digitalent.kominfo.go.id 84

Module-10 - Python Basic-2

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Module-10 - Python Basic-2

Uploaded by

Copyright:

Available Formats

LOGO

Functional Programming tools

• Lambda functions within Python.

Functional programming tools

Functional programming tools

Functional programming tools

• Map def expo(x, y):

Functional programming tools

Functional programming tools

>>> list(map(lambda x: x**2, range(0,11)))

More Data Structures

When to use Lists

# mylist1 and mylist2 point to the same list

# mylist3 and mylist4 point to the same list

mylist5 = []; mylist6 = [] # different lists

Accessing list elements

Slicing and sliding

mylist[start:end] # items start to end-1

mylist[start:end:step] # start to end-1, by step

Slicing and sliding

mylist[-1] # last item in the array

>>> mylist = [34, 56, 29, 73, 19, 62]

>>> mylist = ['a', 'b', 'c', 'd', 'a', 'f', 'c']

When to use sets

Mutable operations continued

set |= other | ...

set &= other & ...

set -= other | ...

Mutable operations continued

When to use tuples

>>> s = "Susan", 19, "CS" # tuple packing

When to use dictionaries

Accessing the dictionary

>>> d1 = {'Age':19, 'Name':"Susan", 'Major':"CS"}

>>> d1 = {'Age':19, 'Name':"Susan", 'Major':"CS"}

Built-in dictionary methods

Note: in, not in, pop(key),and popitem()are also supported.

An additional method supported by OrderedDict is the following:

OrderedDict.popitem(last=True) # pops items in LIFO order

>>> # dictionary sorted by key

>>> # dictionary sorted by value

>>> # dictionary sorted by length of the key string

Python Libraries for Data Science

Many popular Python toolboxes/libraries:

and many more …

Python Libraries for Data Science

 many other python libraries are built on NumPy

Python Libraries for Data Science

 part of SciPy Stack

Python Libraries for Data Science

 allows handling missing data

Python Libraries for Data Science

 built on NumPy, SciPy and matplotlib

Python Libraries for Data Science

 a set of functionalities similar to those of MATLAB

 line plots, scatter plots, barcharts, histograms, pie charts etc.

 relatively low-level; some effort needed to create advanced visualization

Python Libraries for Data Science

 provides high level interface for drawing attractive statistical graphics

 Similar (in style) to the popular ggplot2 library in R

Loading Python Libraries

Press Shift+Enter to execute the jupyter cell