ROLE AND RESPONSIBILITIES
final
• Building data pipelines to bring together information
from different source systems using Python and
NodeJS
• Data pipeline:
• Extract raw data from different
sources to data warehouse
• In staging area, tranform data (clean
data, put it into right data type,
check NULL (remove or fill), check
duplicate, check invalid data value,…)
• Load into analytics area
• Maintain data pipeline, automate
tasks
• Integrate, consolidate and cleanse data and structure
it for use in analytics applications
• Data schema of email marketing data:
• 3 tables: receivers, products, emails
• Receivers: name, age, job,
income,…
• Products: product we market
for – industry, type of
customers, number of
customers each month, price
of product
• Emails: content, subject, kind
of campaign
• Click: click through rate,
conversion rate, device,
position, gender…
• Analyze incoming data sets using Python Pandas
• Assignment 2 - Influence on Job Satisfaction
(FINAL)
• Check data: dtypes, info, describe,
drop -> pandas_profilling
• Fill missing value with mean or 0,…
remove outlier
• Visualize: matplotlib, seaborn
• Preprocessing: label encoding, one
hot encoding
• hypothesis testing
• t-test
• anova
• chi-square
• correlation
• Nominal -> ordinal ->
interval -> ratio
• Assignment 6 - Clustering Boston Marathon
groups final
• Kmean (w and wout PCA, elbow
method to find less squared
distance, find silhouette_score –
greater is better) – but for number
• K-prototype (kmodes) (w and wout
PCA) – for cate
• Assignment 5 - Predict airline arrival delay
• Tree based model: label encode –
Logistic regression – fit 1 and 0, what
is likely
• Non tree-based: one hot
• Naïve bayes - likelihood
• Decision tree – label, find
which decision - overfit
• Random forest – lots of
decision tree
• Gradient boosting –
combine of a lot of classifiers
• Light GBM
• XGBoost
• Support vector: find vector
with the least distance
• Assignment 4 - House price predict with
Linear Regression, Polynomial Regression
and PCA (FINAL)
• Linear regression (find function of 2
orders to fit) with cross validation
(cross the train and test), gridsearch
cv (how many parts to cross)
• Linear regression with polynomial
regression (3 order of function)
• Linear regression with PCA (reduce
dimension)
• Linear regression with PCA and
Polynomial Regression
• Work on the automation of data-flows using Python.
• Extract data from API by installing requests
library
• Build pipeline with Pyspark
• Create spark session
• Read table from db using Spark jdbc
• Define ransform, aggregate data
functions similar to pandas functions
• Define main function
• Use Airflow to schedule
• Drive data management improvements through
features/stories
QUALIFICATIONS AND EDUCATION REQUIREMENTS
• Fluent in conversational English: 7.5 IELTS
• Good understanding of ETL tools and REST-
oriented APIs for creating and managing data
integration jobs
• ETL tools: experience with Hadoop
ecosystem, Pentaho (integrated in
Hadoop),
• Create, maintain Nifi tasks (get
raw data from sources such as
Oracle, FTP, API. Update
attribute, put into rawzone in
HDFS), Pentaho tasks (get from
 HDFS to a temporary table,
transform data, output data to
the right type in official table)
• REST-oriented APIs: Use API to get data
in Nifi
• Basic auth: client send user,
password, grant type to server -
> if correct, server send access
token to client
• Use this use this access token
(bearer <access token>) with
generated signature, send to
server -> get data content from
server
• Use Yahoo Finance API with
Python to get data of FOREX:
simple Python code: import
library -> code: .download (forex
code, start date, end date,
options)
• Some experience using SQL or similar data
processing tools: Use Pentaho to data
processing, use Dbeaver to connect, write, test
code
• Experience in process automation using
MongoDB, Mysql, Python, NodeJS: know Pyspark
• Experience in AWS, GCP, data warehouses and
data lakes is a plus: get raw data put in data lake,
then extract, transform data to data warehouse
for DA team.
• Analytic/data-oriented mindset: have experience
with data analyst (teaching assistant), and data
sciencist (internship for FOREX trading), have
some projects on using Pandas with skicit learn
to use machine learning to analyze data
• Good team worker, service oriented towards
internal clients.
• Restful: REpresentational State Transfer, set of
functions as GET (read), PUT (update), POST
(create), DELETE (delete) equivalent with CRUD,
of HTTP methods, to communicate, instead of
use URL
• API: Application program interface - contract
between the consumer of information (calling
API) and the provider (the response), return JSON
or XML
• RESTful API: standard for designing API for web
application to manage resources (files, pics,
audios, videos…)
• Appication -> API -> Client
NodeJS
• Framework based on JS, to build web application
• Fast processing with non blocking I/O, with
thousands of connections at one time
• Scalable
Audience Serv:
• winner of Deloitte Technology fast 50 Germany
2020 & 2021
• 6 offices, head quarter in Berlin
• 12 years of experience, since 2008
• 23 markets
• Provide marketing solutions (esp email), using
ML, AI
• Mission: exceed client expectations of both
service levels and campaign outcomes

API

• Use Flask or FastAPI to write API

• Get library
• Define class
• @app.get (<address>)
define what to return
• Authentication: jwt
• Send POST request with username and
password to server
• Server create jwt and send (header –
type, crypto algorithm; payload – data in
jwt registy - reserved, public, private;
signature – unique string to make sure
jwt is integrity)
• Client send jwt token with every request,
if valid then get data
• Message: 200 – ok, 401: invalid