Birla Institute of Technology & Science, Pilani

Work Integrated Learning Programmes Division

First Semester 2023-2024

Mid-Semester Test
(EC-2 Regular ANSWER-KEY)

Course No. : DSECLZG529/AIMLCZG529

Course Title : Data Management for Machine Learning
Nature of Exam : Closed Book
Weightage : 30% No. of Pages = 12
Duration : 2 Hours No. of Questions = 5
Date of Exam : 06/03/2021 or 19/03/2021 (FN/AN)
Note to Students:
1. Please follow all the Instructions to Candidates given on the cover page of the answer book.
2. All parts of a question should be answered consecutively. Each answer should start from a fresh page.
3. Assumptions made if any, should be stated clearly at the beginning of your answer.

Q1. Imagine a scenario where a healthcare organization is embarking on a machine learning

project to improve patient outcomes. The organization aims to develop a predictive model
that identifies potential health risks for patients based on their medical records, lifestyle data,
and genetic information. In the context of the data management phases describe how the
healthcare organization would navigate each phase to ensure the success and accuracy of the
predictive model. Highlight specific considerations or challenges that might arise during this
process.
Answer:
In this scenario, the healthcare organization is utilizing machine learning to enhance patient
outcomes through the development of a predictive model.
 Creation:
o Process: Combine medical records, lifestyle data, and genetic information to
create a comprehensive dataset for model training.
o Considerations/Challenges:
 Privacy and Compliance: Ensuring compliance with healthcare
regulations (e.g., HIPAA) and protecting patient privacy during data
creation is critical.
 Data Diversity: Integrating diverse data types requires addressing
interoperability challenges and maintaining data integrity.

 Ingestion:
o Process: Ingest real-time and historical patient data from electronic health
records (EHRs), lifestyle apps, and genetic databases.
o Considerations/Challenges:
 Data Quality: Verifying the accuracy and completeness of data from
various sources to avoid biases in the predictive model.
 Data Security: Implementing robust security measures to protect
sensitive health information during data transfer.

 Processing (Validation, Cleaning, Enrichment):

 o Process: Validate data for accuracy, clean it by addressing missing values or
errors, and enrich it with additional relevant information.
o Considerations/Challenges:
 Data Accuracy: Ensuring accurate medical coding and addressing
discrepancies in the records to maintain the model's reliability.
 Ethical Considerations: Handling sensitive genetic information
ethically and transparently, with patient consent.

 Post-processing (Data Management, Storage, Analysis):

o Process: Implement a robust data management system, store the trained model,
and analyze predictions against actual patient outcomes.
o Considerations/Challenges:
 Data Storage: Selecting secure and scalable storage solutions for large
volumes of patient data.
 Model Interpretability: Ensuring the model's predictions are
interpretable for healthcare professionals to make informed decisions.
This comprehensive approach to data management addresses the specific challenges in the
healthcare sector, emphasizing privacy, compliance, and ethical considerations to create a
reliable and accurate predictive model for improving patient outcomes.
OR
Q1. Imagine a scenario where a manufacturing company is initiating a machine learning
project to optimize its production processes. The company intends to develop a predictive
model that forecasts equipment failures based on historical performance data, sensor
readings, and maintenance records. In the context of the data management phases i.e. -
creation, ingestion, Processing (Validation, Cleaning, Enrichment), Post-processing (Data
Management, Storage, Analysis), elucidate how the manufacturing company would navigate
each phase to ensure the effectiveness and accuracy of the predictive model. Bring attention
to particular considerations or challenges that could emerge throughout this process.
Answer:
The manufacturing company would navigate each data management phase in the context of
the machine learning project to optimize production processes as follows:
 Data Creation:
o Process Overview:
 Gather historical performance data, sensor readings, and maintenance
records from machinery and equipment.
 Ensure data is collected with relevant timestamps for accurate analysis.
o Considerations/Challenges:
 Sensor Calibration: Ensure sensors are calibrated properly to collect
accurate readings.
 Consistency: Address inconsistencies in data collection methods across
different machinery.
 Data Ingestion:
o Process Overview:
 Transfer data from various sources into a centralized storage system or
data warehouse.
  Validate data integrity during ingestion to avoid corrupt or incomplete
datasets.
o Considerations/Challenges:
 Real-time Ingestion: Consider the need for real-time data updates,
especially for critical sensor readings.
 Data Format Compatibility: Ensure compatibility of data formats from
different sources.
 Data Processing (Validation, Cleaning, Enrichment):
o Process Overview:
 Validate data for accuracy and consistency, especially dealing with
outliers.
 Cleanse data by addressing errors, missing values, and inconsistencies.
 Enrich the dataset with additional contextual information.
o Considerations/Challenges:
 Outlier Detection: Implement outlier detection mechanisms to handle
unusual readings.
 Missing Data Handling: Develop strategies for handling missing or
incomplete data without compromising model accuracy.
 Post-processing (Data Management, Storage, Analysis):
o Process Overview:
 Manage data storage efficiently for easy retrieval during analysis.
 Conduct exploratory data analysis (EDA) to identify patterns and
correlations.
 Store clean and enriched data for model training.
o Considerations/Challenges:
 Scalable Storage: Choose a storage solution that can handle the
increasing volume of sensor data.
 Analysis Tools: Utilize advanced analytics tools for effective data
exploration.
Q2. In a financial institution's digital transformation initiative, the organization invests $5
million in upgrading its Data Architecture to enhance data storage, integration, and usage. If
this investment leads to a 20% improvement in data retrieval efficiency and a 15% reduction
in data processing time, calculate the potential cost savings in operational expenses due to the
enhanced Data Architecture. Assume the institution's current annual operational expenses
related to data management are $12 million.
Note: Round your answer to the nearest thousand.
Answer:
To calculate the potential cost savings due to enhanced Data Architecture, we can follow
these steps:
 Calculate the improvement in data retrieval efficiency:
o Improvement in retrieval efficiency = Investment × Improvement percentage
o Improvement in retrieval efficiency=$5,000,000×20%=$1,000,000
 Calculate the reduction in data processing time:
o Reduction in processing time=Investment×Reduction percentage
o Reduction in processing time=$5,000,000×15%=$750,000
 Calculate the total potential cost savings:
 o Total potential cost savings=Improvement in retrieval efficiency+Reduction in
processing time
o Total potential cost savings=$1,000,000+$750,000=$1,750,000
 Now, subtract the potential cost savings from the current annual operational expenses
to find the net savings:
o Net savings=Current annual operational expenses−Total potential cost savings
o Net savings=$12,000,000−$1,750,000=$10,250,000
Therefore, the potential cost savings in operational expenses due to the enhanced Data
Architecture is approximately $10.25 million.
OR
Q2. In a financial institution's digital transformation initiative, the organization invests $5
million in upgrading its Data Architecture to enhance data storage, integration, and usage. If
this investment leads to a 15% improvement in data retrieval efficiency and a 20% reduction
in data processing time, calculate the potential cost savings in operational expenses due to the
enhanced Data Architecture. Assume the institution's current annual operational expenses
related to data management are $2 million.
Note: Round your answer to the nearest thousand.
Answer:
Let's calculate the potential cost savings step by step:
 Improvement in data retrieval efficiency:
o Improvement in retrieval efficiency=$5,000,000×15%=$750,000
 Reduction in data processing time:
o Reduction in processing time=$5,000,000×20%=$1,000,000
 Total potential cost savings:
o Total potential cost savings=$750,000+$1,000,000=$1,750,000
 Net savings in operational expenses:
o Net savings=$2,000,000−$1,750,000=$250,000
Therefore, the potential cost savings in operational expenses due to the enhanced Data
Architecture is approximately $250,000 (rounded to the nearest thousand).

Q3. In the dynamic environment of a technology company, the development team faces a
critical decision regarding the selection of a data format for storing and transmitting
information between applications. The team is engaged in a lively debate, weighing the pros
and cons of text-based formats like JSON or XML against binary formats such as Protocol
Buffers or MessagePack. How would the development team navigate the decision-making
process when choosing between text-based formats (JSON or XML) and binary formats
(Protocol Buffers or MessagePack) for data storage and transmission in the technology
company's ecosystem? Frame your answer by considering the specific use cases, advantages,
and challenges associated with each format. Assess Performance Implications,
Interoperability and Development Ease for each format.
Answer:
Q4. Consider a data integration scenario where a company is transferring and transforming
large volumes of data from various source systems to a data warehouse. The data includes
customer information, sales transactions, and product details.
 ETL (Extract, Transform, Load):
o Extraction:
 The company extracts data from three different source systems: CRM
(Customer Relationship Management), POS (Point of Sale), and ERP
(Enterprise Resource Planning).
 The extraction process takes an average of 2 hours for each source
system.
o Transformation:
 The transformation phase involves cleaning, aggregating, and
enriching the data.
 On average, the transformation process takes 4 hours for each source
system.
o Loading:
 Loading the transformed data into the data warehouse takes 1 hour for
each source system.
 ELT (Extract, Load, Transform):
o Extraction and Loading:
 The company extracts raw data from the three source systems and
loads it into the data warehouse without immediate transformation.
 The extraction and loading process takes an average of 3 hours for
each source system.
o Transformation:
 The transformation phase occurs after the data is loaded into the data
warehouse.
 On average, the transformation process takes 5 hours for each source
system.

a) Calculate the total time taken for the ETL process, considering the sequential nature
of the steps.
b) Calculate the total time taken for the ELT process, considering the shift in the order of
transformation.
c) Compare the total time taken for ETL vs. ELT. Discuss the efficiency of each
approach in terms of reducing the overall processing time.
d) Analyze the availability of data for analysis during the ETL and ELT processes.
Discuss how each approach affects the availability of transformed data for reporting
and analytics.
e) Discuss how the scalability of the data integration process might be impacted by
choosing ETL or ELT.
Answer:
a) ETL (Extract, Transform, Load):
Extraction:
2 hours (CRM) + 2 hours (POS) + 2 hours (ERP) = 6 hours
 Transformation:
4 hours (CRM) + 4 hours (POS) + 4 hours (ERP) = 12 hours
Loading:
1 hour (CRM) + 1 hour (POS) + 1 hour (ERP) = 3 hours
Total Time for ETL:
6 hours (Extraction) + 12 hours (Transformation) + 3 hours (Loading) = 21
hours

b) ELT (Extract, Load, Transform):

Extraction and Loading:
3 hours (CRM) + 3 hours (POS) + 3 hours (ERP) = 9 hours
Transformation:
5 hours (CRM) + 5 hours (POS) + 5 hours (ERP) = 15 hours
Total Time for ELT:
9 hours (Extraction and Loading) + 15 hours (Transformation) = 24 hours

c) ETL takes 21 hours, while ELT takes 24 hours. ETL is more time-efficient in this
scenario, with a shorter processing time.
d) ETL provides transformed data after the entire process, ensuring clean and enriched
data is available for analytics. ELT allows for immediate data availability but requires
additional time for transformation, leading to a delay in access to fully processed data.
e) ETL might face challenges with scalability as the volume of data increases, especially
during the transformation phase. ELT, with extraction and loading occurring first,
might handle increasing data volumes more efficiently, leveraging the scalability of
modern data warehouses.
OR
Q4. Consider a data integration scenario where a company is transferring and transforming
large volumes of data from various source systems to a data warehouse. The data includes
customer information, sales transactions, and product details.
 ETL (Extract, Transform, Load):
o Extraction:
 The company extracts data from three different source systems: CRM
(Customer Relationship Management), POS (Point of Sale), and ERP
(Enterprise Resource Planning).
 The extraction process takes an average of 4 hours for each source
system.
o Transformation:
 The transformation phase involves cleaning, aggregating, and
enriching the data.
 On average, the transformation process takes 3 hours for each source
system.
o Loading:
 Loading the transformed data into the data warehouse takes 1.5 hour
for each source system.
 ELT (Extract, Load, Transform):
o Extraction and Loading:
 The company extracts raw data from the three source systems and
loads it into the data warehouse without immediate transformation.
 The extraction and loading process takes an average of 4 hours for
each source system.
o Transformation:
 The transformation phase occurs after the data is loaded into the data
warehouse.
 On average, the transformation process takes 3 hours for each source
system.

f) Calculate the total time taken for the ETL process, considering the sequential nature
of the steps.
g) Calculate the total time taken for the ELT process, considering the shift in the order of
transformation.
h) Compare the total time taken for ETL vs. ELT. Discuss the efficiency of each
approach in terms of reducing the overall processing time.
i) Analyze the availability of data for analysis during the ETL and ELT processes.
Discuss how each approach affects the availability of transformed data for reporting
and analytics.
j) Discuss how the scalability of the data integration process might be impacted by
choosing ETL or ELT.
Answer:
a) ETL (Extract, Transform, Load):
Extraction:
4 hours (CRM) + 4 hours (POS) + 4 hours (ERP) = 12 hours
Transformation:
3 hours (CRM) + 3 hours (POS) + 3 hours (ERP) = 9 hours
Loading:
1.5 hours (CRM) + 1.5 hours (POS) + 1.5 hours (ERP) = 4.5 hours
Total Time for ETL:
12 hours (Extraction) + 9 hours (Transformation) + 4.5 hours (Loading) = 25.5
hours

b) ELT (Extract, Load, Transform):

Extraction and Loading:
4 hours (CRM) + 4 hours (POS) + 4 hours (ERP) = 12 hours
Transformation:
3 hours (CRM) + 3 hours (POS) + 3 hours (ERP) = 9 hours
Total Time for ELT:
12 hours (Extraction and Loading) + 9 hours (Transformation) = 21 hours
c) ETL takes 25.5 hours, while ELT takes 21 hours. ELT is more time-efficient in this
scenario, with a shorter processing time.
d) ETL provides transformed data after the entire process, ensuring clean and enriched
data is available for analytics. ELT allows for immediate data availability but requires
additional time for transformation, leading to a delay in access to fully processed data.
 e) ETL might face challenges with scalability as the volume of data increases, especially
during the transformation phase. ELT, with extraction and loading occurring first,
might handle increasing data volumes more efficiently, leveraging the scalability of
modern data warehouses.

Q5. Imagine a scenario where a growing e-commerce company is considering two different
data architectures for managing and analyzing its diverse and rapidly expanding data sources.
The company is faced with the decision of choosing between a traditional relational database
architecture and a modern data lake architecture. The goal is to enhance data management,
analytics capabilities, and overall business intelligence.
The company's data sources include customer profiles, transaction records, website
interactions, social media data, and product information.
The traditional relational database architecture is well-established, while the data lake
architecture is seen as a more flexible and scalable option.
a) As the lead data architect, outline the considerations and decision-making factors the
company should evaluate when comparing the traditional relational database
architecture and the modern data lake architecture. Include specific aspects such as
data modeling, schema flexibility, scalability, cost-effectiveness, and analytics
capabilities.
b) Discuss the potential advantages and challenges associated with each architecture in
the context of the company's data landscape.
c) Finally, recommend a suitable data architecture based on the given scenario,
providing insights into how the chosen architecture aligns with the company's current
and future data needs.
Answer:
a)
 Data Modeling and Schema Flexibility:
o Relational Database:
 Strong support for structured data with predefined schemas.
 Well-suited for transactional data like customer profiles and
transaction records.
 May require schema modifications when dealing with new or evolving
data types.
o Data Lake:
 Schema-on-read approach allows flexibility in handling diverse data
types.
 Ideal for handling semi-structured and unstructured data like social
media data.
 Enables agility in adapting to changes in data structure without
predefined schemas.

 Scalability:
o Relational Database:
 Vertical scaling might be necessary as data volume increases.
 May face challenges in handling the variety and volume of data
generated by the diverse sources.
 o Data Lake:
 Horizontal scalability allows for handling large volumes of data
efficiently.
 Better suited for the scalability requirements of rapidly expanding and
diverse data sources.

 Cost-Effectiveness:
o Relational Database:
 Can be cost-effective for well-defined, structured data.
 Costs may increase with scaling, especially in terms of hardware and
licensing.
o Data Lake:
 Cost-effectiveness due to storage scalability and the ability to use low-
cost storage options.
 Initial implementation costs might be higher due to the need for
specialized skills.

 Analytics Capabilities:
o Relational Database:
 Well-suited for traditional SQL-based analytics and reporting.
 May face challenges with complex analytics tasks involving diverse
and unstructured data.
o Data Lake:
 Enables advanced analytics, machine learning, and data exploration.
 Supports a wide range of analytics tools and frameworks, providing
more comprehensive insights.
b)
Advantages and Challenges:
 Relational Database:
o Advantages:
 Proven reliability for structured data.
 Mature ecosystem with well-established tools and practices.
o Challenges:
 Limited flexibility for handling diverse data types.
 Potential scalability issues with the exponential growth of data.
 Data Lake:
o Advantages:
 Flexibility to handle various data types.
 Scalable and cost-effective for large and diverse datasets.
 Support for advanced analytics and machine learning.
o Challenges:
 Complexity in managing and governing diverse data sources.
 Requires skilled personnel for effective implementation and
maintenance.
C)
Recommendation:
Considering the company's rapidly expanding and diverse data landscape, a modern data lake
architecture seems more suitable. The data lake's flexibility, scalability, and support for
advanced analytics align well with the company's needs. However, it's crucial to invest in the
necessary skills and governance processes to ensure effective implementation and
management of the data lake. This choice positions the company for future growth and
ensures the ability to extract valuable insights from various data sources, fostering improved
business intelligence capabilities.
OR
Q5. Imagine a scenario where a rapidly growing e-commerce company is at a crossroads in
its data management strategy, deliberating between a traditional data warehouse and a data
lake architecture. The company's diverse data sources encompass customer profiles,
transaction records, website interactions, social media data, and product information.
In your role as the lead data architect, outline the key considerations and decision-making
factors that the company should meticulously evaluate when comparing the traditional data
warehouse architecture and the data lake architecture. Delve into specific aspects such as data
modeling, schema flexibility, scalability, cost-effectiveness, and analytics capabilities. Bring
to light the potential advantages and challenges tied to each architecture within the context of
the company's intricate data landscape. Lastly, provide a well-informed recommendation for
the most suitable data architecture, elucidating how the chosen approach aligns with the
company's present and future data requirements.
Answer:
a)
In the given scenario, the e-commerce company faces a critical decision between a traditional
data warehouse and a data lake architecture. As the lead data architect, I would meticulously
evaluate several key considerations and decision-making factors to guide the company
towards the most suitable data architecture.
 Data Modeling:
o Traditional Data Warehouse: Offers structured data modeling with
predefined schemas, ensuring consistency and ease of querying.
o Data Lake: Provides schema-on-read flexibility, allowing for the
inclusion of structured, semi-structured, and unstructured data.
 Schema Flexibility:
o Traditional Data Warehouse: Relies on a rigid schema, making it
suitable for well-defined and stable data structures.
o Data Lake: Allows for dynamic schema evolution, accommodating
diverse and evolving data sources.
 Scalability:
o Traditional Data Warehouse: Might face challenges in scaling
horizontally, potentially leading to performance issues as data volume
grows.
o Data Lake: Offers horizontal scalability, seamlessly handling the influx
of large volumes of data, making it suitable for a rapidly growing e-
commerce business.
 Cost-Effectiveness:
 o Traditional Data Warehouse: Often involves higher upfront costs,
especially with proprietary solutions, and may incur additional
expenses as data volume increases.
o Data Lake: Generally more cost-effective, especially in cloud
environments, with a pay-as-you-go model and the ability to store raw
data economically.
 Analytics Capabilities:
o Traditional Data Warehouse: Efficient for structured data analytics,
reporting, and predefined queries.
o Data Lake: Excels in handling diverse data types and empowers
advanced analytics, machine learning, and exploratory data analysis.
b)
Advantages and Challenges:
 Traditional Data Warehouse:
o Advantages: Well-established, mature solutions; optimal for structured
data analytics.
o Challenges: Limited flexibility for unstructured data; potential
scalability and cost challenges with rapid data growth.
 Data Lake:
o Advantages: High flexibility, scalability, and cost-effectiveness; ideal
for diverse and evolving data sources.
o Challenges: Requires careful data governance to maintain quality and
prevent chaos; potential complexity in managing diverse data formats.
c)
Recommendation:
Considering the company's present and future data requirements, the recommendation
is to adopt a Data Lake Architecture. This choice aligns with the e-commerce
industry's demand for handling diverse and dynamic data sources, ensuring
scalability, and empowering advanced analytics. Implementing robust data
governance practices will be crucial to harness the full potential of a data lake while
maintaining data quality and integrity. This approach positions the company
strategically to adapt to evolving business needs, supporting innovation and growth in
the dynamic e-commerce landscape.

==================================================================