Professional Documents
Culture Documents
Technology
For now: Into Databases…!
Christoph Lofi
Ujwal Gadiraju
WIS 1
Christoph Lofi Ujwal Gadiraju
50% 50%
Midterm Endterm
CSE1500: Database Technology Exam CSE1500: Web Technology Exam
Asterios Katsifodimos
3
WIS
Exams
• Midterm Exam focuses on DB
• Endterm Exam focuses on Web
• Each exam can be resit
– either both or individually
4
WIS
More Information
• Prerequisites
– Basic understanding of OOP languages
– YOU NEED TO REGISTER ON BRIGHTSPACE!
• Materials
– Slides (on Brightspace)
– Video Recordings (on Collegerama)
– Extra Materials (on Brightspace)
– Lab assignments
– Discussions on https://answers.ewi.tudelft.nl/categories/1/tags/152
Use at least tags “CS1500” with “23-24”
• Syllabus on Brightspace
– Should cover most questions!
– Course Information - CSE1500 Web and Database Technology (2023/24
Q2) (tudelft.nl)
5
WIS
Literature DB
• Books:
– Fundamentals of Database Systems Elmasri & Navathe
• You can get quite forward with just using the slide deck
• Most things we use can be found in the 6th edition, but there is a
7th edition which is a better investment
– ISBN 1292097612 for 7th edition
– Good book, very dry and theoretical though. Maybe buy it or find somebody who has it to share/
borrow. I guess you could survive without the book, though….
6
WIS
Expectations
• This course is 5 ETCS
– ~140 hours of work to pass
• ~ 10 weeks, ~14 hours per week
– During class weeks:
• 4 hours of class and 4 hours of lab (if you want to)
– 6 hours of self-study / homework?
• No labs and class in exam prep weeks
7
WIS
What is the purpose of what?
• Self-Study (at home or in lab)
– Study how things work in detail
• e.g., how to normalize a schema step by step, how to build a B-tree step by step, etc.
• Use all available material for that, e.g., lecture slides, transcripts, home study videos, books,
internet resources & tutorials, YouTube videos, etc.
• Classes
– Editorialized view on lecture content & core motivations and general approaches
• e.g., why (or why not) would we want to normalize schemas?, why do we care about B-
trees?, etc.
• Labs
– Learn how to apply methodologies
– “Do things on a computer”; “Experience the pain of fiddling with a DBMS”…
• Practice Exams
– Overfit to previous exams….
• Be careful with this!
8
WIS
Web Information Systems
9
WIS
WIS Research
10
WIS
Information, Data
and Knowledge?
“Isn’t that all the same…?”
WIS 11
Information and Data Modelling???
WIS 1212
Information and Data Modelling???
• Another approach:
– "Typically, information is defined in terms of data,
knowledge in terms of information, and wisdom
in terms of knowledge“
– DIKW pyramid:
• Not a formal model
• Early versions liked to omit data, later versions like to omit
wisdom
Rowley, Jennifer (2007). "The wisdom hierarchy: representations of the DIKW hierarchy".
Journal of Information and Communication Science. 33 (2): 163–180. doi:10.1177/0165551506070706. 13
WIS
14
WIS
The DIKW Pyramid
• Data as
– … facts:
• "as being discrete, objective facts or observations, which are
unorganized and unprocessed and therefore have no meaning
or value because of lack of context and interpretation.“
• “raw facts”
– … symbols:
• "sets of signs that represent empirical stimuli or perceptions“
• "recorded (captured or stored) symbols"
– … signals
• "signal readings“
• "sensory stimuli, which we perceive through our senses"
- Zins, Chaim."Conceptual Approaches for Defining Data, Information, and Knowledge" (PDF). Journal of the American Society for Information Science and Technology. 58 (4): 479–493. 2007
- Rowley, Jennifer; Richard Hartley (2006). Organizing Knowledge: An Introduction to Managing Access to Information. Ashgate Publishing, Ltd. pp. 5–6. ISBN 978-0-7546-4431-6.
- Zins, Chaim (22 January 2007). "Conceptual Approaches for Defining Data, Information, and Knowledge". Journal of the American Society for Information Science and Technology. 58 (4): 479–
493. doi:10.1002/asi.20508..
15
WIS
The DIKW Pyramid
• Aspects of Information
– Structural vs functional
• "organized or structured data, which has been processed in
such a way that the information now has relevance for a
specific purpose or context, and is therefore meaningful,
valuable, useful and relevant."
– Symbolic vs subjective
• "the meaning of statements as they are intended by the
speaker/writer”
• "related to meaning or human intention“
– Information as Decrease Of Entropy
• Information Theory: Lower uncertainty on the state of the world
16
WIS
The DIKW Pyramid
• Aspects of Knowledge
– “Knowledge is a fluid mix of framed experience, values,
contextual information, expert insight and grounded intuition
that provides an environment and framework for evaluating and
incorporating new experiences and information.”
– Processed Knowledge
• "synthesis of multiple sources of information over time"
• "a mix of contextual information, values, experience and rules"
– Procedural Knowledge
• "a mix of contextual information, expert opinion, skills and experience"
• "information combined with understanding and capability"
– Propositional Knowledge
• “Knowledge is a thought in the individual's mind, which is characterized by
the individual's justifiable belief that it is true. “
- Wallace, Danny P. (2007). Knowledge Management: Historical and Cross-Disciplinary Themes. Libraries Unlimited. pp. 1–14. ISBN 978-1-59158-502-2.
17
WIS
The DIKW Pyramid
• Wisdom….?
– Very fuzzy…
• “knowing the right things to do“
• "integrated knowledge — information made super-useful“
• "the ability to make sound judgments and decisions
apparently without thought"
- Wallace, Danny P. (2007). Knowledge Management: Historical and Cross-Disciplinary Themes. Libraries Unlimited. pp. 1–14. ISBN 978-1-59158-502-2.
- Chisholm, James; Warman, Greg (2007). "Experiential Learning in Change Management". In Silberman, Melvin L. The Handbook of Experiential Learning. Jossey
Bass. pp. 321–40. ISBN 978-0-7879-8258-4. 18
WIS
Example: Weather Forecast
• Data:
– “Raw observations or facts without context.”
– Temperature readings, humidity levels, wind
speed, and barometric pressure at various
locations.
• Information:
– “Data organized and processed to provide
context.”
– A weather report that combines temperature,
humidity, wind speed, and barometric
pressure data to describe current weather
conditions in a specific region.
19
WIS
Example: Weather Forecast
• Knowledge:
– “Information that is interpreted and understood in a broader context.”
– Meteorological analysis that explains the current weather patterns, predicts
future weather conditions, and identifies the factors influencing the weather.
• Wisdom:
– “Application of knowledge to make informed decisions.”
– Using the meteorological knowledge to make decisions such as advising
the public on potential weather hazards, helping farmers plan their planting
seasons, or guiding businesses in making weather-sensitive decisions.
20
WIS
The Modern World produces a humongous
amount of Data
21
Quiz: Data Processing you Encountered Today?
• Vevox - Dashboard
22
WIS
23
https://www.youtube.com/watch?v=qegdkl2GYrk 24
https://en.wikipedia.org/wiki/Smart_grid 25
https://beaudaniels.com/connected-car 26
https://www.axis.com/blog/secure-insights/what-smart-city/
27
https://www.geovision.com.tw/
28
29
https://deltalife.deltares.nl/september_2022/verzilting_de_trage_ramp
From Data to Information/Knowledge??
AI is here to fix it all!!!?
30
“Artificial intelligence (AI) will be the most powerful
technology in generations for benefiting humanity.”
34
WIS
The “hidden depth” of ML
Data Management
37
Into Databases!
WIS 38
WIS 39
Databases?
https://www.oracle.com/database/what-is-database/#:~:text=Is%20a%20Database%3F-,Database%20defined,database%20management%20system%20(DBMS). 40
WIS
Databases?
https://en.wikipedia.org/wiki/Database 41
WIS
Databases?
https://aws.amazon.com/what-is/database/
42
WIS
Databases?
https://www.techtarget.com/searchdatamanagement/definition/database
43
WIS
Database?
44
WIS
CSE1500 Overview
• Week 1
• Lecture: Introduction to Database Systems
• Lecture: Database Modelling
• Handout Assignment: DB01-Schemas
• Week 2
• Lecture: Database Modelling
• Lecture: Jumpstart SQL Query Language
• Handout Assignment: DB02-SQL
• Deadline Assignment: DB01-Schemas
• Week 3
• Lecture: Intermediate SQL Query Language
• Lecture: Data Models & The Relational Model
• Handout Assignment: DB03-Relational Model
• Deadline Assignment: DB02-SQL
• Week 4
• Lecture: The Relational Model, Functional Dependencies & Normalizations
• Lecture: Roundup & Q/A
• Deadline Assignment: DB02-SQL
• Week 5
• Midterm Exam: Database Technology
45
WIS
CSE1505 Overview
– CSE1505
• Internals of Relational Databases
– Traditional Implementation
– Physical Storage & Physical Models
– Query Processing & Query Optimization
– Transaction Processing
• NoSQL Systems
– Conceptual Difference
– Usage Scenarios
46
WIS
Goal of this course cycle…
• BSc: Web and Database Technologies
– Focus on how to USE databases
• SQL
• Play around with different products
– Learn how to make (simple!) Web applications
• BSc: Information and Data Management
– Focus on how (relational) databases work
– Theory, concepts, and foundations
• BSc: Machine Learning
• BSc Electives: Big Data Processing, Data Mining,
Computational Intelligence
• MSc Lectures
– Specialized topics like scalability, streams, Web, crowd, Data & AI, etc.
47
WIS
Towards the
Relational DBs
WIS 48
What are Data Model Theories?
• A Data Model (Theory) is an abstract model that
describes how data is represented, accessed, and
reasoned about
– Warning: The term “data model” is ambiguous
• A data model theory is a formal description of how data may be
structured and accessed and is independent of a specific software or
hardware.
• A data model instance or schema applies a data model theory to
create an instance for some application (e.g., data models in MySQL
Workbench designer refer to a logical model adapted to the MySQL
database).
• This will haunt us a lot…and it’s nobody’s fault really.
• I try to use data model (theory) and schema during this course to
lower the confusion…
– We re-visit this later!
49
WIS
Data Model Theories
• There are several common data models
– Relational Model
– Key-Value Model
– Document-centered Model
– Graph Model
– Etc.
50
WIS
Relational Model Example
• Core Idea: We store data in Tables!!
movie actor
m_id title year a_id f_name l_name
1 1 T-800
1 2 Sarah Connor
2 1 Julius Benedict
2 3 Vincent Benedict
movie actor
m_id title year a_id f_name l_name
3 Danny DeVito
cast
m_id a_id role
1 1 T-800
1 2 Sarah Connor
Rows Table (or also: “Relation”)
(or also “tuples”)
2 1 Julius Benedict
2 3 Vincent Benedict
52
WIS
Relational Model Example
movie actor
m_id title year a_id f_name l_name
1 1 T-800
1 2 Sarah Connor
2 1 Julius Benedict
2 3 Vincent Benedict
"twins":
{ "title" : "Twins", "linda":
"year" : 1988, { "first_name" : "Linda",
"director" : { "last_name" : "Hamilton",
"first_name" : "Ivan", "movies" : [
"last_name" : "Reitmann" } {
"actors": [ "title" : "The Terminator 2",
{"first_name" : "Arnold", "last_name" : "Schwarzenegger" }, "year" : "1991" }
{"first_name" : "Danny", "last_name" : "DeVito" }, ]
{"first_name" : "Kelly", "last_name" : "Preston" } }
]
}
54
WIS
Key-Value Store Example
Terminator 2
Twins 2
55
WIS
Graph Model Example
Terminator 2 –
mdfs:title mdb:Termiator2 mdfs:year 1991
Judgement Day
mdfs:cast
mdfs:cast
mdfs:f_name Linda
Arnold mdfs:f_name mdb:arni mdb:linda
mdfs:l_name Hamilton
Schwarzenegger mdfs:l_name
mdfs:cast
mdfs:cast
DeVito mdfs:l_name
56
WIS
Data Models
• A Data Model describes data
objects, operations and their effects
• Data Definition Language (DDL)
– How to structure data
• Data Manipulation Language (DML)
– How to add, edit, delete data
58
WIS
Informal Summary
• Database: “A collection of
related data represented (using a data
model and a defined data schema).”
• Database Management System (DBMS):
“A software system managing and maintaining a
database.”
• Data Model: “A formal definition on how to
represent data (in general) and the available
data operations.”
• Data Schema: “A definition of the structure of a
specific database.”
59
WIS
Overview
Relational
Databases
WIS 60
Reltional Databases
• Relational DBs alone: Industry of > €30 Billion a year
• DBs store & manage all (most) transactions in the world
• Simplify the life of application programmers
62
WIS
What are databases?
• Databases are used to manage vast
amount of data
• A database (DB) is a collection of related
data
– Data represents some aspects of the real world
• Universe of discourse
– Data is logically coherent
– It is provided for an intended group of users
and applications
63
WIS
Database Management Systems (DBMS)
id accNo
M N
firstname customer has account balance
type
lastname address
67
WIS
What are Relational Databases?
• Resulting tables (logical “model”)
Customer
ID firstname lastname address
DBMS
Disk 1 Disk 2
AccNo balance
1278945 € 312.10
2437954 € 1324.82
4543032 € -43.03
5539783 € 12.54
71
WIS
Data Independence cont.
• Example: Schema is changed, and table-
space moved without an application noticing
Application
DBMS
Disk 1 Disk 2
72
WIS
Views on the data
• Supports multiple views of the data
– Views provide a different perspective
of the DB
• A user’s conceptual understanding or
task-based excerpt of all data (e.g., aggregations)
• Security considerations and access control (e.g., projections)
– For the application, a view does not differ from a table
– Views may contain subsets of a DB and/or contain
virtual data
• Virtual data is derived from the DB (mostly by simple SQL
statements, e.g., joins over several tables)
• Can either be computed at query time or materialized upfront
73
WIS
Views on the data cont.
• Example view: Projection
– Saving account clerk vs. checking account clerk
Saving View
Original Table AccNo balance
AccNo type balance 1278945 € 312.10
1278945 saving € 312.10 2437954 € 1324.82
2437954 saving € 1324.82 5539783 € 12.54
4543032 checking € -43.03 9134354 € 2.22
5539783 saving € 12.54 9543252 € 524.89
7809849 checking € 7643.89
8942214 checking € -345.17 Checking View
AccNo balance
9134354 saving € 2.22
4543032 € -43.03
9543252 saving € 524.89
7809849 € 7643.89
8942214 € -345.17 74
WIS
Transactions: concurrently accessing data
75
WIS
What is a transaction?
• A transaction is a unit of work, possibly containing multiple
data accesses and updates, that must commit or abort as
a single unit, and follows the ACID principles
– e.g., transfer €100 from account A to account B
• Transactions introduce two problems:
– Recovery: What if the system fails in the middle of execution?
– Concurrency: What happens when two transactions try to access
the same object?
read(A, t);
t := t – 100;
write(t, A);
A = €400 read(B, t); A = €300
B = €200 t := t + 100; B = €300
write(t, B)
76
WIS
Transactions: the ACID principle
• Atomicity
– A transaction is either executed completely (commit) or not at all
(abort).
• Consistency
– A transaction transforms a consistent database state into a
(possibly different) consistent database state.
• Isolation
– A transaction is executed in isolation (i.e., does not see any effect
of other concurrently running (“uncommitted“) transactions).
• Durability
– A successfully completed (“committed“) transaction has a
permanent effect on the database.
77
WIS
Atomicity
• “All or nothing”
– E.g., failures during the transaction should be
handled
read(A, t);
t := t – 100;
write(t, A);
A = €400 ---CRASH!--- A = €300
B = €200 read(B, t); B = €200
t := t + 100;
write(t, B)
read(A, t);
t := t – 100;
write(t, A);
A = €400 read(B, t); A = €300
B = €200 t := t + 50; B = €250
write(t, B)
A + B = €600 A + B = €550
79
WIS
Isolation
• A transaction is executed in isolation i.e., does not see any effect of other
concurrently running (“uncommitted“) transactions.
• Example here: Two people try to transfer €100 from A to B at the same time.
T1 (husband) starts
read(A, t);
t := t – 100;
write(t, A);
T1 pauses
T2 (wife) starts
read(B, t);
A = €400 t := t – 100; A = €300
B = €200 T2 pauses B = €100
T1 restarts
read(B, t);
t := t + 100;
write(t, B)
T1 commits
T2 restarts
write(t, B);
T2 commits 80
WIS
Durability
• Transactions that reported that they finished
must have their results durable.
read(A, t);
t := t – 100;
write(t, A);
A = €400 read(B, t); A = €300
B = €200 t := t + 50; B = €200
write(t, B);
commit
---CRASH!---
81
WIS
Summary?
• Vevox - Dashboard
82
WIS
Summary
• In CSE1500/CSE1505, we care about
– Well-structured schemas, conceptional and logical
• Incl controlling data redundancy
– “Independence” of data and application
– Powerful Declarative Query Languages like SQL
– Query Optimization and Access Path Optimization
• e.g., incl Indexing
– Transaction Management and Recovery
83
WIS
DETOUR
Why should you care about IDM?
84
WIS
DETOUR
The Role of Relational Databases
85
WIS
DETOUR
The Role of Relational Databases
86
WIS
DETOUR
The Role of Relational Databases
WIS
??? 87
DETOUR
88
WIS
WIS
89