SQL Considered

Harmful

Shannon Posniewski
Cryptic Studios

GDC 2008
 Current Relational
DBMSs are
Inappropriate for Low-
Latency, Controlled
Working-Set
Databases
and what we did trying to circumvent
that
and how that
Shannon didn’t work out very well
Posniewski
 Database
Fundamentals

Structured collection of data

“ACID” rules
 Atomicity
Consistency
Isolation
Durability
 Database Types

Relational
Object
Object-Relational
Hierarchical
etc.
 Relational Model

Defined mathematically
Based on
set theory
predicate logic
 Relational Database

Tables
Rows
aka records
aka entities
Columns
values for that entity
 SQL

computer language for

manipulating relational
databases
 Unique and Foreign
Keys

Unique key in one table refers to

other rows in another table.
Used to represent hierarchical
data
Player PlayerNam … AttribN
ID
818 e
Warthog … foo
237 John … bar

Player Item Count

ID
818 Hat 1
818 Brooch 1
237 Monkey 1
818 Paw
Pteradactyl 1
Player Name … Class ID
ID
818 Warthog … 2
237 John … 3

Class ID Name
1 Elf
2 Orc
3 Monkey
 SQL DBMSs

Provide all this and more

Have object extensions, BLOBs,
etc
Used by practically everyone
Safe, tested
Lots of experience
Why not use it?
 The Good

SQL servers optimized for general

case
Usually good at arbitrary queries
Will always “work”
 The Bad

Long query set up time

Self-optimizes on the fly
Unpredictable latency
 The Ugly

Inserts are SLOW

Per row overhead is high on
updates
 Object-Relational
Impedance Mismatch

Hierarchy and references are

typical
Joins and normalization are often
slow
 The Evils of
Normalization

Doing it the “the right way” can

backfire
 This was Bad

Player Stat ID Value

ID
818 Time 2,123
818 Online
Fights Won 700
818 Fights Lost 3
…
818 Stat N Value N
 This was Good

Player ID Stat 0 Stat 1 … Stat N

818 Value
2,123 Value
200 … ValueN
Value
54 675 23 … Value N
237 5,235 60 … Value N

Abandons relational DB
concepts
 Solution: Caching

Built a write-through cache

Reads come from the cache
Writes go to cache and then to
SQL
All active players in memory (fast)
 Not Enough

Still too slow

Cache smoothes out latency
issues
Throughput still a problem
 Reduce Throughput
Needs

Don’t save HP, location, etc

Snapshot instead

ACID is pretty much shot

 Data Ownership

A process “owns” an entity’s data

Vast majority of changes are local
Changes are periodically flushed
to DB
 Cross-Owner
Transactions

Flush both characters

Still a window of failure
We just did our best
Optimized and played
tricks on the
database…

…and stopped getting

the benefits of using
one
It’s just not the right tool
Cryptic DB
 Requirements

Regain ACID (as much as possible)

Low, predictable latency
high read/write throughput
minimize code-database
impedance
friendly to schema changes
easy backup and restore
 Relaxations

Offline character access can be

slower
Domain integrity responsibility of
programmer
General ad hoc queries can be
slower
SQL does not need to be
 StructParser

Structured text definition of data

Generated from C structs
Used for serializing static data
Unifies static and DB data
 Example

AUTO_STRUCT AST_CONTAINER;
typedef struct Entity
{

EntityRef myRef; NO_AST

const ContainerID myContainerID; AST(PERSIST, KEY)
Vec3 pos; AST(PERSIST, SERVER_ONLY, NO_TRANSACT)
…

} Entity;
 Struct Metadata

Generates metadata on structs

memory layout
data types
Generic mutators are possible
Cryptic DB
 Basic Operations
Cryptic DB supports only a few
basic operations

Simple transactions built-in

More complicated things need

code
 Data and Logic
Ownership

A server “owns” the data for an

entity

Knows how to modify these

entities
 Architecture

Transaction
Cryptic DB
Server

Team
Map Server
Server Map Server
Map Server
 Cryptic Engine
Transactions

Local if possible
Dispatched by Transaction Server
if not
AUTO_TRANSACTIONS in C
Field-level locking
 AUTO_TRANSACTION
C code plus special macros
AUTO_TRANSACTION;
enumTransactionOutcome trGiveGold(ATR_ARGS,
NOCONST(Entity) *pEnt1,
NOCONST(Entity) *pEnt2,
int amount)
{
if (pEnt1->gold < amount)
TRANSACTION_RETURN(TRANSACTION_OUTCOME_FAILURE,
”NotEnoughGold”);
pEnt1->gold -= amount;
pEnt2->gold += amount;

return TRANSACTION_OUTCOME_SUCCESS;
}
 Transaction
Server Step 1

Trans(“DoTrade”, A, B)

A DoTrade B
Server 1 Server 2
 Transaction
Server Step 2
1
2
3
4
…

1
Trans(“DoTrade”,
2 A, B)
3
4
…

A DoTrade B
Server 1 Server 2
 Transaction
Server Step 3
1
2
3
4
…

A DoTrade B
Server 1 Server 2
 Transaction
Server Step 4
1
2
3
4
…

B B’
DoTrade

A A’ B
Server 1 Server 2
 Transaction
Server Step 4
1
2
3
4
…

B
dB
dB
B’
B’
A A’ B

Server 1 Server 2
 Transaction
Server Step 4
1
2
3
4
…

A’ B’

Server 1 Server 2
 Abstract Cryptic DB

Get transaction
Log it to journal
Apply change to in-memory DB
Occasionally write out updated DB
file
Mark the journal resolved to that
point
Copyright 2008 Cryptic Studios,
We’re hiring! Apply Online!