You are on page 1of 46

SQL Considered Harmful

Shannon Posniewski Cryptic Studios
GDC 2008

Current Relational DBMSs are Inappropriate for LowLatency, Controlled Working-Set Databases
and what we did trying to circumvent that
Shannon Posniewski and how that didn’t work out very well

Database Fundamentals
Structured collection of data “ACID” rules

Atomicity Consistency Isolation Durability

Database Types
Relational Object Object-Relational Hierarchical etc.

Relational Model
Defined mathematically Based on
set theory predicate logic

Relational Database
aka records aka entities

values for that entity

computer language for manipulating relational databases

Unique and Foreign Keys
Unique key in one table refers to other rows in another table. Used to represent hierarchical data

Player ID 818 237

PlayerNam … e Warthog … John …

AttribN foo bar

Player ID 818 818 237 818

Item Hat Brooch Monkey Paw Pteradactyl

Count 1 1 1 1

Player ID 818 237

Name Warthog John

… … …

Class ID 2 3

Class ID 1 2 3

Name Elf Orc Monkey

Provide all this and more Have object extensions, BLOBs, etc Used by practically everyone Safe, tested Lots of experience

Why not use it?

The Good
SQL servers optimized for general case Usually good at arbitrary queries Will always “work”

The Bad
Long query set up time Self-optimizes on the fly Unpredictable latency

The Ugly
Inserts are SLOW Per row overhead is high on updates

Object-Relational Impedance Mismatch
Hierarchy and references are typical Joins and normalization are often slow

The Evils of Normalization

Doing it the “the right way” can backfire

This was Bad
Player ID 818 818 818 818 Stat ID Time Online Fights Won Fights Lost … Stat N Value N Value 2,123 700 3

This was Good
Player ID 818 54 237 Stat 0 Value 2,123 675 5,235 Stat 1 Value 200 23 60 … … … … Stat N Value Value N Value N Value N

Abandons relational DB concepts

Solution: Caching
Built a write-through cache Reads come from the cache Writes go to cache and then to SQL All active players in memory (fast)

Not Enough
Still too slow Cache smoothes out latency issues Throughput still a problem

Reduce Throughput Needs
Don’t save HP, location, etc Snapshot instead

ACID is pretty much shot

Data Ownership
A process “owns” an entity’s data Vast majority of changes are local Changes are periodically flushed to DB

Cross-Owner Transactions
Flush both characters Still a window of failure We just did our best

Optimized and played tricks on the database… …and stopped getting the benefits of using one

It’s just not the right tool

Cryptic DB

Regain ACID (as much as possible) Low, predictable latency high read/write throughput minimize code-database impedance friendly to schema changes easy backup and restore

Offline character access can be slower Domain integrity responsibility of programmer General ad hoc queries can be slower SQL does not need to be

Structured text definition of data Generated from C structs Used for serializing static data Unifies static and DB data

AUTO_STRUCT AST_CONTAINER; typedef struct Entity { EntityRef myRef; NO_AST const ContainerID myContainerID; AST(PERSIST, KEY) Vec3 pos; AST(PERSIST, SERVER_ONLY, NO_TRANSACT) … } Entity;

Struct Metadata
Generates metadata on structs
memory layout data types

Generic mutators are possible
Cryptic DB

Basic Operations
Cryptic DB supports only a few basic operations Simple transactions built-in More complicated things need code

Data and Logic Ownership
A server “owns” the data for an entity Knows how to modify these entities

Transaction Server Cryptic DB

Team Server

Map Server Map Server Map Server

Cryptic Engine Transactions
Local if possible Dispatched by Transaction Server if not AUTO_TRANSACTIONS in C Field-level locking

C code plus special macros
AUTO_TRANSACTION; enumTransactionOutcome trGiveGold(ATR_ARGS, NOCONST(Entity) *pEnt1, NOCONST(Entity) *pEnt2, int amount) { if (pEnt1->gold < amount) TRANSACTION_RETURN(TRANSACTION_OUTCOME_FAILURE, ”NotEnoughGold”); pEnt1->gold -= amount; pEnt2->gold += amount; } return TRANSACTION_OUTCOME_SUCCESS;

Transaction Server

Step 1

Trans(“DoTrade”, A, B)



B Server 2

Server 1

Transaction Server
1 2 3 4 …

Step 2

1 2 Trans(“DoTrade”, 3 4 …

A, B)



B Server 2

Server 1

Transaction Server
1 2 3 4 …

Step 3


B Server 2

Server 1

Transaction Server
1 2 3 4 …

Step 4


B’ A’ Server 1 B Server 2


Transaction Server
1 2 3 4 …

Step 4

B dB B’ A Server 1 A’

dB B’ B Server 2

Transaction Server
1 2 3 4 …

Step 4

A’ Server 1 Server 2


Abstract Cryptic DB
Get transaction Log it to journal Apply change to in-memory DB Occasionally write out updated DB file Mark the journal resolved to that point

Copyright 2008 Cryptic Studios,

We’re hiring!

Apply Online!