Schema Design Alvin Richards alvin@10gen.

com

Topics
Introduction • Basic Data Modeling • Evolving a schema Common patterns • Single table inheritance • One-to-Many & Many-to-Many • Trees • Queues

So why model data?

http://www.flickr.com/photos/42304632@N00/493639870/

A brief history of normalization
• 1970 E.F.Codd introduces 1st Normal Form (1NF) • 1971 E.F.Codd introduces 2nd and 3rd Normal Form (2NF, 3NF) • 1974 Codd & Boyce define Boyce/Codd Normal Form (BCNF) • 2002 Date, Darween, Lorentzos define 6th Normal Form (6NF) Goals: • Avoid anomalies when inserting, updating or deleting • Minimize redesign when extending the schema • Make the model informative to users • Avoid bias towards a particular style of query

* source : wikipedia

The real benefit of relational
• Before relational • After relational

• Data and Logic combined • Separation of concerns • Data modeled independent of logic • Logic freed from concerns of data design

• MongoDB continues this separation

Relational made normalized data look like this

Document databases make normalized data look like this

Terminology
RDBMS Table Row(s) Index Join Partition Partition  Key MongoDB Collection JSON  Document Index Embedding  &  Linking Shard Shard  Key

DB Considerations
How can we manipulate this data ? Access Patterns ?

• Dynamic Queries • Secondary Indexes • Atomic Updates • Map Reduce

• Read / Write Ratio • Types of updates • Types of queries • Data life-cycle

Considerations • No Joins • Document writes are atomic

So today’s example will use...

Design Session
Design documents that simply map to your application
post  =  {author:  “Hergé”,                date:  new  Date(),                text:  “Destination  Moon”,                tags:  [“comic”,  “adventure”]} >  db.post.save(post)

Find the document
>  db.posts.find()    {  _id:  ObjectId("4c4ba5c0672c685e5e8aabf3"),        author:  "Hergé",          date:  "Sat  Jul  24  2010  19:47:11  GMT-­‐0700  (PDT)",          text:  "Destination  Moon",          tags:  [  "comic",  "adventure"  ]    }     Notes: • ID must be unique, but can be anything you’d like • MongoDB will generate a default ID if one is not supplied

Add and index, find via Index
Secondary index for “author”  //      1  means  ascending,  -­‐1  means  descending  >  db.posts.ensureIndex({author:  1})  >  db.posts.find({author:  'Hergé'})          {  _id:  ObjectId("4c4ba5c0672c685e5e8aabf3"),          date:  "Sat  Jul  24  2010  19:47:11  GMT-­‐0700  (PDT)",          author:  "Hergé",            ...  }

Verifying indexes exist
>  db.system.indexes.find() //  Index  on  ID    {  name:  "_id_",          ns:  "test.posts",          key:  {  "_id"  :  1  }  } //  Index  on  author    {  _id:  ObjectId("4c4ba6c5672c685e5e8aabf4"),          ns:  "test.posts",          key:  {  "author"  :  1  },          name:  "author_1"  }

Examine the query plan
>  db.blogs.find({author:  'Hergé'}).explain() {   "cursor"  :  "BtreeCursor  author_1",   "nscanned"  :  1,   "nscannedObjects"  :  1,   "n"  :  1,   "millis"  :  5,   "indexBounds"  :  {     "author"  :  [       [         "Hergé",         "Hergé"       ]     ]   } }

Query operators
Conditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne,

//  find  posts  with  any  tags >  db.posts.find({tags:  {$exists:  true}})

Query operators
Conditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne,

//  find  posts  with  any  tags >  db.posts.find({tags:  {$exists:  true}})
Regular expressions:

//  posts  where  author  starts  with  h >  db.posts.find({author:  /^h/i  })  

Query operators
Conditional operators: $ne, $in, $nin, $mod, $all, $size, $exists, $type, .. $lt, $lte, $gt, $gte, $ne,

//  find  posts  with  any  tags >  db.posts.find({tags:  {$exists:  true}})
Regular expressions:

//  posts  where  author  starts  with  h >  db.posts.find({author:  /^h/i  })  
Counting:

//  number  of  posts  written  by  Hergé >  db.posts.find({author:  “Hergé”}).count()

Extending the Schema
     
 new_comment  =  {author:  “Kyle”,                                  date:  new  Date(),                                text:  “great  book”}

 >  db.posts.update(                      {text:  “Destination  Moon”  },                        {  ‘$push’:  {comments:  new_comment},                          ‘$inc’:    {comments_count:  1}})

Extending the Schema
 

   {  _id  :  ObjectId("4c4ba5c0672c685e5e8aabf3"),          author  :  "Hergé",        date  :  "Sat  Jul  24  2010  19:47:11  GMT-­‐0700  (PDT)",          text  :  "Destination  Moon",        tags  :  [  "comic",  "adventure"  ],                comments  :  [   {     author  :  "Kyle",     date  :  "Sat  Jul  24  2010  20:51:03  GMT-­‐0700  (PDT)",     text  :  "great  book"   }        ],        comments_count:  1    }
   

Extending the Schema
//  create  index  on  nested  documents: >  db.posts.ensureIndex({"comments.author":  1}) >  db.posts.find({comments.author:”Kyle”})

Extending the Schema
//  create  index  on  nested  documents: >  db.posts.ensureIndex({"comments.author":  1}) >  db.posts.find({comments.author:”Kyle”}) //  find  last  5  posts: >  db.posts.find().sort({date:-­‐1}).limit(5)

Extending the Schema
//  create  index  on  nested  documents: >  db.posts.ensureIndex({"comments.author":  1}) >  db.posts.find({comments.author:”Kyle”}) //  find  last  5  posts: >  db.posts.find().sort({date:-­‐1}).limit(5) //  most  commented  post: >  db.posts.find().sort({comments_count:-­‐1}).limit(1)

When sorting, check if you need an index

Watch for full table scans
>  db.blogs.find({text:  'Destination  Moon'}).explain()     {   "cursor"  :  "BasicCursor",   "nscanned"  :  1,   "nscannedObjects"  :  1,   "n"  :  1,   "millis"  :  0,   "indexBounds"  :  {       } }

Map Reduce

Map reduce : count tags
mapFunc  =  function  ()  {        this.tags.forEach(function  (z)  {emit(z,  {count:1});}); } reduceFunc  =  function  (k,  v)  {        var  total  =  0;        for  (var  i  =  0;  i  <  v.length;  i++)  {                total  +=  v[i].count;  }        return  {count:total};   }
res  =  db.posts.mapReduce(mapFunc,  reduceFunc)

>db[res.result].find()          {  _id  :  "comic",  value  :  {  count  :  1  }  }          {  _id  :  "adventure",  value  :  {  count  :  1  }  }    

Group
• Equivalent to a Group By in SQL • Specific the attributes to group the data • Process the results in a Reduce function

Group - Count post by Author
cmd  =  {  key:  {  "author":true  },                initial:  {count:  0},                reduce:  function(obj,  prev)  {                                prev.count++;                            },            }; result  =  db.posts.group(cmd);
[

                ]

{     }, {     }

"author"  :  "Hergé", "count"  :  1

"author"  :  "Kyle", "count"  :  3

Review
So Far: - Started out with a simple schema - Queried Data - Evolved the schema - Queried / Updated the data some more

http://devilseve.blogspot.com/2010/06/like-drinking-from-fire-hose.html

Inheritance

Single Table Inheritance - RDBMS
shapes table id type
1 area radius d 1 length width

circle 3.14

2

square 4

2

3

rect

10

5

2

Single Table Inheritance MongoDB
>  db.shapes.find()
 {  _id:  "1",  type:  "circle",area:  3.14,  radius:  1}  {  _id:  "2",  type:  "square",area:  4,  d:  2}  {  _id:  "3",  type:  "rect",    area:  10,  length:  5,  width:  2}

Single Table Inheritance MongoDB
>  db.shapes.find()
 {  _id:  "1",  type:  "circle",area:  3.14,  radius:  1}  {  _id:  "2",  type:  "square",area:  4,  d:  2}  {  _id:  "3",  type:  "rect",    area:  10,  length:  5,  width:  2}

//  find  shapes  where  radius  >  0   >  db.shapes.find({radius:  {$gt:  0}})

Single Table Inheritance MongoDB
>  db.shapes.find()
 {  _id:  "1",  type:  "circle",area:  3.14,  radius:  1}  {  _id:  "2",  type:  "square",area:  4,  d:  2}  {  _id:  "3",  type:  "rect",    area:  10,  length:  5,  width:  2}

//  find  shapes  where  radius  >  0   >  db.shapes.find({radius:  {$gt:  0}}) //  create  index >  db.shapes.ensureIndex({radius:  1})

One to Many
One to Many relationships can specify • degree of association between objects • containment • life-cycle

One to Many
- Embedded Array / Array Keys - slice operator to return subset of array - some queries hard e.g find latest comments across all documents

One to Many
- Embedded Array / Array Keys - slice operator to return subset of array - some queries hard e.g find latest comments across all documents - Embedded tree - Single document - Natural - Hard to query

One to Many
- Embedded Array / Array Keys - slice operator to return subset of array - some queries hard e.g find latest comments across all documents - Embedded tree - Single document - Natural - Hard to query - Normalized (2 collections) - most flexible - more queries

One to Many - patterns

- Embedded Array / Array Keys

- Embedded Array / Array Keys - Embedded tree - Normalized

Many - Many
Example:
- Product can be in many categories - Category can have many products

Many - Many
products:
     {  _id:  ObjectId("4c4ca23933fb5941681b912e"),          name:  "Destination  Moon",          category_ids:  [  ObjectId("4c4ca25433fb5941681b912f"),                                          ObjectId("4c4ca25433fb5941681b92af”]}

   

Many - Many
products:
     {  _id:  ObjectId("4c4ca23933fb5941681b912e"),          name:  "Destination  Moon",          category_ids:  [  ObjectId("4c4ca25433fb5941681b912f"),                                          ObjectId("4c4ca25433fb5941681b92af”]}

   
categories:
     {  _id:  ObjectId("4c4ca25433fb5941681b912f"),            name:  "adventure",            product_ids:  [  ObjectId("4c4ca23933fb5941681b912e"),                                        ObjectId("4c4ca30433fb5941681b9130"),                                        ObjectId("4c4ca30433fb5941681b913a"]}

Many - Many
products:
     {  _id:  ObjectId("4c4ca23933fb5941681b912e"),          name:  "Destination  Moon",          category_ids:  [  ObjectId("4c4ca25433fb5941681b912f"),                                          ObjectId("4c4ca25433fb5941681b92af”]}

   
categories:
     {  _id:  ObjectId("4c4ca25433fb5941681b912f"),            name:  "adventure",            product_ids:  [  ObjectId("4c4ca23933fb5941681b912e"),                                        ObjectId("4c4ca30433fb5941681b9130"),                                        ObjectId("4c4ca30433fb5941681b913a"]}

//All  categories  for  a  given  product >  db.categories.find({product_ids:  ObjectId ("4c4ca23933fb5941681b912e")})

Alternative
products:      {  _id:  ObjectId("4c4ca23933fb5941681b912e"),          name:  "Destination  Moon",          category_ids:  [  ObjectId("4c4ca25433fb5941681b912f"),                                          ObjectId("4c4ca25433fb5941681b92af”]}

   
categories:      {  _id:  ObjectId("4c4ca25433fb5941681b912f"),            name:  "adventure"}

Alternative
products:      {  _id:  ObjectId("4c4ca23933fb5941681b912e"),          name:  "Destination  Moon",          category_ids:  [  ObjectId("4c4ca25433fb5941681b912f"),                                          ObjectId("4c4ca25433fb5941681b92af”]}

   
categories:      {  _id:  ObjectId("4c4ca25433fb5941681b912f"),            name:  "adventure"} //  All  products  for  a  given  category >  db.products.find({category_ids:  ObjectId ("4c4ca25433fb5941681b912f")})    

Alternative
products:      {  _id:  ObjectId("4c4ca23933fb5941681b912e"),          name:  "Destination  Moon",          category_ids:  [  ObjectId("4c4ca25433fb5941681b912f"),                                          ObjectId("4c4ca25433fb5941681b92af”]}

   
categories:      {  _id:  ObjectId("4c4ca25433fb5941681b912f"),            name:  "adventure"} //  All  products  for  a  given  category >  db.products.find({category_ids:  ObjectId ("4c4ca25433fb5941681b912f")})   //  All  categories  for  a  given  product product    =  db.products.find(_id  :  some_id) >  db.categories.find({_id  :  {$in  :  product.category_ids}})  

Trees
Full Tree in Document
{  comments:  [          {  author:  “Kyle”,  text:  “...”,                replies:  [                                            {author:  “Fred”,  text:  “...”,                                              replies:  []}                ]}    ] }

Pros: Single Document, Performance, Intuitive Cons: Hard to search, Partial Results, 4MB limit
   

Trees
Parent Links - Each node is stored as a document - Contains the id of the parent Child Links - Each node contains the id’s of the children - Can support graphs (multiple parents / child)

Array of Ancestors
- Store all Ancestors of a node    {  _id:  "a"  }    {  _id:  "b",  ancestors:  [  "a"  ],  parent:  "a"  }    {  _id:  "c",  ancestors:  [  "a",  "b"  ],  parent:  "b"  }    {  _id:  "d",  ancestors:  [  "a",  "b"  ],  parent:  "b"  }    {  _id:  "e",  ancestors:  [  "a"  ],  parent:  "a"  }    {  _id:  "f",  ancestors:  [  "a",  "e"  ],  parent:  "e"  }

Array of Ancestors
- Store all Ancestors of a node    {  _id:  "a"  }    {  _id:  "b",  ancestors:  [  "a"  ],  parent:  "a"  }    {  _id:  "c",  ancestors:  [  "a",  "b"  ],  parent:  "b"  }    {  _id:  "d",  ancestors:  [  "a",  "b"  ],  parent:  "b"  }    {  _id:  "e",  ancestors:  [  "a"  ],  parent:  "a"  }    {  _id:  "f",  ancestors:  [  "a",  "e"  ],  parent:  "e"  } //find  all  descendants  of  b: >  db.tree2.find({ancestors:  ‘b’}) //find  all  direct  descendants  of  b: >  db.tree2.find({parent:  ‘b’})

Array of Ancestors
- Store all Ancestors of a node    {  _id:  "a"  }    {  _id:  "b",  ancestors:  [  "a"  ],  parent:  "a"  }    {  _id:  "c",  ancestors:  [  "a",  "b"  ],  parent:  "b"  }    {  _id:  "d",  ancestors:  [  "a",  "b"  ],  parent:  "b"  }    {  _id:  "e",  ancestors:  [  "a"  ],  parent:  "a"  }    {  _id:  "f",  ancestors:  [  "a",  "e"  ],  parent:  "e"  } //find  all  descendants  of  b: >  db.tree2.find({ancestors:  ‘b’}) //find  all  direct  descendants  of  b: >  db.tree2.find({parent:  ‘b’}) //find  all  ancestors  of  f: >  ancestors  =  db.tree2.findOne({_id:’f’}).ancestors >  db.tree2.find({_id:  {  $in  :  ancestors})

Trees as Paths
Store hierarchy as a path expression - Separate each node by a delimiter, e.g. “/” - Use text search for find parts of a tree
{  comments:  [          {  author:  “Kyle”,  text:  “initial  post”,                path:  “/”  },          {  author:  “Jim”,    text:  “jim’s  comment”,              path:  “/jim”  },          {  author:  “Kyle”,  text:  “Kyle’s  reply  to  Jim”,              path  :  “/jim/kyle”}  ]  } //  Find  the  conversations  Jim  was  part  of   >  db.posts.find({path:  /^jim/i})

Queue
Requirements • See jobs waiting, jobs in progress • Ensure that each job is started once and only once      {  inprogress:  false,          priority:  1,        ...      }

Queue
Requirements • See jobs waiting, jobs in progress • Ensure that each job is started once and only once      {  inprogress:  false,          priority:  1,        ...      } //  find  highest  priority  job  and  mark  as  in-­‐progress job  =  db.jobs.findAndModify({                              query:    {inprogress:  false},                              sort:      {priority:  -­‐1),                                update:  {$set:  {inprogress:  true,                                                                started:  new  Date()}},                              new:  true})    

Remember me?

http://devilseve.blogspot.com/2010/06/like-drinking-from-fire-hose.html

Summary
Schema design is different in MongoDB Basic data design principals stay the same Focus on how the apps manipulates data Rapidly evolve schema to meet your requirements Enjoy your new freedom, use it wisely :-)

download at mongodb.org

We’re Hiring !
alvin@10gen.com
conferences,  appearances,  and  meetups
http://www.10gen.com/events

http://bit.ly/mongo>  

Facebook                    |                  Twitter                  |                  LinkedIn
@mongodb

http://linkd.in/joinmongo

Sign up to vote on this title
UsefulNot useful