Databases for Online Gaming (MMOG)

gklein@versant.com
Manager Engineering, CARE Group

Copyright Versant 2007 All products are trademarks or registered trademarks of the respective companies . The information contained in this document is property of Versant.

Agenda
Introduction Persistent World Strategies
Scalability Distribution High Availability

Copyright Versant 2007 - 2

Versant Corporate Profile

Established 1988, HQ in Fremont, CA 1996 NASDAQ: VSNT Leading provider of 7th generation enterpriselevel object database
Installed base is Global 500 whos-who Focus is on performance and scalability improvement of complex large scale enterprise systems

Offices: U.S., Germany, India, Japan

Copyright Versant 2007 - 3

The main takeaway: Versant has delivered enterprise level solutions to the Global 500 for 14 years.

MMOG
Massively Multiplayer Online Game
is capable of supporting hundreds or thousands of players simultaneously on the Internet is played in a giant persistent (fantasy) world
Role Games (MMORG) Tycoon Games (MMOTG) and First Person Shooter (MMOFPS) Real Time Strategy (MMORTS) Sport Games (MMOSG) and Racing (MMOR) Social Games (MMOSG)

Business Model
Most MMOGs require a monthly subscription fee Some can be played for free
Copyright Versant 2007 - 4

Persistent World
Is a virtual world
is always available world events happen continually
events continue to develop even while some of the players are not playing their character

a player's character can also influence and change a persistent world

The state of the gaming world has to be maintained around the clock
Copyright Versant 2007 - 5

Definition en.wiki.org: A persistent world is a virtual world (often in a fantasy setting) that is used as a setting for a role-playing game, often Online. The world is always available and world events happen continually. The persistency comes from maintaining and developing the state of the gaming world around the clock. Unlike with other games, a persistent world game's plots and events continue to develop even while some of the players are not playing their character. The comparison is to the real world where events occur that are not directly connected to a person, or continue to happen while a person sleeps, etc. Likewise, a player's character can also influence and change a persistent world. The degree to which a character can affect a world varies from game to game. (Since the game does not pause nor create player-accessible back-up files, a character's actions will have consequences that the player must deal with.)

Characteristics of MMOG
Within minutes, players have an uncanny ability to discover the size of the possible space, Will Wright Players are the central focus
+ artificial intelligence used in the older-technology games

Fulltime availability unlimited content Performance, Performance, Performance

Market is highly competitive

MMOG need to adopt strategies to provide tremendous amounts of content to the users
Copyright Versant 2007 - 6

Storing Persistent Worlds

MMOG is one of the most demanding, large scale applications in the world
Evolutionary game models using OO languages
Early models were quite simple and complexity is increasing > 95% written in C++

Scalability High concurrency and transaction throughput

Real Time response
High Load at peak times

Large data set Distributed State management Parallel Processing Data Clustering
Copyright Versant 2007 - 7

High availability

A SUCCESSFUL Massive Multi-Player Online Game (MMOG) falls squarely into the category of one of the most demanding, large scale applications in the world. The MMOG's combined requirements of realtime response, high concurrency and large data sets can bring traditional technologies and architectures to their knees. This point is evidenced by the public pain of successful games like Ultima Online, EverQuest, and World of WarCraft. It is well documented that the effectiveness of the data tier will drive game scalability and system operating footprint and therefore operational expense. Traditional technology in the data tier consists of a combination of File System for static content and Relational Databases (RDB) for dynamic content. It is well documented that under the requirements of real-time response, high concurrency and large data, relational database technology does not scale well. It is also well documented in that the right Object Oriented Database (ODB), for reasons to be discussed here in, does scale well under those MMOG requirements.

Brief History of the Database

Model Hierarchical Network Relational ODBMS Strength Performance Performance Flexibility, queries Performance, flexibility Weakness Flexibility, queries Flexibility, queries Performance Random, ad hoc queries Data Simple Simple Simple Complex

Very important for MMOG problem domains

Copyright Versant 2007 - 8

Storing Objects
Transient Objects

Persistent World
Cache

Persistent Objects

Transparently writes changes back to the Database(s)

Transparently fetches objects from the database(s)

Direct object persistence Vs. Mapping

Versant

Navigational versus query-based access

Copyright Versant 2007 - 9

MMOGS and so the persistent world are mainly developed with C++, leveraging OO concepts. The persistent world and game logic uses the objects and the database provides a transparent access to virtually all objects. The developer can use the OO language constructs like Inheritance associations, collections and polymorphism to model the problem domains and implement it seamlessly. With Versant the persistent objects are loaded either as result of a query or most of the time as part of a navigation. The caching mechanism is very sophisticated and provides the developer with additional controls, so that he can develop own strategies. Once an object is modified it will be automatically be written to the database with the next commit. The objects will be stored as objects and no mapping is required. Versant provides 2 access patterns: navigation and selection. Often many operations are natural navigations like getting associated objects. With RDBMS these navigations have to be translated into query/join statements.

Using the RDBMS

RDBMS
SELECT FROM table1 ...; SELECT FROM table2 ...; convert_tables_to_memory(); BackPack bPack = new BackPack() for(i=0; i<100; i++) { BackPackItemp = new BackPackItem(); bPack.addItem(p) } convert_memory_to_tables(); INSERT INTO table1 ...; UPDATE table2 ...;

In-Memory

Tables

Different in-memory and storage models Significantly degraded performance when (de)composing data Even more conversion code Separate programming and data manipulation languages Only suitable for simple models
Difficult if schema evolution is required
Copyright Versant 2007 - 10

10

ODBMS Solution
In-Memory
BackPack bPack = new BackPack() for(i=0; i<100; i++) { BackPackItemp = new BackPackItem(); b.addItem(p) }

VERSANT

In-memory and storage models are the same Orders of magnitude faster No additional conversion code Improved programmer productivity Host programming language is also DML and DDL
Copyright Versant 2007 - 11

Versant also provides advantages in the area of flexibility in evolving the existing database schema and in creating dynamic game object characteristics. This flexibility is achieved through support of both easier design and model extensibility. Versant deals with object oriented models in the proper polymorphic forms of the language and treats references to related objects orthogonal to the state of the objects involved. This means that you can write game logic to deal with interfaces or abstract classes that are then further derived to create the specialized classes for your game without introducing needlessly complex application and mapping code. Performance optimizations created at the abstract level allow you to add new game classes while keeping your performance optimizations intact. The original game logic incorporates new game object use with minimal code modification. Game object attributes can be easily managed through objectified relationships without introducing complicated coding semantics. A simple design example could be the modeling of a character Backpack. One obvious design when properly using OO would be to model the Backpack as an Item that contains other Items. The design uses both inheritance and self referencibility. This design is handled seamlessly using Versant and retrieval of a Backpack's contained Items requires nothing more than a message send in the game logic code. This same design in a relational form, even if flattening of the Backpack into a single Item table, forces performance draining JOIN operations due to the self referencibility. A more complex extensibility example could be the creation of an entirely new game character by subclassing an existing class that describes general character behavior. The addition of the new character class can be deployed to the database server on the fly. The Versant Object Database will incorporate the new schema and evolve the database automatically, and existing game logic will continue to operate uninterrupted bringing the new character into the game with minimal effort.

11

Example Architecture

NPC (non-playable character) FE Front End BE Back End

Copyright Versant 2007 - 12

The example MMOG Architecture is divided into three main layers: Game Server Layer ( clusterable game engine and the communications layers ) and the Database Layer. The game server is designed over a proxy data tier composing direct and clustered data access services. The architecture is designed to support the addition of new users ( via Game FE machines ) who can play seamlessly across the entire world with all other users within a Game Server Layer, without replicating the entire system hardware. This is in direct contrast to traditional MMOG architectures which scale by replicating the entire world ( hardware infrastructure ) and isolating new users to their given world. This approach enables in game events composed of 100's of thousands of users. Further, additional game server layers can be clustered and although direct game play across server layers is not possible, all game resources like money, land, weapons, etc are common across servers. The design provides for cost effective horizontal scalability enabled via a highly concurrent, super scalable database tier.

12

Scalability
Vertically
Single hardware node can maintain a set capacity of operations und required performance criteria by adding resources to the node Direct correlation to costs , f.e. you can support 5000 concurrent user on a single Xeon CPU server with 2G RAM

Horizontal
When reaching a tolerable threshold of vertical scalability continued capacity of operations expansion can be achieved by adding additional hardware nodes.

Goal:
Maximize vertically scalability Ensure Horizontal scalability to provide unlimited expansion and

Copyright Versant 2007 - 13

When designing an MMOG software system for scalability, one should consider the ability to scale both vertically and horizontally. Vertical scalability means that a single hardware node in the software system can maintain a set capacity of operations under required performance criteria by adding resources to the hardware node. Horizontal scalability means that when reaching the tolerable threshold of vertical scalability, continued capacity of operations expansion can be achieved by adding additional hardware nodes. Maximizing Vertical scalability is important as this generally has a direct correlation to profit margins as a given configuration of hardware and software infrastructure supports a fixed number of users which translates into revenue. Additionally, maximizing vertical scalability reduces the complexity and overhead of maintainability and further improves margins. Ensuring horizontal scalability over vertical scalability means that unlimited expansion is possible with predictable margins. Versant enables developers of MMOG applications to implement both vertical and horizontal scalability through an advanced multi-threaded architecture integrated with a number of specific features described below.

13

Distribute data and workload

Strategies
Reduce the number of active data and users per database
Distributed data and load Offload Obsolete information that will be deleted after a while, migrate them to an archive db Reduce number of conflicting operations (Query vs. Updates)

Data Caching
Caching and In Memory DBs
Copyright Versant 2007 - 14

14

Shards or Grids
Shards
Split game into a number of parallel worlds through clustered servers Distribute data on different databases Users can only see subset
need distributed access or propagation of new data

Grids
F.e. each resource is responsible for an square area in the persistent world
It knows its direct 4 neighbors (right, left, up and down) and if a character moves to another square, the responsibility also moves.
Copyright Versant 2007 - 15

A significant advantage of Versant is the ability to cache all or portions of the database in the game shard memory space and/or proxy data server for real-time read response. This shard caching is critical for meeting the minimized latency requirements of an MMOG and significantly improves vertical scalability by off-loading activity from the database server processes relieving contention for shared resources. The "native" persistence capabilities also provide transparency of access. For example, if a character game object is loaded into the shard, either by id or query, then in the game logic a message is sent to the charter to retrieve a wallet for money, if the wallet did not exist in the shard memory space, Versant will transparently retrieve that wallet from the database server memory space without the application executing any special query code for the wallet retrieval. Game objects are implicitly assigned system global id's so that they can be retrieved without impacting application logic. There are also capabilities in Versant so when a game object is retrieved from the server memory space, a configurable number of related game objects will automatically be retrieved into the shard to guarantee a subsequent call to the database server memory is not required, thereby avoiding unnecessary network calls.

15

A Transparently Distributed Database

Game Shard Cache B D TCP/IP C A

Relationships maintained across persistent stores

B D

Object

Object

ObjectObject Object

ObjectObject C

Object

Object

Versant

Versant

Versant

Copyright Versant 2007 - 16

Versant is a fully distributed architecture. The distribution is essentially transparent to the developer. Objects can reside in multiple persistent stores and be used transparently by a single or multiple clients. Objects in one persistent store may refer to objects in another persistent store and the bean developer can directly access the referenced objects without any additional coding. The only requirement is that your session have a connection to the persistent store where the object resides. Versant will then find it for you. Taking advantage of such transparent distribution allows for seem less implementation of data aging strategies and active data sets. This also allows for advanced partitioning strategies for disk bound systems where expensive options like EMC arrays are not feasible.

16

Parallel Processing with distributed databases (One Shard)

Copyright Versant 2007 - 17

Parallel Processing: Versant provides the ability to incrementally add physical databases to a cluster which are treated as parallel processing nodes in a distributed system. Operations performed on the nodes are optimized using threading and asynchronous communications to provide parallel operations for optimal performance. For example, the Game logic can define many logical groupings of physical databases and treat them each as a virtual datastore. SQL92 type indexed queries can be targeted at a logical database and are executed in parallel within the physical in-memory databases representing the virtual datastore. By controlling the size of any physical database node within the cluster, the game logic can guarantee response times by controlling concurrent access and internal structure traversal times. Unlike in traditional relational databases where queries target a table and its columns, returning records that need to be mapped into your language objects, the Versant database queries target a class and its attributes, returning objects of that class without any intermediate application mapping. Once the queried objects are retrieved into the game logic, access to related objects occurs through the normal message send process of the language. An example is shown in the following figure. A query can be performed across many in-memory databases in parallel to find a particular player. Once the player object is returned by the query, the game logic sends a message to the player to get the related backpack. There can be specialized messages defined so that whenever a backpack is retrieved it's contained items are also automatically retrieved. The message getBackpack is sent to the player and the Versant database will ensure that the related backpack and it's contained items are retrieved into the shard cache. The Versant database handles all of the issues dealing with locality of the related objects in the physical cluster. There is no required code in the game logic to deal with determining where objects are physically located. Despite t he fact that the backpack and its items are not in the same physical database as the player, they are still transparently returned on message send to the shard cache. There is no limit to the type of architectural flexibility gained with this design, allowing any type of partitioning scheme deemed appropriate for your game design. Versant guarantees uniqueness of object identity across the entire cluster and maintains the physical locality orthogonal to the logical identity. Even if external processes are put in place to move objects from one physical database to another, there will be no impact to gaming logic. Versant even handles the logical and physical consistency of backups across the distributed cluster.

17

Distributed State Management (Multiple Shards)

Multiple shards may use same data that can be changed on one side Cache Coherence needs to be ensured

Copyright Versant 2007 - 18

Versant has another feature working in concert with the real-time shard cache that facilitates distributed state management across multiple shards. Shared game objects in the shard memory space can be identified so that changes made to the shared game objects cause cache invalidation notifications to be broadcast to all shards. All shards in the cluster who have those shared objects will then automatically refresh the shared objects from the database server memory to ensure a consistent and up to date game state across the shard cluster. This allows unprecedented horizontal scalability for large numbers of users with controlled cost structure by allowing incremental addition of shards as the player demand and concurrency builds. Further, network communications routing can use load balancing to take advantage of this clustering by reassigning incoming requests for better response or failover without incurring significant performance penalties. In addition, this allows the design of a game where players and related game objects can effectively "walk" shard boundaries for uninhibited game play.

18

Optimizing TX throughput

In Memory database caches data read from disk Typically the logging of tx is the bottleneck (ACID) With RDBMS also the query execution and caching becomes a bottleneck

Copyright Versant 2007 - 19

Transaction throughput: Versant has the unique ability to provide in-memory performance characteristics while maintaining the ACID (Atomicity, Consistency, Isolation, Durability ) properties of an enterprise level database management system. Versant makes this possible by leveraging synchronous in memory databases coupled with the ability to recover in-flight transaction failures. Versant uses asynchronous socket communications to provide highly efficient 2-phase commit protocol across distributed in-memory databases. Further, commit calls to all databases are non-blocking allowing for immediate return to application control with only the overhead of a network call. Contents of commit operations are stored in memory based transaction log buffers which are asynchronously flushed to the disk based logging subsystem by background threads every N ( configurable ) number of commits. Versant maintains durability for transactions by ensuring that transactional content resides synchronously in multiple buffer spaces in a shared nothing architecture. The rapid database server response, due to this architecture, facilitates unprecedented vertical scalability by minimizing contention for internal structures in the server processes and eliminating waiting due to I/O operations. The result is a database node that can support thousands of concurrent update transactions. The game shard manager tracks all CRUD ( create, read, update, delete ) operations in the game logic. When the game logic calls commit, Versant uses asynchronous socket calls to get the changes ( results of the transactional CRUD operations ) to flow towards both in memory databases. The transactional changes are written into piggy backed double buffers . Piggy backed double buffers work in a way that a buffer is always available for writing even when one side of the buffer is being flushed to disk. Once the server threads have written into the piggy back buffers, control is immediately returned to the shard, so that realtime processing can continue. When the configured buffer flush threshold is reached, the piggy back buffers reverse, new transactions are written into the opposite side and an asynchronous thread flushes changes to disk. This process continues during normal operations. If any failure should occur, Versant recovers on the fly as described in the section below on high availability.

19

Fault Tolerant Database Configuration

Copyright Versant 2007 - 22

High Availability: Using Versant, if a failure occurs in a database server process due to network, hardware, software process failure, the shard will transparently receive the failure notification and change it's mode of operation to continue the failed transaction only to the remaining live server. Versant handles in-flight failures from either primary or secondary site transparently. During failover operation, the shard sends additional transaction information to the live server to be used for automatic background restoration of the failed site. The live site forks a polling process that seeks to start the failed database server. When the failed site is reachable, the polling process uses the change set records to automatically resynchronize the failed site with the live site production database server. Once synchronized, the polling process sets to server to notify the shard to begin synchronous operations. The Recovery of the failed site requires no DBA intervention, unless of course something catastrophic at the hardware level has occurred like complete disk failure. If desired, during the failover operation period, the live server can be configured to automatically change modes to provide synchronous buffer flush on commit.

22

Summary

Copyright Versant 2007 - 23

The MMOG application space demands a real-time, high concurrency, large scale, enterprise class database solution. These combined requirements do not fit well with traditional relational technology. The Versant database provides a combination of features specifically designed for the real-time, high scalability requirements of the MMOG application space.

23

For More Information

Visit Versant on the Web at:
http://www.versant.com

Technical Questions
gklein@versant.com

Jobs
jobs@versant.com

Copyright Versant 2007 - 24

24

Even more information

Pushing the technological barriers of game design with IBM technology http://www-306.ibm.com/software/success/cssdb.nsf/CS/MCAG6FXRPV?OpenDocument&Site=default Architecting Scalability for Massively Multiplayer Online Gaming Experiences http://www.gamesconference.org/digra2005/papers/68879030073efafba26fb47d37bb81b9.doc SecondLife Design - Enabling Player-Created Online Worlds with Grid Computing and Streaming http://www.gamasutra.com/resource_guide/20030916/rosedale_pfv.htm Powering next generation gaming with on-demand computing http://www-1.ibm.com/grid/pdf/butterfly.pdf A scalable peer-to-peer network for virtual environments http://www.phys.sinica.edu.tw/~statphys/publications/2006_full_text/S_Y_ Hu_IEEE_JulyAugust_22(2006).pdf Adaptive middleware for multi-player games http://www. cs.cmu.edu/~rajesh/papers/middleware05.pdf Object database -vs- Object-Relational databases http://ce.sharif.edu/courses/84-85/1/ce384/resources/root/Object%20Database%20vs_%20ObjectRelational%20Databases.pdf Comparing the performance of object databases and ORM tools http://www.odbms.org/download/027.01%20Zyl%20Comparing%20the%20Performance%20of%20O bject%20Databases%20and%20ORM%20Tools%20September%202006.PDF
Copyright Versant 2007 - 25

25