Data Center Design Considerations

Chris DiMinico MC Communications cdiminico@ieee.org

10GBASE-T

Contributors:
Jonathan Jew, President, J&M Consultants (Telecommunication Engineering Firm) Co-chair TR42.1.1, Data Center Standard Phil Isaak, Associate, Senior Communications Engineer, Mazzetti & Associates (Engineering Firm) William Baxter, Telecommunications Practice Leader OWP/P (A&E Firm)

10GBASE-T

Telecommunication Infrastructure
Enterprise
Customer Premise

Enterprise
Service provider - Distribution Node -Central Office Service Provider Backbone Customer lease Service provider Local Loop - Distribution -service Node provided -Central Office Customer Premise

ISP Local Loop

Internet Data Center Home SOHO -service provided

ISP
Internet Data Center Home SOHO

DATA/Voice Infrastructure Data Center EFM- Access Networks

10GBASE-T

Campus Network: California State University Data Center -80s

10GBASE-T

Cornell Campus Network, -87

10GBASE-T

Ethernet Networking -LAN/WAN/SAN

10 Gb/s
Internet

WAN
Remote Enterprise

10 Gb/s
WAN Router

Switch

Equipment Room

10 Gb/s 1 Gb/s

Firewall Backbone Switch / Router

1 Gb/s 100 Mb/s

LAN 10 Mb/s
File Servers

SAN

1 Gb/s 10 Gb/s

Web Email Application Servers Servers Servers

10GBASE-T

Generic Cabling - ISO/IEC 11801 - TIA/EIA-568

LAN
TO HC HC IC MC HC TO TO
TP

HC

Horizontal Cable HC 100 m

HC HC IC HC

Intrabuilding backbone 500 m

Interbuilding backbone 2000 m

TO HC TO HC HC HC HC IC HC HC

WAN/RAN/MAN

IC MC

HC

LAN Geographical Span ~3 kilometer

10GBASE-T

TIA-942-Data Center Standard

The standard is being developed by the TIA/TR42 Engineering Committee - subcommittee-TR-42.1.1 Network Distribution Nodes - Project No. 3-0092 Participants include:

Architecture & Engineering Firms Consultants End Users Manufacturers

The standard will become TIA-942 To be submitted for approval by ANSI and CSA Draft 3.0 - expected to be issued by TIA end of November Anticipated ballot closure early January

10GBASE-T

Data Center- Relationship of Spaces

BUILDING SITE BUILDING SHELL
GENERAL OFFICE SPACE TELECOM ROOMS & EQUIPMENT ROOMS for spaces outside data center OFFICE BUILDING SUPPORT SPACE

DATA CENTER
SUPPORT STAFF OFFICES ENTRANCE ROOM(S) DATA CENTER ELECTRICAL & MECHANICAL ROOMS

OPERATIONS CENTER

TELECOM ROOM(S) for data center support spaces

STORAGE ROOMS & LOADING DOCKS

COMPUTER ROOM

10GBASE-T

Data Center Topology

Entrance Room

Entrance Facility

Carriers

Active Equip interconnect

Carriers

Offices, Operations Center, Support Rooms Secondary Entrance Room

Active Equip interconnect MC MC Active Equip interconnect Carriers Carriers

Back bone Cabling HC IC HC IC Horizontal Cabling interconnect Active Equip Storage Area Devices interconnect Active Equip interconnect Active Equip Active Equip HC IC HC IC Zone Distribution interconnect

10GBASE-T

Data Center

Data Center Horizontal cable distance

In a data center, horizontal cabling is the cabling from the horizontal crossconnect (in the main distribution area or horizontal distribution area) to the outlet in the equipment distribution area or zone distribution area.
MC MC Main Distribution Area Backbone Cabling

Horizontal Cabling

HC IC

HC IC

Horizontal Distribution Area Zone Distribution Area Equipment Distribution Area

interconnect

interconnect Active Equip

interconnect Active Equip

Active Equip

10GBASE-T

Physical Layer Design Considerations:

Active Equipment Network topology, including operating distances and numbers of connectors PMD types + copper + optics, mode conditioning patch cords Equipment densities per linear rack unit .

Cabling

Cabling components: cable, connectors, patch cords installation: connecterization cabling density Field testing

Compatibility with existing (installed Network) Total Network Cost

10GBASE-T

Customer Requirements Network Design

e.g., Service Provider, Corporate (enterprise) e.g., Storage Area Network

Site Survey Wiring Concept Design Network Schematic Diagram Copper/Fiber Link Analysis Design Complete Distribution Subsytem Design

Site Diagram

1. BOM 2. Installation Requirements

Cost Analysis

yes

no

10GBASE-T

Data Center: Network Cabling Design Considerations

Placement of active equipment and usage of interconnect versus cross-connects:
Main Distribution Area Backbone Cabling Active Equip interconnect Horizontal Distribution Area Zone Distribution Area Equipment Distribution Area

Active Equip interconnect

MC MC

HC IC

HC IC

Horizontal Cabling
interconnect

interconnect Active Equip

interconnect Active Equip

Active Equip

10GBASE-T

Data Center: Horizontal Cabling Length: J&M Consultants (Engineering Firm)

Data centers recently built by (J&M) type and size 93% of data center horizontal cables are <= 45m 99% of data center horizontal cables are <= 55m
Data Center Type Data Center Size (sq ft) Corporate Corporate Govt Corporate Corporate Internet Corporate Internet Internet Internet Internet Average 5000 8000 10000 20000 20000 40000 45000 60000 60000 80000 100000 0-30 m 100% 90% 70% 70% 90% 60% 65% 35% 55% 55% 55% 67.7% 31-45 m 0% 10% 20% 25% 9% 35% 25% 48% 35% 35% 35% 25.2% 46-55m 0% 0% 10% 5% 1% 5% 8% 15% 8% 8% 8% 6.2% 56-75m 0% 0% 0% 0% 0% 0% 1% 1% 1% 1% 1% 0.5% 76-100m 0% 0% 0% 0% 0% 0% 1% 1% 1% 1% 1% 0.5%

Source: Jonathan Jew, J&M Consultants, Co-chair TR42.1.1- Data Center Standard

10GBASE-T

Data Center: Horizontal Cabling Length: Mazzetti & Associates (Engineering Firm)

Data centers recently built by (M&A) type and size 83% of data center horizontal cables are <= 45m 94% of data center horizontal cables are <= 55m
Data Center Type Corporate Corporate Financial Corporate Internet Internet Internet Average Data Center Size (sq ft) 5000 10000 25000 30000 60000 75000 120000 0-30 m 100% 8% 74% 99% 47% 44% 35% 58.1% 31-45 m 0% 23% 19% 1% 46% 50% 32% 24.4% 46-55m 0% 38% 6% 0% 5% 4% 27% 11.4% 56-75m 0% 31% 1% 0% 1% 1% 5% 5.6% 76-100m 0% 0% 0% 0% 1% 1% 1% 0.4%

Source: Phil Isaak, Associate, Senior Communications Engineer, Mazzetti & Associates (Engineering Firm)

10GBASE-T

Data Center: size versus percentage built

Data Center Size (sq ft) </= 5000 </=15000 >15000 Size Category small midrange large Percentage built 50% 30% 20%

80% of data centers recently built by (J&M) are 15000 sq ft

Source: Jonathan Jew, J&M consultants, Engineering Firm

Data Center Size (sq ft) </=5,000 </=15,000 >15,000

Size Category small midrange large

Percentage built 30% 45% 25%

75% of data centers recently built by (M&A) are 15000 sq ft

Source: Phil Isaak, Associate, Senior Communications Engineer, Mazzetti & Associates (Engineering Firm)

10GBASE-T

Data Center Cabling by Category: Percentage Usage (avg.)

Category Category 5 Category 5e Category 6

1999 10.00 66.67 23.33

2000 1.67 60.00 38.33

2001 0.33 55.00 44.67

2002 0.33 46.67 53.00

2003 0.33 35.00 64.67

Source: Jonathan Jew, J&M Consultants (Telecommunication Engineering Firm) Phil Isaak, Associate, Senior Communications Engineer, Mazzetti & Associates (Engineering Firm) William Baxter, Telecommunications Practice Leader, OWP/P (A&E Firm)

10GBASE-T

Data Center Cabling: Pathways and Spaces

Under Floor Space 10GBASE-T

Overhead Cable Tray

Example of Wire Basket Cable Trays For Cabling Under Raised Floor
24"
RAISED FLOOR TILE 1.375"

DIVIDER
9" 4" 5"

FIBER

18" COPPER CABLING

1"

SUPPORT STRUT

36"

12" 5" 4"

FIBER

18" COPPER CABLING

1"

SUPPORT STRUT

Open Area
13.5"

24" On Center

10GBASE-T