Telecommunications Infrastructure Standard for Data Centers ANSI/TIA-942

Chris DiMinico cdiminico@ieee.org MC Communications

IEEE 802.3 HSSG

Data Center Standards

ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers Co-chairs: Chris DiMinico & Jonathan Jew Published 2005 available through TIA at www.tiaonline.org ANSI/NECA/BICSI-002 Data Center Design and Implementation Best Practices co-chairs: Jonathan Jew & John Kacperski best practices complements TIA-942 2007 target
IEEE 802.3 HSSG

Who Developed TIA-942?

Developed by the TIA TR-42.1.1 Network Distribution Nodes subcommittee as Project No. 3-0092 Participants included: Architecture & Engineering Firms Consultants End Users Manufacturers

IEEE 802.3 HSSG

Purpose of TIA-942
Encourage early participation of telecom designers in data center design process Fill a void by providing standards for planning of data centers, computer rooms, server rooms, and similar spaces. The standard encompasses much more than just telecommunications infrastructure. Close to half of the technical content deals with facility specifications.

IEEE 802.3 HSSG

Purpose of TIA-942
Define a standard telecommunications infrastructure for data centers
Structured cabling system for data centers using standardized architecture and media Accommodates a wide range of applications (LAN, WAN, SAN, channels, consoles) Accommodates current and future protocols (e.g., 10+ GbE ) Replaces unstructured point-to-point cabling that uses different cabling for different applications

IEEE 802.3 HSSG

Purpose of TIA-942
Specifications for data center telecommunications pathways and spaces Recommendations on media and distance for applications over structured cabling Establish a standard for data center tiers to replace several proprietary standards. The TIA data center tier standard is: A tool to evaluate existing data centers A tool to communicate design requirements
IEEE 802.3 HSSG

Unstructured Cabling
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS
Heater - A/C Heater - A/C POMONA A/C UNIT #7 Heater - A/C POMONA A/C UNIT #6

1
DESK
TABLE

STORAGE CABINET

F IL E CABI NET

FI L E CABINET

F IL E CABINET

FIL E CABINET

2
He ater - A/C

TABLE 247-7589

TABLE

C D ROM SCANNER

HP PRINTER

AT &T DEFINIT Y UNIT

CONS OLE

CI SCO ROU TE R

3 4 5 6 7 8

N ETW OR K CABINET H8 NETWORK NETWOR K NETWOR K C ABINET I6 NE TWORK C ABINET I4 NE TWORK NE TWORK CABI NET I2 NETWORK CAB INE T NETWOR K CAB INE T I8 CABINET I7 CABI NET I3 CABI NET I1 N ETW OR K CABINET H7 NETWORK CABINET G7 N ETW ORK CABINET F6

N ETWORK CABINET E7

IBM CAB

NETW OR K CABINET C6

NETWORK CABINET B7 N ETWOR K CABINET B6

N ETW OR K CABINET A6 N ETW OR K CABINET A5 NETWORK CABINET A4 N ETW OR K CABINET A3 N ETW OR K CABINET A2 HALON PANEL

DESK

NETWORK C ABINET E6

IBM 3746-900

IBM 3745 -A11

I5

N ETW OR K CABINET H6

N ETWOR K CABINET G6 NETWORK CABINET G5

N ETW ORK C ABINET F5

NETW OR K CABINET C5 NETW OR K CABINET C4 NETW OR K CABINET C3

NETWORK C ABINET E5

IBM IBM 3745-17A 3745-41D

NETWORK CABIN ET H5

N ETW ORK C ABINET F4

N ETWOR K CABINET B5 N ETWOR K CABINET B4 N ETWOR K CABINET B3 NETWORK N ETWOR K CABINET B2 N ETWOR K CABINET B1

F/X
247-7543
HALON ABORT HALON RELEASE

NETWORK C ABINET E4 NETWORK C ABINET E3

TAPE RACK

TAPE RACK

NETWORK CABIN ET H4 NETWORK CABIN ET H3

NETWORK CABIN ET G4

N ETW ORK C ABINET F3 N ETW ORK C ABINET F2 LAN CABINET F1

IBM 3745 -L13

SAA CABIN ET D2

NETW OR K CABINET C2 NETW OR K CABINET C1

PA C BELL

POMONA A/C UNIT #8

9 10 11 12

G3 NETWORK CABIN ET G2

NETWORK C ABINET E2

NETWORK CABIN ET H2

SAA CABIN ET D1

NETWORK C ABINET E1

IBM DASD

IBM DASD

IBM S-390 R34

NETWORK CABIN ET H1

NETWORK CABIN ET G1

247-7538

13 14
TAPE RACK

C ABI NE T A1

NETWORK CABIN ET

15 16

STORAGE C ABINE T

IBM AS -400

IBM AS-400

INSTALL A CABLE WHEN YOU NEED IT (SINGLE-USE, UNORGANIZED CABLING)

TABLE

LOCKER

STORAGE CABINET

LIEBERT PDU -C

3174 CABINET STORAGE CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

IEEE 802.3 HSSG

Structured Cabling
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS
Heater - A/C Heater - A/C POMONA A/C UNIT #7 Heater - A/C POMONA A/C UNIT #6

1
DESK
TABLE

STORAGE CABINET

TABLE 247-7589

TABLE

FILE CABINET

FIL E CABINET

FIL E CABINET

FIL E CABINET

2
Heater - A/C

CD ROM SCANN ER

HP PRINTER
AT&T DEFINITY UNIT

CISCO ROUTE R CON SOLE

3 4 5 6 7 8

NETWORK CABINET H8 NETWO K R CAB INE T I7 NE TWORK C ABINET I1 NE TWORK CABI NET I8 NE TWORK NETWORK NETWORK CABINET I3 NETWORK CABINET I2 NETWORK C ABI NET I5 CABIN ET I6 CAB IN ET I4 NETWORK C ABINET H7 NETWORK CABINET G7 NETWORK CABIN ET G6 N ETWOR K C ABINET G5 NETWORK CABIN ET G4 NETWORK CABIN ET G3 NETWORK CABIN ET G2 NETWORK CABIN ET G1

FIBER MDF COPPER & CORE MDF

NETWORK C ABINET E7 NETWOR K C ABINET F6 NETWOR K C ABINET F5 N ETWORK CABINET E6 N ETWORK CABINET E5 N ETWORK CABINET E4 N ETWORK CABINET E3 N ETWORK CABINET E2

IBM 3745s
IBM CAB NETWORK C ABINET C6 NETWORK CABINET B7 NETWORK CABINET B6

NETWOR K C ABINET A6 NETWOR K C ABINET A5 NETWOR K C ABINET A4 NETWOR K C ABINET A3 NETWOR K C ABINET A2 H ALON PANEL

DESK

IDF

IBM 3746-900

IBM 3745-A11

NETWORK

NETWORK C ABINET C5 NETWORK C ABINET C4 NETWORK C ABINET C3

IDF
T APE RACK TAPE RACK

C ABINET H6 N ETWOR K C ABINET H5 N ETWOR K C ABINET H4 N ETWOR K C ABINET H3

IBM IBM 3745 -17A 3745-41D

NETWORK CABINET B5 NETWORK CABINET B4 NETWORK CABINET B3

F/X
247-7543
H ALON ABORT H ALON RELEASE

NETWORK CABIN ET F4 NETWORK CABIN ET F3 NETWORK CABIN ET F2 LAN CABINET F1

IBM 3745-L13
SAA CABINET D2

POMONA A/C UNIT #8

9 10 11 12

NETWORK SAA C ABINET D1 C ABINET C1

Mainframe
IBM DASD IBM DASD IBM S-390 R34

IDF

IDF

N ETWOR K C ABINET H2 N ETWOR K C ABINET H1

NETWORK CABINET B2 NETWORK CABINET B1

N ETWORK CABINET E1

IDF

IDF

IDF

IDF

IDF IDF

247-7538

13 14
TAPE RACK

15 16

S TOR AGE C AB IN ET

IBM AS-400

IBM AS-400

STRUCTURED CABLING SYSTEM (ORGANIZED, REUSABLE, FLEXIBLE CABLING)

T ABLE STORAGE CABINET

PAC B ELL C A BIN ET A1

NETWORK C ABINET C2

LOC KER

LIEBERT PDU- C

3174 CABINET STORAGE CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

3174 CABINET

IEEE 802.3 HSSG

Design Elements
Cabling Design Facility Design Network Design Informative annexs: Provide best practices Annex A - Cabling Design Considerations Annex B- Telecommunications infrastructure administration Annex C-Access provider information Annex D- Coordination of equipment plans with other engineers Annex E- Data center space considerations Annex F- Site selection
IEEE 802.3 HSSG

Design Elements
Cabling Design: Copper and fiber cabling performance Connectors, cables, distribution hardware Cabling distances Space management Facility Design: Data center sizing Power distribution methodologies Pathways and spaces HVAC, security, operations, and administration. Flexibility, scalability, reliability and space management
IEEE 802.3 HSSG

Design Elements
Network Design:
Support of legacy systems Enable rapid deployment of new and emerging technologies such as 10 GbE and 10+ GbE copper and fiber applications.

IEEE 802.3 HSSG

Relationship of Spaces
BUILDING SITE BUILDING SHELL
GENERAL OFFICE SPACE TELECOM ROOMS & EQUIPMENT ROOMS for spaces outside data center OFFICE BUILDING SUPPORT SPACE

DATA CENTER
SUPPORT STAFF OFFICES ENTRANCE ROOM(S) DATA CENTER ELECTRICAL & MECHANICAL ROOMS

OPERATIONS CENTER

TELECOM ROOM(S) for data center support spaces

STORAGE ROOMS & LOADING DOCKS

COMPUTER ROOM

IEEE 802.3 HSSG

Data Center Topology

Entrance Room

Entrance Facility

Carriers

Active Equip interconnect

Carriers

Offices, Operations Center, Support Rooms Secondary Entrance Room

Active Equip interconnect MC MC Active Equip interconnect Carriers Carriers

Back bone Cabling HC IC HC IC Horizontal Cabling interconnect Active Equip Storage Area Devices interconnect Active Equip interconnect Active Equip Active Equip HC IC HC IC Zone Distribution interconnect

Data Center
IEEE 802.3 HSSG

TIA-942 Spaces
Entrance Room (ER) - location of interface with campus and carrier entrance facilities Main Distribution Area (MDA) location of main cross-connect (MC) Horizontal Distribution Area (HDA) location of horizontal cross-connect (HC) Zone Distribution Area (ZDA) location of zone outlet (ZO) or consolidation point (CP) Equipment Distribution Area (EDA) location of equipment cabinets and racks
IEEE 802.3 HSSG

Data Center Cabling

MC MC Main Distribution Area Backbone Cabling

Horizontal Cabling

HC IC

HC IC

Horizontal Distribution Area Zone Distribution Area Equipment Distribution Area

interconnect

interconnect Active Equip

interconnect Active Equip

Active Equip

Horizontal cabling is the cabling from the horizontal cross-connect (in the main distribution area or horizontal distribution area) to the outlet in the equipment distribution area or zone distribution area.
IEEE 802.3 HSSG

Horizontal and Backbone Cabling

Recognized Cables: a) 100-ohm twisted-pair cable (ANSI/TIA/EIA-568B.2), category 6 recommended (ANSI/TIA/EIA568-B.2-1) b) multimode optical fiber cable, either 62.5/125 micron or 50/125 micron (ANSI/TIA/EIA-568-B.3), 50/125 micron 850 nm laser optimized multimode fiber is recommended (ANSI/TIA-568-3-1) c) singlemode optical fiber cable (ANSI/TIA/EIA568-B.3) d) 75-ohm (734 and 735 type) coaxial cable (Telcordia Technologies GR-139-CORE)
IEEE 802.3 HSSG

Horizontal cabling distances

The maximum horizontal distance is 90 m independent of media type. The maximum channel distance including equipment cords is 100 m. The maximum cabling distance in a data center not containing a horizontal distribution area is:
300 m for an optical fiber channel including equipment cords. 100 m for copper cabling including equipment cords.

IEEE 802.3 HSSG

Backbone Cabling
Includes cabling from MDA to ER, HDA Optional cabling between HDAs allowed Maximum backbone cable lengths depend on applications to be supported Centralized optical fiber cabling supported with interconnect, splice, or pull-through at the HDA Star topology with no intermediate crossconnects Various topologies permit redundancy and flexibility to support various data center sizes
IEEE 802.3 HSSG

Computer Room & Entrance Room Requirements

Min clear height of 2.6m/8.5 ft Min door size 1m/3ft wide 2.13/7ft high Min dist floor loading 7.2 kPA/150lbf/ft2, recommended min 12 kPA/250 lbf/ft2 20 degrees C to 25 degrees C 40% to 55% relative humidity (reduces ESD) Any sprinkler systems must be pre-action system Common bonding network (CBN) equipotential ground reference

IEEE 802.3 HSSG

Main Distribution Area

Location of Main Cross-Connect (MC), the central point of distribution for data center structured cabling system Centrally located to avoid exceeding maximum distance restrictions (typically for E1s, E-3s, T-1s and T-3s) Install separate racks for Fiber, UTP, and coaxial cable distribution

IEEE 802.3 HSSG

Horizontal Distribution Area

Location of Horizontal Cross-Connect (HC), the distribution point for cabling to equipment distribution area Distribution LAN, SAN, KVM switches and console servers located in HDA MDA may also include an HC for nearby equipment distribution area Number of HDAs depends on the density of cabling and the size of the data center

IEEE 802.3 HSSG

Horizontal Distribution Area

The capacity of the cable tray system and the size of the cross-connect creates practical limits on the size of the HC Guideline is maximum of 2,000 4-pair UTP or coax cable terminations per HDA Arrange patch bays to minimize patch cable lengths and to simplify cable management Separate racks for fiber, UTP, and coax Locate switches and patch panels to minimize patch cord lengths

IEEE 802.3 HSSG

Zone Distribution Area

Rack, cabinet, or under floor enclosure that houses a zone outlet (ZO) or consolidation point (CP) ZO - structured cabling termination for floor-standing equipment that cannot accept patch panels (e.g. mainframes and large servers). CP - intermediate termination point (e.g. cabling to areas where floor plan is uncertain or dynamic) No cross-connects within the ZDA No active equipment shall be located in the ZDA Maximum of 144 connections in a ZDA Maximum of one ZDA within a horizontal cable run

IEEE 802.3 HSSG

Equipment Cabinets
Front rails of cabinets must be recessed to provide adequate room for patch cables and wire managers Recommend 1-to-1 ratio of patching to cable management Arrange switches and patch panels to minimize patching between cabinets & racks Perforated tiles at front of cabinets One edge of cabinets placed at edge of tile
IEEE 802.3 HSSG

Raised Floor
Better appearance than overhead cabling. Allows higher power densities, better control of cooling, and more flexibility in location of cooling equipment Most stand-alone computer systems are designed for cabling from below Coordinate under floor cabling with mechanical & electrical engineers Recommend wire basket cable trays in hot aisles for telecom cabling

IEEE 802.3 HSSG

Example of Wire Basket Cable Trays For Cabling Under Raised Floor
24"
RAISED FLOOR TILE 1.375"

DIVIDER 18"
9" 4" 5"

FIBER

COPPER CABLING
1"

SUPPORT STRUT

36"

12" 5" 4"

18" FIBER COPPER CABLING

1"

SUPPORT STRUT

Open Area
13.5"

24" On Center

IEEE 802.3 HSSG

Overhead Cable Trays

Less expensive than raised floor systems Cable trays can be attached to the top of racks and cabinets (if they are uniform in height) Cable trays suspended from the ceiling provides more flexibility for supporting cabinets/racks of various heights and for adding and removing cabinets/racks. Cable trays can be installed with several layers Coordinate location with lighting, ducts, overhead conduits, overhead power IEEE 802.3 HSSG distribution

Overhead Cable Tray Example

3 Layer cable tray system: Bottom layer signal Middle layer power Top layer fiber Signal Reference Grid in brackets attached to lower layer of trays Fiber patch cables may be in fiber duct attached to threaded rods

IEEE 802.3 HSSG

Infrastructure Administration
Informative annex with TIA-606-A standards compliant labeling scheme for all components. Labeling scheme extended for use in data centers Cabinets and racks labeled by location using tile grid or row/position identifiers All cabinets, racks, patch panels, cables, and patch cords should be labeled
IEEE 802.3 HSSG

Labeling Example
AA AB AC AD AE AF AG AH AI AJ AK AL AM AN AO AP AQ AR AS AT AU AV AW AX AY AZ BA

"X" COORDINATE
BB BC BD

01 02 03 04 05 06 07 08 09 10 11 12 13

Cat 6 x 24

RIGHT FRONT CORNER CABINET AJ05

CABINET AQ03

"Y" COORDINATE

Patch Panel for 24 Cat 6 from 1st Panel in Cab AJ05 to 2nd Panel in Cab AQ03
AJ05-A to AQ03-B Ports 01-24
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

IEEE 802.3 HSSG

Site Selection
Informative annex with guidelines for selection of a site for a data center
Architectural Electrical Mechanical Telecommunications Security Other
IEEE 802.3 HSSG

Facilities Specifications & Tiers

Informative annex with general architectural, structural, electrical, mechanical, and telecommunications recommendations requirements Annex includes detailed architectural, security, electrical, mechanical, and telecommunications recommendations for each Tier Recommended specifications by tier are a uniform way to rate aspects of a data center design and are a starting point for initiating design requirements with qualified architects and engineers

IEEE 802.3 HSSG

Data Center Tiers

Tier 1 basic data center no redundancy Tier 2 redundant components Single distribution path with redundant components Tier 3 concurrently maintainable Multiple distribution paths with only one active Tier 4 fault tolerant Multiple active distribution paths
IEEE 802.3 HSSG

Redundant Topologies
Carriers Carriers Carriers Carriers
Primary Customer Maintenance Hole (Tier 1 and higher)
Tie r3
2

Secondary Customer Maintenance Hole (Tier 2 and higher) Tier 3

Tier 1

Data Center

Tier

Primary Entrance Room (Tier 1 and higher)

Tie r4
r3

Tier 3

Secondary Entrance Room (Tier 3 and higher)

Tier 1

Tier 4

Offices, Operations Center, Support Rooms

Tie

Telecom Room

Main Dist Area (Tier 1 and higher)

Tier 4

Secondary Dist Area (Optional for Tier 4)

Computer Room
Horiz Dist Area Optional Cabling Horiz Dist Area Zone Dist Area Horiz Dist Area

Equip Dist Area

Equip Dist Area

Equip Dist Area

IEEE 802.3 HSSG

Data Center Tiers

Higher tiers correspond to higher availability, but also have higher construction costs Data Center can have different tier ratings for different portions of its infrastructure (architectural, security, mechanical, electrical, telecommunications) The overall rating for the data center is equal to the lowest tier rating Capacity of systems may need to be upgraded to maintain tier rating as data center load increases Human error and operating procedures have a major impact on availability

IEEE 802.3 HSSG

Conclusion
TIA-942 is the first standard to specifically addresses data center infrastructure. Primarily a telecom infrastructure standard, but about half of the content deals with facility requirements. Provides a flexible and manageable structured cabling system using standard media. Builds on existing standards, when applicable Guidelines on a wide range of subjects useful to someone designing or managing a data center. An official tiering standard for determining the quality of a center. A way to objectively compare one center with another.

IEEE 802.3 HSSG