This action might not be possible to undo. Are you sure you want to continue?
By: Peter Harrison
Friday, November 30, 2007 A LinuxHomeNetworking.com © White Paper
About The Author
Peter Harrison has over ten years of data center experience in both the private and public sectors providing Internet and corporate services. He is a Cisco Certified Internetworking Engineer (CCIE) and currently consults on data center design and network architectures. He is also the author of the Linux Quick Fix Notebook published by Prentice Hall now available on Amazon.com. He was the founding president of PCJAM, Jamaica's first computer user group, and was the principal systems engineer responsible for the computerization of the island's tax collection and social security systems. He then sought new opportunities as the western Caribbean representative for a Fortune 500 pharmaceuticals firm and later became the international sales manager for a West Indian rum company. Before moving to Silicon Valley he ran Trinidad and Tobago's first industrial trade office to Latin America. Peter has since worked extensively in the Internet sector deploying large-scale data centers and Web sites. His extensive technical experience combined with his varied business background has helped him create this highly readable guide for project managers, techies, and their bosses. Peter also is the creator of LinuxHomeNetworking.com, a site dedicated to IT white papers and discussion on Linux, Cisco products and data center activities. In his quieter moments, Peter enjoys the art and literature of the Caribbean and Latin America. Long rides on his bicycle provide another guilty pleasure. Peter likes to relax with his family on short weekend trips to the many attractions of the San Francisco Bay Area.
© Peter Harrison 2002-2007, All rights reserved. Unless otherwise stated, the material published within this document is copyright of the author, Peter Harrison. No part of this document, including page design, interior design, cover design and icons may be reproduced or transmitted in any form, by any means, (electronic, photocopying, recording, or otherwise) without the prior consent of the publisher/author. As a sole exception, the author allows purchasers of this document to create a single printed version for their own personal, non-commercial use.
Disclaimer – The Website and Manual
While every effort will be made to ensure that the information contained within this website and manual is accurate and up to date, the author makes no warranty, representation or undertaking whether expressed or implied, nor does it assume any legal liability, whether direct or indirect, or responsibility for the accuracy, completeness, or usefulness of any information.
Disclaimer - Other sites
Hypertext links to sites outside this website are provided as a convenience to users and should not necessarily be construed as an endorsement. Although every care is taken to provide links to suitable material from this site, the nature of the Internet prevents the author from guaranteeing the suitability or accuracy of any of the material that this site may be linked to. Consequently, the author can accept no responsibility for unsuitable or inaccurate material that may be encountered and accepts no liability whether direct or indirect for any loss or damage a person suffers because that person had directly or indirectly relied on any information stored in the hypertext links.
Further, the author is not and can not be responsible for the accuracy or legitimacy of information found elsewhere on the Internet and there is therefore no guarantee or warranty that any of the sites listed will be available at any particular time. The author does not guarantee or warrant any services that might be announced - use at your own risk.
........................................................................................................................................................................58 .................................................................................................................................................................................................................................................................................................................10 Potential Increased Profits...13 Chapter 2...............................31 Data Center Rating Form.............................39 Potential Monthly Profit Improvement ................................................................................................................................................................................................................................................49 DNS Preparation Check Sheet.............................................58 Incident Detail ..................................................................................................................................................................................................................17 Coordination Preparation...............................................................................................................................................................48 Server Preparation Check Sheet .......................................................................................................................................7 Why Relocate Your Web Site? .........................................52 Individual Server Worksheet .............................................................................................53 Individual Server Worksheet (Part II) ..........................................................................................................................................................................................................................Table of Contents Chapter 1...................................................................................................................................................................................................................................................................................................................................................................27 Post Relocation Activities.............................................................................................................................................................................................................................................................................................27 Activities During the Relocation.........................................................................................................................................................................................................................54 Network Enabled Applications List (netstat -a output) .............................................................31 Relocation Check ............................................................................................................39 Relocation versus Data Center Upgrade Capital Outlays ......................................................................................................................................................................................................................................................17 Customer Communications Preparation ........................................................................19 Server Area Preparation...........................51 Post Relocation Check Sheet......................................................................................................................................................................................................................................................................23 DNS Preparation ...............................................................................................................................10 Net Changes in Monthly Expenses ...........................................................................58 Customer Impact ....36 Cost Justification Work Sheet ................................................................................................................................................................................................................14 Data Center Selection Criteria.......25 Conclusion.................................46 Network Preparation Check Sheet.......................................................................26 Chapter 3..............................................................................................................................................................50 Activities During the Relocation Check Sheet..........................................................41 Coordination Preparation Check Sheet ..............................27 Post Relocation Activities .........................................................................................................................................................................25 Transportation Preparation .........................................................................................28 Appendix I ......................58 Customer Information...50 Transportation Preparation Check Sheet ..........................................55 Data Circuit Check Sheet...........58 Incident Summary..............................43 Customer Communication Check Sheet ...........................................................54 Contractor Qualification Check Sheet .........................................................................14 Preparing for Server Relocation................12 Conclusion..................................................................................22 Server Preparation............................................14 The Relocation Project Plan.32 ISP Rating Form ................................................................8 Factors That Affect Virtual and Self-Hosting ..............................................7 When to Migrate From Virtual Hosting ...........................................................................57 Post Mortem Analysis Sample Form.................................................................................................................................................................................................................................................................................................20 Network Preparation ..........................56 Vendor / Purchasing Check Sheet .................................................................................................................................................................................................................................................................................11 Capital Outlays ............45 Server Area Preparation Check Sheet ..............................40 Monthly Costs .................................................................9 How to Analyze Migration Costs..........................................54 Additional Server Routes...7 When to Migrate Between Data Centers ...............................................
...64 Border Gateway Protocol ...................................................................................................................................................60 Non-Internet (Carrier) Services .................................................................................................................................................................................................67 .......65 Determining a BGP Autonomous System Number ........................................................................................................................61 Table A3.................61 Data Circuit Provisioning.................................................................................................................................................................................................................................64 Table A3...62 IP Address Ownership.....................59 Corrective Action Plan............................................60 How to Choose a Data Center ISP .........................60 Data Circuit Pricing ..........................................................................................Common Data Circuit Terminologies .............................................................................................................................................1 ....................................................................................................................................................Common BGP Routing Terms..........66 Conclusion.................................................................................................................................................................................................................................................................................................63 Routing Protocols ........2 ....................................59 Appendix II ............................................61 Data Circuit Types...................................................................66 Administrative Tasks Needed to Advertise BGP Routes ........................60 Internet Services...........................................................Root Cause Analysis......................
It covers common reasons for such actions. the factors to consider in choosing a data center. please feel free to contact me via the LinuxHomeNetworking. .com Web site. from software and servers to Internet access and data storage. This book focuses on the need to relocate a website of servers to a new physical location where the destination data center is managed by a third party. Now almost every business has considered or uses the Internet as a sales. The supply and demand of IT services is therefore very fluid as the business and technology cycles change. has created an explosive growth in the use of computers in day to day life. It is a short guide aimed at the busy professional who needs to be aware of the most important operational activities that need to be done and gives many concrete examples with explanations of many terms used in these types of projects. marketing or operational tool. The result is a constant reevaluation of IT costs and functionality with the server farm often being at the center of the analysis. the actual server migration and post relocation activities. If you have any comments or suggestions about this document.Introduction The commoditization of many aspects of IT infrastructure.
the next challenge is to determine the daily tasks you will need to do yourself and those you intend to outsource to third party service providers. Here are some typical examples: 1. The rest of the book will cover the logistical problems of data center selection and preparation. Plan we ll. should be sufficient. database. Embark on it when your service provider threatens the future growth of your company. by category. performance or functionality problems. This can cost about $100 per month. There may be delays in completing your requests especially if you are one of many customers on a server. Support is usually only given through instant messaging. Examples of this . Always remember that the decision to migrate your Web site should be strictly based on business needs. go slowly. Each VHP web server potentially handles hundreds of web sites. which start at about $10 per month. inform your customers and minimize your exposure to downtime risk at every step of the way. the migration itself. This chapter will describe a number of scenarios in which physically migrating a Web site can become desirable and will cover ways of determining a financial justification for doing so. have a backup plan. web mail or message board forums software suite. then the price can reach as much as $150 for an entry level service. and if you need the person to buy the product using a shopping cart. The cost advantage of this service declines as you require additional high end services or customization. only use proven stable technologies. planning. blog. Customization usually occurs through a standard web GUI interface which is usually geared towards altering the work flow features of the software and not its overall performance. 2. This requires web pages to be generated dynamically using application server software that queries a database. The service provider also has to ensure that the upgrade won’t affect any of the other websites and this can add delays. If you decide to migrate to self-hosting. you can lease a dedicated server for $200 per month in a collocation data center and if you choose to use Linux. In comparison. your software procurement costs would be negligible. Additionally. Self hosting in this scenario can become desirable if you already have a capable IT staff with sufficient resources to complete the project wi thin your budget and on time. You will constantly find yourself adjusting these responsibilities for a variety of reasons and you may even have to consider migrating your Web site between physical locations to achieve these goals.Chapter 1 Why Relocate Your Web Site? Businesses that need to have a Web presence usually begin with a cheap virtual hosting provider (VHP) that constantly aim to reduce their costs via standardization. in a particular price range or from a specific manufacturer. The challenge is to determine the point at which do-it-yourself self-hosting becomes more attractive than using a VHP. testing and post migration procedures. Unpredictable Software Updates: With virtual hosting you are dependent on your service provider to provide software updates or patches to fix security. For a simple website with the aim of providing supplemental information to newspaper or web advertising then basic virtual hosting services. there may be times when you need to implement software that needs to be installed external to your home directory that isn't supported by your hosting provider. There are many cases in which this can become painfully obvious. When to Migrate From Virtual Hosting The lack of support for customization is the VHP's greatest weakness. with access to only a single type of application server. shopping cart. Expensive Shopping Carts: Sometimes you want visitors to be able to search your website for a list of available products by name.
There may be the need to link your shopping cart order entry system. 1. engineering. 4. There are many other scenarios in which a VHP may become undesirable but these examples have provided some of the main ones you will most likely face. allowing FTP access to the virtual server potentially opens the door to unrestricted file transfers to all sites on the server that share the same IP address. There may also be cases in which your section of the floor just has poor circulation and other better ventilated . 5. sub assemblies and finished products. In these cases the justification for self-hosting becomes even stronger. This is because the desire for improved services extends beyond an individual server and encompasses the entire data center facility. Lack of Redundancy: Many businesses rely on a web presence for the majority of their revenues and cannot afford to have extended periods of downtime. Your service provider may not have adequate computer room air-conditioning (CRAC) units to cool the entire floor space and may be unwilling to upgrade due to financial constraints such as a lack of funding or an anticipated inadequate return on investment. If you can't get adequate support for billing. If you don’t want to risk this type of exposure. Poor Cooling: As a data center becomes increasingly occupied the thermal load it needs to handle becomes greater. This may be viewed as a security risk for your neighbors. then consider self-hosting. then an in house solution may be better. This is a useful offering if you need to take an application offline for maintenance. 7. The load balancer regularly probes your servers and automatically steers traffic away from any server that appears to be malfunctioning or down. Insufficient Language Capabilities: VHPs often provide technical support in only a few languages. Here are some common reasons for considering this option whether the facility is owned by your company or provided by a third party. The next section discusses how migrating existing self-hosted Web site can have very different issues than those associated with migrating from a VHP. but the time they choose may be inconvenient to you. This requirement may open a vulnerability to other web sites. With the use of load balancing devices it is possible to spread your web hits across two or more servers. Inflexible Security Services: You may want your applications to run on unique TCP/IP ports and be accessible only to certain IP address ranges or you may want communications with these ranges to be fully encrypted over a virtual private network (VPN).8 Relocating Servers Between Data Centers - © LinuxHomeNetworking. 8. 6. Server Overload: With hundreds of websites on a server. 3. you run the risk of slow response times due to one of the URLs owned by another company suddenly becoming popular. and correct especially in a shared environment where you don't have access to many systems tools. but it may expose more of your business to this provider thereby increasing your risk. with the inventory system of a supplier. When to Migrate Between Data Centers The decision to migrate an existing self-hosted site to a new physical location often has very different criteria than those associated with migrating from a VHP. The cause of this latency is often difficult to determine. Poor Site Availability: All service providers need to schedule maintenance of their servers. which contains all your customers' credit card information.com include a new database product and centrally managed server logins using LDAP. For example. They may also have unreliable equipment that adversely affects your site's performance and may not have adequate backups of your data in the event of catastrophic failure. Restricted Supplier Management: You may require highly customized reporting or have complicated inventory listings which have to track parts. and customer care services in your preferred language. Many virtual hosting providers don't offer such a service to individual customers. Your virtual hosting provider may be able to do this. This will usually require some form of VPN or firewall service that your provider may not offer.
2. both the status quo and upgrades of the service could threaten the reliability of your electricity supply.Why Relocate Your Website? 9 locations within the facility are too small or geographically fragmented to comfortably accommodate your servers. but it should also include accurate entrance logs whether they be manual or electronic. the networking gear could be poorly maintained or over-utilized. In all these cases. 8. Disaster Recovery: A disaster can debilitate your company especially if all your IT resources are in a single location. Be especially careful if the unsuitable data center is your own. Unreliable Data Networks: The data center you use may have a number of connectivity shortcomings. the use of unique keys for all cabinets and customers. Even the simplest things can have a huge impact. The internet service providers (ISPs) used may be unreliable. that's as much as 20 microwave ovens running 24x7. Using a New ISP: Sometimes the Internet IP addresses your site uses came bundled with the data circuit provided by your ISP. Inadequate Floor Space: The most obvious shortcoming is a lack of floor space but people often forget that. then it may be time to consider your alternatives such as negotiating better monthly rates from your existing provider or simply go to the competition. Sometimes data center space can be used more productively for another purpose. 1. 3. insufficient UPS backup. 7. snow storms. the recurring costs there could be more than the ones you currently incur. floods. staff could be inadequately trained increasing the risk of human error. maintenance work that could cause outages may be unannounced or fall outside your desired time frame. you may be forced to change your addresses. Unstable Power: This can cover a variety of factors from fluctuating voltage. employee . video recording of physical access to your area. a server room located in a downtown office where rents are high may be more cheaply located in an industrial area nearby. Data Center Consolidation: Companies often inherit data centers when they purchase other companies. If you are forced to use a new ISP. and earthquakes. Servers and Web sites can be repurposed or retired to reduce overhead costs and management complexity. Insufficient Power: As servers become smaller and more powerful their demand for power can become enormous. Leaking water pipes in floors above your data centers. or they can be more localized in cases such as hazardous spills and lightening strikes. It is now possible to install up to 80 servers in a standard 19" x 72" x 36" rack with a power load of over 20 kilowatts. insufficient circuits in your server area and nonredundant power feeds to your server area. For example. and the equipment used my be obsolete making it unable to provide desired features that could improve your site's performance. Unsuitable Physical Security: You may feel that this is limited to access to your server area. ignored preventative maintenance. floor space used for servers could be better used for other purposes. cabling work could be sloppy. It is for this reason that some IT managers may use the opportunity to consider a complete re-evaluation of the data center strategy which may include a migration of servers to a more suitable data center. in a data center owned by the business itself. faulty fire alarms that force complete building evacuations. 5. 4. 2. Catastrophes can be caused by huge events such as hurricanes. High Cost: If your data center meets all your needs and yet costs more than a comparable facility nearby. and the deployment of suitable fire detection and suppression equipment. Though the cost of making improvements to your data center may be much greater than the cost of moving the servers to a data center owned by a third party. You may want to consider migrating if any of these factors don't meet your requirements. This can be a complicated problem which will require many of the steps used in a physical relocation. 6. Factors That Affect Virtual and Self-Hosting There are some relocation needs that affect companies that do either virtual or self-hosting.
and risk analysis for doing so can be difficult to quantify. performance and cost expectations forcing you to seek alternatives such as new outsourcing providers or to take the work in-house where you have more control. 4. may be beyond the budget and capabilities of their organizations. backups and networking services more cheaply in another facility. Switching to a more expensive fully managed hosting provider that specializes in customizations may have to be considered in such a case. At this time create a pilot project using only the most essential customization and if successful. For example it may be possible to outsource some back office operations such as monitoring. Migration cost analysis should focus on three broad areas. consulting. As part of these plans they purchase software that can facilitate expansion into new markets or save costs and they may even consider testing the limits of existing IT resources to achieve these goals. Obsolescence: Outdated facilities can increase your recurring maintenance costs but they can also not meet industry standards or government regulations. 5. though desirable. then it may be best to adjust your requirements to fit the services offered by a virtual hosting provider. The cost justification. Potential Increased Profits Most businesses try to forecast their profit growth. Cost Reduction: New technologies and business processes can reduce many IT recurring expenses but can also require relocation of your servers. namely time. This may require the use of additional IT tools that are more efficient. The reasons for relocating can be complex and should be approached carefully. the potential for increased profits. Sometimes the driving force to invest in IT isn't profit growth. expected reductions in monthly expenses and the capital outlay to carry out the migration. How to Analyze Migration Costs If you decide to do self-hosting. you should also consider its consumption of your business resources. This will be addressed in the sections to follow. Also. or you have capable staff that would be better utilized expanding the business. talent and money. but there are resource costs related to installation. The financial cost of the equipment is obvious.com sabotage triggered by layoffs. With an existing IT staff. In such cases relocation may be cheaper than upgrading.10 Relocating Servers Between Data Centers - © LinuxHomeNetworking. Your existing facilities may not be able to accommodate these increased demands and you may find yourself having to expand beyond your existing boundaries. have better performance or have more features. If you are a small company with limited IT staff. as mentioned before. Failed Outsourcing: Your IT business partners may fail to meet your disaster recovery. You can reconsider self-hosting in future when the customization needs of the company are more pressing. 3. 6. gradually migrate over to a production version of the pilot site. Convert your pilot to a general testing and staging area and then add modifications to the production site when you are satisfied they work. This can sometimes easily be calculated by determining the impact per hour of downtime on sales and the expected amount of downtime during . less costly. the strain would be less but if the company is small the price of customization could be a high proportion of your business overhead expenses thereby making self-hosting uneconomical. These are covered next. Sometimes businesses should accept the fact that self-hosting. training. some overhead costs can be lowered by reducing the number of data centers or servers you operate. but guaranteed revenue. Changing Customer Demands: Increased competition and the opening of new markets can shift the demographics of your customer base. and overloaded circuit breakers can all contribute to downtime that could be avoided by having a failover data center. security and long term maintenance. staff shortages. Calculating the potential lost profits by not investing in information systems can also be a deciding factor.
You may decide on a cheap data center located an hour away by car but if a device begins to intermittently fail. 8. The upkeep of CRAC units. sometimes during rush hour. Many third party data centers include the cooling cost into their power charges unless your installation requires major expansion of their facilities. the duration of outages can jump dramatically and so will the cost of lost revenue. In the event of data center consolidation you may have less staffing needs too. but always try to determine the cost of commuting to your new server location. Floor Space: In your own server room this is a fairly straightforward cost. Cooling: As power consumption increases so do your cooling requirements. Power: Electrical costs in your own server room may be directly tied to actual consumption. 7. security systems. You may also have to consider additional IT services such as networking. 5. replacement tapes or other media. DNS and database services. You may have to hire additional employees. 4. Once you have this information it becomes easier to calculate the expected return on the IT investment in terms of increased profits or liberated cash flow.Why Relocate Your Website? 11 the financial year due to inadequate resources. and firewall and intrusion detection services. it may also be included in one or more maintenance contracts. Net Changes in Monthly Expenses Net expense reductions can be calculated as the eliminated monthly costs that stem from the migration subtracted from the forecasted monthly expenses in the new facility. In some cases you may be offered an existing caged area which may not exactly meet your needs or you may have to sign a long term contract before the facility will agree to creating a custom space for you. Staffing: Self-hosting will inevitably require additional staffing resources. Security: Physical security expenses will largely be covered under various security system maintenance contracts and the hiring of security guards. This may be provided as a contractual service or handled by yourself. With a data center provider you may be offered floor space as a full computer cabinet or only part of it. standby generator plants and fire systems as well as the retention of janitorial and security guard services may need to be included. Recurring expenses to consider in both the new and old facilities should include the following: 1. 9. Equipment Leasing: Migrating to a new facility may also require leasing additional hardware to accommodate the move. but wh en things go wrong or you have new physical installation work to do the cost of travel delays can become significant. If you are using a third party data center. off site storage. It may also be a good time to renegotiate the . but third party data centers usually charge for power by the number of circuit breakers and/or power outlets you decide to use. UPSs. 2. either as staff or contractors. Some data center providers will also throw in equipment leases at favorable rates to get your business but they won't allow you to leave with the equipment should you decide to go to a competitor. Travel Time: It may seem trivial. Most website maintenance can be handled remotely. Bandwidth: When you convert to self-hosting the cost of your Internet connection will most likely become more variable in nature as it will fluctuate with the amount of bandwidth you use each month. Maintenance Contracts: Most people primarily think of computer equipment maintenance but if you manage your own server room there will be many other maintenance figures to consider. Your budgeting should include the monthly cost of backup services. software update and anti-virus services. or you may have to train existing staff members continuously and raise their salaries to match their new skill set. 3. In this case you may be expected to either directly contribute to the expansion or commit to a long term contract. Data security is similar. Appendix II discusses many of the factors in selecting a data center ISP in more detail. 6. then this may be included in your floor space expenses.
These could include project management and data migration and recovery skills your organization may not already have. 3. 4. Contractor Fees: Contractors may not just be used to install leasehold improvements but they could provide services specifically for the move. 6. 2. 5. standby generator plants. You may also be forced to use new servers which may add unexpected problems. This then can be used to justify the one time capital outlays required to get the job started. Capital Outlays There are a number of one time costs that will be encountered at the beginning of the migration that you will have to account for. This could include additional power.12 Relocating Servers Between Data Centers - © LinuxHomeNetworking. Leasehold Improvements: You may have install the infrastructure to support your web site being moved to the new location if you don't have it already or if it's not supplied by your new hosting facility. 10. This could add additional hidden expenses to the project. others may want to account for it as part of the moving cost. Subtracting the net change in monthly expenses from your profit projections will provide an overall expected monthly cash flow from the project. fencing. Systems Development: In a virtual hosting environment the software used to make the management of your site easy is rolled into the cost of the service. patch panel cabling. cameras. the data backup formats used at the old and new facilities may be different forcing you to lease a tape backup unit just in case you have to restore data in the event of a server failure during the migration. cooling. but you may require your staff to get several foundation training courses to allow them to handle the new tasks they will be expected to achieve. For example. and mount it in the truck. keyless entry systems and raised flooring to name a few. on which the computer equipment will be placed.com replacement of existing obsolete equipment with newer models. Equipment Acquisition and Temporary Leases: If you are moving between data center providers you may be faced with incompatible equipment at the new location. Transportation: The cost of physically moving should also be included. Relocation or expansion activities will also increase staffing costs due to the need for increased numbers of employees. These include the following: 1. With the management of your website more firmly within your control you may also find yourself having to integrate its operation more closely with other business systems within the organization. With self-hosting this could become an additional expense as you may have to begin a continuing project to provide the same features to your internal users. Staffing: In the event of data center consolidation you may be faced with increased staff costs due to layoffs or rapid attrition which may demand unexpected re-hiring costs. Some people will treat it as a recurring expense. You may also be forced to rent or buy boxes in which to transport your equipment and other miscellaneous items. racks and shelving. computer cabinets. It is easy for a profitable change of web service providers to be weighted down by the allocation of expenses not directly related to the activity. Overtime: Most web site migrations are usually done after business hours at night and on weekends which may require the payment of shift premiums or additional vacation time to compensate your employees for the inconvenience of the working on the project. You could use a professional moving firm or you could rent a truck. The software you currently use may be incompatible with newer technologies or you may not have any . 7. backup systems. Training: This is a frequently over looked expense that is often rolled into monthly costs. This could affect your operational costs at the new location and should be factored into your calculations. Always try to separate the migration savings from the additional expenses of this type of work. fire systems. UPSs. In the latter case you may also have to rent secured shelving.
even if it's informal. Penalty Fees: Some contracts have early termination fees and these may apply to some of your IT infrastructure. This could force you to purchase brand new versions of the software before you can proceed. Chapter 2. but also from technical professionals who are capable of decommissioning IT infrastructure. Once you have an idea of the capital investment required for the migration. Be aware of the cost and plan your migration to possibly limit the extent of your obligations. 10. This may add further delays and costs. power feeds and networking gear. raised flooring. If the return on investment is better than the interest you can get at a bank. Conclusion The decision to migrate your Web site can be difficult. This could require the removal of racks.Why Relocate Your Website? 13 remaining staff that knows how to reinstall your existing software on the new hardware. Cleanup Costs: You may have to restore the original server area to its original condition. Even if the project has marginal profitability. You may have to get professional cleaning help not just from janitorial services. then the project is probably worthwhile. The extra time it takes to do it could save your company thousands of dollars in the long run. 9. improved flexibility and reduced costs with that less complicated management and technical skill requirements. it may be justifiable on the basis of making your company more responsive to market pressures especially if you are moving from a VHP. Movers Insurance: You may have to insure some of your equipment against damage that could occur during the relocation. Always get a professional opinion. plan carefully. . and always be aware of the potential risks of the decision you make. you can compare it with the expected returns in the form of monthly savings or increased profitability. If you decide to do it. Whatever you choose to do. You have to weigh the benefits of increased control. cabinets. This type of financial analysis is often ignored and yet it is so important in making a reasonable decision to switch service providers. "Preparing for Server Relocation" will provide a lot of guidance in completing a successful project. 8.
Don't sign a contract with an ISP where you are held hostage to unsuitable or otherwise inadequate connectivity. basic monitoring and power. The immediate vicinity of the site is also important. recreational attractions in the area. and the availability of amenities such as schools and parks in the neighborhood. You should monitor how traffic patterns affect the ease of accessibility to the site to see whether they are unsuitable. This reduces the risk of outages due to a technical failures as well as construction and landscaping accidents. Rainwater should drain away from the building and then off site to prevent localized flooding. 2. It is also extremely important to visually verify the type of connectivity you have. These are called collocation providers. reducing the risk physical attack.Chapter 2 Preparing for Server Relocation The rationale for deciding to relocate or consolidate data centers was discussed at length in Chapter 1 "Why Relocate Your Web Site?" This chapter explains in detail the criteria you should use to select your new data center and create a project plan. A lot of information will be covered and the numerous action items mentioned are included in the work sheets in Appendix I to help make the process easier. Ensure that the building has sufficient excess capacity to handle future growth. Your employees may have other personal interests in the location such as the presence of reasonably priced housing nearby. There are many factors related to the facility and its services that need to be considered that are often overlooked. hurricanes and earthquakes. As expected. Communications: The facility should have access to multiple ISPs with the cable entering from different points of the building. The first only supply computer room floor space. Locations close to hazardous production facilities and aircraft flight corridors should be avoided. Location: The data center should be positioned away from zones at risk from natural disasters such as flooding from rivers and dams. The second group provides more comprehensive management that may include all possible IT services related to your site including systems development. It should also be no closer than a quarter kilometer away from major highways and railroads to reduce the evacuation risk from toxic spills. the UPS feeds a network of power distribution units (PDUs) to supply each section of the floor with a series of circuit breaker panels. Electrical: Power should be supplied from multiple feeds from different substations. access to an ISP. These are called managed hosting providers. There are a wide range of varying service levels in between and the interpretation of the terms within the industry can often be very loose. the selection of a suitable data center will play an important role in any data center or web farm relocation project. Verify the roof access rights in the event you need to have a satellite or microwave line of sight antenna installed. Make sure that every rack or cabinet you intend to use has access to outlets from at least two PDUs and that each PDU is operating at no more than 45% so that it can handle the full load of the other one if it fails. access to public transportation. Data Center Selection Criteria There are two broad categories of data center providers. . These include: 1. 3. This is discussed in greater detail in the appendix. In large facilities. Always request a very specific list of the services your data center provides as part of your selection process. The facility should also be able to run without interruption if its largest standby generator or UPS are offline for maintenance. Be certain that both the ISPs that enter the building and the types of data circuits they can provide are suitable. In high security environments the building should be surrounded by embankments and perimeter fencing.
This helps to deter ID card fraud. then the sprinklers release the water only after a pre-defined temperature has been reached. A polling cycle of five . If you require only one link. Some will only have a demarcation point where ISPs have placed their equipment. ask whether testing includes the use of a load bank to simulate the power consumption of the data center. When multiple links are required. Inquire about whether the UPSs have ever supplied the full load of the data center and when last the system. Appendix II will cover many frequently used terminologies and scenarios to help you evaluate your options better. Standby generators can be regularly started without revealing any apparent problems. then you'll need to configure a single default gateway on your network equipment to get to the Internet.How to Choose a data Center ISP 15 Request a history of outages or other irregularities in the feeds from the site's utilities and ask how you'll be notified by the facility of any electrical maintenance work to be carried out by either themselves or their providers. Detailed discussion of typical network connectivity issues usually requires the services of a network engineer and is beyond the scope of this chapter. 6. The facility's staff should also be automatically notified by monitoring equipment of any disruptions in the power supply to the area. Also in this case. but they should be linked to an alarm panel that graphically shows the location of the fire on the building's floor plan. Connectivity can become more complex than it first appears. Fire Protection: Not only should there be smoke and heat detectors. the server area should also be isolated using fire proof doors. Network Monitoring: It is often taken for granted that your data center provider continuously monitors its equipment for failure. verify this. This helps to prevent problems due to extinguisher and CRAC unit leaks. Racks of densely packed servers and data storage can be power hungry. If your data center is situated on raised floor tiles.Appendix II . Some data centers also link visitor ID cards with a person's biometric information through the use of a palm reader. Here the pipe lines are pre-filled with pressurized air to reduce the risk of flooding during normal operation. be on the lookout for water leaks. Water only enters the piping after an alarm signal has been detected. 5. If possible. Ask how quickly the generators respond to an electrical outage and how long the UPS batteries can last. you'll need to configure a dynamic routing protocol on your network equipment. Network Connectivity: Not all data centers will provide you with Internet connectivity. Ask about the frequency of the checks. This should include offices and common areas being isolated from the data center floor. False alarms are minimized by requiring two events to occur before the system is activated. you should ask whether there are liquid detectors underneath. Verify that the power per square foot that the data center can provide meets your needs. These systems are less damaging than water based ones but they are usually designed for fires of short duration. mandatory visitor/employee registration or electronic ID access and interior/exterior video surveillance. Larger fires will often require a pre-action water based system. This is an industry standard method of fire prevention and it should be on your checklist. The first line of defense should be a gaseous system that suffocates the fire by displacing the oxygen in the air. Cooling: Most data centers try to maintain a 75F/25C air temperature. Investigate how frequently the equipment is tested and how often it is maintained. On your plant tour be on the lookout for computer room air conditioning (CRAC) units that squeak or rattle loudly as it could be a sign of poor maintenance. Security: Verify that there is 24/7 security enforcement. This will automatically calculate which of the many links will get to the data to its final destination most quickly. including the batteries. You will then have to contract with the ISPs to extend a data circuit to your server area. was last maintained. 7. 4. There are different types of data circuits requiring varying types of adapters on your network equipment. 8. It can also be used to bias traffic to and from your web site on the cheapest ISP link and will automatically fail traffic over to the remaining ISP circuits if one of the other circuits fail. make sure that the cabling lies in trays above the floor out of harms way from minor flooding. Condensation from CRAC units should be drained away immediately through piping.
With authoritative services. Ask about the number of failed polls that will trigger an alarm and whether they too will automatically generate a ticket. the data center floor should use anti static tiles to reduce the risk of electrostatic shock damaging your equipment components. or "traps". security. floor space rental and custom services. ask about the procedures for updating DNS. Some facilities allow you to reserve the area immediately surrounding your server area for future expansion.16 Relocating Servers Between Data Centers - © LinuxHomeNetworking. This protects the data from catastrophic events at the main data center. you may have to invest in data conversion services with a third party. error and data throughput rates. Also ask about the times during which scheduled maintenance is done and the types of notifications that are provided. Request a summary of escalation procedures used when problems occur and whether there is a formalized means of documenting and permanently fixing problems. and server information related to your site. 12. behind different firewalls. 11. they should not be directly above the area. in multiple locations to prevent your web site from being affected by one of the servers going down. Facility Cost: This factor can present itself in many different ways to include pricing for bandwidth. The facility should also use SNMP (Simple Network Management Protocol) to track CPU. The data centers in your vicinity may not meet all the criteria but the list should allow you to reduce your final candidates to a manageable number. Also ask about the types of checks done. 10. If not. It is a good idea to determine what the total costs would be over the time period you expect your current website architecture to be used as the costs can be presented as recurring and/or one time expenses. Lower recurring costs can easily give the perception of cheaper operating expenses but the price may become unfavorable when higher setup fees are taken into account. 9. such as high CPU usage. They should also store most of the data at a secured secondary facility. steam lines. Data Backups: The backup system you are using at your current location may be different from the one used at the new facility. Ask about the availability of such services and possibly more complex ones through an as-needed contractual or longer term retainer based agreements. Verify whether the new facility can handle data backed up using your software on your backup media. DNS: Ensure that your data center uses multiple DNS servers. cooling.com minutes or less is generally acceptable. It is possible for SNMP enabled systems to send notifications. billing. Water pipes. occurs. when components fail. ICMP (Internet Control Message Protocol) or "ping" tests only check basic network connectivity and server response. Floor Space: For improved safety. contact. Good backup services usually store data for a predetermined period of time before reusing the media. memory. the lead time for requesting changes and the format of the DNS data the provider will need to enter it into their systems. This information should be fed into some form of a job ticketing system that will ensure that the problem is fixed quickly. or a predefined event. Customer Support: Ask about the availability of a web portal through which you can view statistics. power. This could be the source of difficulties if you have to restore historical data during or after the relocation due to server failure or human error. The work that will follow will demand a lot more from your IT support staff and will have to be carefully coordinated as we will see in the following sections. From time to time you may need simple services such remote hands on help in rebooting a server or changing a backup tape. 13. but also authoritative DNS services to handle Internet queries for your Web domain. You should also determine whether the location has sufficient floor space to handle our current and future needs. Remember that this is a perfect wish list. Also. . Data center selection is only the first phase of the physical planning for the relocation and will largely be the responsibility of your facilities and networking teams. bathrooms. Some facilities will provide not just caching DNS for the exclusive use of your servers. Verify that this type of extra data security exists. kitchens and other sources of moisture should all be located a safe distance away.
They can still play important roles in the preparation steps. You'll probably need to start with a number of meetings to inform each of the affected groups about the project. Procedures Documentation: There are three types of procedural documentation that will have to be up to date. networking and backups. As part of the definition of the roles and responsibilities. There should be persons to lead transportation. server shutdown/startup. An often overlooked role is one of the "gofer". The second would obviously be the documentation for systems that will change after the project is over. or to find the software CDs that "must be somewhere over there in those boxes". These would include step by step commands that a technician or engineer would need to execute as part of the process. Constantly changing priorities can be disruptive. There are many aspects of the migration that need to be thought about prior to arranging the first meeting. The third type is equally important. Disaster Recovery Team: Have a group of persons assigned to disaster recovery. be prepared to schedule daily and sometimes twice daily meetings to achieve your goals. Set up meetings on these days to determine whether the project or sub-project should be aborted. Project Management: Have a single overall project manager for the activity. As the deadline date draws near or as the complexity increases. The project manager will inevitably be distracted by other events and this will help to ensure that forgotten technical issues don't threaten the success of the migration. 3. . Spreadsheets can track static information well but do relatively poorly in monitoring the status of dynamically changing deadlines. customer communications and the locking of the doors at both the old and new facilities once the relocation is complete.How to Choose a data Center ISP 17 The Relocation Project Plan Detailed logistical planning of all the steps related to the relocation needs to be started well in advance of the deadline date. but should be given a reduced workload during the relocation itself that will allow them to dedicate their time to such activities. Plan to include deadlines after which time no further changes may be made. It is the documentation of the steps each participant is expected to do during the relocation. Some of the most pressing ones will now be discussed.Appendix II . The first relates to those used by your existing systems which won't change as a result of the relocation. These persons don't necessarily have to be sitting idly by waiting for something to break. It should include staff that is familiar with systems administration. application testing. These will have to be followed by project planning meetings in which roles and responsibilities are assigned and progress reports given. Remember to give them some small reward when it's all over as it is one of the most thankless jobs. some participants will need to have a detailed task list to help prevent them from making errors. Coordination Preparation There are a number of things that need to be considered prior to setting up specific functional groups for each aspect of the relocation. 4. 2. someone who will go for anything that you have forgotten. Roles and Responsibilities: Create an activity checklist that assigns each member in the team clearly defined roles and time frames in which to get activities done. These are discussed next. networking. pickup catered food. This person can cross this activity off their check list when the tasks are completed for better control of the change process. database administration. It could be to buy cables you forgot to order. continued as planned or given a preparation time extension. Specifically assign someone the task of keeping track of the problem equipment that may fail. If the project starts to become complicated invest the time in tracking it with software tools such as Microsoft Project. 1.
items may have to be returned or replaced. 8. They should be qualified. make sure your daytime skeleton staff is capable of handling regular business issues and can relieve the night staff of some of the technical problems that may arise. Inventory: You will have to do a complete inventory of all the equipment to be moved. Interpersonal Communications: Make preparations to have a permanent conference bridge open so that all members of the team can be better coordinated in the event of a crisis. 11.18 Relocating Servers Between Data Centers - © LinuxHomeNetworking. Create a shortlist of scenarios during the actual relocation under which the project cannot go forward. and if possible. Also identify a point of no return at which you cannot roll back your changes. Plan of Retreat: Create a plan of retreat in the event that things go dreadfully wrong. A sample purchasing check list for purchases is provided in Appendix I. The actual required information for each type of equipment will be covered later in the chapter and accompanying check lists are provided in Appendix I. so ensure that it has the capability of creating word processing or spreadsheet versions of the reports for more universal distribution. ensure that these servers are functioning correctly. This will also have to include "before" and "after" data related to the network connectivity and physical location of each device. Not everyone will have access to the application. This list should be distributed to the entire team. Record this information in a database if possible. 9. Relocation Date and Time: Determine the best time for the migration. These are some of the general preparatory tasks that need to be done and you may have to add a few that cater to your unique needs. technicians and contractors. For nighttime moves. experienced and authorized by the manufacturers they represent. their home phone numbers. Contractors should also use the correct tools and be able to test the quality of their work. 12. If things go wrong. and/or over an extended period. 13. It will allow for very flexible reporting including individual status and data sheets for each device. allow for overtime. Verify that there won't be delays due to rush hour traffic or road maintenance at the planned time. you may forget to order something. Have some mechanism of informing everyone of the decision. A brightly colored sticky note stuck to a server and the rack or cabinet in which it is located is usually sufficient. Equipment Leases: Some relocations cannot afford any downtime at all and you be forced to purchase or lease equipment to create a duplicate environment at the new location. 6. mobile. In this case create a minimum list of servers that need to be functioning for the website to be adequately operational. You may want to adapt it to a spreadsheet format to make it easier to share with your vendor. 14. You may have to assign the acquisition of such equipment to a team lead and adjust your budget accordingly. Create a list that defines the sequence in which servers should be returned. Participant Lists: Have a complete list of participants in the relocation. Practice Migrations: Plan to do a practice relocation of some non critical servers to see whether you are really prepared for the full scale operation. catered food and possibly compensatory time off. Items have varying delivery lead times. There are many things to track. 7. It should also include contact information for all third parties involved with the activity such as movers. Failed Equipment Identification: Have some way of marking equipment that isn't working. and deadlines may shift. It is a good first step before proceeding with more specific plans. Make sure all active participants at the time of the migration all have mobile phones on their person. A check list for contractors is provided in Appendix I. Vendors / Purchasing: One of the most difficult aspects of a data center move is the coordination of purchases from your vendors. This makes it very easy to identify broken equipment from a distance.com 5. If it takes place at night. 10. Contractors: You may need to use contractors to do some of the work your staff may have neither the time nor ability to do. Make everyone aware of this process. . This should include their work.
2. You should also have someone monitoring the web media and blogosphere to get feedback on what people are saying. simply reference your updates web page. have a video or audio clip of the company President/CEO making a factual statement about the matter that expresses remorse.Appendix II . Focus on the most important ones to provide them with personalized verbal service assurances. expected features and associated benefits of the change. Plan to have this message delivered in person to your most valued customers. The message should explain the reasons. Contingency Planning: Create a communications plan in the event of a failure. business prospects. Provide regular email updates to employees via email. Significant events outside this schedule should be reported immediately. and analysts. the press. Use this updates page to correct misstated facts neither making excuses nor being defensive. Targeted Messages: Have a single message with varying degrees of detail for each customer group depending on their information needs. Customer Notifications: Provide ample warning of the impending activities so that your customers can plan for the change. Use a company wide all-hands meeting to provide a first hand situation status. Provide sufficient information to make all affected parties aware that you're taking the matter seriously. This should also include contact information for your crisis spokesperson. If the issue has public visibility. Make sure it is delivered to all internal departments that could be affected by the planned activities. > > > > > > > > . The callers will have to be prepared to take some abuse. When comments are necessary. It should be brief as there will be lots of work to do. 3. Here is a list of guidelines I strongly recommend be followed: > > Create a web page or blog to provide updates on the progress of problem resolution. Create lists of potential questions. personalized verbal assurances and updates can be valuable. Use the Q&A discussion points. Don't forget to contact customers that are not impacted. For example. and a promise to have quick resolution. Senior management should make some calls too to gain first hand understanding of the pain everyone is feeling. web surfers may need to know that your site may be unavailable for maintenance for a specific time period but business partners may need to know about any new procedures the change may create. Only authorized persons do such limited commentary on blogs. Have senior management call important sales prospects to provide justifications to choose your company in spite of the disruption. In times of crisis. Consider these activities as part of the plan whether the relocation is successful or not. Have them allow the customers vent while sticking to the script. Describe the extent of the problem and a statement that addresses any possible concerns of those who may not be affected. answers and discussion points for those parties that are affected. Attempt to give every customer a personal phone call. Update this page at predictable stated intervals even when there's nothing to report. and who's saying it. Send e-mails to all other nonimpacted customers with statements about service continuity.How to Choose a data Center ISP 19 Customer Communications Preparation Notifying your internal and external customers of the expected changes will be critical to the success of the project. not affected. 1.
This is especially important for systems with dual power supplies that should be plugged into separate power sources. Develop an easy to use post-mortem template that can be used to document any failures related to the migration. Make sure each rack or cabinet can receive power from redundant PDUs and that there is adequate excess capacity on the PDUs to support not only your server farm but also the failure of one of the PDU units. . Some power hungry devices may need unusual voltages or electrical connectors. Make sure power cables are labeled with the name of the server at both ends too. This information should be made available to all members of staff involved in the migration and to customers who may demand detailed explanations of the cause of disruptions. The rule of thumb is that each watt of power consumed by a server requires a watt of cooling. 2. dates and times the events occurred. Cooling: Verify that the area has an adequate number of CRAC units to cover your anticipated power load. A poorly prepared area can cause unacceptable delays and even a complete site shutdown. Always include them as part of any relocation plan to ensure a more complete success. 4. The problem usually occurs when converting from fixed racks to ones in which the servers are mounted on rails. Follow these steps to reduce your risk. chilled and then returns to the server area under the floor blowing up into the cabinets through vents in the floor.20 Relocating Servers Between Data Centers - © LinuxHomeNetworking. error messages. hot air is extracted from the room. Labeling: This is very important. You may also find yourself in a situation where the overall cooling requirements of the server farm are within the specification of your combined CRAC units but certain concentrations of servers within the farm could overtax the capacity of individual units. double check this information ahead of time. Verify that the servers in each cabinet or rack won't overload the power circuits supplying them.com > Contact key press and analysts. The floor under the tiles should therefore be clean and generally clear of obstructions such as cabling and ducting. the final solution and steps that will be taken to prevent the recurrence of the problem. Have the facility's management prove that you are getting UPS protected power in your area. Get a total figure and an estimate of what you expect to consume per rack or cabinet. They will appreciate hearing directly from you even if they remain highly critical. Power Management: Your servers and disk storage will consume the most amount of power in your environment. Once the migration is completed you'll have to test air temperatures and humidity to ensure they meet the requirements of your equipment. If possible baffles can be placed under each CRAC unit to guide the air flow in the direction of the cabinets it needs to cool. Server Area Preparation The health of your server farm depends on the quality of your physical infrastructure. It should include the persons involved. Do an audit of the power consumption to determine how much you will require in the new location. This allows you to slide the servers out into the aisles for better access but requires cables that can stretch the distance. It can be very frustrating to arrive at the new facility to discover that the server power cords used at the old location are too short for the racks in the new one. Lessons Learned: Some aspects of the migration may fail and documenting the issues can lead to much better experiences in future. 3. In some cases the floor under the cooling zone of the unit may need to be sealed off to force the air only to the required cabinets. Plan to spread these high power density racks across the server floor to help balance the load across all CRAC units. In raised computer room flooring. Verify that you have sufficient quantities of adequate cables. Ensure the power outlets are labeled with the PDU and circuit breaker number. reasons for the failure. 1. The establishment of clear channels of communication with your customers is always important but especially so during projects of high risk.
Make room for monitors and their KVM (Keyboard.How to Choose a data Center ISP 21 Make provisions to have all servers labeled on the front and the back to reduce the risk of incorrect cabling and likelihood of making a mistake with a hard (power cycle) reboot in the event of an unexpected server failure. You may have to install patch panels to connect the server racks and cabinets to those containing your network equipment. This port is in turn connected to an equivalent port on a patch panel in the network rack. With raised floors. You would then connect your server to a patch panel port in its rack with a standard network cable. This can affect the correct channeling of the cooling airflow through the servers and can usually be avoided through better patch panel layouts ahead of moving time. the air blown up through the cabinets is insufficient to cool the servers and perforated floor tiles need to be placed in the cold aisles for added cooling. Remember that perforated tiles located in hot aisles are counter productive as they will help to cool air the servers never use. Sometimes with raised flooring. cabinet 4. Each zone will have a number of rows of racks and/or cabinets. Also make sure that all network cables are labeled at both ends. This equipment will need to be identified and located accordingly. It is for this reason that CRAC units should be placed in line with the hot aisles so that the air can easily be extracted from them. Server cabinets come in a variety of widths. CRAC units extract air through filters on the top of the unit and blow chilled air through vents at the bottom. 7. video monitor. Split up the server area into zones serviced by the same patch panels or switches. Make sure that the . As expected. Sometimes the walls between adjacent cabinets are removed to facilitate cabling. Without the blanking panels. Finally.Appendix II . row 11. Mobile Monitors: Video monitors on carts should also be available at the new facility for troubleshooting servers that need to be removed from racks or cabinets. By using another standard length cable you can extend the connection to your network gear from the network rack patch panel. You should also label patch panels in a similar way so that 1-11-4 p7-2 would refer to the 2nd port of the 7th patch panel in cabinet 1-11-4. 5. the CRAC unit vents are physically below the floor level blowing air up into the server cabinets. When regular flooring is used. Servers on opposite sides of an aisle should either face each other or be back to back. Diagrams: Create diagrams that map the precise layout of servers in the racks and pre-install shelving and rack kits at the expected locations. A location number such as 111-4 could mean zone 1. This will make all power and network cabling connections reside neatly on one side of the racks. the most common one being 19 inches wide. It is a good idea to put the heavier equipment at locations near the bottom as this will make them easier to insert and remove. Network Cabling: Determine your network cabling requirements based on your server layout. the usual swirling vortices of exhaust air can easily be blown back to the front of the rack through the spaces. some types of equipment may be too heavy for regular server cabinets and will require the use of racks as an alternative source of support. This creates a better cooling environment as the hot power supply exhausts of one server won't be sucked in by the front facing air of the server behind it. mouse) switches also. You may have to plan for the purchase and installation of such a system. 4. Remember to consider the use of both copper and fiber connections. Rack Usage and Orientation: Install the servers in the same direction in the racks. How do you start to number? Numbering schemes for cabinets and racks are usually straight forward. Remember to make the aisles wide enough to allow people to easily mount and dismount servers in them. In this case the baffles and sealed floor techniques mentioned in this chapter would help channel the air flow better. you may require ducting to blow the chilled air into the cold aisle. Copper Ethernet cable used for 100 Mbps communication can be no longer than 100m in length. When cabinets are used you should insert unperforated blanking panels in any spaces between the servers to better channel the cooling air from the front of the cabinet to the hot aisle in the back. the servers should be stacked vertically. 6.
These circuits should not only be sized to capably . Diagrams: Ensure you have a complete set of network diagrams that include each server and network device that will be relocated.22 Relocating Servers Between Data Centers - © LinuxHomeNetworking. 3. Verify that the network links have been installed and tested prior to your migration date. Glass fiber cables for servers are delicate in comparison to copper. and ISP circuit number. Facility Access: Verify that each person that is going to have access to the server area has key access and parking rights at the data center beforehand. move it to the new location. Preconfigure: Setup your new network equipment at the target data center and test connectivity ahead of time. Connectivity should include tests from the Internet and practice servers at the new location. Bundled data cables should be wrapped together with Velcro. Data Circuits: Keep close track on the provisioning of data circuits for the new location so that they are installed prior to the migration date. The entire relocation depends on the proper preparation of the server area but fortunately you can save time by simultaneously preparing for other aspects of the move. Tools: Make sure each person responsible for the racking of the servers has a correct set of tools. Equipment List: As mentioned previously. Another equally important reason is to help in server farm relocations. do a complete inventory of all your networking equipment. Create a comprehensive list of all the important networking information that will change as a result of the move. cables shouldn't hang in the air or be stretched taught. access control lists. 10. tie wraps to make it easier to add additional wiring to the bunch. Have separate drawings that clearly show how the network cables plug into the switches from each server. Some sites require T1 data circuit links to credit card facilities or VPNs to remote offices. Network Preparation One of the most obvious reasons for having redundant network hardware is to help protect against hardware failure causing your Internet connectivity to fail. 4. Make sure your routing. does not exceed this length. such as database servers. 2. Verify that you'll be able to switch monitoring from your old server address to the new one seamlessly. there are more preparations that need to be done. Internet Connectivity: Ensure that your Internet connectivity to the area has been secured. via your patch panel system. The bundles should be run to the sides of racks and cabinets so as not to impede airflow. should be directly connected to different switches. These will be explored next. switch port. To further reduce the risk of damage. 1. route. Multimode fiber has a maximum distance of 2Km when running at 100 Mbps and between 220m and 500m when running at 1 Gbps. and not plastic. 8. 9. VPN tunnels and firewall rules all take the IP addressing scheme you will be using at the new location into account. Have many and also have lots of charged replacement batteries. That isn't all. and preconfigure it in anticipation of the server migrations. Make sure these are in place and tested before the move too. Wherever possible ensure that they are run in separate cable trays to help prevent possible damage. They should include every IP address. 5. Also make sure that the power cables run in separate trays or conduits from the data cables to reduce the risk of damage and electrical interference. Servers that play similar roles. The most noticeable time saving tools will be electric screwdrivers. Redundancy allows you to shutdown network equipment. gateway.com combined length of your connections. This will help illustrate whether too many of your servers are vulnerable to the failure of a single network device. Monitoring: Special attention should be given to network monitoring.
TCP/IP ports: It may seem tedious. If possible. This can be recorded in a simple spreadsheet and would include IP addresses. 4. Windows servers provide the option of having a default gateway per NIC. Create a comprehensive list of all important server information that will change as a result of the move. In this case. Logging: Many managed networks have centralized error logging and authentication servers. This will be covered in Chapter 3. In Linux and UNIX systems there is only a single place to enter this value. 9. The new ones will help protect against hardware failure in your new facility. This would be attached to the server so that the engineers working on it would be able to instantaneously reconfigure it when it arrives at the new location. It should also include information such as the server's name and serial number for inventory purposes. 6. it will help to identify inter-network application dependencies between servers which can be used to determine the servers that should be relocated together as part of the same group. Server List: Do a complete inventory of all your servers. Each server should also have its own separate worksheet document that contains all its relocation information. Get printouts of these for each server as part of the server's more comprehensive post migration system check. Server Preparation Server preparation for the migration is probably the most complicated task because there are usually many of them with each running multiple applications that rely on the functioning of varying components.Appendix II . Follow these simple steps to make the job easier. backup server IP addresses. 3. It will also be helpful in more precisely restricting the TCP/IP access between servers on your new networks. routing gateways. 1. VPNs: Some corporate offices use VPNs to gain access to their Web server farm. Make sure that you have a data restoration unit at the remote location that will be able to restore your data from your backup media using the . but get a printout of all the TCP/IP ports on which the server is listening and also which clients have active connections to your server. Routing Tables: Note the routing tables of all servers before the migration using the netstat -nr command. Most operating systems can also give you a snapshot of services or applications that should be running on startup.How to Choose a data Center ISP 23 handle your expected data transfer rates but also tested at various times of the day to ensure your ISP has met their contractual commitments. "Post Relocation Activities". Data Backups: Archive all your server data. This is especially important for noting the default gateways and also for analyzing routes on servers with either multiple NICs or routers. 8. subnet masks. including Windows. This can be done with the netstat -a command in most operating systems. Some types of equipment require modem lines to provide emergency out of band technician access in the event of an emergency. Make sure your relocated network devices can continue to do so. Determine what the new routes should look like at the new location and note it down. make sure that only one NIC has a default gateway configured. This would require the additional installation of one or more POTS telephone lines. terminate some test VPN tunnels on the network equipment at the new location ahead of time. Finally. This will help you to identify the applications that should be running before and after the relocation and can be used as a quick check to detect any unexpected failures. 7. The old ones will be helpful if you have to quickly roll back the work to the original data center. Samples of both documents are available in Appendix I. both old and new prior to the relocation. 2. Backups: Create copies of all your network configurations. Cabling: Make sure that network cables are all re-labeled to reduce the risk of human error when the servers are reconnected to their new network. the switch ports the servers will use and server rack locations. Once the migration is complete you'll have to plan for recreating a redundant network architecture in the new location. Note: A server should have only one default gateway.
This will require you to account for each application within your environment which should also include their interdependencies with every other application. the order in which they are accessed in the RAID set and the type of RAID being used. In these cases data restoration by tape can be excessively long which can make this an option of last resort. Large databases are often stored on storage area network (SAN) and network attached storage (NAS) devices and are too large for the USB solution. Sometimes co-dependent servers have to booted in a special order. It will help to focus the minds of the team members in the event of multiple failures.0 connections could simplify smaller backup and restoration work. Don't limit this to just network connectivity. RAID BIOS Settings: An often ignored item is the server's BIOS settings. The regular parameters are usually easy to determine as the defaults are usually sufficient. 8.com same software. Customized Procedures: In many cases applications on servers have to be started or shutdown in a particular sequence. 5.24 Relocating Servers Between Data Centers - © LinuxHomeNetworking. Plan well and you should be able to have a successful project. Always remember to do practice data backups and restorations for key servers and applications. 11. The real problem is with the BIOS metadata on hardware RAID cards. the firewall rules that protect them and any application batch or cron jobs they rely on. Application Code Surveys: Test to make sure your applications don't use IP addresses to access information on remote servers but use DNS names instead. The relocation may force you to change the IP addresses of devices and could cause some programs to fail unless this precaution has been taken. 9. Base Equipment List: Create a minimum list of servers that absolutely have to be up and running in order to maintain the web farm's functionality. their software interfaces. Another option is to restore the data ahead of time on the new NAS / SAN equipment located at the new facility. have a set of spare servers that can be used for spare parts or complete system replacements in case of failure. 6. The audit will also help to determine the groupings of systems that should move together. Schedule a server reboot before the relocation and enter the RAID controllers BIOS setup to record this information. The general steps required to prepare your servers for the migration are not hard but the process can become difficult due to the sheer volume of information you need to track. At any moment in time the data bases at the two locations will be synchronized. a sudden jolt of a RAID card's loose onboard battery backup could cause you hours of downtime. This often cannot be guessed. . You can then set up a data circuit between the old and new data centers so that any new transactions can be replicated between the two. Functionality Testing: Create a short list of tests for each server that will be used to verify that it is functioning correctly. Spare Parts: If possible. Without this simple plan. It may also influence the order in which the servers are relocated to the new data center. This metadata lists all the drives in each RAID set. but also check that all the required applications have started correctly and that some simple business operations can be successfully completed. Migration Timetable: Perform a careful audit can help to determine the number of days over which the migration should be spread and the sequence of the server moves. 10. With SAN and NAS data bases you may need to lease a duplicate device and clone your data to it. Disaster recovery can be much faster as the secondary device will be preconfigured to replace the failed primary one. 7. The recent advent of portable external hard disks using USB 2. You will need to document these special procedures wherever they exist and make note of them in your project plan.
Appendix II . Transportation Preparation A relocation would certainly not succeed without adequate transportation therefore it should be planned well.org if the new Web server is going to also be your new mail server. Just edit your /etc/hosts file on your Web browsing Linux PC to make www. for example www. Server Based Testing: Set up your test server in house. If anything goes wrong. Your best alternative is to request your existing service provider to set the TTL on your web site.org.How to Choose a data Center ISP 25 DNS Preparation If the relocation requires the IP addresses of your site or servers to change then you'll have to make plans to adjust your DNS settings during the relocation.my-web-site. .org refer to its own IP address. Once the propagation is complete.org even though DNS hasn't been updated. Plan to change your DNS TTL at least a week before the expected migration to limit it risking the success of your project. Prepare to Switch: Once testing is completed. The second is that every DNS entry has an associated time to live (TTL) value which defines how long DNS caching servers should store the entry for local use before being required to query the entry's authoritative DNS server to see whether there have been any changes. This file is usually given a higher priority than DNS. coordinate with your Web hosting provider to update your domain registration's DNS records for www.my-web-site. In the case of Windows. Remember to also update the DNS entries for your mail servers. in the DNS zone file to a very low value.org if the new Web server is going to also be your new mail server. Client Based Testing: Test the server from a remote client. it will take only one minute to see the results of the final DNS configuration switch to your new server.site. 5. DNS management is probably the easiest task to accomplish but poor DNS planning can unexpectedly delay your project with your only recourse being to sit and wait for the changes to propagate. they are generally located in a different section of the DNS zone file and can be easily overlooked.org to point to your new Web server at the time of the relocation. not that of the www. hence you can use them to predefine some DNS lookups at the local client level only. you can then revert to the old configuration.my-web-site. You may also want to add an entry for mail. The first is that it will take at least 48 hours for any DNS change to propagate across the Internet. Set The TTL: There is no magic bullet that will allow you to tell all the caching DNS servers in the world to simultaneously flush their caches of your zone file entries.org map to the IP address of the new server. say one minute.my-web-site.myweb. here is what needs to be done: 1. As the TTL is usually set to a number of days. the file would be C:\WINDOWS\system32\drivers\etc\hosts. 2. 4. You can test the server running as www.my-web-site. There are two things to remember with DNS.org is really hosted on itself.my-web-site. it will take at least three to five days for all remote DNS servers to recognize the change. Edit the /etc/hosts file to make www.my-web-site. 3.my-web-site. Here are some factors to consider. Check All Domains: Make sure similar steps are taken for all your DNS domains. With this in mind. Test your server based applications from the server itself. You may also want to add an entry for mail. therefore the test server wi ll begin to think that www. Your client will usually refer to these files first before checking DNS.org site that is currently in production. This should include mail. knowing it will rapidly recover within minutes rather than days. and so on. Web.
Do not stack servers one on top of the other as this increases the risk of damage. Sample check lists. Get access to hand carts or wheeled dollies on which the servers can be manually pushed within the buildings. Finally. Internal Logistics: Servers can be heavy. Cost shouldn't be the only factor. rent ramps to reduce the need to manually transfer servers at the various stages of transportation along the way. renting a truck or using professional movers will be much faster.com 1. and the claims settlement customer service of each moving company. make sure the mover clearly explains the extent of your exposure. 2. Investigate the reputation. Reserve Transportation: If you choose to rent or use movers. Moving Company Selection: Get multiple quotations from movers. Determine whether the company belongs to a trade organization that requires a code of ethics and operation. reliability. . have guarantees that the transportation will arrive on time.26 Relocating Servers Between Data Centers - © LinuxHomeNetworking. "Post Relocation Activities" will begin by discussing what needs to be done during the relocation and will end with a number of activities that need to be completed once the project appears to be over. You should also ensure that the servers are securely fastened to prevent shifting in transit. Make adequate preparations in advance. preferably with each provider giving a guaranteed maximum price for the job. less prone to equipment damage and easier to insure. Check to make sure the selected moving company carries the required insurance coverage. Visit the mover's office to verify they have a business. 5. Carefully consider whether the staff are people you want to do business with. but the few dollars spent on ensuring proper transportation can save thousands in potential down time. Some may have to be bolted to shipping pallets and mechanically moved. Insurance: You may have to insure the equipment prior to the relocation. Transportation is often given the least amount of thought and servers will inevitably be carried on the back seat of cars. Conclusion The preparative tasks for a server farm relocation can be complex but with the right tools and planning it can be very manageable. Physical Protection: For large quantities of servers you'll need to have racks or shelving preinstalled to accommodate as many servers as possible. you should request that it be fixed or replaced to the limits of their liability. If they break or lose something. Avoid this as much as possible. 4. Chapter 3. and post mortem forms are available in Appendix I. You may be tempted to save money here. 3. There can be many clauses to this type of coverage. Most importantly. performance record. easier to track. moving van cleanliness and safety. it specifically outlines what to do if things start to go wrong. If practical. Some of the more delicate devices may have to be specially wrapped for their protection in bubble wrap or foam. staff training. some vendors with full coverage maintenance contracts my stipulate that their staff be the only persons authorized to package the equipment.
"successfully tested" and "failed" with only one color being assigned at a time. Assign a technical lead to be responsible for the full recovery of the situation even if the solution wanders outside of their area of expertise. the booting up of the servers in the new location and testing. The red. Those responsible for the shutdown of the old site. DNS Updates: You'll have to synchronize the update of your DNS entry with the movement of the servers. Troubleshooting and Recovery: Things will go wrong.org to point to your new Web server. Activities During the Relocation It may appear that with good planning the relocation should progress smoothly. The use of email to track the minute by minute progress of a problem is generally counter productive as many of the participants will not have access to email during the activities. . It may be something as simple as a sticky note. As the TTLs were set to one minute previously. These should include status colors for "untested". Team Allocations: You'll probably need to split your team into four groups. 18. Make sure you have a conference call bridge open throughout the activity period. green and yellow traffic light colors are one obvious choice. In times of crisis real time communication is always best. The testing teams should report the status of each device periodically. or self adhesive dots that are commonly available in office supply stores. This chapter outlines some precautions you'll need to take. 14. white board or some other highly visible tracking system to monitor the basic status of every device being migrated. Move the servers in the predefined order on your check list. you'll be able to see results of the migration within minutes. That is largely true but the challenge is in the coordination of the activities when they go right and when they go wrong. It is also a good practice to have an equivalent managerial lead for overall responsibility of the issue. Customer Communications: Send out a notification to all affected parties that the migration is about to begin. This makes the transfer of troubleshooting responsibility from group to group much smoother. Delete any test entries in your server /etc/hosts files to make sure they don't unexpectedly interfere with future migrations. What should you do? It may not be as straightforward as it first appears. Assign someone to issue general status messages when milestones in the troubleshooting have been reached. 15.my-web-site. The advantage of doing this is that it ensures that there is at least one technical person on the line who has a complete understanding of the history of the problem. 16. the physical movement of the servers. Try to limit the attendees to only managers once the initial problem has been identified as your technical staff will function best if left relatively undisturbed during troubleshooting. The abundant use of the "Reply All" button should be avoided as many team members will miss one of the many email threads.Chapter 3 Post Relocation Activities The deadline day has finally arrived. They should place one of three markers on each system. make sure everyone has access to a mobile phone. Coordinate with your DNS provider to update your domain registration's DNS records for www. If more than one problem occurs create another conference bridge and assign another pair of technical and managerial leads to reduce confusion. 17. you can set the TTL back to the original value to help reduce the volume of DNS query traffic hitting your DNS server. Testing: The overall project manager should have a check list. Once the migration is complete. This can cause delays.
19. Without good contingency planning the success of the project could quickly falter. The steps outlined previously should help to further guarantee that your preparations help to achieve your objectives. 3. 8. Thorough Testing: Test to make sure your customers and partners can gain access to the expected areas of your server farm. Monitoring: Verify that the monitoring and error logging are working correctly and that your predefined basic functionality testing is progressing as planned. Simulate some known but non fatal errors and see whether your systems can detect them correctly. This would include maintenance contracts. leases and data circuits. If not. Relocate Redundant Equipment: In many migration plans. test maximum data throughput rates. Monitoring and Reporting: Verify that your logging and monitoring are working correctly. Redundancy Testing: Plan to test the redundancy of the site. Post Relocation Activities Relocation activities don't end with the passing of the deadline date. execute your roll back plan to return your servers to the original data center. and you have passed your point of no return. 5. do practice data backups and restorations. A simple alternative is to maintain your investment in these central switches. Service Level Verification: Verify that all the promises made by the management of the new facility have been met.com If things are going really badly. focus your resources on getting your minimum set of servers functioning as quickly as possible. There is one type of simultaneous re-architecture that can gain many rewards with relatively small risk. Plan for this eventuality by reading your equipment handbooks and having a copy of the valid configurations close at hand. The new equipment may also have to be configured for the IP address. Diagrams and procedures are often tedious to change but the time spent confirming the accuracy of your information can greatly improve the efficiency of your operations staff and make troubleshooting much quicker. Examine Architecture Changes: It is generally not a good idea to incorporate major changes in the network architecture of your server farm as part of the relocation.28 Relocating Servers Between Data Centers - © LinuxHomeNetworking. There are many other factors to consider. It will tend to increase the complexity of the tasks and the likelihood of failure. You will need to execute more detailed testing later and this will be covered in sections to follow. This may be as simple as disconnecting one of your Internet links and seeing whether traffic reaches your site through an alternative ISP. confirm the use of redundant power feeds. 7. the newly inserted equipment may overwrite the configuration of its partner with the configuration used at the old location. A project to re-architect your farm after the relocation is much more desirable. This can make cabling costs high. logging. half of a redundant pair of devices is moved first as part of the main migration with the second device in the pair being moved at a later date. 4. contractor services. Older server farms tend to have all servers directly connected to pairs of high capacity switches located in a central location. Servers can then be connected to the switch in their rack . Contract Negotiations: Contracts relating to the old server facility will have to be terminated or reapplied to the new location. 6. Documentation: Modified environments usually require updated documentation. 1. make sure the information in your online customer care portal is correct. 2. This can create unexpected problems. It could also be more complicated such as doing a database cluster failover or network device reboot. and test the facility's response time to do standard changes. Measure temperatures. Run through some predefined transactions to test the health of your applications in the new environment. It's never a good thing to merely assume that your architecture is redundant. monitoring and authentication schemes used at the new facility. for example. but have them connect to smaller "pizza box" sized switches in each rack or cabinet.
13. for example on the lawn. but the failure of one of these small switches is less likely to threaten your entire site and if they do fail. lower cost and increased speed. but it may be worth investigating for newer projects. a pot-luck. You can largely avoid this problem by using Gigabit instead of 100 megabit per second fast Ethernet trunk uplinks. or "trunk". This would include moving trucks. Without their inclusion the project would only be an economic victory not a morale building exercise. in a nearby park or at a restaurant. Also check that the temperature at the front and rear of each server conform to the requirements of the manufacturer. 12. This should compare the results to the initial business case. With Gigabit links this is usually an unlikely scenario. confirmation that only authorized persons have physical and network access to your systems. There is the risk of having increased points of failure.How to Choose a data Center ISP 29 and the switch can then aggregate. and there is a sufficient budget allocation. The cabling becomes much cheaper and less complicated. the required networks/VLANs to the main central switches. conformity to schedule and cost estimates and feedback from team members and key stakeholders. Project Audit: You should conduct a post-move audit and review of the data center relocation project. 9. If the migration required weeks of late nights in preparation and execution. Infrastructure Checks: Verify the ambient temperature and humidity in the aisles meet the specifications of the CRAC unit vendor. Expansion can be done using cheaper commodity switches instead of more expensive blades that need to be fitted into the central departmental switches. This prevents data from bouncing around from rack to rack as it passes from firewalls. The lessons learned from the data center move/relocation project can be used to enhance future projects. Incentives such as extra time off. and regularly scanned for viruses if applicable. 10. It is important to recognize the talents and new skills learned by the members of the . Special events aren't usually viewed as being special if they are held in the usual everyday surroundings. the addition of a new rack only requires one or two cable runs to the central switches and short standard length cables connect the server to their local switch. and checks to see whether firewall rules need to be adjusted to match the new needs of the organization. This solution may not be suitable for your existing environment. a fun day. cinema tickets for the whole office to see the latest blockbuster movie. a single saturated trunk to one of the racks could take your entire site offline instead of only a small group of servers. Rental Returns: Remember to return all items rented as part of the migration. They had to bear the absence of loved ones working under stress over extended periods of time. The audit will also highlight areas of day to day processes that will need attention and so it should be used as a means of improving the universal strategic goals of improved simplicity. commendations for people who genuinely saved the day. correctly patched. Security Audits: Now that the relocation is complete you should perform a complete security audit which should include a remote network vulnerability scan. In extreme cases you may have to consider using separate switches for Internet facing and internal networks/VLANs. verification that all installed software is authorized. Very little routing changes are required in this architecture. in departmental switches the backplane or master bus can usually handle this type of situation easily. the replacement cost and time is usually less. dollies. 14. If this is not done. an outdoor barbeque. They should have an element of surprise and most importantly should be held outside of the office. Remember that in this architecture the firewalls. 11. Party Time! When it's all over have some form of recognition of the hard work done by all your staff. load balancers and all other networking equipment except the rack based switches should be directly connected to the departmental switches. or a small party can all be arranged at little extra cost. to load balancers and then to servers. Another risk is that backup traffic on one network can choke out Web traffic traveling on the same trunk. but prevention is always better than the cure. There is also the complexity of managing additional switches but the use of automated network device backup software such as RANCID can significantly reduce some aspects of administration. Documentation Storage: Store all documentation related to the project in electronic form in one location so that it cal be reused and accessible by all. invitations to the event should be extended to the immediate families of your staff. trolleys and shelving.Appendix II .
Conclusion Server relocations can be daunting projects but the tools and knowledge provided in this book should allow you to ably complete the project with few unexpected complications.com company and the tolerance of their families. careful planning is critical to success and should be taken seriously by everyone involved. as with any project. Remember that. The relocation is only finished once there is complete peace of mind.30 Relocating Servers Between Data Centers - © LinuxHomeNetworking. These are a few of the activities that need to be executed after the dust has settled but they are no less important than any other task. .
They cover all the preparation.Appendix I Relocation Check This appendix has samples of all the various forms you may need as part of the relocation. execution and cleanup tasks mentioned in earlier chapters. .
6. 4. Power feeds from multiple substations Building has sufficient excess power capacity for growth Adequate UPS coverage for expected load Staff automatically alerted when poor power line conditioning is detected. 12.Total (W x S) Item Description Comments Item Weight (W) Score (S) SubTotal (W x S) ________________________ ________________________ Engineer: ___________________ Contact: ___________________ 1. 11. 10. The data center with the highest grand total will be the most desirable. When visiting the data center give each item a score (S) from 1 to 10. 14. External flood risk Proximity to highways. Note: Data center ISP related factors are covered in the ISP rating form Data Center Name: Date: Score (S) Sub. 3. At the end of the visit multiply W and S to get a sub total. 2. 17.32 Relocating Servers Between Data Centers - © LinuxHomeNetworking. Greater than 250 meters Proximity to railways.com Data Center Rating Form In this table. give each item a weight (W) from 1 to 10 depending on how important it is to you. 8. Greater than 250 meters Proximity to hazardous production facilities Proximity to reasonably priced housing Proximity to good schools Proximity to recreational areas Connectivity to public transportation Proximity to airport flight paths Grounds susceptible to localized flooding Embankments and security fencing surrounding the building Availability of roof access rights. 9. 16. 7. 5. 13. Sufficient extra . UPS N+1 redundancy. 15.
29. 21. Liquid detectors under the raised floor tiles. Standby generators are regularly tested under load UPSs are regularly tested Air temperature at 75F / 25C Signs of leakage or rattling from the CRAC units. 24. 20. 18. Biometric access to data center. 30. 28. 26. Offices and common areas isolated for data center floor. Facility has sufficient power to support server load expansion into the foreseeable future. Job tickets automatically created when facility's equipment fails. Redundant PDUs supply your desired floor space. Facility can support the power per square foot required by all our equipment. 37. 35. . 25. 31. 23.How to Choose a data Center ISP 33 Item Description Comments Item Weight (W) Score (S) SubTotal (W x S) units to allow for offline maintenance without jeopardizing coverage. 36.Appendix II . 27. 32. Each rack or cabinet is fed by both PDU units. 34. Redundant PDUs are less than 50% loaded. Pre-action fire suppression system. 22. Mandatory visitor registration or electronic ID access. Graphical fire alarm panel with map of data center floor. Availability of web based customer 19. Lead times for change requests meet your minimum response times. Smoke and heat detectors. Standby generator N+1 redundancy. Sufficient extra units to allow for offline maintenance without jeopardizing coverage. 33.
53. 42. Data center area of the facility protected by fire proof doors. Power and cooling costs Floor space costs Monitoring costs Automatic data backup services provided Backup media compatible with existing systems at current data center. 54. 55. 49. kitchens and bathrooms located away from the computer area. 46. Technicians on site to provide "remote hands" access to servers. Problem escalation procedures meet your needs.34 Relocating Servers Between Data Centers - © LinuxHomeNetworking. 43. Facility provides access to multiple desired ISPs or data services carriers. 39. 48. 47. 57.com Item Description Comments Item Weight (W) Score (S) SubTotal (W x S) care portal. Antistatic tiles used on floor Water pipes. Backup media with valid data is stored off site Backup media is reused after an acceptable period in off site storage. 56. 50. 52. 44. Site has 24/7 video surveillance Site has biometric or some other form of keyless entry system to gain access to the server area. 40. Availability of 24/7 remote hands access. Facility allows you to reserve neighboring space for future expansion. . Facility allows all your required types of data circuits Facility allows access to all your required ISPs and data circuit carriers. 41. 38. steam pipes. 51. ISPs and carriers at facility have sufficient capacity to meet your future 45.
(Reduce single point of failure risk) Facility's CFA process meets timing. 58.How to Choose a data Center ISP 35 Item Description Comments Item Weight (W) Score (S) SubTotal (W x S) needs for each type of circuit. (Reduce single point of failure or pricing negotiation risk) Facility local loops use diverse paths into facility. 61. . procedural and cost requirements. The provisioning of data circuit cross connects meets timing.Appendix II . 60. procedural needs. Facility has multiple ISP / carrier local loop options. Grand Total 59.
com ISP Rating Form In this table. 11. 13. 8. give each item a weight (W) from 1 to 10 depending on how important it is to you. 14. 3. 16. The data center with the highest grand total will be the most desirable. Adequate lead times for DNS change requests. 6. 10. Lead times for regular change requests meet your minimum response times. Use of SNMP traps to generate job tickets automatically. Availability of web based customer care portal. 12. 7. Multiple DNS servers in multiple geographic locations. CIR acceptable CIR can be incremented to meet your projected needs Signs of congestion on network that would prevent the ISP form meeting the contractual CIR 2. 15. Bandwidth costs (Local Loop) Bandwidth costs (95th percentile) Lead time for provisioning data circuit. 5. 9.36 Relocating Servers Between Data Centers - © LinuxHomeNetworking. When visiting the data center give each item a score (S) from 1 to 5 At the end of the visit multiply W and S to get a sub total. 5 minute or better network monitoring polling cycle Number of failed polling cycles that trigger alarms (Usually 3) Job tickets automatically created when polling fails? SNMP based network monitoring. Data Center Name: ________________________ Engineer: ___________________ Date: Item Description ________________________ Comments Contact: ___________________ Item Weight (W) Score (S) SubTotal (W x S) 1. . Problem escalation procedures meet your needs. 4.
How often are error rates monitored to trigger maintenance? At what error rate triggers mandatory maintenance? What is your procedure for scheduling technician visits to fix line degradation problems? Do you provide full BGP routes? We have our own BGP AS number with our own networks. Will you be able to advertise these to the Internet if we peer with you? Do you have fully redundant paths to each of your BGP peering transit providers and the router to which we will be peering? We may need to adjust BGP routing to amount of bandwidth we push through your circuits. Are NOC staff certified in the equipment they monitor? NOC escalation procedures meet your needs. 32. E1. 19. Modem connection) to gain access to their equipment if their data circuit fails.) Does ISP have out of band access (eg.How to Choose a data Center ISP 37 Item Description Comments Item Weight (W) Score (S) SubTotal (W x S) 17. 18. 30. 26. 21. OC3. (T1. 25. DS3.Appendix II . 27. . 22. Ethernet. 31. HSSI. 24. 34. Type of data circuit to be provided. 20. Do you support AS path prepending? Is your networking infrastructure fully protected with UPS and standby generator power? Who do you peer with for Internet transit? Facility allows all your required types of data circuits for this ISP Facility allows access to this ISP / data circuit carriers. SLA guaranteed uptime (Percent) Scheduled maintenance windows meet your timetable needs. 33. 23. 28. HSSI. etc. 29. Scheduled maintenance windows give adequate advance notification.
38 Relocating Servers Between Data Centers - © LinuxHomeNetworking.com Item Description Comments Item Weight (W) Score (S) SubTotal (W x S) 35. ISP / data circuit carrier has sufficient capacity at facility to meet your future needs for each type of circuit. Grand Total .
How to Choose a data Center ISP 39 Cost Justification Work Sheet Calculate the expected monthly change in profit and expenses over the expected life of the servers in the new data center and off set this against the expected startup capital outlay costs to determine the economic merit of the relocation/consolidation project. Liberated cash flow due to consolidated operations Grand Total . Eliminated risk to profits by moving to a more suitable data center. Description Expected increase in profits due to increased sales facilitated by improved data center activities. or systems outages) 3. Potential Monthly Profit Improvement Done By: ________________________ Date: ___________________ Item 1. Comments Old Facility New Facility Net Change 2.Appendix II . (Less revenue loss due to infrastructure being incapable of supporting customer needs.
5. 7. .40 Relocating Servers Between Data Centers - © LinuxHomeNetworking. 2. 6. 10. 11. 4. Description Leasehold improvements cost Transportation moving costs Overtime expenses Increased staff hiring expenses Contractor fees Training fees Equipment acquisition Temporary leases for the relocation or upgrade Cleanup costs Penalty fees Staff layoff expenses SUB Grand Total Comments Old Facility New Facility Net Change 9. 3. 8.com Relocation versus Data Center Upgrade Capital Outlays Done By: ________________________ Date: ___________________ Item 1.
22. . 3. 5. 21. 4. 9. 13. 15. Description Air conditioning lease Air conditioning maintenance Data backup equipment leases Data backup services DBA services Fire systems maintenance Floor space lease ISP fees Janitorial services Network equipment leases Network management services Power Data backup replacement media expenses Security services fees Security systems maintenance Server leases Server maintenance contracts Server systems administration services Software licenses Software maintenance contracts Staffing expenses Standby generator lease Standby generator maintenance Comments Old Facility New Facility Net Change 14.How to Choose a data Center ISP 41 Monthly Costs Done By: ________________________ Date: ___________________ Item 1. 8. 11. 16. 2. 18. 6. 10. 17.Appendix II . 23. 7. 20. 19. 12.
42 Relocating Servers Between Data Centers - © LinuxHomeNetworking.com Item 24. 27. Description Systems development expense UPS maintenance fees Web site monitoring expenses Staff expense Grand Total Net Change Comments Old Facility New Facility Net Change Description Old Facility New Facility Net Change . 26. 25.
Conference call bride arranged Best time and date for relocation identified Team aware of method to be used to identify malfunctioning equipment. 17. etc. Point of no return identified. 19.Appendix II . 11. Kosher. 12. 2. Halal. 20. 21. 18. Description Project manager assigned Networking team lead identified Transportation team lead identified Server area team lead identified Server team lead identified Customer communication lead identified. 10. Veggie. Catered food lead identified (Pizza. 9. and when. 5. .How to Choose a data Center ISP 43 Coordination Preparation Check Sheet Done By: ________________________ Date: ___________________ Item 1. 13. 14.) Post mortem team created to analyze any failures in the process. Roll back plan created that includes the sequence in which equipment will be returned. 6. 3. Application audit performed for all servers to account for interdependencies with other applications. 7. Priority Person Responsible Deadline Status 8. created. Gofer identified Disaster recovery team lead identified All members of the relocation team will have functioning mobile phones on the day or the migration. 15. 4. Short list of scenarios during which the relocation will have to be rolled back has been created. Sequence list of which equipment will be moved. Server list / inventory spreadsheet has been created and distributed to all teams. Minimum list of functional servers for successful migration created. 16.
com Item 22. Description Procedures documentation completed for each server / network device Equipment that will have to be leased or purchased to facilitate the relocation has been obtained Priority Person Responsible Deadline Status 23.44 Relocating Servers Between Data Centers - © LinuxHomeNetworking. .
5. 3. Short list of customers to be kept updated by phone created. Post mortem template identified Priority Person Responsible Deadline Status 2. . 6.How to Choose a data Center ISP 45 Customer Communication Check Sheet Done By: ________________________ Date: ___________________ Item 1. Internal customers alerted of plan External customers alerted of plan Draft notifications to be used in the event of failure and success created. 4.Appendix II . Description Single message of varying degrees of detail created for each customer group.
Power supplied is of the correct voltage using the desired connectors. 2. Shelving and rack kits for servers pre installed in cabinets and racks. 9. 3. 14. 10. 15. Servers in racks and cabinets won't overload the power circuits that supply them. Racks and cabinets clearly indicate the server orientation to create hot and cold aisles.46 Relocating Servers Between Data Centers - © LinuxHomeNetworking. Antistatic tiles installed on floor Priority Person Responsible Deadline Status 4. Adequate power supplied to each rack and cabinet Each PDU in the server area can handle the failure of the other redundant one supplying the area.com Server Area Preparation Check Sheet Done By: ________________________ Date: ___________________ Item 1. Description Racks and cabinets installed in new location. 19. Data circuits for internet connectivity have been run to the racks to be used by the networking equipment. 12. Staff doing the server "Rack and Stack" work have adequate tools especially electric screwdrivers. 16. Rack mounted keyboards and monitors installed in cabinets or racks. 18. Verify that you have power cables in sufficient quantities and adequate lengths. 17. Monitors on carts available for troubleshooting servers that need to be removed from racks. 8. Mouse) switches preinstalled. KVM (Keyboard. 7. . 5. Video Monitor. Diagrams created to show the server layout in the racks or cabinets. Power cables are labeled at both ends. 6. Power outlets are labeled to indicate the PDU that supplies them. Keys given to all persons who require access to the server area. 11. 13. Adequate cooling provided for expected server heat load.
29. 30. 33. 24. Raised floor under the cooling zone of each CRAC unit is sealed to force air through desired cabinets. Server area is large enough to accommodate future expansion. 36. Cable lengths through the patch panels are under 100m in length. Copper data cables. Aisles between racks wide enough to allow people to easily mount and dismount servers in them. Patch panels installed correctly Patch panels have the correct connectors. Perforated floor tiles only used in cold aisles. 27. fiber data cables and power cables run in separate conduits or trays. Data cables not hanging in the air or pulled taught. steam pipes. 34. 32.How to Choose a data Center ISP 47 Item 20. . Adjacent server cabinets do not have their walls removed. 23. kitchens and bathrooms located away from the computer area. Few air flow obstacles under the raised floor area. 22. 31. 35. 26. Baffles used under the raised floor area to improve airflow. Priority Person Responsible Deadline Status 21. Glass fiber and copper cables run in separate cable trays. 28.Appendix II . 25. Description Water pipes. Server area has 24/7 video surveillance Server area has biometric or some other form of keyless entry system access system. Patch panel cable bundles run to the sides of racks and cabinets so as not to impede airflow.
13. 9. POTS lines installed for out of band access. Network devices are labeled in the front and back. Test VPNs to remote offices have been created.48 Relocating Servers Between Data Centers - © LinuxHomeNetworking. Each server is labeled on the front and back. 10. Sample configurations have been created for all networking devices with the new network addresses to be used at the new location. The standby devices in redundant pairs of networking equipment at the old location have been shutdown. All network cables labeled at both ends. 12. 4. 5. shipped to the new location and preconfigured. 6. Procedures documentation completed for each server / network device Data circuit capacity requirements identified for both production Web access and possible data replication between data centers. Complete network diagrams of the new location have been created. 11. Description Network equipment with dual power supplies are plugged into electrical circuits from each PDU. 8. DB servers) are connected to redundant switches in a 50/50 fashion. Groups of servers with similar function (eg. Priority Person Responsible Deadline Status 2. . 3. 7.com Network Preparation Check Sheet Done By: ________________________ Date: ___________________ Item 1. Data circuits to be used by the network gear have been tested.
DB servers) with single power supplies are connected to redundant PDUs in a 50/50 fashion. 17. Documentation of the sequence in which servers should be booted or applications started) Snapshots of the actively running services or applications for each server has been created. Sequence of how the servers will be returned moved in the event of a roll back identified. Groups of servers with similar function (eg. 18. 4.available (replication scenario) Spare servers identified as possible sources of spare parts or as "hot standby" devices. . 16. All RAID BIOS settings recorded. Practice data backups and restorations done for key servers and applications. 14.How to Choose a data Center ISP 49 Server Preparation Check Sheet Done By: ________________________ Date: ___________________ Item 1. 9. Each server's "netstat -nr" output has been recorded with the expected output at the remote location prepared. Minimum list of servers for correct site functionality created. (eg. Applications vetted to make sure they refer only to DNS names not IP addresses. Each server's "netstat -an" output has been recorded. Equivalent amount of production data storage Priority Person Responsible Deadline Status 2. 5. 13. 8. 15. 19. Sequence of servers to be moved identified. Procedures documentation completed for each server / network device Amount of data to be backed up per server has been determined. Description Servers with dual power supplies are plugged into electrical circuits from each PDU. 12. Special operating procedures for key servers created. All server power cords labeled at both ends. 6. 10. Server data backed up. 3. Testing short list created for each server. 11. 7.Appendix II .
not internally. Transportation Preparation Check Sheet Done By: ________________________ Date: ___________________ Item 1. New DNS zone files created with new IP addresses. Special packaging for the more delicate items.com DNS Preparation Check Sheet Done By: ________________________ Date: ___________________ Item 1. 3. Priority Person Responsible Deadline Status 2. 7. Priority Person Responsible Deadline Status 4.50 Relocating Servers Between Data Centers - © LinuxHomeNetworking. . 6. 3. Dollies and carts available at both locations for moving the servers across the floor. (TTLs should remain at 5 minutes) Testing of application using new DNS entries in the test server's hosts file completed. Items under maintenance contract prepared for transportation by the equipment vendor. 4. 8. 5. Boxes purchased or rented for moving equipment and miscellaneous items. Description Moving truck rented / Professional movers hired Mover has adequate insurance coverage Adequate method for stacking and securing servers in the truck found. 2. Equipment insurance up to date. Description TTL on old DNS zone files set to 5 minutes and propagated to the Internet. Testing of website using new DNS entries in the web browser client PC's hosts file completed.
Appendix II . Description Notifications sent to all customers that the activity is about to start. 3. . DNS using the new zone files has been propagated to the Internet. Emergency conference call bridge has been created for communication with team.How to Choose a data Center ISP 51 Activities During the Relocation Check Sheet Done By: ________________________ Date: ___________________ Item 1. Priority Person Responsible Deadline Status 2.
Test data backups of servers successful. . Description All power cables labeled at both ends All network cables labeled at both ends Moving truck. All equipment at the new location has been restored to fully redundant operation. DNS TTLs on new zone files have been reverted to a couple days. 19. carts. 17. 11. Security audits planned and executed. Customers and partners can get adequate access to the servers at the new location. dollies. 5. 2. 7. 8. 3. Blanking panels inserted between servers Server cable bundles run to the sides of racks and cabinets so as not to impede airflow. Testing of data throughput rates at new location completed. 10. boxes and other transportation related items returned. 15. Redundancy testing of new server environment completed. 13. 16. Ambient air temperature within design of the CRAC unit Ambient air humidity within design of the CRAC unit Intake temperature at the front of servers in the rack is within specification.52 Relocating Servers Between Data Centers - © LinuxHomeNetworking.com Post Relocation Check Sheet Done By: ________________________ Date: ___________________ Item 1. Priority Person Responsible Deadline Status 4. Contracts related to old facility terminated or re-negotiated for the new facility. 9. 14. 18. Error logging and monitoring occurring correctly. Exhaust temperature at the front of servers in the rack is within specification. 12. 6. New diagrams and other documentation created for adequate description of the new server environment.
11.: Backup Done: Date: Old Data Center Item 1. 7. 2. 8. 12.Appendix II . 4. Description Default gateway Server rack location NIC #1 IP address NIC #1 Subnet Mask NIC #1 Connected to switch named: Backup server IP address DNS server #1 IP address DNS server #2 IP address NIC #1 Connected to switch port number: NIC #1 Connected to VLAN number: NIC #2 IP address NIC #2 Subnet Mask NIC #2 Connected to switch named: NIC #2 Connected to switch port number: NIC #2 Connected to VLAN number: Attached Devices New Data Center . 14. 13. 16. 3.How to Choose a data Center ISP 53 Individual Server Worksheet Server Name: Make: Serial Number: Operating System: Engineer: Backup Done: Model: Inventory No. 10. 9. 6. 5. 15.
Relocating Servers Between Data Centers -
Individual Server Worksheet (Part II)
Additional Server Routes
Item 1. 2. 3. 4. Network Subnet Mask Gateway
Network Enabled Applications List (netstat -a output)
(Will be same as that expected in New Data Center)
Old Data Center
Old Data Center
Appendix II - How to Choose a data Center ISP
Contractor Qualification Check Sheet
Done By: ________________________ Date: ___________________
Item 1. 2. 3. 4. 5. 6.
Description Contractor has sufficient experience in the industry. Contractor has sufficient certified staff. Contractor has provided evidence of staff certification Contractor has the correct installation and testing tools Contractor lists the type and manufacturer of all testing tools Contractor is willing to guarantee the quality of their work in writing and back it with a one year warranty. Contractor has provided proof of doing similar work previously. Contractor has tested their work to your satisfaction and provided a test plan as proof.
Relocating Servers Between Data Centers -
Data Circuit Check Sheet
Done By: ________________________ Date: ___________________
Circuit Type T1 PRI DS3 GigE
T1 PRI DS3 GigE
T1 PRI DS3 GigE
T1 PRI DS3 GigE
T1 PRI DS3 GigE
T1 PRI DS3 GigE
T1 PRI DS3 GigE
T1 PRI DS3 GigE
11. 10. 7. 14. 16. 12. 6. 13. 3. . 15. 2. 4. 8. 9. 5.Vendor / Purchasing Check Sheet Done By: ________________________ Date: ___________________ Item Shipment Group Product Order Quantity Shipped Quantity Status Scheduled Shipped Date Date Shipped Shipment Method Data Center Ticket Number Tracking Number Number of Pieces 1.
escalation or communication are referenced. . However. Ensure all parties involved in the troubleshooting.Post Mortem Analysis Sample Form Customer Information Customer Name Location Date Author Ticket Number(s) Incident Date Post Mortem # Responsible Support Team List all affected customers Data Center where incident occurred. site down – Escalated to Networking – Networking troubleshooting event – Confirmed with Customer site up Customer Impact Duration of time the customer was impacted and aspect of the site that was not available. Report Date Engineer working issue Related work order or trouble ticket numbers Date incident occurred Assigned by the IR Administrator Engineer’s Department Incident Summary A brief. even after the load-balancer failed over to the secondary. from 5:24 pm to 7:24 pm. factual description of the issue: Example: We received an alert from the monitoring system for customer X indicating sites were unavailable. The troubleshooting process indicated that the primary local-director (load balancer) had failed over to the secondary. There were also traps that indicated VLAN state changes on the Internet facing VLAN of the load-balancer. but do not use the individual’s names Example: Eastern Standard Time: Date: xx/xx/2004 05:24pm 05:29pm 05:30pm 05:34pm 07:24pm – Event noted – Project Manager paged. the sites were still not accessible Incident Detail Details from the ticket with time stamps (include event start time. Example: Customer sites were inaccessible for 2hrs. escalation times & end time when customer confirmed resolution).
it came active but was unable to pass traffic. Customer care representative will schedule a meeting to review site specs with customer in 2 weeks. which were not setup for ether-channel and were in the wrong VLAN. When the device failed over to the secondary device. Ensure complex technical terms and company specific terminology is defined for the customer.Appendix II . . the sites were accessible Corrective Action Plan Corrective actions taken to immediately address the issue and any planned actions to prevent reoccurrences and estimated completion dates of planned actions. Example: o o o o Interfaces of the local-director were connected to the correct switch ports that had ether channel setup. Example: The local-director load balancer has etherchannel setup on its interfaces.How to Choose a data Center ISP 59 Root Cause Analysis Underlying factor causing the incident. When the primary device was made active. Will run an audit of the Customers net work diagrams to confirm correct setup. The fail over load-balancer had its interfaces plugged into switch ports. Tested Fail over and the status on the local-directors show no errors.
Internet circuits typically require you to commit to a minimum data rate and charge a variable fee for usage above that rate to a defined maximum. Pricing 2. This will be discussed in more detail next. This is frequently proportional to the distance between the exchange and your facility. This chapter will discuss the many technical factors that govern the selection process in detail. First let's discuss these factors in more detail. They then discard the top 5% of the samples with the highest utilization. You may also be able to reduce your 95th percentile rate by committing to a longer contract or by convincing your ISP that you will be generating sufficient traffic to justify a bulk discount. 4. 3. The ISP samples your data rate every five minutes and sorts all the sample readings for the month from high to low. Configuring a routing protocol to use with your ISP. Check lists are also included in Appendix I to help you make a better decision and facilitate your monitoring of the status of all the required tasks. you may also be charged a local loop rate which amounts to a monthly fee that covers your connection from your facility to the nearest telephone / Internet exchange. Determining the type of data circuit to use. "Preparing for Server Relocation". There is no variable component. One of the advantages of this billing method is that it allows you to download files. . You are then billed at the rate of the highest sample that remains. Internet Services A very common ISP billing technique used is called the 95th percentile method. not the average of those remaining. a usually bandwidth intensive process. Here the internet service provider provides an absolute maximum data rate. for up to about an hour a day without it affecting your bill. also known as a committed information rate (CIR). In addition to bandwidth. but you are billed based on actual usage. Data Circuit Pricing Pricing varies depending on the type of service you purchase. These factors include: 1.Appendix II How to Choose a Data Center ISP In Chapter 2. Non-Internet dedicated point to point services from data carriers usually charge a fixed fee that allows transfers up to the maximum data rate. Sometimes this fee is also related to your CIR and you may find that you can reduce this monthly fixed cost my negotiating a lower CIR. I outlined the importance of ISP preparation. Deciding on whether to use IP addresses issued by your ISP or addresses owned by your company.
048 Mbps link that can be split into up to 32 x 64 Kbps channels. International standard for transmitting digitized voice over optical fiber circuits. OC-12). The most commonly used data circuit technologies include those listed in Table A3. There are a number of SONET circuit types. (OC-3. A DS3 can be fractionalized with up to 30 T1 circuits. Channels can be aggregated together to increase throughput up to the E1 maximum in a configuration called a "fractional E1".1. The TCP/IP packets are inserted into ATM packets which are then placed on the SONET circuit. Channels may be used for voice or data. You will most likely need a POS interface on your router if you intend to transmit TCP/IP. Optical version of Ethernet that operates at 1 Gbps. Data traveling on SONET networks use asynchronous transfer mode (ATM) formatted packets which were originally designed to carry voice traffic. Primarily copper based version of Ethernet that operates at 100 Mbps. This means a T3 can up to 672 x 64 Kbps voice/data channels.1 . Fast Ethernet Gigabit Ethernet (Fiber) Gigabit Ethernet (Copper) . Methodology carrying TCP/IP traffic over SONET networks.736 Mbps. Typically runs over fiber optics. Channels may be used for voice or data. the most commonly used ones being optical carrier levels 3 and 12 mentioned in this table. Term Description E1 T3 DS3 HSSI Packet Over SONET (POS) SONET Synchronous Optical Network. Typically runs over copper 2. Typically runs over copper. ATM interfaces may look the same.544 Mbps link that can be split into up to 24 x 64 Kbps channels.Appendix II . Circuit configured to carry DS3 formatted data at up to 44. The complexities of a CIR and 95th percentile are usually absent.How to Choose a data Center ISP 61 Non-Internet (Carrier) Services Data carriers typically will charge a flat fee for circuits with a pre-defined maximum data rate. Typically runs over copper. Channels can be aggregated together to increase throughput up to the T1 maximum in a configuration called a "fractional T1". Two channels are used for signaling.Common Data Circuit Terminologies Term T1 Description 1. Table A3. but are designed to strictly carry traditional voice traffic. Two circuits are used for signaling. the equipment available to your ISP in the area and the capabilities of your networking equipment. Voice over IP (VoIP ) or Internet packets on an OC type circuit. You will also be charged a local loop rate. Copper based version of Ethernet that operates at 1 Gbps. See T3 High Speed Serial Interface capable of supporting up to 52 Mbps. Data Circuit Types The selection of the type of data circuit to be used will depend upon the amount of bandwidth you expect to use. Typically runs over fiber optics.
In most cases neighborhoods are grouped into geographic zones which receive data communication services from a central office (CO). Usually offers a maximum bandwidth of 2 Mbps. In most cases the ISP manages the antenna equipment and hands off a physical cable link to their customer. . You can: 1. The connection between a CO and your business or home is often called the local loop. This means that the incoming downstream data rate from the Internet is different from the reverse outgoing upstream speed. Select your data circuit with care. Inbound Web browser queries don't use a lot of data bandwidth. COs can also be called telephone or Internet exchanges. AT&T) that owns the wiring infrastructure all the way to the neighborhoods' homes and businesses. An adequate solution for low volume web sites. Use a factional or channelized service and expand your usage one channel at a time till the maximum capacity of the circuit is reached. Remember that there are many ways to plan the expansion of your data circuit bandwidth. Whenever possible always opt for symmetrical services for your business. They reserve symmetrical data circuits for businesses which usually need high bandwidth to both surf the web and serve Web pages and more reliable guaranteed service. 2. You should be most concerned about the upstream speed for your Web site to the Internet. Therefore it is possible to purchase services from an ISP who then has to negotiate multiple local loops for the circuit to finally reach its backbone infrastructure.com Term Wireless Circuits Description Uses a variety of methods to transmit data through the air. In order to provide services to all neighborhoods in a city. Order a high speed circuit and throttle it with a lower valued CIR. a dedicated point to point data circuit between two neighborhoods should have a local loop in neighborhood "A". This link may be of any of the circuit types mentioned in this table. The backbone should then provide services to the CO in neighborhood "B". Wireless links can be quickly installed but they tend to be subject to interference that can reduce their reliability. The ISP will usually provide the business with a fixed range of Internet addresses as part of the service. Increasing the CIR increases your bandwidth. but the Web pages that contain the outbound replies do. Usually the CO is owned and operated by a single incumbent carrier (eg. Ideally. Internet service providers provide asymmetric services for residential users and the downstream rate is almost always higher than the upstream. A wrong decision could inhibit the growth of your business. 3. Competing carrier can sometimes arrange with the incumbent to provide competing services over the wired infrastructure for a fee. especially with DSL and cable modem circuits. Some ISPs purposely restrict traffic to web servers on their cable links for this reason DSL Cable Modem It is good to note that data services are sometimes asymmetrical in nature. there need only be a single local loop to your ISPs Internet infrastructure. Your bandwidth usage could also affect video quality of other subscribers. Data Circuit Provisioning You should always be aware of the environment in which data circuit providers work. which then connects to the carrier's backbone network. Unsuitable for high bandwidth websites as your circuit is shared by many other subscribers who could affect your performance. Add more circuits. For Internet services.62 Relocating Servers Between Data Centers - © LinuxHomeNetworking. residential customers usually get a dynamic address allocation which is unsuitable for most businesses. Not all ISPs are present in all COs. ISPs may have to negotiate interconnections between COs. which connects to the remote business via another local loop.
a server room or the location of their PBX) will have their own IDF for their own equipment. Remarkably. This could force you to reassign brand new IP addresses to your servers. IDF (Intermediate distribution frame): In buildings with multiple tenants it is common to extend connectivity from the MPOE to each tenant's premises. circuits that are securely shared with other customers. It can include physical information such as location. IP Address Ownership In a data center environment you will normally request a block of IP addresses (the data equivalent of a telephone number) from your ISP for use by your servers. but you'll need to have a cross-connect created to link your server room's IDF to the MPOE's MDF. Make sure you understand how your desired types of circuits and carriers will gain access to the facility before making a final data center decision. you will lose the IP addresses they assigned to you. such as a channelized DS3. MPOE (Main Point of Entry): Carriers and ISPs need to deliver data circuits to a specific room at a business address. You will almost certainly have to coordinate your engineers and those of the carrier to harmonize and test their configurations before data flows correctly. (eg. You should also realize that not all data centers allow access to all carriers and in some cases there may be only a limited number of circuit types available.Appendix II . DLR (Design Layout Record): This document describes all the details of the circuit path from one end to the other. With the knowledge of these terminologies you should be in a much stronger position when talking to your ISP and carriers. Coordinating the installation and troubleshooting activities of one ISP can be difficult. The more local loops you have. in other words. Work cannot proceed without a LOACFA for the local loops. Data center Cross-connects: A carrier or ISP will deliver your circuit to the MPOE. If you cancel your ISP data circuit. Secondly the carrier that issues the document also provides a facility assignment (CFA) which indicates the specific interconnection point within the CO for the other carrier to use. row. Always consider applying for your own IP addresses from your Regional Internet Registry (RIR). which can help reduce the complexity of the document. data centers often charge for this accessibility on a per circuit basis. Connectivity between gear in the MDF and the IDF is usually achieved by using patch panels. A DLR can also mention interconnections with other known circuits. There is a disadvantage to this. You should always verify that a DLR has been created on time in order for it not to hold up the rest of your operation. It is typically the same room in which all telephone lines enter the building. but FOC is a common term used in the industry. Always ask what additional tasks will be required after the FOC date. FOC (Firm Order Commitment): It may sound rude. floor. It is the date your carrier will commit to having a fully functional circuit delivered to you. Your equipment will usually connect to the MDF gear through a patch panel provided by you carrier / ISP. the more LOA-CFAs are required. This rack and equipment is usually the property of the carrier / ISP. It can be an unexpected hidden cost. Each tenant location. It is very possible for carriers to test their local loops correctly but make a mistake on the CFA with an incorrect cross-connection. The ISP will assign a range of addresses to you and will configure their equipment to route traffic to this range via the data circuit they provide.How to Choose a data Center ISP 63 It is best to minimize the number of local loops in your circuit design. The relationship between carriers and ISPs in a CO leads to a variety of additional terminologies you'll need to know: LOA-CFA (Letter of Authority and Customer Facility Assignment): This document does two things. Here is a useful list of RIRs you can use for your area: . It can also refer to virtual circuits. it allows a carrier to have access to another carrier's facility to do work (LOA). rack. Firstly. Extending this to multiple ISPs can be tricky. It is important to keep a very close eye on this process. MDF (Main Distribution Frame): Is usually a rack in the MPOE in which carriers will install the equipment required to terminate the circuit's local loop coming from the CO. panel and port.
It will provide insights into what can be done if things go wrong during your data center relocation. AfriNIC (African Network Information Centre) . This is the usual option when only a single link is provided. APNIC (Asia Pacific Network Information Centre) . When new IP addresses are required. LACNIC (Regional Latin-American and Caribbean IP Address Registry) . The first is by providing a simple default gateway through which all network traffic should pass. Border Gateway Protocol BGP is a dynamic protocol that can be adjusted relatively easily to influence traffic to and from your site in order to reduce bandwidth costs when your ISPs charge different rates. ISPs usually use two methods to provide internet access to their clients. or to divert excess traffic from an overloaded circuit to a lesser utilized one. This section will provide an overview for project managers of the most common Internet routing challenges data center based web sites face. This section will cover BGP for use by project managers in some detail and Table A3.Asia Pacific region 3.Americas and Southern Africa 4. This minimizes the impact of forced IP address changes on your operation. RIPE NCC (Réseaux IP Européens Network Coordination Centre) . Unlike the configuration of a static route that can never change even if a link fails.com 1. you can just modify the DNS name to map to the new address.2 summarizes many of the terms that will be used later. The circumstances for doing so are slightly different for each affiliate but the main factors are that you can prove that your routing policy is different from that of your ISP and/or that your connectivity requires links to multiple ISPs.Latin America and some Caribbean 5. . BGP routes adjust themselves automatically depending on the availability of network links to reach target destinations.64 Relocating Servers Between Data Centers - © LinuxHomeNetworking.Africa region 2. Routing Protocols Internet routing can be quite complicated and you will often need a network engineer to configure your equipment to get access. ARIN (American Registry for Internet Numbers) .Europe and surrounding areas will recognize your operation as being similar to that of an ISP and will assign you your own AS and IP addresses. The second method relies on the border gateway protocol (BGP) and is used primarily when Internet connectivity is provided via multiple ISP links. If you cannot obtain your own IP block then you will have to ensure that all your applications use DNS names to refer to other servers in your environment and not their actual IP addresses.
BGP routers update their neighbors of changes they detect in the Internet. Routes are exchanged with other ASs in the form of "I am AS number X and in my AS I have the following networks". Prepending can be applied on a per link basis so that internet routers will feel that the AS path to your network on your preferred link is much shorter than the one to your network via the less favorable link. This results in BGP routers having long lists of all the available networks on the Internet tied to a sequence of ASs that need to crossed in order to get to each one. Autonomous systems are assigned blocks of IP addresses which the ISP advertises to neighboring autonomous systems. each ISP is provided with a BGP autonomous system (AS) number from the Internet Assigned Numbers Authority (IANA). One method uses AS path prepending in which you repeatedly add on your own AS number in your BGP advertisements. ISPs can advertise default routes to your routers via BGP. it then notifies its remaining neighbors of the failure and instructs them to remove their routes to the failed AS from their routing tables. A method in which an ISP.Appendix II . you can also have them advertise a unique multi-exit discriminator (MED) metric value in the advertisements on each link which will bias BGP on your router to route its traffic on one link versus the other. If the links terminate on the same router you can use a system of weighting to route traffic completely over one link versus the other.How to Choose a data Center ISP 65 Table A3. Inbound traffic can be influenced too. . This can be useful when your equipment doesn't have sufficient memory to store the entire Internet routing table. A unique number assigned to the ISP for their autonomous system. If a BGP router loses connectivity with a peer responsible for advertising networks for a particular AS. Autonomous System Number (ASN) AS Path AS Path Prepending A sequential list of ASs traffic must pass through in order to reach its destination network. so when the links terminate on different routers within your control you'll need to use BGP's local preference feature help them negotiate the preferred link. Local Preference Multi-Exit Discriminator (MED) With BGP.Common BGP Routing Terms Term BGP Autonomous System (AS) Description A BGP routing management domain usually owned by ISPs. Weights aren't exchanged between routers. No matter what types of routes you receive you can influence how traffic leaves your site with a number of commonly used techniques. with multiple links to your AS. A method of biasing the desirability of routes to the internet via a particular link in favor of an alternative one within your BGP AS. The methods previously discussed only refer to outbound traffic.2 . This sequence is called an AS path list. can bias your routers to select one of its paths to the Internet over another. When both your links are provided by the same ISP. This lengthens your AS path list without making the traffic pass through any additional ASs. A method of repeatedly adding on your AS number to the beginning of the AS path to your network to bias traffic away from the link that is advertising the prepended list. The ISP is also responsible for the routing of traffic destined to their AS or passing through it to another AS. Some manufacturers recommend a minimum of 512 KB of RAM to support full routes. The ISP then associates the IP addresses it owns to this AS.
valid. (aggregated by 8176 172. Once they receive this they will configure their equipment and then provide you with: 1.193. Administrative Tasks Needed to Advertise BGP Routes Network engineers will have to configure BGP on their routers. but the project manager will have to contact the ISP to make sure they are prepared to receive your routes.aol. Your network engineer will then be able to configure your equipment to provide correct BGP connectivity to the Internet. IP address assignments for both your and their equipment connected to the circuit. The volume of traffic passing through your AS could cripple your data circuits too.aol. 2.aol.218. Their BGP AS number. metric 46.aol. and select "BGP". Another common example would be a situation in which BGP automatically passes most of your outbound traffic over your most expensive link.86.com has address 205.com.126.96.36.199 (216.com belongs.aol.aol. not "BGP summary". you can use local preferences to make traffic to Hotmail. 3. This means to get to mail.168. The final AS.66 Relocating Servers Between Data Centers - © LinuxHomeNetworking. the networks you wish to advertise and the possibility of using AS path prepending. internal. MSN and Yahoo! go through the original link. TELIA (1299) AOL (1668) 8176. atomic-aggregate Community: 6939:2000 The output shows that the AS path list is 1299. if not there are automated tools such as netflow on routers. then 1688 and finally AS 8176. that can provide more accurate insights. Note: When you use multiple ISPs. you will find yourself paying for traffic that wasn't destined to your site. Log on to one of the looking glass routers listed on the site.188. Say for example you have to email a weekly newsletter to thousands of customers but the additional traffic saturates one of your ISP links. You can usually guess the most popular ISPs from which web surfers would be coming. You will now have to do this again for all the other IP addresses returned by the hosts command. If this happens. is the AS for mail. In this case I'll attempt to determine the AS to which the IP address of mail. and webalizer on web servers. Enter one of the AOL IP addresses.252.44.com the looking glass router had to pass through AS 1299. Use a search engine to find a site that will provide access to "BGP looking glass" routers. 8176.252.12.aol. make sure your network engineer's BGP configuration guarantees that data traveling from ISP #1 to ISP #2 doesn't pass through your AS.119 mail.GENUITY/BBN (1)) 213.com has address 64. localpref 100.248. in this case 64. You can bias traffic on a per-network and per-AS basis.249 [root@bigboy tmp]# 2. A data circuit. . Click on the submit button and you will get output looking like this. This can be very useful.21.com These modifications don't have to apply to all Internet routes. You can also figure out an AS number manually using the method in the following section. 8176.12.53 (metric 7) from 216.com [root@bigboy tmp]# host mail.com has address 64.119.149) Origin IGP.249 mail. You can use some of the techniques mentioned to make BGP favor the cheaper link for your traffic until your safe link bandwidth threshold is reached. You will have to inform them ahead of time of your AS number. 3. Use the nslookup or host command on a Windows or Linux server to determine the IP address of mail. 1688.160.com mail. Determining a BGP Autonomous System Number Determining the AS number of an ISP or mail service manually is usually straight forward.aol. 1. but traffic to AOL and Gmail pass through the other.
How to Choose a data Center ISP 67 Conclusion Data center ISP selection is a very important part of any relocation activity. This Appendix has provided a summary of the issues that need to be addressed in the process and will make the overall task much simpler to complete.Appendix II . Activity checklists are provided in Appendix I to help facilitate this further. .
A LinuxHomeNetworking.com © White Paper .