Concerns- Network Team.

1. Switch TC-A4-D02 replacement likely to cause the biggest impact. The reason for this is due to
there being no space in rack to put a new switch in. (Server Team agreed to help us during the
Change Window)

2. N3K there are a number of areas where there could be potential issues.

 Testing cross connect links.

 Ensuring the QSFP arrive on time to carry out this work?

 Planning outage for additional work (N3K)

PHASE-1

Pre-work prior to the DAY on Site.

1) Increase the priority STP value on the Nexus 9K TC (STP value 12288 – 20480)

2) Configure trunk link between core switch and 9K allowing all VLANs in TC. (Link between D01
and TC-NEW-CORE). This need doing at the time when the 1xGiG link between TC and AM are
down

AM- Pre-work On the DAY prior to outage.

1) Make sure that TC is active on all network devices FTD/ASA’s/F5.

2) Take stats of Network

3) Disconnect the cross site links x2 1GB. (Potential downtime at AIM). AM side will create a L2
port-channel on 9k . At this point a port channel will be created on NEW-CORE-9K connectivity
between the 2 sites should establish.
 4) Disconnect the cross site links x2 1GB. (Potential downtime at AIM). Plug them in to NEW-CORE-
N9K connectivity between the 2 sites should establish. Follow point 2 (Pre-work prior to the Day
on Site)

5) Once 9NK in AIMES is cabled and connected up and running. Failover to Aimes. (Make AM active
(manually failover, failover firewall/FTD/F5/u-smart Fw. Now at this time AM is fully up and
running.)

TC- Actual Work – OUTAGE

1) At 22:00 connect TC-A4-D02 source port 18 (old-port 47) to Destin (New) port 45 on the N9K
switch (HVG Link) Down time here

2) Thereafter move port 10 on existing core to N9K switch port 10 (Internet) Down time here

3) Move cables 41 and 42 on core switch to 9k. (ASR)

4) At 22:05 uSmart connectivity will re-establish. (Aimes will be active VPN-tunnels)

NOTE – Any traffic not replicated in Aimes will be down still.

5) Cable 9k switch moving the remaining cables from Core to new 9k including all remaining cross
connects.

6) Old Core switch should have no cables connected.

7) Failover all network devices to Telecity and check connectivity.

8) Test connectivity and compare with results prior to change.

PHASE-2 (Optional but strongly recommanded).

1. Get rid of no STP vlan configurations on the Core/Access switches. (Example no spanning tree
vlan x,y,z)

2. Force N9K-TC to become Root STP Primary and N9K-AM become Root STP secondary. (STP
reconvergence potentially hiccup in network.)

9) Test connectivity and compare with results prior to change.

Action plan for replacing two core switches in Telecity and Aimes

The two switches are essential in supporting the business, providing access to core and uSmart services
along with inter-site connectivity between Data Centers.

Access to all servers and services will be lost while the switches are disconnected from the network.
A 4 hour and 30 minutes maintenance window will be required although actual downtime could be 1
hours[with a best effort of 10 minutes].

We should be in position to start the pre-work at 21:00, looking to stop services at 22:00.

The work in Telecity will take longer to complete, due to rack space restrictions and overhanging cables.

Overall service downtime would therefore depend on the time taken to complete the work in Telecity
only.

This approach would result in one period of service disruption.

Furthermore, the new links between Data Centers can be tested and confirmed as working before
starting the work to replace the switches.

Before any site visit

Raise change to help co-ordinate involvement from other departments (Done)

Check which servers or services need to be shutdown/stopped prior to beginning any work, and any
requirements to consider such as shutdown/ startup sequence of events to be detailed in Annex C.
(Done)

Confirm the rack to rack fiber cross-connects are installed in Telecity - S2-D05 to B5-D01, S1-D04 to B5-
D01 and A4-D02 to B5-D01 and Aimes - E12-D01 to E13-D02. (Done).

Confirm site to site cross-connects are installed and signed off. (TBC).

Confirm all SFP, XFP and fibers have been received. (waiting for QSFP)
A visit to Telecity to complete a site survey would be beneficial but may not be possible. We may need
to power down other devices, either network or servers, to create space to allow the switch to be
replaced. Mike G to advise on whether any server would need to be gracefully shutdown rather than a
hard power down. (DONE)

Contact details to be added to a list in annex C.

Decide what notice period customers will need as this will influence the Date we propose for the work
to be carried out.

Account/Project managers to ensure all customers are informed as to the time and duration of the
maintenance window - feeding back any customer comments to the Network Team.

Site visits to be arranged once the final "GO" decision from Phil has been approved [may be influenced
by customer feedback or objections].
Visit to Chorley Office to pre-configure switches

Configure the new switches and test interoperability.

Port descriptions for the new switches, will be shared separately with network team - Visio site diagram
will have a new tab detailing all switch port descriptions.

Explore capability of new switches with regard to number of VLAN's and PVST limitation and configure
accordingly.

Configure inter-site links into a port-channel and assign SVI's for testing now and when first connected in
the Data Centers.

Segregate the switches, components and tools for each site:

Telecity

 1 x switch
 2 x power cables
 45 x 1GB SFP's [three spare]
 11 x 10GB XFP's
 11 x fiber patch cords
 cage nuts
 label maker and screwdriver
 PPE ?

Aimes

 1 x switch
 2 x power cables
 46 x 1GB SFP's [one spare]
 5 x 10GB XFP's [one spare]
 4 x fiber patch cords [one spare]
 cage nuts
 label maker and screwdriver
 PPE ?

pre-configure TenGigabit interfaces in both sites

Telecity

               D09 Te1/1/1, D04 Te1/1/1 and D02 Te1/1/1 [leaving as no shut]

              

               *** discuss with Mike Goulding and agree the best option for dealing with D05 Te1/1/3

                              removing from port-channel 10 may result in dropped traffic from TC-SAN-SW1

                              but the interface cannot be used unless it is removed from the port-channel

               if links to TC-SAN-SW1 can be shutdown on Manchester SAN then we can proceed in advance

                              D05 remove Te1/1/3 from port-channel 10 and reconfigure interface [leaving as no
shut]

               else continue to use two 1GB links between switches - this is not the preferred option

                              D01 Gi1/0/30 to D05 Gi1/0/35 AND D01 Gi1/0/46 to D05 Gi1/0/48 [spare 1GB SFP's will
be available to cover this outcome]

                             

Aimes                  
               E13-D02 Te1/1/1 [leaving as no shut]

On the day

Yahya and Steve will carry out the work in Telecity - Dave and Faz will carry out the work in Aimes [not
fixed and can be amended].

Sheraz on hand in the office [or home] to co-ordinate the work and to carry out testing or instigate
troubleshooting.

Ensure a laptop and console cable available at both sites

Before installing the switches into the racks, test connectivity over the new fiber links between sites -
identify and match ports [i.e. 47 to 47, 48 to 48] [30 minutes to complete]

carry out labelling and pre-work

Telecity  [possibly 2 hours to complete]

D01

cable in Gi1/0/44 - label cable D01 port 17

cable in Gi1/0/17 - label REMOVE

cable in Gi1/0/43 - label REMOVE

cable in Gi1/0/45 - label REMOVE

cable in Gi1/0/48 - label REMOVE

cable in Gi1/0/47 - label REMOVE

all remaining cables to be labelled to reflect existing port Gi1/0/X - TC-B5-D01 port X [too time
consuming to include the destination]

               ***Depending on SAN discussion

               ***possibly cable in Gi1/0/46 - label cable D01 port 46 to D05 Gi1/0/48 OR label REMOVE

               ***possibly cable in Gi1/0/30 - label cable D01 port 30 to D05 Gi1/0/35 OR label REMOVE

label fiber D01 port 43 to D09 Te1/1/1 [both ends]

label fiber D01 port 44 to D04 Te1/1/1 [both ends]

label fiber D01 port 45 to D02 Te1/1/1 [both ends]

               ***possibly label fiber D01 port 46 to D05 Te1/1/3 [both ends] OR will stick to using two 1GB
links

label fiber D01 port 47 to AM-E12-D01 port 47 [both ends]

label fiber D01 port 48 to AM-E12=D04 port 48 [both ends]

D05

               ***possibly cable in Gi1/0/35 - label cable D05 Gi1/0/35 to D01 port 30 OR label REMOVE

               ***possibly cable in Gi1/0/48 - label cable D05 Gi1/0/48 to D01 port 46 OR label REMOVE

               ***possibly cable in Te1/1/3 - label REMOVE OR ignore this action

               ***possibly label fiber D05 Te1/1/3 to D01 port 46 [both ends]

D09

cable in Gi1/0/48 - label REMOVE

no fiber required - same rack as D01 so fiber already labelled

D04

cable in Gi1/0/48 - label REMOVE

label fiber D04 Te1/1/1 to D01 port 44

D02

cable in Gi1/0/18 - label REMOVE

label fiber D02 Te1/1/1 to D01 port 45

Aimes  [possibly 1 hour to complete] simultaneous with Telecity work

E12

cable in Gi1/0/47 - label AM-E12-D01 port 22

cable in Gi1/0/22 - label REMOVE

cable in Gi1/0/48 - label REMOVE

cable in Gi1/0/47 - label REMOVE

all remaining cables to be labelled to reflect existing port Gi1/0/X - AM-E12-D01 port X [too time
consuming to include the destination]

label fiber AM-E12-D01 port 46 to AM-E13-D02 Te1/1/1

label fiber AM-E12-D01 port 47 to TC-B5-D01 port 47

label fiber AM-E12-D01 port 48 to TC-B5-D01 port 48

E13

cable in Gi1/0/36 - label REMOVE

label fiber AM-E13-D02 Te1/1/1 to AM-E12-D01 port 46

stop all applications and power down servers as specified in annex A

disconnect all cables [trying to preserve position and cable run]

power off firewall and possibly F5 in Aimes prior to AM-E12-D01 being replaced

Replace the switches and power up  [30 minutes to complete]

connect the fibers first  [1 hour for all cables/fibers]

Telecity

connect fibers in D09 Te1/1/1, D04 Te1/1/1, D02 Te1/1/1 and ***D05 Te1/1/3

then connect D01 ports 43, 44, 45, 46, 47 and 48

Aimes

connect fibers in E13-D02 Te1/1/1

then connect E12-D01 ports 46, 47 and 48

check that all switches are accessible via CLI.

Both Sites

Connect the remaining cables and check the port states match the pre-change condition

Turn on firewall and possibly F5 in Aimes - check synchronisation.

Carry out testing as per annex B  [possibly 30 minutes].

Start all applications and servers as specified in annex A.

Remove all cables labelled REMOVE [poss 12 in Telecity, 4 in Aimes].

Roll back

This would need to be a complete reversal of the work carried out for switch related failures.

10GB link failures - it may be possible [after some reconfiguration] to roll back to using 1GB connections.

1GB link failures – would need to be assessed but unlikely to be carried so full roll back may be required.

Annex A

Server and Services Shutdown - TBC

Annex B.

Testing strategy – Sheraz to provide

Annex C

Contact details - TBC