Case Study - Refreshing A CS 10 Test Development System With Fresh Production Data

Product: OpenText Content Server
Version: 10.0.0
Task/Topic: Deployment
Audience: Administrators
Platform: SQL Server 2008, Windows 2008 R2
Document ID: 500154
Updated: September 12, 2013
Case Study
Refreshing an Existing Content Server
10.0.0 Test or Development System with
Fresh Production Data
Contents
Introduction .................................................................................................................4
Case Study Scenario .................................................................................................. 5
Current Production Environment: ........................................................................... 5
End Goal: ......................................................................................................... 7
Step One: Snapshot Production Database ............................................................... 8
OUTAGE WINDOW BEGINS........................................................................... 8
Shut Down the Production Environment .......................................................... 8
Disable Notifications ...................................................................................... 11
Prepare the Index for migration to Test Environment ........................................... 15
Acquire the Metadata Memory Settings ......................................................... 16
Stop Processes .............................................................................................. 21
Obtain a Snapshot of the Production Database ................................................... 27
Back up the Content Server 10.0.0 Index ............................................................ 30
Migrate the Production 10.0.0 Test EFS............................................................... 33
Bring the Production Environment Back Online ................................................... 33
Start the Search and Indexing processes ...................................................... 34
Start Index Engines ................................................................................. 35
Start Extractor Process ............................................................................ 37
Start Document Conversion Process ...................................................... 37
Start Update Distributor ........................................................................... 38
Start Enterprise Prospector Importer ....................................................... 38
Start Memcached Processes ......................................................................... 39
Start Admin and User Help Data Sources ...................................................... 40
Enable Notifications .............................................................................................. 43
OUTAGE WINDOW ENDS ................................................................................... 48
Step Two: Deploy Database Snapshot to Test SQL Server................................... 49
On the Test Environment ...................................................................................... 49
Disable Agents and Notifications ................................................................... 50
On the Front-End Servers .............................................................................. 50
On the Back-End OTCS Primary Admin Server ............................................ 51
On the dedicated OTCS Agents and Notifications Server ............................. 52
Stop the Tier-One Servers ............................................................................. 53
Shut Down the Tier-One Load Balancer ................................................. 53
Stop SEA Server services ....................................................................... 54
Stop the Tier-Two Servers.............................................................................. 55
Stop the Tier- Two Load Balancer ........................................................... 55
Stop the Front-End Server services ........................................................ 56
Stop Agents and Notifications Server Services.............................................. 57
Disconnect from the Test Database ............................................................... 58
Restore the Production Database and EFS Snapshot to the Test
Environment ................................................................................................... 59
2
Restore the Production Database Image to the Test SQL Server ................. 60
Migrate the Production 10.0.0 Test EFS............................................................... 63
Restore EFS, fix permissions and sharing..................................................... 64
Connect the Database to Content Server 10.0 .................................................... 65
Reconcile the Production and Test 10.0 opentext.ini File Settings ................ 66
Connect to the Database ............................................................................... 68
Migrate the Production Index to Content Server 10.0 .......................................... 72
Reconfigure the Index Process and Location Pathways ............................... 72
Resynchronize the Admin Servers ................................................................. 78
Replace Production Signature Files with new OTCS 10 Signature
Files ......................................................................................................... 83
Delete the Original OTCS 10 Test Environment Index ............................ 84
Restore the Production Index .................................................................. 85
Turn on Search Debug Logging ..................................................................... 86
Resynchronize all Index Partitions ................................................................. 87
Match Metadata Memory Settings with the Production Environment
Settings .................................................................................................... 88
Start Indexing Processes Individually ............................................................ 91
Start Index Engines ................................................................................. 92
Start Extractor Process ............................................................................ 94
Start Document Conversion Process ...................................................... 94
Start Update Distributor ........................................................................... 95
Start Enterprise Prospector Importer ....................................................... 95
Start Memcached Processes ......................................................................... 96
Start Admin and User Help Data Sources ...................................................... 97
Verify the Enterprise Index ............................................................................. 99
Update the Secondary Admin Server opentext.ini file. ......................... 101
About OpenText ...................................................................................................... 106
3
Introduction
This document provides a technical overview of the steps for replicating and
refreshing the data from an OpenText Content Server (OTCS) 10.0.0 production
environment over to an existing OTCS 10.0.0 test or development environment. This
case study is not intended to replace the manuals distributed with the OTCS, but
rather to supplement the information in the official manuals. It is targeted for Content
Server Administrators (the person or people responsible for deploying, administering,
and maintaining Content Server). This guide presumes that you possess the
following knowledge and skills:
• An understanding of Content Server’s purpose within your organization and how
it will be used
• Familiarity with each server’s operating system, and any legacy products or
systems that will be integrated with OTCS
• Web server administration skills
• Database administration skills (or a database administrator to assist you)
Your environment may differ in optional modules, patches, database and operating
system platforms as well as architecture. Information here is specific to this particular
case study scenario.
Customers who are considering a refresh of test data on their replica of their
production environment for testing and development purposes should consider
consultation with our Technical Services Group.
DISCLAIMER:
This Case Study contains procedures and sample SQL code. All
procedures and sample code in this area are specific to the
scenario presented in the Case Study and are delivered as is, for
educational purposes only. This Case Study is presented as a
guide to supplement official Open Text product documentation.
Procedures on deployments using other operating systems and
database application platforms may differ in some areas.
Deployments with additional optional modules may also add to the
complexity.
4
Case Study Scenario
Current Production Environment:

The OTCS 10.0.0 SP2 deployment consists of ten servers. The servers are divided
into three tier levels. Each tier will be a dedicated security zone, separated by
firewalls.
Tier one (the Access Tier) consists of three Secure Extranet Architecture (SEA)
servers, deployed behind a load balancer. The SEA servers are deployed on the
Tomcat 6 Web Application server, along with the Java SDK, on the Windows 2008 R2
platform.
Tier two (the Application Tier) consists of four front-end servers located behind a
secondary load balancer. Each server has a single instance of OTCS with 15 threads
each.
Tier three (the Data Tier) consists of an additional three servers making up the back
end. Two of the servers (Primary or Default admin server and Admin B or secondary
admin server) are utilized for full-text indexing and Search. The remaining server is
dedicated to running Agents and Notifications. The servers will not be load balanced.
The External File Store will reside on a NetApp server. The database will be on
Microsoft® SQL Server, and the Enterprise Index will be housed locally on the
primary Admin server.
The system is bilingual (English and French), and the current patch level is Update
10.
Figure 1
- Production OTCS
10.0 Architecture
5
Current Test Environment:
The Test Content Server 10.0.0 environment was built over one year ago.
The test environment architecture is an exact replica of the production environment.
The deployment also consists of ten servers, divided into three tier levels.
Tier one (the Access tier) consists of three Secure Extranet Architecture (SEA)
servers, deployed behind a load balancer. The SEA servers are deployed on the
Tomcat 6 Web Application server, along with the Java SDK, on the Windows 2008 R2
platform.
Tier Two (the Application Tier) consists of four front-end servers located behind a
secondary load balancer. Each server has a single instance of OTCS with 15 threads
each.
Tier three (the Data Tier) consists of an additional three servers making up the back
end. Two of the servers (Primary or Default admin server and Admin B or secondary
admin server) are utilized for full-text indexing and Search. The remaining server is
dedicated to running Agents and Notifications. The servers will not be load balanced.
The External File Store will reside on a NetApp server. The database will be on a
Microsoft SQL Server, and the Enterprise Index will be housed locally on the primary
Admin server.
The system is bilingual (English and French), and the current patch level is Update
10.
Figure 2
- Test Environment
OTCS 10.0
Architecture
Because the Content Server 10.0.0 test environment was built one year ago, it will be
necessary to first ensure the installed optional modules also match the production
6
environment. This should be the case since it is a best practice to install and test any
new optional modules and patches on the test environment first. Once a new or
updated version of an optional module has been cleared on the test environment, it is
then deployed to production.
All hotfixes and modular patches also match the production environment.
If there is any discrepancy between the production and test environment, these will
have to be addressed first. Perform whatever tasks are necessary to ensure the test
environment is an exact replica of the production environment before proceeding any
further.
End Goal:
The test environment is still deployed on an image made over a year ago. Over the
past year however, the production environment has grown with numerous
documents, projects, workflows, users etc. which have been added, deleted and
changed.
The goal is to update the test environment with a fresh copy of the database and
external file store from the production environment for continued testing and
development purposes.
Note: In the event your environment is using Archive Server for

storage, please refer to the following Knowledge Base article:
Article Number: 18366967 - Archive Server - Cloning an
Archive Server Computer
In addition to the above article, there are also changes required
on Content Server when Archive Server is present. If this scenario
is more applicable to your situation, please Contact OpenText
Customer Support for further assistance.
7
Step One: Snapshot Production Database
For this section, we visit the production environment.
OUTAGE WINDOW BEGINS

To ensure that the images of the database, EFS, and index snapshot all match, the
first step will be to temporarily lock out the users by shutting down the load balancer
and SEA servers on the production environment. Schedule this event during a
regular maintenance window or, better yet, over a weekend. Large deployments may
take an extended period of time to back up and copy data over for the test
environment.
Shut Down the Production Environment

Stop the Load Balancer for Tier One.
Load Balancer
(SSL Accelerator Card)
SEA Servers
1 Instance per server TIER ONE
Tomcat 6
JSDK (Access Tier)
Shared view cache
Load Balancer
Front End Servers TIER TWO

1 Instance per server
15 threads per instance
(Application Tier)
Figure 3
Full-Text Indexing and Agents and

Search Servers Notifications Server
Index stored locally on Primary TIER THREE
Admin Server (Data Tier)
3 Partitions
3 Index Engines
2 Search Federators
6 Search Engines
INDEX EFS DATABASE
8
Shut down the Tier-One SEA servers
Figure 4
9
Stop the Load Balancer for Tier Two
Figure 5
At this point, the next step is to disable all notifications.
10
Disable Notifications
1. Log on to the Content Server Administration page, and in the Notification
Administration section, click the Configure Notification link.
Figure 6
2. Click Disable in the Enable Notifications section to stop Notifications from

running. At the bottom of the page, click Submit.
Figure 7
11
3. Click Configure Scheduled Activities. Make note of, or make a screen capture
of, all the settings. This may be required when the time comes to re-enable
Notifications.
Figure 8
4. Click Disable for all items. Click Submit when complete.
Figure 9
5. Restart the Content Server service.
Figure 10
12
6. Click Continue.
Figure 11
With the Notifications now turned off, shut down the individual Content Server
services on each of the Tier-Two front-end Content Server servers.
Figure 12
13
Stop the Content Server service and disable it on the dedicated Agents and
Notifications Content Server.
Figure 13
With only the Admin server service running on Admin Server B, and both Content
Server services running on the primary Admin server, the production index processes
will now be stopped in preparation for copying to the test environment.
14
Prepare the Index for migration to Test Environment
7. To prevent a potentially large number of notifications from being issued while the
search migration process is performed, disable the email alerts on the production
OTCS 10.0 instance. This setting is brought forward from production OTCS 10.0
to the test OTCS 10.0 instance. To do this, in the Search Administration section
of the Content Server Administration page, click Configure System Object E-
Mail Delivery.
Figure 14
8. Under the Recipients and Options section, clear the Active check box, and
then click Apply. Click Admin Home.
Figure 15
15
9. In the Content Server Administration page, in the Search Administration
section, click the Open the System Object Volume link. On the Content Server
System page, click the Enterprise Data Source Folder link.
Figure 16
Acquire the Metadata Memory Settings

In the event any of the settings differ from the test environment, or as a method of
double-checking that the settings match, do the following:
10. On the Functions menu for the Enterprise Partition Map, click Properties >
Metadata Memory Settings.
Figure 17
11. Record the settings on this screen.
Figure 18
16
12. Under the Action row, click on the Edit button on the top. Record the values in
this screen. (Use screen captures and scroll down the list, if necessary.)
Figure 19
13. Click the Back button on the browser to return to the previous screen, and click
the Edit button under the Action column, on the second row this time.
Figure 20
17
14. Record the values in this screen. (Use screen captures and scroll down the list, if
necessary.)
Figure 21
15. Click the Back button on the browser to return to the previous screen, and click
the Edit button under the Action column, on the third row this time.
Figure 22
18
16. Record the values in this screen. (Use screen captures and scroll down the list, if
necessary.)
Figure 23
17. Click the Back button on the browser to return to the previous screen, click the
Up arrow, and then click Content Server System to return to the System page.
Figure 24
19
18. At this point, further content indexing will cause the instances to be out of synch.
To prevent this, stop the Extractor and any non-Enterprise data sources (such as
Directory Walker, Spider, or XML Activators). Click Enterprise Data Source
Folder.
Figure 25
19. Click Enterprise Data Flow Manager.
Figure 26
20. Monitor and wait until the iPools have all reached a zero size (count of 0 in the
Pending column for Interchange Pools) for each data source. In the event that
an iPool transaction cannot be completed, this will become a special case for
correction later. Record the details, back up the non-zero iPools, and then flush
the iPools.
Figure 27
20
Stop Processes
When the database settings are migrated into the test environment, the new system
will try to restore the saved state of the various processes. It is very important that the
state of the system is idle when it is moved to the test environment. Ensure that all of
the following processes are stopped using the Content Server Administration page:
• Extractors
• Directory Walkers
• XML Activators
• Document Conversion Services
• Admin servers
• Update Distributors
• Index Engines
• Search Federators
• Search Engines
At this point, the operating system tools should also be used to stop the Content
Server Admin services. The objective is to prevent the test environment from
inadvertently connecting to another running instance of an OTCS 10.0.0 Content
Server Admin service.
21. On the Functions menu for Enterprise Extractor Process, click Stop.
Figure 28
22. Browse up one level to Enterprise Data Source Folder, and click Suspend on
the Functions menu for Enterprise Data Flow Manager.
Figure 29
21
23. On the Functions menu for Enterprise Search Manager, click Stop.
Figure 30
24. On the Functions menu for Enterprise Partition Map, click Stop.
Figure 31
25. Browse to the Content Server System page. There are still processes running.
These are from the Admin Help and Help Data Sources, as well as the
Memcached Processes. Click Admin Help Data Source Folder.
Figure 32
22
26. On the Functions menu of the Admin Help Data Flow Manager, click Suspend.
Figure 33
27. On the Functions menu of the Admin Help Search Manager, click Stop.
Figure 34
28. Browse to the Content Server System page. Click Help Data Source Folder.
Figure 35
23
29. On the Functions menu of the Help Data Flow Manager, click Suspend.
Figure 36
30. On the Functions menu of the Help Search Manager, click Stop.
Figure 37
31. Browse to the Content Server System page. Click Memcached Processes.
Figure 38
24
32. On the Functions menu of Memcached 1, click Stop.
Figure 39
Repeat for the Memcached 2 and Memcached 3 processes.
33. Browse to the Content Server System page. Ensure that no other processes
are running.
Figure 40
34. Stop the Content Server and Content Server Admin services on the primary
Admin server.
Figure 41
25
35. Stop the Content Server Admin service on the secondary Admin server.
Figure 42
The database, EFS and index are now ready for copying to the test environment.
Load Balancer
SEA Servers
Tomcat 6
JSDK
SHUTDOWN STATE TIER ONE
(Access Tier)
Shared view cache
Load Balancer
Front End Servers

SHUTDOWN STATE TIER TWO
15 threads per instance
(Application Tier)
Figure 43
Full-Text Indexing and
Search Servers Agents and
1 Instance per server Notifications Server
Index stored locally on TIER THREE
Primary Admin Server (Data Tier)
3 Partitions
3 Index Engines
2 Search Federators
SHUTDOWN STATE
6 Search Engines
INDEX EFS DATABASE
26
Obtain a Snapshot of the Production Database
Using the built-in backup utility provided by SQL Server is the easiest way to take a
snapshot of the production database.
NOTE: If your production Content Server instance is using

external file storage (rather than internal BLOB storage), you will
also need to restore a copy of the external file store. The
database, file store and index backup should be synchronized
with each other (backed up simultaneously).
36. Back up the entire Content Server database to a file or tape. To use the built-in
backup facility in SQL Server 2008, launch SQL Server Management Studio, and
log on as sa. Right-click the production database, and then click Tasks > Back
Up.
Figure 44
27
37. In the General tab, ensure that the backup type is Full and that the Destination
is set to Back up to Disk, and then click Add. Next, click Options on the left.
Figure 45
NOTE: When restoring a database from one system to another,

make sure that the path to the mdf and ldf files is valid,
because in most cases it will not be.
38. On the Options pane, enable Overwrite all existing backup sets and Verify
Backup when finished. Click OK.
Figure 46
28
39. Click OK when the database backup completes.
Figure 47
40. The database backup file resides on the production SQL Server for now.
Figure 48
29
Back up the Content Server 10.0.0 Index
The production index will be copied over to a temporary folder on the test
environment’s primary Admin server. It will not be in place during the initial migration.
For clarity, see the figures below each step.
41. On the primary Admin Server, create a new folder called prod_backup.
Figure 49
30
42. Copy the index folder (housing the enterprise, adminhelp and help folders)
from the production system to the prod_backup folder on the OTCS 10.0 server
computer.
Figure 50
43. Return to the production system, browse to the C:\OPENTEXT\config\ folder,

and copy the LLFieldDefinitions.txt and
LLFieldDefinitions_EL.txt files to the prod_backup folder on the new
OTCS 10.0 server computer.
Figure 51
31
44. Paste the LLFieldDefinitions.txt and LLFieldDefinitions_EL.txt
files into the prod_backup folder on the new OTCS 10.0 server computer.
Figure 52
32
Migrate the Production 10.0.0 Test EFS
Next, deploy the corresponding test OTCS 10.0.0 EFS to the new location on the
NetApp appliance, but under a temporary folder name (because the test environment
is still in an active state at this point).
Figure 53
NOTE: In this scenario, the EFS consists of only one folder. Some
deployments may have multiple folders which will need to be
deployed in the same manner.
Bring the Production Environment Back Online
45. Start the Content Server services on the production primary Admin server.
46. Start the Admin server service(s) on the secondary Admin server.
33
Figure 54 illustrates the current state of the environment. Only the Admin servers are
running.
Figure 54
Start the Search and Indexing processes

47. Log on to the Content Server Administration page, browse to the Search
Administration section, and click the Open the System Object Volume link.
Figure 55
34
48. Click the Enterprise Data Source Folder link. Start the Enterprise Search
Manager.
Figure 56
49. Click the up arrow to return to the Enterprise Data Source Folder. Click
Enterprise Data Flow Manager.
Figure 57
Start Index Engines

50. At this point, the Admin server, DCS, and the Index Engines can be started.
Indexing processes (Extractors) should not be started yet; verify that the
Extractor is stopped. The Update Distributor must be started first (do not use the
convenience function that starts the Update Distributor and all the Index
Engines). On the Functions menu of the Enterprise Update Distributor, click
Properties > Index Engines.
Figure 58
35
51. On the Functions menu of the Enterprise Index Engine for Partition 1, click
Properties > Specific.
Figure 59
52. In the Actions section, click Start to start the Index engine for Partition 1.
Figure 60
53. Once the Index Engine for Partition 1 is running, return to the Enterprise Index
Engine for Partition 1 page. Click the Back button on your browser and repeat the
above steps for each Index Engine.
Figure 61
36
54. Once all the Index Engines are running, click the Navigate To menu and select
Figure 62
Start Extractor Process
55. On the Functions menu of the Enterprise Extractor Process, click Start.
Figure 63
Start Document Conversion Process
56. On the Functions menu of Enterprise Document Conversion, click Start.
Figure 64
37
Start Update Distributor
57. On the Functions menu of the Enterprise Update Distributor; select

Properties, Specific, and then click Start in the Actions section.
Figure 65
Start Enterprise Prospector Importer
58. Go up one level. On the Functions menu of Enterprise Prospector Importer,

click Start.
Figure 66
38
Start Memcached Processes
59. Browse to the Content Server System page. Click Memcached Processes.
Figure 67
60. On the Functions menu for Memcached 1, click Start.
Figure 68
Repeat for Memcached 2 and Memcached 3.

61. Return to the Content Server System page. All Processes for the Enterprise Data
Sources and Memcached should now be listed as running with no errors.
Figure 69
Next, the Admin Help and Help Data Source processes will be started.
39
Start Admin and User Help Data Sources
62. On the Functions menu for the Admin Help Search Manager, click Start.
Figure 70
63. On the Functions menu for the Admin Help Data Flow Manager, click Resume.
Figure 71
64. Return to the Content Server System page. The Help Data Source processes will
be started next. Click Help Data Source Folder.
Figure 72
40
65. On the Functions menu of the Help Search Manager, click Start.
Figure 73
66. On the Functions menu of the Help Data Flow Manager, click Resume.
Figure 74
Return to the Content Server System page. All Search and Indexing processes are
back online for production.
Figure 75
41
Start up the Agents and Notification Content Server service. Figure 76 below
illustrates the current state of the system.
Figure 76
Start up the individual Content Server service on each front-end server one at a time
and confirm connectivity to the database, EFS and Admin servers.
Figure 77
42
Enable Notifications
67. Return to the Content Server Administration page and, in the Notification
Administration section, click Configure Scheduled Activities.
Figure 78
68. Using the notes or screen captures that you made earlier, enable all items that
were originally selected. Click Submit when complete.
Figure 79
43
69. Click the Configure Notification link.
Figure 80
70. Click Enable in the Enable Notifications section to start Notifications. At the
bottom of the page, click Submit.
Figure 81
Figure 82
44
72. Click Continue.
Figure 83
73. Return to the Search Administration section, and click Configure System
Object E-Mail Delivery.
Figure 84
74. Under the Recipients and Options section, select the Active check box, and
then click Apply. Click Admin Home.
Figure 85
45
Start up the Tier Two Load Balancer and ensure that the four front-end Content
Servers are in the load balancer pool and that all is functioning normally.
Load Balancer
SHUTDOWN STATE
SEA Servers TIER ONE
(Access Tier)
Load Balancer

(Application Tier)
Figure 86
Agents and
Notifications Server
Full-Text Indexing and TIER THREE
Search Servers (Data Tier)
DATABASE EFS INDEX
46
Start up the individual SEA Server services on each SEA server one at a time and
confirm connectivity through the SEA servers to the database, EFS and Admin
servers.
Load Balancer

(Access Tier)
Load Balancer

(Application Tier)
Figure 87
Agents and
DATABASE EFS INDEX
47
Start up the Tier One Load Balancer and ensure that the three SEA servers are in the
load balancer pool and that all is functioning normally.
Load Balancer

(Access Tier)
Load Balancer

(Application Tier)
Figure 88
Agents and
DATABASE EFS INDEX
The production environment is now back online.
OUTAGE WINDOW ENDS
48
Step Two: Deploy Database Snapshot to Test
SQL Server
On the Test Environment

The OTCS 10.0 database schema used for testing will be replaced with an updated
copy of the current OTCS 10.0.0 production database schema. Since the two
schemas are identical and the end result will only be the actual content being
updated, the database connection information for the servers that will be offline is not
going to change.
Figure 89
49
Disable Agents and Notifications
The first step is to disable Agents and Notifications across the cluster.
On the Front-End Servers

For each front-end server do the following:
1. Back up and then edit the opentext.ini file
2. Scroll down to the [loader] section. Comment out the lines
load_distributedagent=distributedagent
load=sockserv;javaserver
#;agents;notify;wfagent
#load_relagent=relagent
#load_distributedagent=distributedagent
#load_daagent=daagent
#load_verify=verifyAgent
This configuration will stop the distributed agent from loading.
3. Close and save the opentext.ini file.
Figure 90
- OTCS 10 Agent
Distribution Map
50
On the Back-End OTCS Primary Admin Server
2. Scroll down to the [loader] section. Comment out the lines
load_verify=verifyAgent:
This configuration will stop the verify agent from loading.
The current state of the Agent deployment is illustrated below:
Figure 91
- OTCS 10 Agent
Distribution Map
51
On the dedicated OTCS Agents and Notifications Server
2. Scroll down to the [loader] section. Move and comment out agents,
notify and wfagent from the top line, and then comment out the remaining
lines:
All Agents and Notifications are now disabled across the cluster.
Figure 92
- OTCS 10 Agent
Distribution Map
Next, all non-essential servers in the cluster will be shutdown.
52
Stop the Tier-One Servers
Shut Down the Tier-One Load Balancer
Stop the Load Balancer for Tier One.
Figure 93
53
Stop SEA Server services
Once the Load balancer has been stopped, shut down each individual SEA service
and disable it on each of the Tier-One SEA servers.
Figure 94 illustrates the state of the OTCS 10.0 web farm after the shutdown of the
Tier-One servers.
Figure 94
54
Stop the Tier-Two Servers
Stop the Tier- Two Load Balancer
Stop the Load Balancer for Tier Two. Figure 95 illustrates the state of the OTCS 10.0
web farm after the shutdown of the Tier-Two load balancer.
Figure 95
55
Stop the Front-End Server services
Once the Load balancer has been stopped, shut down the individual Content Server
services and disable them on each of the Tier-Two front-end Content Server servers.
Tier-Two servers.
Figure 96
56
Stop Agents and Notifications Server Services
Stop the Content Server service and disable it on the dedicated Agents and
Notification Content Server.
Agents and Notification server.
With the remaining two Admin servers running, the web farm is ready for the
migration.
Figure 97
57
Disconnect from the Test Database
1. Log on to the test environment primary Admin server’s Administration page. In
the Database Administration section, click the Change Current Database link.
Figure 98
2. Log on as the Admin user.
Figure 99
3. The Deselect Content Server Database page then appears. By default, the
Enterprise Data Source Folder is selected for deletion. Leave this selected
because the Search Index it is referring to belongs to the test database. Click
Continue.
Figure 100
58
4. Close the browser window and stop the Content Server services on the primary
Admin server and the Content Server Admin service on the secondary Admin
server.
Figure 101
Restore the Production Database and EFS Snapshot to the

Test Environment
The next step is to restore the production database snapshot and External File Store
image to the test environment servers.
Figure 102
59
Restore the Production Database Image to the Test SQL Server
5. Log on to SQL Server Management Studio on the test SQL Server as sa and
connect to the test database. Right-click the test database, and then click Tasks
> Restore > Database.
Figure 103
6. In the Source for restore section, enable From device and then click the ellipsis
button ( ).
Figure 104
60
7. Select File in the Backup media menu, and then click Add.
Figure 105
8. Locate the production database backup file and click OK. Enable Restore, and
then click Options on the left.
Figure 106
61
9. Enable Overwrite the existing database and Leave the database ready to
use. Click OK.
Figure 107
10. The staged database “test” will now be replaced by the backup from production.
Figure 108
62
11. Once the database restore is complete, click OK.
Figure 109
Migrate the Production 10.0.0 Test EFS

Stop sharing on the test environments’ EFS, then rename the folder (in this example
OLD_EFS).
Next, rename the Prod_EFS folder (which contains the latest copy of the production
EFS) to efs. Share the folder and assign read/write permissions to the OTCS user.
Figure 110
NOTE: In this scenario, the EFS consists of only one folder. Some
deployments may have multiple folders which will need to be
deployed in the same manner.
63
Restore EFS, fix permissions and sharing
Once the EFS from production has been restored in place, the next step is to
reconfigure the new test database by redirecting it to the new EFS path. The first step
is to check where the EFS is currently referenced:
SELECT * FROM KINI WHERE IniSection='Livelink.LogicalProviders' AND

IniKeyWord='Default'
12. The results will show where the EFS is currently referenced:
{'EXTSHARED','\\\\hal9000\\efs\\'}
To redirect it to the test external file store, run a statement similar to the following
example:
UPDATE KINI SET IniValue='{''EXTSHARED'', ''\\\\coutts1\\efs\\''}' WHERE

IniSection= 'Livelink.LogicalProviders' AND inikeyword = 'Default'
NOTE: If this were an Oracle database, you would need to

“commit” the SQL statement.
13. Check to ensure that the path to the EFS has been changed. Run the query
statement:
SELECT * FROM KIni WHERE IniSection='Livelink.LogicalProviders' AND

IniKeyWord='Default'
14. The result should show that the corrected path to the EFS entry in the database
is currently pointed at the production file store on the server (in this example,
coutts1).
{'EXTSHARED','\\\\coutts1\\efs\\'}
64
NOTE: A new system GUID has been added to Content Server
core with Update 7. This system GUID is stored in the KIni
database table so it will be the same across a cluster.
When copying a database (for example to create a test instance
based on a production instance) if this value exists in your
system, then it needs to be deleted from the KIni table. A new
one will be automatically regenerated when a call is made to
retrieve the system GUID.
DELETE FROM KIni WHERE IniSection='Livelink' AND

IniKeyword='GUID'
Connect the Database to Content Server 10.0

The database is now ready to be connected to the test environment instance.
15. Open Windows Explorer and navigate to the C:\opentext\config\ folder.

Back up and then delete the following files:
otadmin.cfg
otadmin.cfg.old
otadmin.pid
otadmin.pid.old
search.ini
NOTE: Do not delete the otadmin.pwd file. This is required in

order to connect to the database and log on to the local Admin
server.
Figure 111
16. Repeat the above procedure on the secondary Admin server.
65
17. From the prod_backup folder, copy the LLFieldDefinitions.txt and
LLFieldDefinitions_EL.txt files to the C:\opentext\config\ folder,
replacing the original files.
Figure 112
Reconcile the Production and Test 10.0 opentext.ini File Settings

The opentext.ini file settings must be present before the database is connected,
so that the migration process can move the system settings into the database.
18. Rename the production opentext.ini file prod_opentext.ini, and copy it to the
OTCS 10.0 config folder.
19. Make a backup of the test OTCS 10.0 opentext.ini file and change its name
(to opentext.ini.ori, for example).
20. Using a text editor (for example, WinMerge or TextPad), open both the
prod_opentext.ini and opentext.ini files and compare them side by
side.
21. Look for any settings in the prod_opentext.ini file that do not appear in any
form in the OTCS 10.0 opentext.ini file.
66
22. Once complete, save the changes to the opentext.ini file and close it.
NOTE:
One section of the opentext.ini files which must be compared
and merged is the [scheduleactivity] section. Because you
disabled the Agents and Notifications on the OTCS 10 production
environment for this migration, these settings will need to be
manually matched in the OTCS 10 test environments’
opentext.ini file with the settings in the production database
image.
Change only the settings that need to be matched with the
production section. Leave any new settings in their current state.
PRODUCTION VERSION TEST VERSION
[scheduleactivity] [scheduleactivity]
1000=0 1000=1 (change to 0)
4000=0 4000=1 (change to 0)
4100=0 4100=0
4101=0 4101=0
4102=0 4102=0
8900=0 8900=1 (change to 0)
8999=0 8999=1 (change to 0)
9000=0
9000=1 (change to 0)
9001=0
3502=1
8000=1
3601=0
9001_Instances=1
9001_SeparateAdmin=false Unless you have
configured multiple agents,
ignore the 9001_* settings
67
Connect to the Database
23. Close the prod_opentext.ini file and retain it. If any issues arise, this will
assist you and/or OpenText Customer Support in determining if any
settings were missed.
24. Start both the Content Server and Content Server Admin services on the
primary Admin server.
Figure 113
25. Start only the Content Server Admin service on the secondary Admin server.
Figure 114
26. On the primary Admin server, log on to the Content Server Administration page.
The Database Administration page appears. Click the Select Existing Database
link.
Figure 115
68
27. In the RDBMS Type field, click Microsoft SQL Server, and then click Continue.
Figure 116
28. Complete the required connection information to the database, and then click
Continue.
Figure 117
29. Enter the Admin password for the user Admin in the database you are connecting
to and then click Log-in.
Figure 118
30. Click Continue.
Figure 119
69
31. The Admin Server Configuration page appears.
Figure 120
The information presented here is being read from the database, therefore by default
the system is set to the actual production Admin server. In the example above, the
Admin server is listed as unavailable because the test environment is in an isolated
subnet.
The critical step here is to recognize and properly enter the updated Admin server
host name (not localhost, but rather the actual host name of the test system) and
port information before the UPDATE button is clicked.
(In our example, coutts1 is the test environment’s primary production Admin
server. The host name of the secondary Admin server—Admin B— is seaview).
32. Replace hal9000 (the production environment’s primary Admin server’s name)
in the Host Name field with coutts1 (the OTCS 10 test environment’s Admin
server’s name). Leave the Port Number field as is—remember it was installed
using the same ports as the production environment. Enter the Admin server’s
password for the OTCS Admin server (livelink by default), and click Accept.
Figure 121
70
33. Replace area51 (the production environments’ secondary Admin server’s name)
in the Host Name field with seaview (the OTCS10 test environment’s
secondary Admin server’s name). Leave the Port Number field as is—remember
it was installed using the same ports as the production environment. Enter the
Admin server’s password for the OTCS Admin server (livelink by default),
and enable Accept.
Figure 122
Figure 123 illustrates the current state of the test OTCS 10 environment:
Figure 123
71
Migrate the Production Index to Content Server 10.0
The new interface which was built for migrating the index during the upgrade process
will be used this time to connect an existing index (the production index) from another
instance to a different environment (the test environment).
Currently, the actual production index copy resides in a backup folder
(prod_backup). The original test environment index is still in place. In order for the
migration to be successful, new signature files need to be generated. These new
signature files will then be deployed to the production index copy while it is in the
backup folder before it is restored to its final destination.
At this point, the Search Index services should not be running. If they are, stop them.
At a minimum, you must ensure that no processes are running that might trigger
indexing; verify that the Extractor and Search Index engines are stopped.
Ensure that the DCS Server ports and paths are correct. These settings are imported
from production, but the new environment may require changes to these values.
Verify that the selected ports are available and change them if necessary. These
settings apply to Hit Highlighting and View as HTML, but do not affect indexing.
Reconfigure the Index Process and Location Pathways

34. Open a new browser tab for each of the four links in the Configure and Migrate
the Search System page: Document Conversion Service and Path
Management for each of the Admin servers.
Figure 124
72
35. On the Path Management page, in the Process Paths section, ensure that the
path to the Search Index is correct. In our example, the path
\\coutts1\index\enterprise\<index#> is correct for Admin server B.
Figure 125
36. In this case, the server name has changed (from hal9000 to coutts1) in the
path. Adjust each corresponding path, and then click Update Process Paths.
Figure 126
73
37. Since the Document Conversion Server is assigned to run on the primary Admin
server and not Admin B, check the Document Conversion Service page to
ensure that the Enable check box is clear. Close each browser tab when
complete.
Figure 127
38. Once again in the Process Paths section of the Path Management page, this
time on the primary Admin server, ensure that the paths (in this case,
\\coutts1\index\enterprise\<index#>) to the Search Index are correct.
Figure 128
74
39. In this case, the server name has changed in the path. Adjust each
corresponding path, and then click Update Partition Location Manager Paths.
Do not close this tab yet.
Figure 129
40. Verify that the paths in the Process Paths section are correct. If any changes
are required (in this case, only the server name needs to be updated), start with
the Admin Help Data Source Folder section. When that section is complete,
proceed to the next section.
Figure 130
75
41. Update the paths in the Enterprise Data Source Folder section (only the server
name needs to be updated). When that section is complete, proceed to the next
section.
Figure 131
42. Update the paths in the Help Data Source Folder section (only the server name
needs to be updated). Finally, update the path to the Other Processes section.
When you have finished, click Update Partition Process Paths to make the
necessary changes, and then close the browser tab.
Figure 132
76
43. Next, check the Document Conversion Service page to ensure it is set to
Enable and the port paths are correct. If any change is required, click Update,
and then close each browser tab.
Figure 133
44. On the Configure and Migrate the Search System page, click Continue.
Figure 134
45. Click Admin Home.
Figure 135
77
46. On the Content Server Administration page, in the Search Administration
section, click the Open the System Object Volume link.
Figure 136
47. Confirm that no processes are running at this point.
Figure 137
Resynchronize the Admin Servers

48. In the Admin Servers section, on the Functions menu for Default, click
Resynchronize.
Figure 138
78
49. Click OK to confirm.
Figure 139
50. Once the resynchronization completes, click Continue.
Figure 140
51. Do not be alarmed if any error messages appear. There is a known limitation in
the Admin server synchronization process that occasionally prevents some
processes from properly resynchronizing. The first synchronization joins the
processes to the Admin server. Synchronize the Admin server a second time. The
second resynchronization generates the final signature file. On the Functions
menu for Default, click Resynchronize.
Figure 141
79
52. Click OK to confirm.
Figure 142
Figure 143
54. Next, on the Functions menu for Admin B, click Resynchronize.
Figure 144
55. Click OK.
Figure 145
80
Figure 146
57. Synchronize the secondary Admin server a second time. On the Functions menu
for Admin B, click Resynchronize
Figure 147
58. Click OK.
Figure 148
81
Figure 149
60. There should be no errors present. The resynchronization process has created
new signature files for the Search Index. It is absolutely essential to retain these
files.
Figure 150
61. Stop the Content Server service and Content Server Admin service on the
primary Admin Server. Stop the Content Server Admin service on the secondary
Admin server.
Figure 151
82
Replace Production Signature Files with new OTCS 10 Signature Files
62. Browse to the Index folder on the primary Admin server, (in this case
C:\index\) and open the enterprise folder.
Figure 152
63. The signature file contains the name of the original Admin server. This is normal.
Open the new index1 folder, copy the newly generated signature file, and then
overwrite and replace the corresponding signature file in the index1 folder in the
prod_backup folder. Repeat this for the other two partitions. See figure 153
below for reference.
EXAMPLE:
Old signature file: hal9000X2099X4919215421X4123.txt

New signature file: hal9000X2099X1245129194X3214.txt
Figure 153
83
Repeat steps 71 and 72 for both the …adminhelp/index1/ and …help/index1/
folders.
Delete the Original OTCS 10 Test Environment Index
64. Once the signature file has been backed up, browse back to the Search Index
folder location (in this case C:\Index\) and delete the enterprise folder.
Figure 154
84
Restore the Production Index
65. Browse to the prod_backup folder and copy the enterprise folder from the
Production system to the OTCS 10.0 Search Index location (in this case the
C:\index\ folder).
Figure 155
The copy of the Production index is now in place for the final stage of the migration.
Figure 156
85
Turn on Search Debug Logging
66. Open the opentext.ini file. Within the [options] section, add the following
line:
wantDebugSearch=TRUE
Save and close the file.
Figure 157
67. Start the Content Server services on the primary Admin server.
Figure 158
68. Start only the Content Server Admin service on the secondary Admin server.
Figure 159
86
Resynchronize all Index Partitions
To bring the search.ini files on both Admin servers in sync, you need to
resynchronize each partition individually.
69. Log on to the Content Server Administration page. In the Search Administration
section, click the Open the System Object Volume link. On the Content Server
System page, click Enterprise Data Source Folder.
Figure 160
70. Click Enterprise Partition Map.
Figure 161
71. Click Partition 1, and then click Resynchronize.
Figure 162
87
72. Repeat steps 70-71 for Partition 2 and Partition 3.
73. Repeat steps 70-71 for the Admin Help Partition 1 and the Help Partition 1.
Restart all the Admin servers when you have finished.
Match Metadata Memory Settings with the Production Environment

Settings
It is now necessary to ensure that the Metadata Memory Settings on the new OTCS
10 Admin server match the settings from the Production Admin server exactly.
Reference the screen captures of the Production system made earlier.
Figure 163
74. Go up one level, and then start the Enterprise Search Manager.
Figure 164
75. On the Functions menu of the Enterprise Partition Map, click Properties >
Metadata Memory Settings.
Figure 165
88
76. Ensure that the settings match those from the Production screen captures.
Figure 166
77. Under the Action row, click the Edit button on the top row for the Read / Write
partitions. Ensure that the values in this screen match those from the Production
screen captures.
Figure 167
89
78. Click the Back button on the browser to return to the previous screen. Click the
Edit button, on the second row this time, under the Action column for the
Update Only partitions.
Figure 168
79. Click the Back button on the browser to return to the previous screen. Click the
Edit button, on the third row this time, under the Action column for the Read
Only partitions.
Figure 169
90
Start Indexing Processes Individually
NOTE: Starting the entire search grid at the same time places a
high load on your system, but if your deployment has only a few
partitions, it is an acceptable approach. The procedures outlined
in these steps optimize the overall time for large search indexes
that contain many partitions.
80. Click the Back button in the browser, to return to the Metadata Memory Settings
tab of the Enterprise Partition Map properties page.
Figure 170
81. Click the up arrow to return to the Enterprise Data Source Folder. Click
Figure 171
91
Start Index Engines
82. At this point, the Admin server, DCS, and the Index Engines can be started.
Indexing processes (Extractor) should not be started yet; verify that the Extractor
is stopped. The Update Distributor must first be started (do not use the
convenience function that starts the Update Distributor and all the Index
Engines). On the Functions menu of the Enterprise Update Distributor, click
Properties > Index Engines.
Figure 172
83. On the Functions menu of the Enterprise Index Engine for Partition 1, click
Properties > Specific.
Figure 173
92
84. In the Actions section, click Start to start the Index Engine for Partition 1.
Figure 174
85. Once the Index Engine for Partition 1 is running, click the Back button on the
browser to return to the Enterprise Update Distributor page. Repeat the above
steps for each Index Engine.
Figure 175
93
86. Once all the Index Engines are running, click the Navigate To menu and select
Figure 176
Start Extractor Process
87. On the Functions menu for Enterprise Extractor Process, click Start.
Figure 177
Start Document Conversion Process
88. On the Functions menu for Enterprise Document Conversion, click Start.
Figure 178
94
Start Update Distributor
89. Click Enterprise Update Distributor. Click Start in the Actions section.
Figure 179
Start Enterprise Prospector Importer
90. Go up one level. On the Functions menu of the Enterprise Prospector

Importer, click Start.
Figure 180
95
91. Browse to the Content Server System page. All processes should now be
running.
Figure 181
Start Memcached Processes

The Memcached processes collect the information that generates the new Faceted
Browsing feature in Content Server 10.0.0.
92. On the Functions menu for Memcached 1, click Start.
Figure 182
Repeat for Memcached 2 and Memcached 3.
96
93. Return to the Content Server System page. All processes are now listed as
running, with no errors.
Figure 183
Start Admin and User Help Data Sources
94. On the Functions menu for the Admin Help Search Manager, click Start.
Figure 184
95. On the Functions menu of the Admin Help Data Flow Manager, click Resume.
Figure 185
97
96. Return to the Content Server System page. The Help Data Source processes will
be started next. Click Help Data Source Folder.
Figure 186
97. On the Functions menu of the Help Search Manager, click Start.
Figure 187
98. On the Functions menu of the Admin Help Data Flow Manager, click Resume.
Figure 188
98
Return to the Content Server System page. All Search and Indexing processes are
back online for production.
Figure 189
Verify the Enterprise Index

99. On the Functions menu of the Enterprise Data Source Folder, click
Maintenance.
Figure 190
100. Enable Configure search index verification, and then click OK.
Figure 191
99
101. Enable Run Once and then enable the check box for the nearest next hour.
Alternatively, enable all of the check boxes.
Figure 192
102. At the bottom of the page, click Update. Click the Admin Home link when
complete.
Figure 193
103. Once the report has run, 1 Item will appear adjacent to the Verification
Reports folder. Click the folder to open it.
Figure 194
100
104. Select the bulk actions check box for the report, and then click Zip & Download.
Figure 195
105. Once the report is zipped, click Download Zip File. Save the ZIP file to a
location for viewing in Microsoft Excel.
Figure 196
106. Open the report file using Microsoft Excel. The key elements to look for in the
report are indications that something is missing from the Search Index.
Figure 197
The database, EFS and index are a replica of production.
Update the Secondary Admin Server opentext.ini file.

Up to this point, the Content Server service has never been started on the secondary
Admin server. The information in the opentext.ini file for the database connection
needs to be updated before you start it up (if it is ever to be started up). Open the
opentext.ini file on the primary Admin server and copy this entire section:
[dbconnection:test]
servtype=MSSQL
servname=coutts1
database=test
username=OTCS
encrypteduserpwd=aiPQvuXdfeXh5cgVATpPWQ==
101
Replace the same section on the secondary Admin server’s opentext.ini file with
this section. In the event the Content Server service is ever started on this secondary
Admin server, the database connection information will be correct.
Start Agents and Notifications across the Test Environment

All agents and notifications are in a disabled state. The [loader] section in the
opentext.ini file of every server is configured as illustrated below:
Figure 179
- OTCS 10 Agent
Distribution Map
102
On the dedicated OTCS Agents and Notifications Server
107. Scroll down to the [loader] section. Remove the comment and cut and paste
agents;notify;wfagent to the end of the top line. Uncomment the
remaining lines except for the load_verify=verify agent line:
load=sockserv;javaserver;agents;notify;wfagent
load_relagent=relagent
load_daagent=daagent
109. Restart the Content Server (OTCS) Service.
Figure 180
- OTCS 10 Agent
Distribution Map
103
On the Back End OTCS Primary Admin Server
5. Scroll down to the [loader] section. Remove the comment from the line
load_verify=verifyAgent:
load_verify=verifyAgent
This configuration will load the verify agent only on the primary Admin server.
7. Restart the Content Server (OTCS) Service
Figure 181
- OTCS 10 Agent
Distribution Map
No agents are configured to run on the secondary Admin server.
104
On the Front-End Servers
The Distributed Agent will be deployed to run on each of the front-end servers. For
each front-end server, do the following:
8. Scroll down to the [loader] section. Remove the comment from the line
This configuration will load only the distributed agent.
6. Restart the Content Server (OTCS) Service.
All Agents and Notifications are running across the cluster.
182
- OTCS 10 Agent
Distribution Map
The refresh of the database, EFS and index in the text environment is now complete.
105
About OpenText
OpenText is the world’s largest independent provider of Enterprise Content
Management (ECM) software. The Company's solutions manage information for all
types of business, compliance and industry requirements in the world's largest
companies, government agencies and professional service firms. OpenText supports
approximately 46,000 customers and millions of users in 114 countries and 12
languages. For more information about OpenText, visit www.opentext.com.
CHAMPION TOOLKIT 106

Visit online.opentext.com for more information about OpenText solutions. OpenText is a publicly traded company on both NASDAQ (OTEX) and the TSX (OTC) Copyright © 2013 by OpenText Corporation. Trademarks or registered
trademarks of OpenText Corporation. This list is not exhaustive. All other trademarks or registered trademarks are the property of their respective owners. All rights reserved.

Case Study - Refreshing A CS 10 Test Development System With Fresh Production Data

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Case Study - Refreshing A CS 10 Test Development System With Fresh Production Data

Uploaded by

Copyright:

Available Formats

Product: OpenText Content Server

Current Production Environment:

Note: In the event your environment is using Archive Server for

OUTAGE WINDOW BEGINS

Shut Down the Production Environment

Front End Servers TIER TWO

Full-Text Indexing and Agents and

INDEX EFS DATABASE

At this point, the next step is to disable all notifications.

2. Click Disable in the Enable Notifications section to stop Notifications from

4. Click Disable for all items. Click Submit when complete.

5. Restart the Content Server service.

Acquire the Metadata Memory Settings

11. Record the settings on this screen.

19. Click Enterprise Data Flow Manager.

Repeat for the Memcached 2 and Memcached 3 processes.

Front End Servers

INDEX EFS DATABASE

NOTE: If your production Content Server instance is using

NOTE: When restoring a database from one system to another,

43. Return to the production system, browse to the C:\OPENTEXT\config\ folder,

Bring the Production Environment Back Online

Start the Search and Indexing processes

Start Index Engines

Start Extractor Process

Start Document Conversion Process

56. On the Functions menu of Enterprise Document Conversion, click Start.

57. On the Functions menu of the Enterprise Update Distributor; select

Start Enterprise Prospector Importer

58. Go up one level. On the Functions menu of Enterprise Prospector Importer,

60. On the Functions menu for Memcached 1, click Start.

Repeat for Memcached 2 and Memcached 3.

71. Restart the Content Server service.

Front End Servers TIER TWO

DATABASE EFS INDEX

SEA Servers TIER ONE

Front End Servers TIER TWO

DATABASE EFS INDEX

SEA Servers TIER ONE

Front End Servers TIER TWO

DATABASE EFS INDEX

The production environment is now back online.

OUTAGE WINDOW ENDS

On the Test Environment

On the Front-End Servers

The current state of the Agent deployment is illustrated below:

Next, all non-essential servers in the cluster will be shutdown.

2. Log on as the Admin user.

Restore the Production Database and EFS Snapshot to the

Migrate the Production 10.0.0 Test EFS

SELECT * FROM KINI WHERE IniSection='Livelink.LogicalProviders' AND

UPDATE KINI SET IniValue='{''EXTSHARED'', ''\\\\coutts1\\efs\\''}' WHERE

NOTE: If this were an Oracle database, you would need to

SELECT * FROM KIni WHERE IniSection='Livelink.LogicalProviders' AND

DELETE FROM KIni WHERE IniSection='Livelink' AND

Connect the Database to Content Server 10.0

15. Open Windows Explorer and navigate to the C:\opentext\config\ folder.

NOTE: Do not delete the otadmin.pwd file. This is required in

16. Repeat the above procedure on the secondary Admin server.

Reconcile the Production and Test 10.0 opentext.ini File Settings

30. Click Continue.

Reconfigure the Index Process and Location Pathways

45. Click Admin Home.