Cray Network Subsystem (CNS) Software Installation and Administration

S–2366–16

© 2002–2005 Cray Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Cray Inc. U.S. GOVERNMENT RESTRICTED RIGHTS NOTICE The Computer Software is delivered as "Commercial Computer Software" as defined in DFARS 48 CFR 252.227-7014. All Computer Software and Computer Software Documentation acquired by or for the U.S. Government is provided with Restricted Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7014, as applicable. Technical Data acquired by or for the U.S. Government, if any, is provided with Limited Rights. Use, duplication or disclosure by the U.S. Government is subject to the restrictions described in FAR 48 CFR 52.227-14 or DFARS 48 CFR 252.227-7013, as applicable. Autotasking, Cray, Cray Channels, Cray Y-MP, GigaRing, LibSci, UNICOS and UNICOS/mk are federally registered trademarks and Active Manager, CCI, CCMT, CF77, CF90, CFT, CFT2, CFT77, ConCurrent Maintenance Tools, COS, Cray Ada, Cray Animation Theater, Cray APP, Cray Apprentice2, Cray C++ Compiling System, Cray C90, Cray C90D, Cray CF90, Cray EL, Cray Fortran Compiler, Cray J90, Cray J90se, Cray J916, Cray J932, Cray MTA, Cray MTA-2, Cray MTX, Cray NQS, Cray Research, Cray SeaStar, Cray S-MP, Cray SHMEM, Cray SSD-T90, Cray SuperCluster, Cray SV1, Cray SV1ex, Cray SX-5, Cray SX-6, Cray T3D, Cray T3D MC, Cray T3D MCA, Cray T3D SC, Cray T3E, Cray T90, Cray T916, Cray T932, Cray UNICOS, Cray X1, Cray X1E, Cray XD1, Cray X-MP, Cray XMS, Cray XT3, Cray Y-MP EL, Cray-1, Cray-2, Cray-3, CrayDoc, CrayLink, Cray-MP, CrayPacs, Cray/REELlibrarian, CraySoft, CrayTutor, CRInform, CRI/TurboKiva, CSIM, CVT, Delivering the power..., Dgauss, Docview, EMDS, HEXAR, HSX, IOS, ISP/Superlink, MPP Apprentice, ND Series Network Disk Array, Network Queuing Environment, Network Queuing Tools, OLNET, RapidArray, RQS, SEGLDR, SMARTE, SSD, SUPERLINK, System Maintenance and Remote Testing Environment, Trusted UNICOS, TurboKiva, UNICOS MAX, UNICOS/lc, and UNICOS/mp are trademarks of Cray Inc. Apache is a trademark of The Apache Software Foundation. CERT is a trademark of Carnegie Mellon University. Dell and PowerEdge are trademarks of Dell Computer Corporation. ESS-800 is a trademark of SBS Technologies, Inc. FLEXlm is a trademark of Macrovision Corporation. GNU is a trademark of The Free Software Foundation. Linux is a trademark of Linus Torvalds. Mac OS is a trademark of Apple Computer, Inc. NFS, Solaris and Sun are trademarks of Sun Microsystems, Inc. in the United States and other countries. Red Hat is a trademark of Red Hat, Inc. UNIX is a trademark of The Open Group. All other trademarks are the property of their respective owners. The UNICOS, UNICOS/mk, and UNICOS/mp operating systems are derived from UNIX System V. These operating systems are also based in part on the Fourth Berkeley Software Distribution (BSD) under license from The Regents of the University of California.

Record of Revision
Version
1.0

Description
December 2002 Supports the CNS 1.0 version software for the Cray Network Subsystem (CNS) used with Cray X1 and Cray GigaRing systems. June 2003 Supports the CNS 1.1 release software for the Cray Network Subsystem (CNS) used with Cray X1 and Cray GigaRing systems. December 2003 Supports the CNS 1.2 release software for the Cray Network Subsystem (CNS) used with Cray X1 and Cray GigaRing systems. May 2004 Supports the CNS 1.3 release software for the Cray Network Subsystem (CNS) used with Cray X1 and Cray GigaRing systems. November 2004 Supports the CNS 1.4 release software for the Cray Network Subsystem (CNS) used with Cray X1 series and Cray GigaRing systems. March 2005 Supports the CNS 1.5 release software for the Cray Network Subsystem (CNS) used with Cray X1 series and Cray GigaRing systems. September 2005 Supports the CNS 1.6 release software for the Cray Network Subsystem (CNS) used with Cray X1 series and Cray GigaRing systems.

1.1

1.2

1.3

1.4

1.5

1.6

S–2366–16

i

Contents

Page

Preface
Accessing Product Documentation Conventions . . . . . . . . . . . . Reader Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ix ix x xi 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 5 . . . . . . . . . . . . . . . . . 5 5 5 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 7 7 8 8 8 8 9 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 11 11
iii

Introduction [1]
Emphasis for the CNS 1.6 Release Description of the CNS . . .

Software Enhancements [2]
Upgraded CNS Kernel to the Linux 2.4.21-32.0.1.EL kernel distributed by Red Hat Enhanced CNS configuration to support IPv6 Added Support for the zebra Daemon . . . . . . . . . . . . . . . . . . . . . .

Compatibilities and Differences [3]
Release Upgrade Support . . . . . . . . . Changes to iptables Affect DNS and NTP Servers for the CNS CNS Comparison to the Cray L7R Dell PowerEdge 2850 Serial Console

Connecting To Console From Dell Poweredge 2850 Drac4 cns_gen_config changes . . . . . . . . . . . . .

tip command change for Dell PowerEdge 1650 and Dell PowerEdge 2650 cns_gen_config executed by default

Limitations [4]
cns_gen_config Command Does Not Configure Static Routes cns_config restore Sometimes Does Not Work Properly PCI Slot Ethernet Card Naming Convention
S–2366–16

.

.

.

.

Cray Network Subsystem (CNS) Software Installation and Administration
Page

Side Effects of the tcp_assistd Daemon

.

.

.

.

.

.

. .

. . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

12 13 13 14 14 14 15

Accumulation of tcp_assistd Daemon "Dead" Connections

Upgrade/Install Sometimes Fails After Abnormal CNS Shutdown The tcp_assist Daemon Does Not Support IPv6 Proxy ARP Routing for IPv6 Not Supported . . . . . . . . . .

Initial Installation Process Does Not Clean Up a Temporary Directory

Documentation [5]
CrayDoc Documentation Delivery System Accessing Product Documentation Books Provided with This Release Ordering Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

15 15 16 16 17 19

Related Documentation Not Provided with This Release

CNS Release Package [6]
Hardware and Software Requirements Contents of the Release Package Licensing . . . . . . . . . . . Ordering Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19 21 21 22 25

Customer Services [7]
Technical Assistance with Software Problems CRInform System Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

25 25 26 27 27 29

Cray Service Bulletin (CRSB) Cray Public Website

Installation and Configuration [8]
Preparing for CNS Software Installation, Upgrade, or Configuration Configuring Software for a New CNS Configuring Network Routing
iv

. . . . . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

32 33 33 35
S–2366–16

. . .

. .

. .

. . .

. . .

. . .

. . .

. . .

Generating the Configuration for a New CNS .

Contents
Page

Upgrading Existing CNS Software

.

.

.

. .

. . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

. . . . .

35 39 42 43 43 45

Performing an Initial CNS Software Installation Reverting to the Previous CNS Configuration Supporting UNICOS/mp Installation . . Using the Shell During a CNS Software Installation .

CNS Functional Overview and Administration [9]
Introduction to the CNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Overview of CNS Networking CNS Security . . . . .

45 47 48 49 50 50 50 51 52 52 53 53 54 54 54 55 55 56 58 58 59

Automatic Failover and Fibre Channel IP Bonding Connection Methods for Accessing the CNS CNS Accounts and Passwords CNS root User Account and Password Resetting the CNS root Password RAC root Password . . . . . . . . . . CNS crayadm User and Password CNS Private Ethernet Firewall CNS Software Security Features CNS Message Logging . . .

Remote Access Controller (RAC) Maintenance Port Function cns_rac Command Functions Logging In to the RAC Maintenance Port The connect Command The exit Command . . . . . . . . .

Using the CNS RAC Maintenance Port Functions The serveraction Command

Advanced Configuration Topics [10]
NTP for the CNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Setting Up the Local Time Zone of the CNS
S–2366–16

59 59
v

Cray Network Subsystem (CNS) Software Installation and Administration
Page

CNS Routing Configuration

.

.

.

.

.

.

. .

. .

. .

. .

. .

. .

. . .

. . .

. . .

. . .

. . .

. . .

. . . .

. . . . .

. . . . .

60 60 61 61 62 63 63 64 64 65 66 66 69 69 71 72 73 73 73 74 74 75 76 76 76 79 79 79 79 80
S–2366–16

Configuring a Default Static Route on the CNS Example 1: Example 2:

Using gated to Configure a Default Dynamic Route for the CNS

/etc/gated.conf Default Configuration File for a Fibre Channel Network /etc/gated.conf Default Configuration File for a HIPPI Network . . . .

Using zebra to Configure a Default Dynamic Route for the CNS with IPv6

Example 3: /etc/zebra/zebra.conf and /etc/zebra/ripngd.conf are the Default IPv6 Configuration Files for a Fibre Channel Network . . . . . . . . . . . . Configuring Static Routing to the CNS from the Cray Mainframe Example 4: Example 5: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sample static-route.options File for a Cray X1 series Mainframe Sample gated.conf File for a Cray GigaRing Mainframe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Configuring Site Network Routers

Configuring a Proxy Address Resolution Protocol Interface Configuration Using cns_gen_config Example 6:

Interface Configuration Output of cns_gen_config

Configuring Bonded Interfaces for Automatic Failover Example 7: Example 8: Example 9: Example 10: Example 11: Configuring Ethernet Interfaces . . . . . .

cns_gen_config Output When Configuring a Bonded Interface

cns_gen_config Output When Configuring a CNS Interface for a FSS cns_gen_config Output When Configuring CNS Ethernet Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . cns_gen_config Output When Configuring a Fibre Channel Interface cns_gen_config Output When Configuring a HIPPI Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Configuring Fibre Channel or HIPPI Interfaces

Configuring CNS Interfaces That are not Cabled to the CNS Specifying the CNS number Installing the Configuration Initial Installation Configuration Prompts

Backing Up and Restoring the CNS Configuration Backing Up the CNS Configuration Restoring the CNS Configuration

Remote Access Controller (RAC) Maintenance Port Setup and Configuration
vi

Contents
Page

Setting Up the CNS RAC Maintenance Port Configuring the CNS RAC Maintenance Port Example 12:

.

. .

. .

. .

. . .

. . .

. . .

. . .

. . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

80 82 85 86 89

Configuring Private Ethernet for Cray GigaRing Systems

Setting Up the SWS to Act as a bootp Server for the CNS

CNS Troubleshooting [11]
What to Do When the CNS is Unable to Boot Accessing the CNS For Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

91 92 93 93 94 95 95 97 98 99

Troubleshooting CNS Access to Cray Mainframes Troubleshooting CNS Fibre Channel Connections

Example 13: Fibre Channel Link Messages in /var/log/messages on the Cray X1 Series System . . . . . . . . . . . . . . . . . . . . . . . . . . Troubleshooting CNS HIPPI Connections Troubleshooting CNS Access to Site Networks Controlling the tcp_assistd Daemon . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Troubleshooting Remote Host Access to and from Cray Mainframes

Appendix A CNS Hardware Connections
CNS (Dell 1650) CNS (Dell 2650) CNS (Dell 2850) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99 101 102 105

Appendix B HIPPI Network Configuration
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Instructions for Configuring the CNS HIPPI Interface Verification . . . . . . . . . . . .

105 105 108 108 109

Configuration Requirements for the Cray Mainframe for GigaRing Systems

Index Figures
Figure 1. Figure 2.
S–2366–16

CNS Function

.

.

.

.

.

.

.

.

. .

. .

. .

. .

. .

. .

. .

. .

. .

. .

. .

. .

. .

. .

2 30
vii

CNS Network Configuration Worksheet

Cray Network Subsystem (CNS) Software Installation and Administration
Page

Figure 3. Figure 4. Figure 5. Figure 6. Figure 7. Figure 8. Figure 9. Figure 10. Figure 11. Figure 12. Figure 13. Figure 14. Figure 15. Figure 16. Figure 17. Figure 18. Figure 19. Figure 20.

Mainframe Throughput Increased by Large MTU CNS in a Cray X1 Series System Environment CNS in a GigaRing System Environment Fibre Channel IP Bonded Interface . . . . . . . . . . . . . . . . .

. . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . . .

. . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . .

46 47 48 49 68 83 85 90 91 93 96 97 99 100 101 102 103 103

Example of CNS Network Proxy ARP Configuration Connecting to the RAC from the CWS Connecting to the RAC from the SWS

The CNS with Ethernet Connections in a Cray X1 Series Environment CNS Access to the Cray Mainframe CNS Access to Hosts . . . . . . . . . . . . . . . . . . . . . . Routing to Utilize the CNS CNS Front View (Dell 1650) CNS Rear View (Dell 1650) CNS Front View (Dell 2650) CNS Rear View (Dell 2650) CNS Front View (Dell 2850) CNS Rear View (Dell 2850) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The CNS with HIPPI Connections in a Cray GigaRing System Environment . . . . . . . . .

Tables
Table 1. Table 2. Table 3. CNS Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 16 31 Books Provided with This Release

Legend for CNS Network Configuration

viii

S–2366–16

Preface

The information in this preface is common to Cray documentation provided with this software release.

Accessing Product Documentation
With each software release, Cray provides books and man pages, and in some cases, third-party documentation. These documents are provided in the following ways: CrayDoc The Cray documentation delivery system that allows you to quickly access and search Cray books, man pages, and in some cases, third-party documentation. Access this HTML and PDF documentation via CrayDoc at the following locations: • The local network location defined by your system administrator • The CrayDoc public website: docs.cray.com Man pages Access man pages by entering the man command followed by the name of the man page. For more information about man pages, see the man(1) man page by entering:
% man man

Third-party documentation Access third-party documentation not provided through CrayDoc according to the information provided with the product.

S–2366–16

ix

Cray Network Subsystem (CNS) Software Installation and Administration

Conventions
These conventions are used throughout Cray documentation: Convention command Meaning This fixed-space font denotes literal items, such as file names, pathnames, man page names, command names, and programming language elements. Italic typeface indicates an element that you will replace with a specific value. For instance, you may replace filename with the name datafile in your program. It also denotes a word or concept being defined. This bold, fixed-space font denotes literal items that the user enters in interactive sessions. Output is shown in nonbold, fixed-space font. Brackets enclose optional portions of a syntax representation for a command, library routine, system call, and so on. Ellipses indicate that a preceding element can be repeated. Denotes man pages that provide system and programming reference information. Each man page is referred to by its name followed by a section number in parentheses. Enter:
% man man

variable

user input

[] ... name(N)

to see the meaning of each section number for your particular system.

x

S–2366–16

Preface

Reader Comments
Contact us with any comments that will help us to improve the accuracy and usability of this document. Be sure to include the title and number of the document with your comments. We value your comments and will respond to them promptly. Contact us in any of the following ways: E-mail: docs@cray.com Telephone (inside U.S., Canada): 1–800–950–2729 (Cray Customer Support Center) Telephone (outside U.S., Canada): +1–715–726–4993 (Cray Customer Support Center) Mail: Software Publications Cray Inc. 1340 Mendota Heights Road Mendota Heights, MN 55120–1128 USA

S–2366–16

xi

Introduction [1]

This document provides an overview of the Cray Network Subsystem (CNS) release 1.6 used with Cray X1 series or Cray GigaRing systems. This information is intended for system administrators receiving their first release of this product or upgrading from a previous release and assumes the administrator has a good understanding of Cray system administration. This publication does not describe hardware, software, or installation of related products, such as the Cray mainframe, Cray Workstation (CWS), Cray system workstation (SWS), or products not provided through Cray. This information is intended for system or network administrators and software support personnel. It assumes the user has a good understanding of Ethernet, Transmission Control Protocol/Internet Protocol (TCP/IP) networking, High Performance Parallel Interface (HIPPI) networks, IP over Fibre Channel, and Cray system administration. Refer to the CNS errata, if one accompanies the release, Cray Field Notices (FNs), if any, or contact your Cray representative for the latest updates or corrections to the product or documentation. Note: If you are familiar with the CNS and the release, you can go directly to Chapter 8, page 29 for software installation and configuration instructions. Cray continues to improve the CNS. The CNS is an important component of the Cray high performance network strategy, and network performance is a very high priority for Cray. You can expect the CNS to continue to evolve.

1.1 Emphasis for the CNS 1.6 Release
The key reasons for this release are: • Upgrading the CNS kernel to the Linux 2.4.21-32.0.1.EL kernel distributed by Red Hat to support the latest security updates • Enhancing CNS configuration for the support of IPv6 • Adding the zebra daemon to support dynamic routing for IPv6

S–2366–16

1

Cray Network Subsystem (CNS) Software Installation and Administration

1.2 Description of the CNS
The CNS contains hardware and software that functions as a router, passing all packet traffic between site networks and the Cray mainframe. The key feature of the CNS is its ability, for most TCP/IP connections, to act as a specialized gateway to handle the smaller packet sizes used on the site network and the larger packet sizes used by the Cray mainframe. This improves network performance between the Cray mainframe and site networks. The CNS is compatible with the following systems and the operating systems that run on them: • Cray X1 series systems with a Fibre Channel host network connection • Cray GigaRing systems with a HIPPI host network connection Note: The CNS HIPPI host network interface is not supported for Cray X1 series systems. Note: Cray does not support sharing a CNS between Cray mainframes or partitions.

Cray mainframe Communicates with Cray mainframe using a larger packet size or maximum transmission unit (MTU) over Fibre Channel or HIPPI CNS Communicates on behalf of the Cray mainframe to other hosts using a smaller network packet size or MTU Site network

host 1

host 2

Figure 1. CNS Function
2 S–2366–16

Introduction [1]

For more detail about CNS functions, see Chapter 9, page 45. CNS Terminology: Table 1 shows the terminology used to describe the CNS.

Table 1. CNS Terminology Term CNS Meaning Refers to the entire product, that is, CNS hardware device (platform), including the host bus adapter (HBA) and network interface cards (NICs), and its software and firmware. Refers to the system (hardware device) on which the CNS software runs; for example, the Dell server. Refers to the complete collection of software that runs on the CNS platform, including Linux and Cray provided software. Refers to the Remote Access Controller maintenance port facility used mainly for remote console access to the CNS.

CNS platform CNS software

RAC

S–2366–16

3

Cray Network Subsystem (CNS) Software Installation and Administration

4

S–2366–16

Software Enhancements [2]

The following software enhancements have been made since the last release of this CNS software. Note: Documentation changes since the last release of this software are described in Chapter 5, page 15.

2.1 Upgraded CNS Kernel to the Linux 2.4.21-32.0.1.EL kernel distributed by Red Hat
The CNS kernel has been upgraded to Red Hat kernel version 2.4.21-32.0.1.EL. This kernel version contains the latest security updates for the Red Hat Enterprise 3.0 release.

2.2 Enhanced CNS configuration to support IPv6
All software packages for the CNS have been upgraded to provide support for Internet Protocol Version 6 (IPv6). IPv6 fixes a number of problems in the current version Internet Protocol, IP Version 4 (IPv4), such as the limited number of available IPv4 addresses. IPv6 also adds many improvements to IPv4 in areas such as routing and network autoconfiguration. IPv6 is expected to gradually replace IPv4, with the two coexisting for a number of years during a transition period. The cns_gen_config, cns_config, and cns_install scripts have been upgraded to support the Router Advertisement Daemon (radvd). The radvd daemon is used configure the Cray X1 mainframe IPv6 address. The radvd daemon will advertise an IPv6 prefix to the Cray X1 series system. The IPv6 prefix and radvd daemon are configured from the cns_gen_config script. Note: The IPv6 feature is supported only on Cray X1 series systems. IPv6 is not supported on Cray gigaring systems.

2.3 Added Support for the zebra Daemon
The zebra daemon has been added in the CNS 1.6 release to support dynamic routing for IPv6.

S–2366–16

5

Cray Network Subsystem (CNS) Software Installation and Administration

6

S–2366–16

Compatibilities and Differences [3]

This chapter describes compatibility issues and functionality changes to be aware of when using this software after upgrading from the previous release of this software.

3.1 Release Upgrade Support
Software release upgrades preserve much of the previous configuration information on the CNS. Cray has tested and supports upgrades to the CNS 1.6 release from the CNS 1.4 and 1.5 releases.

3.2 Changes to iptables Affect DNS and NTP Servers for the CNS
Changes made to iptables prevent any connection to the CNS other than from the Private Ethernet. Connections from site networks are forwarded to the Cray mainframe; no other connections are allowed to the CNS from any other network. The implication for sites using an external Domain Name System (DNS) and Network Time Protocol (NTP) server for the CNS is that DNS and NTP will not work if you try to configure them to access the site network instead of the CWS; only gated can send a packet directly to the CNS. Sites using ntpd and DNS services from a system on an external network will have to make appropriate adjustments to their /etc/resolv.conf and /etc/ntp.conf files after upgrading to the CNS 1.3 release or later.

3.3 CNS Comparison to the Cray L7R
For those customers who are familiar with the Cray L7R, the CNS is a follow-on to that product. This section describes the compatibilities and differences between the Cray L7R and the CNS. The CNS is similar to Cray L7R but with numerous improvements. The CNS and Cray L7R are functionally almost identical. Both are designed to function as a gateway router for Cray systems and to improve network performance. The differences between them are in the details of the implementation: • The CNS uses an internal SCSI disk rather than an NFS-mounted root file
S–2366–16 7

Cray Network Subsystem (CNS) Software Installation and Administration

system, which was used on the Cray L7R. This improves the administrative flexibility and independence of the CNS. • The CNS is based on Dell x86 servers rather than the Alpha based hardware used with the Cray L7R. • For the CNS, the internal daemon names and administrative details have been changed to make them more reflective of their actual functions and purposes. More scripts and standard Linux mechanisms have been adopted for maintenance and control.

3.4 Dell PowerEdge 2850 Serial Console
The Dell 2850 PowerEdge has a new version of Remote Access Controller (RAC). This new version RAC called the DRAC4 uses the serial interface for its console connection for the DRAC. The BIOS on the Dell PowerEdge 2850 has been changed to allocate the serial interface to the DRAC. Because of this change, the external serial interface to the Dell 2850 PowerEdge is inoperable. In addition, the tip command from the CWS or Cray system workstation (SWS) will not work with a Dell PowerEdge 2850.

3.5 Connecting To Console From Dell Poweredge 2850 Drac4
The command to connect to the console from the RAC on the Dell PowerEdge 2850 has changed. When you telnet into the RAC on a 2850, the command to connect to the CNS console is connect com1.

3.6 tip command change for Dell PowerEdge 1650 and Dell PowerEdge 2650
The command to connect to the serial console on the CNS that use the Dell PowerEdge 2650 and Dell PowerEdge 1650 has changed. The command is now tip -57600 /dev/ttya or tip -57600 /dev/ttyb.

3.7 cns_gen_config changes
The cns_gen_config script has changed include the following options: • Ability to enable IPv6 on the CNS • When IPv6 is enabled, prompts for IP version 6 addresses
8 S–2366–16

Compatibilities and Differences [3]

• Ability to configure IPv4 default gateway • Ability to configure IPv6 default gateway • Ability to configure the radvd daemon • Ability to run chkconfig on the zebra daemon

3.8 cns_gen_config executed by default
The cns_gen_config script is executed by default on every upgrade to CNS 1.6 release. This change is due to the addition of the IPv6 feature.

S–2366–16

9

Cray Network Subsystem (CNS) Software Installation and Administration

10

S–2366–16

Limitations [4]

This chapter describes significant limitations with this CNS software release.

4.1 cns_gen_config Command Does Not Configure Static Routes
The cns_gen_config command does not have the ability to generate static routes, so the prompts for this do not occur. Until cns_gen_config is capable of generating static routes, you must edit the /cray/etc/cns.conf file to set the CNS default route for site networks. For more information, refer to Section 10.3.1, page 60.

4.2 cns_config restore Sometimes Does Not Work Properly
The cns_config command can be used to back up or restore a CNS configuration. Under some circumstances, after backing up a CNS configuration and then restoring that configuration to a CNS running CNS 1.2 or later release software, the configuration might be restored incorrectly. This problem is caused by incorrect handling of the /etc/sysconfig/network-script/ifcfg-bond0 configuration file. The following command restores a CNS configuration from the OLDCNS file: Note: This example, uses cns0; the CNS number you are working on may differ.
cns0# cns_config restore OLDCNS

To ensure correct configuration, you must always run the following two commands after running the cns_config restore command:
cns0# cns_gen_config (answer the questions posed and use defaults as appropriate) cns0# cns_config install

4.3 PCI Slot Ethernet Card Naming Convention
Cray does not support or recommend adding or removing PCI cards from the CNS.
S–2366–16 11

Cray Network Subsystem (CNS) Software Installation and Administration

The CNS software attempts to ensure that the network interface names are predictable. However, the interfaces on PCI cards are determined by the order in which they are discovered by the Linux operating system. If the number of Gigabit Ethernet interfaces is changed or the PCI cards are moved to different PCI slots, the interfaces do not necessarily retain the same interface names. This is important to remember when adding or removing network interface cards. The two built-in Gigabit Ethernet interfaces on the motherboard are always named eth0 and eth1 (Section A.1, page 99 and Section A.2, page 101). For example, the PCI slots of the Dell 2650 CNS platform are numbered 1, 2, and 3, respectively. Slot 1 is closest to the bottom of the CNS and slot 3 is closet to the top of the CNS. If Gigabit Ethernet network interface cards are installed in the slots, they will be named in this same order. If the first Gigabit Ethernet interface is added to slot 2, it will be the first interface found and will be named eth2. If only a single Gigabit Ethernet interface is installed in slot 2, that interface will be named eth2. If another Gigabit Ethernet interface is then added to slot 1, the Linux operating system determines that the interface in slot 1 is named eth2, and the interface in slot 2 is named eth3.

4.4 Side Effects of the tcp_assistd Daemon
In the special case where a TCP connection is opened between a Cray X1 series mainframe and a site endpoint, the CNS handles this connection via the tcp_assistd daemon. The tcp_assistd daemon, a proxy invoked with the tcp_assist command, allows the Cray X1 series system side of the connection to maintain large MTU size (packet size) while using smaller packet sizes on the site side. In almost all respects, the behavior of the CNS and tcp_assist is transparent to the user. However, for those connections that are handled by the tcp_assistd daemon, there are some side effects for the system administrator to consider: 1. Certain obscure socket semantics are not preserved by tcp_assistd. Occasionally a specialized networking application behaves differently when the CNS is involved. KEEPALIVE and SO_LINGER are two known examples where the proxy does not preserve the requested semantics. Cray has verified that these differences do not cause problems for the common networking applications of X11, ftp, telnet, rsh, ssh, and many others. If you believe that your specialized, custom networking application is being adversely affected by the CNS and its tcp_assistd daemon, please contact the Cray Customer Support Center (see Section 7.1, page 25). Have information
12 S–2366–16

Limitations [4]

available about the specific network application, protocol options, and packet traces (tcpdump or equivalent) to facilitate analysis of the problem. 2. The tcp_assistd daemon performs an active proxy function and handles the setup and tear-down of the incoming and outgoing connections to the Cray X1 series system. As a result, port scans of the Cray X1 series system from the site network, or of the site network from the Cray X1 series system, will appear as if all of the ports are open even though they are not usable. Ports appearing to be open might be a concern to sites that depend on port scans to ensure security. Workaround: If you do not need the improved performance of TCP/IP provided by the TCP-assist function, the system administrator can remove both of these limitations by disabling the tcp_assistd daemon. However, this changes the CNS into a simple router, which can severely degrade TCP/IP performance. See Section 11.6, page 98 for information about disabling the tcp_assistd daemon.

4.5 Accumulation of tcp_assistd Daemon "Dead" Connections
In the CNS 1.1 release, a change was made to disable a timer in the tcp_assistd daemon that disconnected idle socket connections after a certain interval. If an endpoint of a connection that is being handled by tcp_assistd stops responding, the tcp_assistd daemon is not always notified and the process handling the connection sometimes does not terminate. Over time, these "dead" connections can accumulate. This can happen when a workstation communicating with the Cray X1 series system through tcp_assistd crashes. This is usually not of great concern, unless a large number of these "dead" connections accumulate. If in the highly unlikely event that too many of these "dead" connections accumulate, to clean out the old processes and prevent new ones from persisting, you can re-enable the idle timer by editing the script in /cray/bin/tcp_assist. Contact your Cray representative for detailed instructions if you must do this.

4.6 Upgrade/Install Sometimes Fails After Abnormal CNS Shutdown
When the CNS does not shut down cleanly, such as after a power failure, and the
S–2366–16 13

Cray Network Subsystem (CNS) Software Installation and Administration

CNS is then booted with the CD supplied for an upgrade, the upgrade scripts sometimes report that the hard disk needs to be reformatted when in fact it does not. This occurs because fsck operation fail to run on the partitions before mounting them. Workaround: The workaround for this problem is to reboot the CNS from the hard disk, and shut it down cleanly (that is, a normal shutdown) before rebooting from the CD.

4.7 Initial Installation Process Does Not Clean Up a Temporary Directory
An initial installation creates a /tmp/INSTALL directory as a "scratch" area and does not remove it. This makes it look like the directory that was created manually on a previous installation and is still there. In fact, this is not true.

4.8 The tcp_assist Daemon Does Not Support IPv6
The tcp_assist daemon relies on the Network Address Translation (NAT) feature of the Linux kernel. The Linux kernel does support the NAT feature for IPv6.

4.9 Proxy ARP Routing for IPv6 Not Supported
The CNS does not support Proxy ARP routing for IPv6.

14

S–2366–16

Documentation [5]

This chapter describes the documentation that supports the CNS 1.6 release.

5.1 CrayDoc Documentation Delivery System
The CrayDoc documentation delivery system, along with product documentation, is provided with each Cray software release. The CrayDoc software runs on any operating system based on UNIX systems or systems like UNIX including Mac OS X, Linux, BSD, and anywhere else that Perl and Apache can be compiled from source code with freely available (GNU) tools. The installation and administration of the CrayDoc server software and Cray documentation are described in CrayDoc Installation and Administration Guide.

5.2 Accessing Product Documentation
With each software release, Cray provides books and man pages, and in some cases, third-party documentation. These documents are provided in the following ways: CrayDoc The Cray documentation delivery system that allows you to quickly access and search Cray books, man pages, and in some cases, third-party documentation. Access this HTML and PDF documentation via CrayDoc at the following locations: • The local network location defined by your system administrator • The CrayDoc public website: docs.cray.com Man pages Access man pages by entering the man command followed by the name of the man page. For more information about man pages, see the man(1) man page by entering:
% man man

Third-party documentation Access third-party documentation not provided through CrayDoc according to the information provided with the product.

S–2366–16

15

Cray Network Subsystem (CNS) Software Installation and Administration

5.3 Books Provided with This Release
The books provided with this release are listed in Table 2, which also notes whether each book was updated and whether it is also provided in hard copy. Most books are provided in HTML and all are provided in PDF. Note: If an errata for the release is required, it includes changes identified after the documentation for this release was packaged. If present, the errata is provided in printed hardcopy format in the release package. Also, contact your Cray representative for other possible late problems published in Field Notices (FNs).

Table 2. Books Provided with This Release Book Title Cray Network Subsystem (CNS) Software Installation and Administration 1 (this manual) Common Installation Tool (CIT) Reference Card CrayDoc Installation and Administration Guide Number S–2366–16 S–2218–20 S–2340–40 Updated Printed Yes No No Yes No No

5.4 Related Documentation Not Provided with This Release
The following documents contain additional information that may be helpful in setting up your CNS environment; they are not provided with this release but are supplied with other products purchased from Cray. They can be ordered on a CrayDoc CD from the Cray Software Distribution Center (see Section 5.5, page 17). Release overviews and installation guides can be ordered in printed format. These Cray documents are related to the CNS used with Cray X1 series systems: • Cray Workstation (CWS) Release Overview • Cray Workstation (CWS) Installation Guide • Cray X1 Series System Configuration and CWS Administration • Cray X1 Series System Overview • UNICOS/mp System Administration
1 16

This document is also provided in printed form.
S–2366–16

Documentation [5]

• UNICOS/mp Networking Facilities Administration Note: The UNICOS/mp Fibre Channel IP bonding driver is included with the UNICOS/mp release package. The UNICOS/mp bfc command is used to manage bonded Fibre Channel interfaces. The UNICOS/mp Fibre Channel IP bonding driver is documented in the UNICOS/mp bfc(8) and bondedfibrechannel(7) man pages and in UNICOS/mp Networking Facilities Administration. These Cray documents are related to the CNS used with Cray GigaRing systems: • SWS-ION Release Overview • SWS-ION Administration and Operations Guide • Cray Scalable I/O Functional Overview • UNICOS Networking Facilities Administrator's Guide • UNICOS/mk Networking Facilities Administration

5.5 Ordering Documentation
To order Cray software documentation, contact your Cray representative or contact the Cray Software Distribution Center in any of the following ways: E-mail: orderdsk@cray.com Telephone (inside U.S., Canada): 1–800–284–2729 (BUG CRAY), then 605–9100 Telephone (outside U.S., Canada): +1–651–605–9100 Fax: +1–651–605–9001 Mail: Software Distribution Center Cray Inc. 1340 Mendota Heights Road Mendota Heights, MN 55120–1128 USA
S–2366–16 17

Cray Network Subsystem (CNS) Software Installation and Administration

18

S–2366–16

CNS Release Package [6]

This chapter contains the following information about the CNS 1.6 release: • Hardware and software requirements • Contents of the release package • Licensing • Ordering software

6.1 Hardware and Software Requirements
Mainframe Software: A CNS can be used with Cray systems running any actively supported release of the UNICOS/mp, UNICOS, and UNICOS/mk operating systems (with the required network interfaces, such as Gigabit Ethernet (fiber-optic cable), Gigabit Ethernet (copper electrical cable), or HIPPI). Note: Although the CNS supports Fibre Channel IP bonding for failover, the UNICOS/mp Fibre Channel IP bonding driver is included with the UNICOS/mp release package. General availability of this feature occurred with the UNICOS/mp 2.4 release. Note: CNS version 1.6 supports IP version 6. To connect from CNS to the Cray X1 series mainframe using IP version 6, UNICOS/mp version 3.0.34 or later is required. Host Interface and Cray Mainframe: The Cray mainframe type dictates the type of supported host interface: HIPPI for GigaRing systems and Fibre Channel for Cray X1 series systems. CNS Network Interface Hardware: This CNS software release supports the following network interface hardware for the two current CNS platforms available: Note: Cray may choose to provide hardware from other vendors for reasons of performance, availability, and vendor support capabilities. • Dell 1650 PowerEdge server

S–2366–16

19

Cray Network Subsystem (CNS) Software Installation and Administration

This device includes two 10/100/1000Base-T copper onboard Ethernet interfaces. One is used for the private Ethernet network. There is also a server management (Ethernet) port. This device can also support up to two PCI interface cards: – One HIPPI network interface card (NIC) for connection to the Cray mainframe HIPPI network or the site HIPPI network – One Gigabit Ethernet NIC, either fiber-optic or copper electrical cable • Dell 2650 PowerEdge server This device includes two 10/100/1000Base-T copper onboard Ethernet interfaces. One is used for the private Ethernet network; the other is reserved for file system server (FSS) access. There is also a server management (Ethernet) port. This device can also support up to three PCI interface cards: – One HIPPI NIC for connection to the Cray mainframe HIPPI network or the site HIPPI network or one single or dual Fibre Channel host bus adapters (HBA) connection to the Cray X1 series mainframe – One Gigabit Ethernet NIC, either fiber-optic or copper electrical cable • Dell 2850 PowerEdge server This device includes two 10/100/1000Base-T copper onboard Ethernet interfaces. One is used for the private Ethernet network; the other is reserved for file system server (FSS) access. There is also a server management (Ethernet) port. This device can also support up to three PCI interface cards: – One HIPPI NIC for connection to the Cray mainframe HIPPI network or the site HIPPI network or one single or dual Fibre Channel host bus adapters (HBA) connection to the Cray X1 series mainframe – One Gigabit Ethernet NIC, either fiber-optic or copper electrical cable – One 10 Gigabit Ethernet NIC, fiber-optic. Note: 10.Gigabit Ethenet is supported only on Cray X1 series systems. 10 Gigabit Ethernet is not supported on Cray Gigaring systems. The CNS requires the following networking components supplied by Cray depending on system type:
20 S–2366–16

CNS Release Package [6]

• For Cray X1 series systems: One or more Fibre Channel HBA and cables to the Cray X1 series system. • For Cray GigaRing systems: – Cray HIPPI node (HPN-1 or HPN-2), node subrack (NSR-1), and peripheral cabinet (PC-10) – ESS-800 HIPPI switch or modem RAC Maintenance Port Ethernet Connection: A Cray supplied Ethernet cable (Field Change Order (FCO) 4043) is required to access the RAC from the CWS or SWS.

6.2 Contents of the Release Package
The release package includes: • CNS software, which consists of: – Cray specific software and drivers – Dell Remote Access Controller firmware version 3.12 or 1.10. – Portions of Red Hat Enterprise Desktop 3.0 with the Linux 2.4.21-32.0.1.EL kernel distributed by Red Hat Note: If you receive a new CNS, the CNS software (including firmware) for it has already been installed on the CNS. A CNS software CD is included, in case you should ever need it. • CrayDoc software suite and the documentation, described in Section 5.1, page 15.

6.3 Licensing
The CNS is covered under a software license agreement for Cray software. Upgrades to this product are provided only when a software support agreement for this Cray software is in place.

S–2366–16

21

Cray Network Subsystem (CNS) Software Installation and Administration

CNS licensing depends on the type of mainframe system used with your CNS: Cray X1 series systems A CNS is required for all Cray X1 series systems. Thus, contractual rights to receive CNS software are included in the UNICOS/mp license agreement for Cray X1 series systems. To receive upgrades to your CNS software, you must have your UNICOS/mp license and a signed support agreement for the UNICOS/mp software. Cray GigaRing systems Cray supports the CNS for use with Cray GigaRing systems through a HIPPI interface. The CNS software for Cray GigaRing systems is covered by a separate CNS software license, which is not dependent on the license for the operating system of your Cray GigaRing system. Cray ships the CNS for Cray GigaRing systems with the CNS software already installed. This software includes a license agreement that need not be signed by customers; customers are bound by the license terms and conditions by virtue of using the software product or any documentation. To receive upgrades to your CNS software for a CNS used with a Cray GigaRing system, you must also have a signed support agreement for your CNS. This product does not require FLEXlm licensing. Customers outside the United States and Canada must sign a Letter of Assurance before software can be shipped to them. For questions about whether you have signed this agreement, or questions about which software requires this letter, send e-mail to crayinfo@cray.com.

6.4 Ordering Software
This release package is distributed by order only to customers who have signed a license agreement for the Cray software that includes this product. The most current revision of the release package is supplied. To receive any upgrades to a given Cray product, the customer must also have a signed support agreement for this Cray software.

22

S–2366–16

CNS Release Package [6]

You can order the release package from the Cray Software Distribution Center in any of the following ways: E-mail: orderdsk@cray.com CRInform (for subscribers): crinform.cray.com Click on the Order Cray Software link. Telephone (inside U.S., Canada): 1–800–284–2729 (BUG CRAY), then 605–9100 Telephone (outside U.S., Canada): +1–651–605–9100 Fax: +1–651–605–9001 Mail: Software Distribution Center Cray Inc. 1340 Mendota Heights Road Mendota Heights, MN 55120–1128 USA Software will be shipped by ground service or 5-day international service.

S–2366–16

23

Cray Network Subsystem (CNS) Software Installation and Administration

24

S–2366–16

Customer Services [7]

This chapter describes the customer services that support this release.

7.1 Technical Assistance with Software Problems
If you experience problems with Cray software, contact your Cray service representative. Your service representative will work with you to resolve the problem. If you choose to have full- or part-time support on site, your on-site personnel are your primary contacts for service. If you have elected not to have on-site support, please call or send e-mail to the Cray Customer Support Center: E-mail: support@cray.com Telephone (inside U.S., Canada): 1–800–950–2729 (CRAY) Telephone (outside U.S., Canada): +1–715–726–4993 CRInform (for subscribers): crinform.cray.com You can also create a Request for Technical Assistance (RTA) and track and search RTAs and Software Problem Reports (SPRs) online if you are a CRInform subscriber, as described in Section 7.2.

7.2 CRInform System
The CRInform system is the information and problem-reporting system for Cray customers who are CRInform subscribers. You are a CRInform subscriber if your site has a software license agreement and software support agreement. Access CRInform at: crinform.cray.com Ask your system administrator for your password. Some of the things a subscriber to CRInform can do include: • Report software problems (SPRs)
S–2366–16 25

Cray Network Subsystem (CNS) Software Installation and Administration

• Request technical assistance (RTAs) • Communicate with other Cray system users • Read about software problems reported at other sites • Learn about solutions to various problems • Order Cray software • View Cray Service Bulletin The CRInform program automatically logs events pertinent to your Cray system site as news items, so you do not have to search through the system for new information. The logged events include Software Problem Report (SPR) or Request for Technical Assistance (RTA) activity, new orderable software, new issues of the Cray Service Bulletin, new field notices (FNs), new software release documents, new software problem fix information, new marketing information, and new CRInform program information. You can also get automatic e-mail notification of any or all of the news items.

7.3 Training
To find out more about Cray training, contact your Cray representative or contact us in any of the following ways: E-mail: wwwtng@cray.com Web: www.cray.com/training/ Fax: +1–715–726–4991 Mail: Technical Training Cray Inc. P.O. Box 6000 Chippewa Falls, WI 54729–0080 USA

26

S–2366–16

Customer Services [7]

7.4 Cray Service Bulletin (CRSB)
The CRInform site provides access to the online Cray Service Bulletin, also called CRSB, which contains descriptions of software problems, information about service procedures or agreements, and announcements of product upgrades and future products for the private use of Cray customers.

7.5 Cray Public Website
The Cray public website offers information about a variety of topics and is located at: www.cray.com

S–2366–16

27

Cray Network Subsystem (CNS) Software Installation and Administration

28

S–2366–16

Installation and Configuration [8]

The procedures in this chapter describe what you need to do to get your Cray Network Subsystem (CNS) running with the CNS 1.6 software. Perform only those procedures that apply to the situation at your site: • Preparing (Section 8.1, page 32) • Configuring Software for a New CNS (Section 8.2, page 33) • Upgrading Existing CNS Software (Section 8.3, page 35) • Performing an Initial CNS Software Installation (Section 8.4, page 39) • Reverting to the Previous CNS Configuration (Section 8.5, page 42) • Using the Shell During a CNS Software Installation (Section 8.6, page 43) • Supporting UNICOS/mp Installation (Section 8.7, page 43) Note: These procedures assume that all required electrical and communication lines are properly connected (see Appendix A, page 99 for the CNS hardware connections).

!

Caution: Cray recommends that you change any default passwords supplied by Cray and use complex passwords for root and crayadm as soon as possible. For advanced and special configuration topics, refer to Chapter 10, page 59. Use Figure 2, page 30 as a worksheet for your CNS network configuration during the CNS software installation or upgrade process.

S–2366–16

29

Cray Network Subsystem (CNS) Software Installation and Administration

Fibre Channel or HIPPI connection to Cray mainframe

IPv4 address/Netmask IPv6 address/Subnet

A
Fibre Channel: /etc/config/netif.options /etc/config/ifconfig_#.options HIPPI: /etc/config/interface

Fibre Channel or HIPPI stand-alone subnetwork
CNS Fibre Channel or HIPPI

IPv4 address/Netmask IPv6 address/Subnet

SWS Usually 10.0.124.200 CWS 10.0.104.1

B
CNS Firewall

Private Ethernet
CWS CWS/SWS

CNS

Gigabit Ethernet

10.0.109.N +1

cnsN N=
(N=CNS number) SWS

IPv4 address/Netmask IPv6 address/Subnet

C
Gigabit Ethernet
Gateway router

Usually 10.1.124.N+200
(N determined from bootp)

IP address

Routing Policy Changes

D E

Gateway Router

Site Network

Figure 2. CNS Network Configuration Worksheet
30 S–2366–16

Installation and Configuration [8]

Table 3. Legend for CNS Network Configuration Key A and B Description A and B are the IPv4/IPv6 addresses on the unique subnetwork established for communication between the CNS and the Cray mainframe. A is the externally visible address of the Cray mainframe. Normally sites configure this subnetwork with only four IPv4 addresses. Cray recommends that this be a subnetwork of the Gigabit Ethernet subnetwork that is directly connected to the CNS. The Cray X1 series mainframe uses router solicitation to configure its IPv6 address. Router solicitation packets are sent over the fibre channel network to the CNS and received by the radvd daemon. The radvd daemon responds to the Cray X1 series mainframe with its IPv6 prefix. The Cray X1 series mainframe configures the fibre channel interface with an IPv6 address using prefix received from the CNS. IPv4/IPv6 addresses are configured on the CNS using the cns_gen_config command. C IPv4/IPv6 address assigned from the site Gigabit Ethernet network. Note: This is not the Cray mainframe IP address. D E The IPv4/IPv6 address of the default route for the CNS. This is the IP address of the gateway router on the Gigabit Ethernet network. Routing policy changes. The site administrators decide how the site routers recognize the path to the Cray mainframe. Cray provides two routing methods, static and dynamic. For static routing, on the site routers you must manually assign a static entry that designates the CNS as the path to the Cray mainframe Fibre Channel or HIPPI network for both IPv4 and IPv6. For dynamic routing, configure gated on the CNS to announce to the site network routers that the CNS is the path to the Cray mainframe Fibre Channel or HIPPI network for IPv4. The zebra daemon on the CNS is used the announce to the site network routers that the CNS is the path to the Cray mainframe Fibre Channel network for IPv6. N For Cray X1 series systems, the CNS number determines the (static) IP address (10.0.109.N+1) of the CNS on the Cray X1 series mainframe private Ethernet network. For example, for cns0, where the CNS number (N) is 0, the CNS name is cns0 and the IP address is 10.0.109.1. For Cray GigaRing systems, the IP address for the CNS private Ethernet eth0 interface on systems using an SWS is set via bootp protocol. Most private IP addresses on an SWS will be 10.1.124.N+200. Where N is a unique numerical value in the bootp configuration. Refer

S–2366–16

31

Cray Network Subsystem (CNS) Software Installation and Administration

Key

Description to Section 10.5, page 85 for configuring the bootptab file when connecting a CNS to a Cray GigaRing system. A CNS configured this way would be accessible from the CWS or SWS as cns0. Note: The administrator must ensure that the CWS /etc/hosts file has an entry for cns0. Note: For Remote Access Controller (RAC) name and IP address, refer to Section 10.4.2, page 82. For sites with a Cray X1 series system, refer to UNICOS/mp Networking Facilities Administration to configure the qfaX interface on the Cray X1 series system, where X is the ordinal number of the interface. For sites with a GigaRing system, refer to UNICOS Networking Facilities Administrator's Guide or UNICOS/mk Networking Facilities Administration to configure the ghippiX interface on the GigaRing system, where X is the ordinal number of the interface.

8.1 Preparing for CNS Software Installation, Upgrade, or Configuration
Before you begin the software installation, upgrade, or configuration of a CNS for your site, perform this procedure. 1. Read the entire release overview portion of this document and Errata (if one was shipped with the release package). An Errata contains important information that may be critical to a successful CNS software installation, upgrade, or configuration. 2. Prepare a console connection to the CNS and make sure all equipment involved in this connection is powered on and booted if necessary. You can choose one of these methods for console connection depending on what is best for your site: Note: For hardware connections, refer to Cray Workstation (CWS) Release Overview for the CWS, SWS Solaris Operating System and Devices Installation Guide for the SWS, and Appendix A, page 99 for the CNS. • Use the CNS serial port and serial cable to connect to the CWS serial port. • Use a local VGA monitor with a keyboard and attached cable(s) to connect to the matching connectors on the CNS itself. • Use the CNS private Ethernet port and Ethernet cable to connect to the
32 S–2366–16

Installation and Configuration [8]

CNS via the CWS private Ethernet network and a window on the CWS (for upgrades only). • Use the CNS RAC maintenance port (only after the RAC firmware has been updated) and Ethernet cable to connect to the CWS via the private Ethernet network and a window on the CWS. Refer to FCO 4043 for the Ethernet cable to the RAC and Section 9.6.3, page 55 to configure for remote console access. 3. Use Figure 2, page 30 as a worksheet (not usually needed for upgrades). Obtain the configuration information (shown as A through D on the worksheet) from your local networking administrator. Record that information in the spaces provided in that figure. In addition, the site's network administrator needs to configure item E in Figure 2 by following the instructions in Section 10.3.1, page 60. 4. Obtain the CNS software CD from the release package (not needed for configuring software on a new CNS). 5. Back up the current CNS configuration. Refer to Section 10.3.8, page 79 for more information.

8.2 Configuring Software for a New CNS
Use this procedure if you received a new CNS shipped from Cray. This procedure only requires that you configure your site's network because the CNS ships with its software suite preinstalled. 8.2.1 Generating the Configuration for a New CNS Configure the software for your new CNS by following the prompts displayed from running the cns_gen_config command. The prompts lead you through network interface configuration and suggest reasonable defaults for commonly used options. Note: The N in cnsN is the CNS number. You must change N to match the number of the CNS on which you are working. 1. Perform the preparation procedures listed in Section 8.1, page 32.

S–2366–16

33

Cray Network Subsystem (CNS) Software Installation and Administration

2. Log in to the CNS as root:
login: root passwd: initial0

3. At the prompt, enter:
[root@cnsN /root]# cns_gen_config

After the introductory output, each network interface will be configured. For each of these, you need to answer several questions. Use the information you recorded for C in Figure 2, page 30. Note: For details and examples of configuring a bonded interface, Ethernet interfaces, Fibre Channel interfaces, and setting the CNS number, refer to Section 10.3.5, page 69. 4. When cns_gen_config is finished, install the configuration (stored in /cray/etc/cns.conf) by entering:
[root@cnsN /root]# cns_config install

5. Answer no if you get the following warning:
Warning: there is an existing, non-standard configuration file for interface eth0. Do you want to keep it? [ n ] n

6. Find the appropriate time zone information file for your location in the /usr/share/zoneinfo directory. Set the local time zone for your CNS by creating a symbolic link to the appropriate zoneinfo file, which is done by executing these commands as root:
[root@cnsN etc]# cd /etc [root@cnsN etc]# rm -f localtime [root@cnsN etc]# ln -s /usr/share/zoneinfo/zoneinfo_file localtime

where zoneinfo_file is the name of the time zone information file, for example America/Chicago (case is important) for Chicago in the U.S.A. For more details about setting the CNS time zone and reviewing the setting, refer to Section 10.2, page 59.

34

S–2366–16

Installation and Configuration [8]

7. To use the newly configured CNS software, reboot the CNS as follows: Note: Rebooting the CNS will cause existing proxy connections to fail or hang. It is best to do this while the Cray X1 series network is up.
[root@cnsN /root]# reboot

8. Log in to the CNS as root:
login: root passwd: initial0

!

Caution: Cray recommends that you change the initial root password. Site policies should include periodically changing the root password on the CNS and Remote Access Controller (RAC). A secure root password is a necessary part of CNS security and should not be overlooked. 9. Change the CNS root and crayadm passwords now using the passwd and passwd crayadm commands, respectively. 10. If you are configuring a CNS in a GigaRing system, use Section 10.5, page 85 to configure the private Ethernet. You must configure the network routing. Proceed to Section 8.2.2.

8.2.2 Configuring Network Routing Configure the network routing interfaces to support your CNS. Use the routing policy information gathered in box E of Figure 2, page 30 to set up routing according to the instructions in Section 10.3, page 60. This completes the CNS software configuration.

8.3 Upgrading Existing CNS Software
Use this procedure to upgrade existing software on your CNS. You can do this over the network because it does not require physical access to the CNS nor does it require disruption of CNS service until the CNS is rebooted to bring it into service with the upgraded software. The CNS 1.6 release supports and has been tested for upgrades from the CNS 1.4 and 1.5 releases.
S–2366–16 35

Cray Network Subsystem (CNS) Software Installation and Administration

Perform the following steps to upgrade the current CNS software to the CNS 1.6 release. Note: The N in cnsN is the CNS number. You must change N to match the number of the CNS on which you are working. Note: If you are re-installing the CNS 1.6 release, and therefore have already upgraded the CNS firmware, you can use the CNS maintenance port (RAC) for re-installation and subsequent release upgrades. For RAC usage and more information, refer to Section 9.6, page 54 and the cns_rac(8) man page. 1. Perform the preparation procedures listed in Section 8.1, page 32. 2. Connect to the CNS using one of these methods: • From the CWS or SWS:
cws/sws$ telnet cnsN

• From the CWS, enter:
cws$ tip -57600 /dev/ttya

• From the SWS, enter:
sws$ tip -57600 /dev/ttyb

• From CNS RAC: see Section 9.6, page 54 for complete details. 3. Log in to the CNS as root. 4. Locate a directory on the CNS that has at least 500 MB of free space, for example, /usr/tmp. Check whether an existing INSTALL directory is present. If so, remove it and make a new one; otherwise, create an INSTALL directory there and go to the new directory:
cnsN# cnsN# cnsN# cnsN# cd /usr/tmp rm -rf INSTALL mkdir INSTALL cd INSTALL

5. Choose one of the following methods to enable access to the tar file containing the CNS software for the release upgrade: • Place the CNS software CD in the CNS CD-ROM drive and, as user root, enter:
cnsN# mount /dev/hda /mnt/cdrom cnsN# cd /mnt/cdrom/cray 36 S–2366–16

Installation and Configuration [8]

• Place the CNS software CD in the CD-ROM drive of the CWS or SWS and enter:
cws/sws$ cd /cdrom/cdrom0/cray

6. Copy the file CNS-pkg-1.6.tar from the CNS software CD to the INSTALL directory on the CNS using one of the following methods: • If using the CD mounted on the CNS, enter:
cnsN# cp CNS-pkg-1.6.tar /usr/tmp/INSTALL

• If using the CD mounted on the CWS or SWS, from the CNS enter:
cnsN# ftp cws Name (cws:root): crayadm Password: ftp> cd /cdrom/cdrom0/cray ftp> bin ftp> get CNS-pkg-1.6.tar ftp> quit

7. Use the tar command to unpack the CNS-pkg-1.6.tar file by entering:
cnsN# tar -xvf CNS-pkg-*.tar

8. Perform the installation upgrade by entering:
cnsN# ./cns_install upgrade

The upgraded software is placed on one of the alternate root (/root_a, /root_b, or /root_c) directories on the CNS disk. 9. You might be prompted about the network interfaces on your machine. Answer the prompts as needed. See Section 8.2.1, page 33 for more information, if needed. 10. When the process completes, a shell prompt displays. Select the newly installed root as the default boot partition by entering:
cnsN# ./cns_install select next

Note: If this is not done, the upgraded software will not be used on the next CNS boot. 11. Use one of the following methods to unmount the CNS software CD, so that the CNS does not boot from it:

S–2366–16

37

Cray Network Subsystem (CNS) Software Installation and Administration

Unmount the CD on the CNS by entering:
cnsN# cd / cnsN# eject

Unmount the CD on the CWS or SWS by entering:
cws/sws$ cd / cws/sws$ eject

12. To run the newly upgraded CNS software, reboot the CNS by entering:
cnsN# reboot

13. Log in to the CNS as root. 14. Update the Remote Access Controller (RAC) firmware version using the cns_rac command:
cnsN# cns_rac firmware

This sample shows some of the output obtained when the firmware needed upgrading:
cnsN# cns_rac firmware cns_rac: Firmware version of RAC 2.10 cns_rac: Is different than supported version 3.12 Do you want to update the RAC firmware [ yes ]? yes ... cns_rac: racreset complete ... cns_rac: Firmware update complete ... Stopping pppd (RAC) services: [ OK ] cnsN#

15. Set the RAC IP address by entering: Note: Because the CNS number is used to determine the RAC IP address, the CNS number must be unique (Section 10.4.2, page 82). In a Cray X1 series system environment, for example, with cns0:
cns0# cns_rac setip cns_rac: RAC IP address 10.0.109.101 for CNS0 will be set … cns_rac: RAC IP successfully set to 10.0.109.101 cns0#

38

S–2366–16

Installation and Configuration [8]

In a Cray GigaRing system environment, for example, with cns0:
cns0# cns_rac setip cns_rac: RAC IP address 10.1.124.240 for CNS0 will be set … cns_rac: RAC IP successfully set to 10.1.124.240 cns0#

16. Set the RAC password by entering:
cnsN# cns_rac password [password] … cns_rac: setting RAC root login password complete cnsN#

where password is your desired password. 17. If you are upgrading the software on a CNS in a GigaRing system, use Section 10.5, page 85 to configure the private Ethernet. This completes the software upgrade procedure.

8.4 Performing an Initial CNS Software Installation

! !

Caution: An initial installation (also known as a clean, cold, scratch, or destructive installation) removes all previous information from the CNS. Caution: Cray recommends that you change the initial root password. Site policies should include periodically changing the root password on the CNS and Remote Access Controller (RAC). A secure root password is a necessary part of CNS security and should not be overlooked. Perform an initial installation to: • Recover from a catastrophic disk failure resulting in lost or corrupted software • Totally replace all CNS software for any release, as requested by your system administrator or Cray support, such as to install the latest CNS release on a CNS currently running CNS version 1.0.0 Use the following procedure to perform an initial installation of CNS software. You need access to the software release CD. The procedure is described for the following CNS access setups: • Direct-attached VGA monitor with a keyboard

S–2366–16

39

Cray Network Subsystem (CNS) Software Installation and Administration

• Serial communications port to the CWS or SWS • CNS RAC Console connection 1. Perform the preparation procedures listed in Section 8.1, page 32. 2. If you will work from a VGA monitor with an attached keyboard, after ensuring that the monitor and keyboard have power and are connected to the CNS, insert the CNS software release CD into the CNS media drive and power on or reboot the CNS using the CNS power button. If working from a CWS or SWS, do Step 3. 3. If you will work from the CWS or SWS: a. b. c. Verify that the serial cable is plugged in to both the CNS and CWS or SWS. Log in to the CWS or SWS as crayadm. Connect to the CNS from the CWS or SWS as described here: • From the CWS, enter:
cws$ tip -57600 /dev/ttya

• From the SWS, enter:
sws$ tip -57600 /dev/ttyb

d. Log in to the CNS as root. e. f. Insert the CNS software release CD into the media drive on the CNS. Reboot the CNS by entering:
[root@cnsN /root]# reboot

4. At the LILO Boot Menu, select Serial Port if working from a CWS/SWS serial console connection or VGA + keyboard if working from a VGA monitor and attached keyboard or CNS RAC console. Note: Serial Port, which is the CWS/SWS serial line, is the default and is automatically selected in 5 seconds. 5. At the prompt for type of installation, enter: I Note: Case is not significant.
I)nitial install, U)pgrade, or S)hell [ Upgrade ]? 40 S–2366–16

Installation and Configuration [8]

6. At the prompt to reformat and partition:
A reformat and partition of /dev/sda is required in order to continue. ALL INFORMATION ON THE DISK WILL BE ERASED. Do you wish to continue? Continue [ Yes ]?

!

Caution: Choosing to continue will erase all information on the disk. Press Enter for the default Yes to continue or enter No to stop the installation. Case is not significant and you can choose to abbreviate an entry using a Y for Yes or a n for No. 7. Answer the time prompt:
Is the current local time 'DDD MMM NN hh:mm:ss yyyy '? [ Yes ]

where DDD is the day of the week, MMM is the month, NN is the numeric day of the month, hh is the hour, mm is the minute, ss is the second, and yyyy is the year. If the time is correct, press Enter to continue or enter N for prompts to correct the time. 8. Answer the questions regarding the interface configuration and CNS number. Refer to Section 10.3.6, page 76 for details and examples. 9. When the following text is displayed, the CNS initial installation has completed successfully:
The CNS has been installed successfully. Booting 1.6 in 20 seconds

The CNS install CD will be ejected from the CNS cdrom drive. The CNS will automatically reboot to CNS version 1.6. 10. Change the CNS root and crayadm passwords by using the passwd and passwd crayadm commands, respectively. 11. Find the appropriate time zone information file for your location in the /usr/share/zoneinfo directory. Set the local time zone for your CNS by creating a symbolic link to the appropriate zoneinfo file, which is done by executing these commands as root: Note: The N in cnsN is the CNS number. You must change N to match the number of the CNS on which you are working.
S–2366–16 41

Cray Network Subsystem (CNS) Software Installation and Administration

[root@cnsN etc]# cd /etc [root@cnsN etc]# rm -f localtime [root@cnsN etc]# ln -s /usr/share/zoneinfo/zoneinfo_file localtime

where zoneinfo_file is the name of the time zone information file, for example America/Chicago (case is important) for Chicago in the U.S.A. For more details about setting the CNS time zone and reviewing the setting, refer to Section 10.2, page 59. 12. If you are performing an initial installation of software on a CNS in a GigaRing I/O system, use Section 10.5, page 85 to configure the private Ethernet. This completes the CNS software initial installation procedure.

8.5 Reverting to the Previous CNS Configuration
Each time the CNS upgrade process is run, the next root partition (/root_a, /root_b, or /root_c) is populated with the contents of the new installation. The previous root partition remains and can be reselected if you encounter problems with the new installation. To revert to the previous CNS installation (the previous root partition), select the last partition that had been installed on the CNS with the cns_install select command: Note: The N in cnsN is the CNS number. You must change N to match the number of the CNS on which you are working.
cnsN# cns_install select last Found CNS version 1.1 on root_c. Warning: LBA32 addressing assumed Added Ser_1.1_b Added VGA_1.1_b * Added Ser_1.2_c Added VGA_1.2_c cnsN#

The asterisk on the line containing VGA_1.1_b indicates that this is the root partition that will be used by default the next time the CNS is booted. Reverting to the previous configuration does not affect the RAC firmware version.

42

S–2366–16

Installation and Configuration [8]

8.6 Using the Shell During a CNS Software Installation
The shell can be used for troubleshooting CNS installation problems. Perform the following procedure to use the shell while booted from the software CD: 1. Start an Initial CNS Software Installation (Section 8.4, page 39). 2. At the prompt for type of installation, type in S, which selects the shell option.
I)nitial install, U)pgrade, or S)hell [ Upgrade ]?

Note: Case is not significant. 3. Enter commands, as desired, to work with items on the CNS software release CD. For troubleshooting information, refer to Chapter 11, page 89. 4. Exit the shell and halt the CNS by entering:
exit 1

Note: Entering exit or exit 0 continues the installation. 5. When Power down appears on the display, press the CNS power button. Wait about 5 seconds and press it again to power up the CNS. Immediately press the eject button on the CNS media drive multiple times until the CD is ejected. The CNS should boot using the new software on its disk drive. Note: If using the VGA monitor and keyboard, press Ctl-Alt-Del then remove the software release CD from the CNS.

8.7 Supporting UNICOS/mp Installation
The CNS supports UNICOS/mp installation by providing a direct route between the CWS and the Cray X1 series mainframe. The cwsroute open command establishes the route by allowing the CWS access to the Cray mainframe directly through the CNS without going through the site network. The cwsroute close command re-imposes the full firewall restrictions. Some sites have a direct network connection between the site network and the CWS. If so, it is important to perform the cwsroute open command before modifying the route between the CWS and the Cray system. If the route is changed before the cwsroute open command is performed, the Cray system

S–2366–16

43

Cray Network Subsystem (CNS) Software Installation and Administration

will become inaccessible from the CWS until the cwsroute open command is invoked. Use the cwsroute command to establish a direct route between the CWS and the Cray X1 series mainframe with the CNS as its default gateway by performing the following procedure. Note: In this procedure, replace CRAYADDR with the IP address of the Cray mainframe (refer to A in Figure 2, page 30). The N in cnsN is the CNS number. You must change N to match the number of the CNS on which you are working. This CNS is assumed to be connected to the Cray mainframe and is the default gateway for that Cray X1 series system. 1. Log in to the CWS, then use telnet to get to the CNS over the private Ethernet. 2. Login to the CNS as root. 3. Enter:
cnsN# cwsroute open CRAYADDR

4. On the CWS, add the route from the CWS to the Cray X1 series mainframe by entering:
cws# route add CRAYADDR cns0

5. Install UNICOS/mp (refer to the UNICOS/mp Installation Guide). 6. On the CWS, delete the route from the CWS to the Cray mainframe:
cws# route delete CRAYADDR cns0

7. Log in to the CNS and enter
cnsN# cwsroute close CRAYADDR

44

S–2366–16

CNS Functional Overview and Administration [9]

This chapter is a functional overview of the CNS and its network environment and includes the following information: • CNS introduction (Section 9.1, page 45) • CNS networking overview (Section 9.2, page 47) • Automatic failover (Section 9.3, page 48) • CNS security (Section 9.4, page 49) • CNS message logging (Section 9.5, page 54) • Remote Access Controller (Section 9.6, page 54)

9.1 Introduction to the CNS
The CNS provides connectivity between networks, as does any Internet Protocol (IP) router, passing all packet traffic between site networks and the Cray mainframe. For this reason, IP Forwarding is enabled on the CNS. This performance improvement for IPv4 only is accomplished, as illustrated in Figure 3, page 46, by handling most TCP/IP connections so that the Cray mainframe can use large network packet sizes while the site network uses smaller packet sizes. The CNS connects to a Cray X1 series mainframe through a Fibre Channel connection. The CNS connects to a GigaRing based mainframe through a High Performance Parallel Interface (HIPPI) connection. The CNS also connects the Cray mainframe to site networks (for example, a Gigabit Ethernet network). Note: Cray does not support sharing a CNS between Cray mainframes or partitions. Refer to Section 9.4, page 49 for an overview of the security issues relating to the CNS.

S–2366–16

45

Cray Network Subsystem (CNS) Software Installation and Administration

Cray mainframe Communicates with Cray mainframe using a larger packet size or maximum transmission unit (MTU) over Fibre Channel or HIPPI CNS Communicates on behalf of the Cray mainframe to other hosts using a smaller network packet size or MTU Site network

host 1

host 2

Figure 3. Mainframe Throughput Increased by Large MTU

The mechanism for translating the large maximum transmission units (MTUs) of the Cray mainframe into the small MTUs of the site network is the tcp_assistd daemon. This daemon is a process running on the CNS that manages all of the TCP connections passing through the CNS. It performs appropriate network address translation (NAT) operations on the packets in these TCP streams to make the translation transparent to both the Cray mainframe and the endpoint on the site network. Because of this, the CNS becomes responsible for managing the flow of data going between the Cray mainframe and the small MTU Ethernet host. Therefore, the CNS must process all TCP traffic (all IP packets using the TCP protocol) that passes between the Cray mainframe and any Ethernet host establishing a connection with the Cray mainframe through that CNS. An ordinary router provides only connectivity; there is no provision available to change the size of a TCP segment. However, the CNS also has the ability to take many small TCP segments, destined for the Cray mainframe, and create a new larger TCP segment, then send the data on to the Cray mainframe. In the

46

S–2366–16

CNS Functional Overview and Administration [9]

opposite direction, the CNS receives one large TCP segment and sends it out as several smaller independent TCP segments. Note: User Datagram Protocol (UDP), Internet Control Message Protocol (ICMP), and other non-TCP protocols are simply routed through the CNS without change. They are not assisted by the CNS as with TCP. Basically, the CNS operates as a router, passing all packet traffic between site networks and the Cray mainframe.

9.2 Overview of CNS Networking
Figure 4 illustrates how a Cray X1 series system uses a CNS in a Gigabit Ethernet network environment.

Cray X1 series mainframe

CNS

Site Gigabit Ethernet

CWS

KEY Serial line (RS-232) Fibre Channel Gigabit Ethernet (fiber or copper) Cat 5 UTP cable
Figure 4. CNS in a Cray X1 Series System Environment

Figure 5 illustrates how a GigaRing system uses a CNS in a Gigabit Ethernet network environment.

S–2366–16

47

Cray Network Subsystem (CNS) Software Installation and Administration

Cray mainframe

NSR-1
HPN-1/HPN-2

HIPPI Switch or HIPPI Modem

CNS

Site Gigabit Ethernet

SWS

KEY GigaRing Parallel HIPPI, copper Serial HIPPI, fiber Gigabit Ethernet, fiber Cat 5 UTP Serial line (RS232)

HPN - HIPPI node NSR - Node subrack

Figure 5. CNS in a GigaRing System Environment

9.3 Automatic Failover and Fibre Channel IP Bonding
The Fibre Channel IP bonded interface allows two Fibre Channel links between a Cray X1 series mainframe and a CNS to be treated as a single logical interface (illustrated in Figure 6), providing a network connection failover capability. To properly configure a bonding interface, you must configure a bonding interface on the CNS and configure a corresponding UNICOS/mp bonding interface on the Cray X1 series system. For an example of the procedure output, refer to Section 10.3.5.1, page 71.

48

S–2366–16

CNS Functional Overview and Administration [9]

Cray X1 series mainframe Primary IP/FC Alternate IP/FC

CNS_0
Gigabit Ethernet

to site network

KEY Fibre Channel (FC) IP Gigabit Ethernet (fiber or copper) Internet Protocol

Figure 6. Fibre Channel IP Bonded Interface

A bonding interface will enslave all of the Fibre Channel network interfaces on the CNS. This allows you to use the Fibre Channel as either a single bonded interface or as two individual interfaces. Once bonding is configured, no Fibre Channel interface will be individually usable on the CNS. Note: If the CNS is booted when the Cray X1 series Fibre Channel interfaces are down, there is a chance that the bonded Fibre Channel network interface will not function. Subsequent rebooting of the Cray X1 series mainframe after the CNS is up should not cause this problem. If the CNS is booted while the Cray X1 series mainframe is down, reboot the CNS after the Cray X1 series mainframe is up to restore network functionality.

9.4 CNS Security
This section provides an overview of the security issues relating to the CNS.
S–2366–16 49

Cray Network Subsystem (CNS) Software Installation and Administration

9.4.1 Connection Methods for Accessing the CNS Controlling physical access to the CNS is the main security concern. Although the CNS does require a password for root access, if there is physical access to the CNS, the CNS could be rebooted, giving root access to the CNS without being required to provide the root password. Four connections are supported for administrators to access the CNS for maintenance and control: • RS-232 serial line—CNS access through the serial connection from the CWS or SWS (or another device having RS-232 capability, such as a laptop computer or simple serial console) by using the tip(8) program. • PS2/VGA connectors—CNS access through directly connected keyboard and monitor to the VGA and PS2 connectors found on both the front and back panels of the CNS. • Private Ethernet—CNS access through the CWS or SWS private Ethernet connection (for example, by logging in to the CWS or SWS and then using telnet to access the CNS). • Remote Access Controller (RAC)—CNS access through an Ethernet cable connection to the CWS or SWS. Only the root user has access to the RAC. The RAC firmware upgrade supplied with the CNS 1.3 or later release must be applied before you can use this access method. 9.4.2 CNS Accounts and Passwords CNS 1.2 and later releases come with the root and crayadm login accounts installed by default. However, there are some differences regarding passwords (see Section 9.4.2.1 and Section 9.4.2.4).

!

Caution: Cray recommends that you change the initial root password. Site policies should include periodically changing the root password on the CNS and Remote Access Controller (RAC). A secure root password is a necessary part of CNS security and should not be overlooked.

9.4.2.1 CNS root User Account and Password The root account is intended to be the one used by the administrator. For your site security, it is important that you change the root account password immediately after completing the CNS installation.

50

S–2366–16

CNS Functional Overview and Administration [9]

Once CNS access is gained, an administrator can log into the CNS as root using the current root password. The initial root password is initial0. 9.4.2.2 Resetting the CNS root Password If the CNS root password is lost or forgotten, use the following procedure to reset this password: 1. Establish a physical connection from the CWS or SWS to the serial port on the CNS by using the supplied cable. If you have problems, refer to Section 11.2, page 92. 2. To establish a serial connection from the CWS to the CNS:
cws$ tip -57600 /dev/ttya

To establish a serial connection from the SWS to the CNS:
sws$ tip -57600 /dev/ttyb

3. Press Enter to ensure that a prompt displays and the serial line is working. 4. Use the CNS Power button to power-cycle the CNS and force the CNS to reboot. 5. When the LILO prompt appears, press the Tab key followed by the Enter key within 5 seconds. This displays a list of options:
LILO Boot Menu Ser_1.1_b VGA_1.1_b Hit any key to cancel timeout --:-Use arrow keys to make selection Enter choice & options, hit CR to boot

where the 1.1 in this example is the release version, which will change with each release, and where _b is the root from which the CNS will boot (the other possible choices for the root devices are a and c). Use the arrow key to select the first option, which is always Ser_1.1_b. The selected option then appears after the boot prompt:
boot: boot: Ser_1.1_b

S–2366–16

51

Cray Network Subsystem (CNS) Software Installation and Administration

Note: You have 5 seconds to enter the word single once the boot prompt appears to force the CNS to boot to single user mode, or the system will follow the default boot process. Add a space and then the word single to force the CNS to boot to single user mode:
boot: boot: Ser_1.1_b single

6. When the CNS system completes the boot to single user mode and you see the shell prompt (that is, sh-2.04#), enter the following command and follow the prompts to reset the root password:
passwd

7. Reboot the CNS system by typing:
reboot

At this point, the CNS will reboot and the root password is reset. 9.4.2.3 RAC root Password Access to the RAC requires a password. The initial password for root on the RAC is initial0. To change the RAC root password after accessing the CNS as root, enter:
[root@cns# root]# cns_rac password [password]

where password is your desired RAC root password. 9.4.2.4 CNS crayadm User and Password The crayadm account is intended for future use in CNS software for instances where root access is not desired, but access to the CNS is required. The crayadm account is installed without a password. In other words, the system administrator must explicitly assign a password for this user before anyone can use the crayadm account. Note: CNS releases, 1.1, 1.1.1, and 1.1.2, came with a crayadm user account that had crayadm pre-assigned as a password. If upgrading from one of these releases, it is important for security reasons that you change this default password immediately after completing the CNS software release upgrade.

52

S–2366–16

CNS Functional Overview and Administration [9]

9.4.3 CNS Private Ethernet Firewall To ensure CNS security, Cray restricts network endpoint connections to the CNS. Although network connections to the Cray mainframe are routed through the CNS, no direct network endpoint connections are permitted to the CNS itself from either the Cray mainframe or from the site network. A network endpoint connection to the CNS is only possible via the private Ethernet network, which is connected to the CWS for Cray X1 series systems or Cray system workstation (SWS) for GigaRing systems. The CNS is configured to allow TCP/IP access through its private Ethernet network interface, eth0. The CWS or SWS accesses the CNS through this network. Warning: Cray recommends that you do not connect this private (internal) network to the site (public) network by way of the CWS or SWS. This private network has access to internal components of the Cray mainframe. Numerous potential security issues can be eliminated by simply not attaching the CWS or SWS to the larger site (public) network. This private configuration limits access to the CWS or SWS and the CNS to only those who have physical access to them. The Linux iptables(8) command is used to implement firewall functionality by using packet filtering techniques. Packets attempting a direct connection to the CNS are dropped. This eliminates all of the commonly used methods of direct attack on the CNS. It also eliminates most of the other security problems caused by bugs or administrative error. Cray believes that this technique offers good security on the CNS. Security for the Cray mainframe, however, is not controlled by the CNS in any way. Network security for the Cray mainframe and for the site network is the responsibility of the administrators of those systems. 9.4.4 CNS Software Security Features These CNS software items affect security for the CNS: iptables ip6tables Connections from site networks will be forwarded to the Cray mainframe, but no connections will be allowed to the CNS itself except from the Private Ethernet. This has implications for sites using an external Domain Name System (DNS) and Network Time Protocol (NTP) server for the CNS (see Section 3.2, page 7).

S–2366–16

53

Cray Network Subsystem (CNS) Software Installation and Administration

NTP and DNS configuration files that have been locally modified will not be carried forward to the CNS 1.6 release upgrade. The CWS will act as the NTP and DNS server. Linux kernel The CNS software uses the Linux 2.4.21-32.0.1.EL kernel distributed by Red Hat, which contains the latest security updates from Red Hat. CERT advisories Cray tracks the relevant security issues, such as CERT (Computer Emergency Response Team) advisories, and responds to them as part of normal CNS support. Additional changes and updates are made available as necessary.

9.5 CNS Message Logging
For Cray X1 series systems and GigaRing systems, the CNS logs messages to the /var/log/messages local CNS log file. Cray X1 series systems also log messages to the /opt/craylog/ops.log file on the CWS.

9.6 Remote Access Controller (RAC) Maintenance Port Function
This section describes the setup, configuration, and usage of the Remote Access Controller (RAC), the standard CNS maintenance port. 9.6.1 cns_rac Command Functions The Cray /cray/bin/cns_rac command configures the RAC feature on the CNS by running the Dell racadm commands for these RAC administrative functions: setip firmware info password Sets the RAC maintenance port IP address, network mask, and gateway Upgrades CNS RAC firmware Displays the RAC firmware version Change the RAC root password

54

S–2366–16

CNS Functional Overview and Administration [9]

For more information about the cns_rac command, refer to the cns_rac(8) man page. 9.6.2 Logging In to the RAC Maintenance Port After the RAC is set up and configured (Section 10.4, page 80), an administrator can log into the RAC directly from the CWS or SWS. After logging in to the CWS or SWS, you can access the RAC by entering: Note: This example uses cns0 for cnsN (where N is the CNS number) in a Cray X1 series environment. The IP address of your CNS, especially when used with Cray GigaRing systems, may be different from this example. Refer to Section 10.4.2, page 82 for determining the RAC IP address.
cws$ telnet cns0-rac cws$ telnet 10.0.109.101 Trying 10.0.109.101... Connected to 10.0.109.101. Escape character is '^]'. Dell Embedded Remote Access Controller (ERA/O) Firmware Version 3.0 (Build 10.06) login: root Password: initial0 [root]#

!

Caution: Cray recommends that you change the initial root password. Site policies should include periodically changing the root password on the CNS and Remote Access Controller (RAC). A secure root password is a necessary part of CNS security and should not be overlooked. You can change the RAC password by using the cns_rac password command. For more information about the cns_rac command, refer to Section 9.6.1, page 54, Section 10.4.1, page 80, and the cns_rac(8) man page.

9.6.3 Using the CNS RAC Maintenance Port Functions The RAC has two main functions in CNS administration: • Connect to the console to perform administrator tasks

S–2366–16

55

Cray Network Subsystem (CNS) Software Installation and Administration

• Power-cycle or reset the CNS After logging in to the RAC as root (Section 9.6.2, page 55), entering help or ? provides this RAC subcommand information: Note: Some of these subcommands are not supported.
[root]# help ? help -- displays the commands with a one-line description help command -- displays usage statement for specified command logout quit exit -- logs the user out and then prints a new login prompt setled -- set the state of the LEDs on a module <NOT SUPPORTED> getled -- display the LED settings on a module <NOT SUPPORTED> getsysinfo -- display general RAC and system information getmodinfo -- get module config and status information getsensors -- display RAC sensor readings and information <NOT SUPPORTED> getsensorinfo-- get sensor status for the specified sensors <NOT SUPPORTED> serveraction -- execute a graceful or hard server reset or power-on/off/cycle getraclog -- display RAC Log entries getraclog -i -- display the total number of entries in the RAC Log gettracelog -- display Trace Log entries gettracelog -i -- display numbers of entries in the Trace Log settracelog -- set Trace Log flags settracelog -s -- display the current Trace Log setting getsel -- displays SEL entries getsel -i -- displays the total number of entries in the system event log connect -- connect to com2 serial port or video text console racadm -- execute a racadm subcommand (allowed only for username:root)

These three RAC subcommands are the most useful for CNS administration: connect, serveraction, and exit. 9.6.3.1 The connect Command You can use the connect command to gain access to the CNS system console. To get the complete syntax of the command, enter:
[root]# ? connect connect [-b] [-u] [-s] <module>

56

S–2366–16

CNS Functional Overview and Administration [9]

Enter the connect video command to obtain a CNS console:

!

Caution: This does not work if the CNS is already in video mode, so do not install or enable gdm or Window Manager, because it takes over the VGA monitor and interferes with the RAC getting the console. Note: If you press the space bar quickly before the key mapping information is cleared from the screen, the information remains displayed for easy reference. To continue after reading the information, press the space bar again. For Dell PowerEdge 2650 CNS, enter:

[root]# connect video

For Dell PowerEdge 2850 CNS, enter:
[root]# connect com1 Connected to video. To end, use the key sequence "<CR>~." Press the spacebar to pause... ( Changes to: Press the spacebar to continue... KEYMAPPING FOR CONSOLE REDIRECTION: Use the "<ESC>R<ESC>r<ESC>R" key sequence for <CTRL-ALT-DEL> Use the "<ESC><n>" key sequence for the Function <n> Key. for <n> = 1,2,3, ..., 9 Use the "<ESC>0" key sequence for the Function 10 Key. Use the "<ESC>!" key sequence for the Function 11 Key. Use the "<ESC>@" key sequence for the Function 12 Key. Use the "<ESC>h" key sequence for the Home Key. Use the "<ESC>k" key sequence for the End Key. Use the "<ESC>+" key sequence for the Insert Key. Use the "<ESC>-" key sequence for the Delete Key. Use the "<ESC>?" key sequence for the Page Up Key. Use the "<ESC>/" key sequence for the Page Down Key. Use the "<ESC>^Cx" key sequence for <CTRL-x> where: ^C is <CTRL-c>, and x is one of h, i, j, m. Use the "<ESC>^Ax" key sequence for <ALT-x> where: ^A is <CTRL-a>, and x is any letter, a through z. … Red Hat Linux release 9 (Shrike) Kernel 2.4.22 on a 2-processor i686

S–2366–16

57

Cray Network Subsystem (CNS) Software Installation and Administration

cns0 login:root Password: Last login: Fri Apr [root@cns0 root]#

2 11:01:51 from cws

Now, you are logged in to the CNS and can run CNS commands. To exit the CNS command mode and return to the RAC login, press Enter before typing the ~. escape command. It might be necessary to enter this escape sequence more than once to effect the escape for logging out of the RAC, as shown in this example for cns0:
[root@cns0 -bash: ~.: [root@cns0 -bash: ~.: [root@cns0 [root]# root]# <CR>~. command not found root]# <CR>~. command not found root]# <CR>~.

9.6.3.2 The serveraction Command You can use the serveraction command to power-cycle and reset the CNS. To obtain the complete syntax of the command, enter:
[root]# ? serveraction serveraction [-s <sysNum>] [-d <delay>] [-w <cycleWait>] <action> where <action> is one of the following: powerup, powerdown, powercycle, hardreset, graceshutdown, gracepowercycle, gracereboot

Note: The graceshutdown, gracepowercycle, and gracereboot commands are not supported. For example, to power down the CNS associated with the RAC after 10 seconds from when the command is received (one second is the default), and later power it back up when it is powered off, enter:
[root]# serveraction -d 10 powerdown … [root]# serveraction powerup

9.6.3.3 The exit Command Use the exit command to exit a session.
58 S–2366–16

Advanced Configuration Topics [10]

This chapter provides information on a variety of topics, including Network Time Protocol (NTP), setting the CNS time zone, routing, interface configuration with cns_gen_config, Fibre Channel interface bonding, and installation prompt details.

10.1 NTP for the CNS
NTP is implemented via the ntpd daemon, and is used to synchronize the time used on the CNS to the time used on the CWS. The CNS gets its time from the ntpd server on the CWS using NTPv4 protocol. Cray recommends that the time updates for the CNS come from the CWS and sets this as the default configuration. It is important that the time stamps in the log file are consistent with that of the CWS.

10.2 Setting Up the Local Time Zone of the CNS
The geographical location of a CNS determines the local time zone desired for that CNS. You set the local time zone by creating a symbolic link to the appropriate zoneinfo file. Find the appropriate time zone information file for your location in the /usr/share/zoneinfo directory. Note: There are numerous choices for various locations around the world. For example, Mendota Heights, Minnesota, USA is in the same time zone as Chicago, Illinois, so the file to select is /usr/share/zoneinfo/America/Chicago (case is important). To establish the time zone as America/Chicago on cns0, for example, as root execute these commands with the appropriate zoneinfo file:
[root@cns0 etc]# cd /etc [root@cns0 etc]# rm -f localtime [root@cns0 etc]# ln -s /usr/share/zoneinfo/America/Chicago localtime

After establishing the time zone, you can use the date command to display the appropriate abbreviation for your time zone, as shown for cns0 in this example:
[root@cns0 etc]# date Sun Apr 25 17:48:59 CDT 2004

S–2366–16

59

Cray Network Subsystem (CNS) Software Installation and Administration

10.3 CNS Routing Configuration
Proper operation of the CNS networks requires you to configure the network routing. To establish network communication, you must set the proper routes to and from the: 1. CNS (Section 10.3.1, page 60) 2. Cray X1 series mainframe (Section 10.3.4, page 64) 3. Site network routers (Section 10.3.4.1, page 66) In a TCP/IP network, there is no restriction on the route that an IP packet must take when it is sent out over the network. As long as the packet gets to its intended destination, there are no limitations on which communication nodes route the packet. This is not true for the CNS; the CNS must process all TCP packets going in to and out of the Cray mainframe. This means that routing must be configured to ensure that: • The CNS is the last hop for any TCP packet destined for the Cray mainframe. • The CNS is the first hop for any TCP packet being sent by a Cray mainframe.

!

Caution: It is the site's responsibility to configure their IP routing to enforce this requirement. Otherwise, connections to and from the Cray mainframe might hang when a route dynamically changes on a network external to the Cray mainframe. The CNS uses a static IP address on the CWS private Ethernet network. For more information, refer to "Adding a CNS to the CWS Private Administration Subnetwork" in the Cray Workstation (CWS) Installation Guide (S–2331–23 or later release) and "Private Administration Subnetwork" in Cray X1 Series System Configuration and CWS Administration (S–2332–23 or later release).

10.3.1 Configuring a Default Static Route on the CNS Routing between the CNS and the site network can be configured statically or dynamically. To configure default static routes on the CNS, use the cns_gen_config command and answer the resulting prompts. To configure additional static routes, edit the /cray/etc/cns.conf file and

60

S–2366–16

Advanced Configuration Topics [10]

add the following line just prior to the line that contains the END%ROUTES text, so it looks similar to this:
any net 192.168.0.0 netmask 255.255.255.0 gw IP or any 192.168.0.1 gw IP

where IP is the IP address of the site default router on the eth2 network. After editing the cns.conf file, use the cns_config install command to write and save the configuration. Once the configuration is written, you must reboot the CNS for the route to take affect. 10.3.2 Using gated to Configure a Default Dynamic Route for the CNS The gated gateway routing daemon is used on the CNS to dynamically configure network routes. The gated daemon handles multiple routing protocols: RIP, BFP, EGP, HELLO, and OSPF. Example 1: /etc/gated.conf Default Configuration File for a Fibre Channel Network This example shows the /etc/gated.conf default configuration file for gated using the RIP protocol and announces the CNS as the path to the Fibre Channel network via the lpfn0 interface to the receiving router on the Gigabit Ethernet network for the eth2 interface. This example will also take in RIP updates from the network to which eth2 belongs. The static entry preserves the static default route. The traceoptions parameter creates a gated log file that logs all gated actions.
traceoptions "/var/tmp/gated.log" replace size 100k files 2 all ; rip yes { interface all noripin noripout ; interface all version 2 ; interface eth2 ripout ripin version 2 multicast ; # interface lpfn0 ripout ripin version 2 multicast ; }; static { default gateway 192.168.240.1 retain; }; import proto rip { all ; S–2366–16 61

Cray Network Subsystem (CNS) Software Installation and Administration

default restrict ; }; export proto rip { proto rip { all ; default restrict; }; proto direct interface eth0 restrict; proto direct interface eth1 restrict; proto direct interface eth2 restrict; proto direct interface hip0 restrict; # proto direct interface lpfn0 restrict; proto direct ; proto static metric 1; };

Example 2: /etc/gated.conf Default Configuration File for a HIPPI Network This is an example that announces the HIPPI network to the routers listening to RIP version 2 on the Gigabit Ethernet network for the eth2 interface.
traceoptions "/var/tmp/gated.log" replace size 100k files 2 all ; rip yes { interface all noripin noripout ; interface all version 2 ; interface eth2 ripout ripin version 2 multicast ; # interface lpfn0 ripout ripin version 2 multicast ; }; static { default gateway 172.30.7.1 retain; }; import proto rip { all ; default restrict ; }; export proto rip { proto rip { all ; default restrict; }; proto direct interface eth0 restrict; proto direct interface eth1 restrict; proto direct interface eth2 restrict; # proto direct interface hip0 restrict; proto direct interface lpfn0 restrict; 62 S–2366–16

Advanced Configuration Topics [10]

proto direct ; proto static metric 1; };

For more detailed information about gated configuration refer to: http://www.mark-itt.ru/Collection/gated/ 10.3.3 Using zebra to Configure a Default Dynamic Route for the CNS with IPv6 The zebra routing daemon is used on the CNS to dynamically configure network routes for IPv6. The zebra daemon handles multiple routing protocols: RIPv1, RIPv2, RIPng, OSPF, OSPF6, BGP4+, and BGP4-. Example 3: /etc/zebra/zebra.conf and /etc/zebra/ripngd.conf are the Default IPv6 Configuration Files for a Fibre Channel Network This example shows the /etc/zebra/zebra.conf and /etc/zebra/ripngd.conf default configuration files for the zebra daemon to use the RIPng protocol and announce the CNS as the path to the Fibre Channel network via the lpfn0 interface to the receiving router on the Gigabit Ethernet network for the eth2 interface. The second line in the /etc/zebra/zebra.conf file adds a IPv6 default route.
/etc/zebra/zebra.conf: hostname cns7 ipv6 route 2000::/3 2001:408:4000:4f0::1 eth2 /etc/zebra/ripngd.conf: hostname cns7-ripng ! router ripng network eth1 ! This is the route we are advertising: route 2001:408:4000:1411::/64 ! this is where we are sending it: distribute-list local-only out eth2 ! log syslog hostname cns7-zebra ! log syslog ! S–2366–16 63

Cray Network Subsystem (CNS) Software Installation and Administration

For more detailed information about zebra configuration refer to: wwww.zebra.org. 10.3.4 Configuring Static Routing to the CNS from the Cray Mainframe For Cray X1 series systems: You must set a default route from the Cray X1 series mainframe to the CNS in the UNICOS/mp /etc/config/static-route.options file, as shown in Example 4. Example 4: Sample static-route.options File for a Cray X1 series Mainframe This example shows the UNICOS/mp /etc/config/static-route.options file that establishes the default static route to the CNS from the Cray X1 series mainframe.
# # # # # # # # # # static-route.options The network startup script, /etc/init.d/network, invokes this script to set static routes. Site-dependent static routes should be put here. Read `man route`. Since $ROUTE and $QUIET are set in /etc/init.d/network, it is convenient to use `route` commands similar to the following here:

# $ROUTE $QUIET add -net 10 192.0.2.3 # or # # $ROUTE $QUIET add 192.168.1.1 192.0.2.5 $ROUTE $QUIET add default 172.30.8.50

Edit the static-route.options file and add the site-assigned IP address for the Fibre Channel network.

64

S–2366–16

Advanced Configuration Topics [10]

After you configure the static-route.options file, you can reboot the Cray X1 series system for the routes to take affect. As an alternative to rebooting the Cray X1 series system, as the superuser you could set the default route with the route command: where cnsN-FCIP is the IP address of the CNS on the Fibre Channel network.
cray# route add default cnsN-FCIP

For Cray GigaRing systems: You can set the Cray GigaRing mainframe static routing default in the /etc/gated.conf file as shown in Example 5. Refer to UNICOS Basic Administration Guide for Cray J90se and Cray SV1 Series Systems, UNICOS Networking Facilities Administrator's Guide, or UNICOS/mk Networking Facilities Administration, whichever is appropriate for your site. For more information on the use of the /etc/gated.conf file with Cray GigaRing systems, refer to the UNICOS or UNICOS/mk gated-config(5), gated(8), and tcpstart(8) man pages. Example 5: Sample gated.conf File for a Cray GigaRing Mainframe For Cray GigaRing systems using the UNICOS or UNICOS/mk operating systems, you must set the default route from the Cray mainframe to the CNS in the Route section for static of the gated.conf file, similar to this example:
# # # # # # # SN0000 - gated.conf - Edition 129 [Mon Oct 14 10:36:58 CDT 2002] Created by Configuration Generator Rev. 80.60

/etc/gated.conf

# # Interfaces #

section

# # Definition section # # # Protocol section S–2366–16 65

Cray Network Subsystem (CNS) Software Installation and Administration

# rip yes { interface all noripout ; } ; redirect yes ; # # Route section # static { default gateway 172.30.8.50 } ; # # Control section #

;

10.3.4.1 Configuring Site Network Routers Your site network routers require a configuration update (C to D in Figure 7, page 68) after the Cray mainframe and CNS routes have been configured. This is to specify to the site network routers that the CNS is the direct route to the Cray mainframe. There are two methods to do this: • Configure a static route by creating a static route entry on your site network routers. • Specify dynamic routing by configuring a routing protocol on the router and having the CNS send updates to the router via gated. 10.3.4.2 Configuring a Proxy Address Resolution Protocol As an alternative to configuring a static route for a site router, you can configure the CNS to use proxy address resolution protocol (ARP) for the path to the Cray mainframe (D to A in Figure 7, page 68). Note: Proxy ARP is not supported for IPv6. To configure proxy ARP, the IP addresses for the network connecting the CNS (lpfnnn) to the Cray mainframe (qfann) (see Figure 7, page 68) need to be taken from the Gigabit Ethernet network. The CNS-to-Cray mainframe network must
66 S–2366–16

Advanced Configuration Topics [10]

be a separate subnetwork different from the Gigabit Ethernet network even though the IP addresses come from the Gigabit Ethernet IP range of addresses. After the IP addresses have been configured on the CNS and the Cray mainframe, turn on proxy ARP on the CNS. To turn on proxy ARP, log in to the CNS as root and enter these echo commands: Note: This example uses cns0; your CNS number may differ. For Cray X1 series systems:
cns0#>echo 1 > /proc/sys/net/ipv4/conf/eth2/proxy_arp cns0#>echo 1 > /proc/sys/net/ipv4/conf/lpfn0/proxy_arp

For Cray GigaRing systems:
cns0#>echo 1 > /proc/sys/net/ipv4/conf/eth2/proxy_arp cns0#>echo 1 > /proc/sys/net/ipv4/conf/hip0/proxy_arp

You can add these echo commands to the /etc/rc.local file so that the proxy ARP setting will be set whenever the CNS is rebooted. Figure 7 shows an example configuration where proxy ARP can be used. Note that the IP address and network mask of the Cray mainframe subnetwork is a subset of the IP address and network mask of the Gigabit Ethernet network. The Gigabit Ethernet network has a range of 192.168.240.0 through 192.168.240.255. The Cray mainframe subnetwork range is 192.168.240.76 through 192.168.240.79. This is accomplished by setting the network mask to 255.255.255.252 on the Cray mainframe subnetwork versus 255.255.255.0 on the Gigabit Ethernet network. Boxes A, B, and C in Figure 7 show examples of these IP addresses and network masks. For sites with a Cray X1 series system, see UNICOS/mp Networking Facilities Administration to configure the qfaX interface on the Cray X1 series system, where X is ordinal number of the interface. For sites with Cray GigaRing systems, see UNICOS Networking Facilities Administrator's Guide or UNICOS/mk Networking Facilities Administration to configure the ghippiX interface, where X is the ordinal number of the interface.

S–2366–16

67

Cray Network Subsystem (CNS) Software Installation and Administration

FIbre Channel or HIPPI connection to Cray mainframe

IP address/Netmask 192.168.240.78 255.255.255.252

A

Fibre Channel: /etc/config/netif.options /etc/config/ifconfig_#.options HIPPI: /etc/config/interface

Fibre Channel or HIPPI stand-alone subnetwork
CNS Fibre Channel or HIPPI

SWS Usually 10.0.124.200 CWS 10.0.104.1

IP address/Netmask 192.168.240.77 255.255.255.252

B

CNS Firewall
CNS
Gigabit Ethernet

Private Ethernet
CWS/SWS CWS

10.0.109.N +1 cnsN N=
(N=CNS number)

SWS

IP address/Netmask 192.168.240.36 255.255.255.0 Gigabit Ethernet
Gateway router

C

Usually 10.1.124.N+200
(N determined from bootp)

IP address

Routing Policy Changes

192.168.240.1

D

E

Gateway Router

Site Network

Figure 7. Example of CNS Network Proxy ARP Configuration
68 S–2366–16

Advanced Configuration Topics [10]

10.3.5 Interface Configuration Using cns_gen_config This section provides information and examples to help you use the cns_gen_config command for network interface configuration. Example 6 shows an example of the output resulting from running the cns_gen_config command, cns_config install command, and the reboot required to activate the new configuration. The subsections following this example break down each of the steps involved.

!

Caution: Cray recommends backing up the CNS configuration before changing it with the cns_gen_config command. Refer to Section 10.3.8, page 79 for more information. Example 6: Interface Configuration Output of cns_gen_config In this sample output, the network setup is such that most of the questions presented by running the cns_gen_config command were answered by pressing Enter (taking the defaults). The exceptions to taking all the defaults were the Configure interface bond0 and Configure interface eth1 prompts, which were answered with a yes. The user then had to specify an IP address and a network mask for each interface. Note: For more information about bonding, refer to Section 9.3, page 48. This sample output shows a bonded Fibre Channel (FC) interface bond0 was configured for lpfn0 and lpfn1 and it shows the configuration of a file system server (FSS) interface, both of which may be less commonly used at most sites. Note: This example uses cns0; your CNS number may differ.
cns0# cns_gen_config Please answer the following questions about your configuration. Interface names starting with "eth" are ethernet. Interface names starting with "hip" are HIPPI. Interface names starting with "lpfn" are Fibre Channel. Defaults (if available) are provided in [brackets]. Do you want to configure IPV6 on [ yes ]? bond0 is a pseudo-device used to bond individual FC interfaces together to provide failover. Answering yes here will cause any FC interfaces to be "enslaved" to

S–2366–16

69

Cray Network Subsystem (CNS) Software Installation and Administration

bond0. This option requires corresponding configuration changes on the X1. You should configure bond0 only if you have made the appropriate changes on the X1. If unsure, answer 'n'. Configure interface bond0 [ no ]? y What is the IP address on this interface ? 12.34.56.78 What is the NETMASK on this interface ? 255.255.255.0 What is the IPV6 address ? 2001:408:4000:1408::5/64 The X1 IPV6 address is configured from the CNS using stateless autoconfiguration. Stateless autoconfiguration specifies ONLY the PREFIX portion of the IPV6 address. What is the IPV6 prefix for the X1 ? 2001:408:4000:1408::/64 bond0 configuration complete. (skipping eth0 - reserved private ethernet interface) eth1 is normally used to connect a File System Server If you have an FSS, say yes. If you don't know what this is, answer no. Configure interface eth1 [ no ]? y How is the IP acquired (none (static), dhcp) [ static ]? What is the IP address on this interface [ 192.168.10.20 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? What is the MTU on this interface [ 1500 ]? eth1 configuration complete. Configure interface eth2 [ yes ]? How is the IP acquired (none (static), dhcp) [ static ]? What is the IP address on this interface [ 10.0.3.91 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? What is the IPV6 address ? 2000:5::71/32 What is the MTU on this interface [ 1500 ]? eth2 configuration complete. Configure interface lpfn0 [ yes ]? Adding lpfn0 as slave to bond0 lpfn0 configuration complete. 70 S–2366–16

Advanced Configuration Topics [10]

Configure interface lpfn1 [ yes ]? Adding lpfn1 as slave to bond0 lpfn1 configuration complete. Do you want to configure IPV4 default route [ yes ]? What is the IP address for the route [ 10.0.3.88 ]? What is the interface for the default route [ eth2 ]? Do you want to configure IPV6 default route [ yes ]? Each CNS must have a unique number. Please indicate the CNS number from 0-19 [ 0 ]? The Gated daemon is used to configure dynamic routing for IPV4 Do you want to configure the gated daemon on [ no ]? The Zebra daemon is used to configure dynamic routing for IPV6 Do you want to configure the zebra daemon on [ no ]? yes Copying existing static-routes file to config Copying existing static-arp file to config

The following warning, which requires no for an answer, might display:
Warning: there is an existing, non-standard configuration file for interface eth0. Do you want to keep it? [ n ]

The CNS was rebooted as shown here to use the newly configured CNS software:
[root@cns0 /root]# reboot

10.3.5.1 Configuring Bonded Interfaces for Automatic Failover To properly configure a bonded interface (for automatic failover), you must configure a bonded interface on the CNS and configure a corresponding UNICOS/mp bonded interface on the Cray X1 series system. For more information about the Fibre Channel IP bonded interface and failover, refer to Section 9.3, page 48 and UNICOS/mp Networking Facilities Administration. If failover is not desired, refer to Section 10.3.5.2, page 73 for information about configuring Ethernet interfaces.
S–2366–16 71

Cray Network Subsystem (CNS) Software Installation and Administration

To configure a bonded interface, use the cns_gen_config command (see Example 7). An IP address and a network mask (NETMASK) are required just as for configuring other network interfaces. After the configuration, you must run cns_config install and then reboot the CNS for the bonding to take effect. Example 7: cns_gen_config Output When Configuring a Bonded Interface This example shows information, questions, and default answers display while running cns_gen_config to generate the bond0 interface for the master CNS configuration file when the CNS is attached to a Cray X1 series mainframe by two bonded Fibre Channel (FC) interfaces (lpfn0 and lpfn1).
Please answer the following questions about your configuration. Interface names starting with "eth" are ethernet. Interface names starting with "hip" are HIPPI. Interface names starting with "lpfn" are Fibre Channel. Defaults (if available) are provided in [brackets] bond0 is a pseudo-device used to bond individual FC interfaces together to provide failover. Answering yes here will cause any FC interfaces to be "enslaved" to bond0. This option requires corresponding configuration changes on the X1. You should configure bond0 only if you have made the appropriate changes on the X1. If unsure, answer 'n'. Configure interface bond0 [ no ]? y What is the IP address on this interface ? 12.34.56.78 What is the NETMASK on this interface ? 255.255.255.0 What is the IPV6 address ? 2001:408:4000:1408::5/64 The X1 IPV6 address is configured from the CNS using stateless autoconfiguration. Stateless autoconfiguration specifies ONLY the PREFIX portion of the IPV6 address. What is the IPV6 prefix for the X1 ? 2001:408:4000:1408::/64 bond0 configuration complete. …

72

S–2366–16

Advanced Configuration Topics [10]

10.3.5.2 Configuring Ethernet Interfaces If you intend to use a file system server (FSS), you must configure the interface for the CNS (refer to Example 8). Example 8: cns_gen_config Output When Configuring a CNS Interface for a FSS This example shows partial output of the cns_gen_config command for configuring a CNS interface for the FSS interface to a Cray X1 series system (connecting the Cray X1 series system to the FSS using the CNS eth1 interface). Note: This is just an example. The values displayed on your system for IP address and NETMASK may be different.
… eth1 is normally used to connect a File System Server If you have an FSS, say yes. If you don't know what this is, answer no. Configure interface eth1 [ no ]? y How is the IP acquired (none (static), dhcp) [ static ]? What is the IP address on this interface [ 192.168.10.20 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? What is the MTU on this interface [ 1500 ]? eth1 configuration complete. …

You must configure the Ethernet interfaces for the CNS (refer to Example 9). Example 9: cns_gen_config Output When Configuring CNS Ethernet Interfaces This example shows the information, questions, and default answers that display while running cns_gen_config to generate the master CNS configuration file when the CNS is attached to a Cray X1 series mainframe via an Ethernet interface. Answer them as follows: Note: This is just an example. The values displayed on your Cray X1 series system or Cray GigaRing system for IP address and NETMASK will be different.
… Configure interface eth2 [ yes ]? y How is the IP acquired (none (static), dhcp) [ static ]? What is the IP address on this interface [ 10.0.3.91 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? S–2366–16 73

Cray Network Subsystem (CNS) Software Installation and Administration

What is the IPV6 address ? 2001:408:4000:804::5/64 What is the MTU on this interface [ 1500 ]? eth2 configuration complete.

Press Enter to use the default value (in brackets). For yes or no questions, the first character is sufficient (y or n is enough). 10.3.5.3 Configuring Fibre Channel or HIPPI Interfaces You must configure the Fibre Channel interfaces for Cray X1 series systems or HIPPI interfaces for Cray GigaRing systems. These are presumed to be the CNS network interfaces to the Cray mainframe. To be directly connected to the Cray mainframe means to be set up as a Cray mainframe network interface. For these network interfaces, the tcp_assistd daemon will handle TCP connections, resulting in improved performance on many operations. Note: If bonding has been configured as described in Section 10.3.5.1, page 71, any lpfn (Fibre Channel) interfaces that are configured will be enslaved to the bonding driver. These Fibre Channel interfaces will therefore not be individually visible to the CNS, as indicated in the enslaving of lpfn0 and lpfn1 to bond0 in this portion of cns_gen_config output:
… Configure interface lpfn0 [ yes ]? y Adding lpfn0 as slave to bond0 lpfn0 configuration complete. Configure interface lpfn1 [ yes ]? y Adding lpfn1 as slave to bond0 lpfn1 configuration complete. …

You must configure Fibre Channel (Example 10) or HIPPI (Example 11) interface connections, if present. Example 10: cns_gen_config Output When Configuring a Fibre Channel Interface This example shows partial output of the cns_gen_config command for configuring a Fibre Channel interface for a Cray X1 series system.

74

S–2366–16

Advanced Configuration Topics [10]

Note: This is just an example. The values displayed on your system for IP address and NETMASK may be different. You would use the information you recorded for B in Figure 2, page 30 for the configuration.
… Configure interface lpfn0 [ yes ]? y Is this interface directly connected to the Cray [ yes ]? What is the IP address on this interface [ 10.0.3.91 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? What is the IPV6 address ? 2001:408:4000:1408::5/64 The X1 IPV6 address is configured from the CNS using stateless autoconfiguration. Stateless autoconfiguration specifies ONLY the PREFIX portion of the IPV6 address. What is the IPV6 prefix for the X1 ? 2001:408:4000:1400::/64 lpfn0 configuration complete. Configure interface lpfn1 [ yes ]? y Is this interface directly connected to the Cray [ yes ]? What is the IP address on this interface [ 10.0.1.91 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? What is the IPV6 address ? 2001:408:4000:1408::5/64 The X1 IPV6 address is configured from the CNS using stateless autoconfiguration. Stateless autoconfiguration specifies ONLY the PREFIX portion of the IPV6 address. What is the IPV6 prefix for the X1 ? 2001:408:4000:1401::/64 lpfn1 configuration complete. …

Example 11: cns_gen_config Output When Configuring a HIPPI Interface This example shows partial output of the cns_gen_config command for configuring a HIPPI interface for a Cray GigaRing system. Note: This is just an example. The values displayed on your system for IP address and NETMASK may be different. You would use the information you recorded for B in Figure 2, page 30 for the configuration.
… Configure interface hip0 Is this interface directly connected to the Cray [ yes ]? y How is the IP acquired (none (static), dhcp) [ none ]? S–2366–16 75

Cray Network Subsystem (CNS) Software Installation and Administration

What is the IP address on this interface [ 192.168.242.58 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? hip0 configuration complete. …

10.3.5.4 Configuring CNS Interfaces That are not Cabled to the CNS Answer no for CNS interfaces that are not cabled (whether bonded or not bonded).
… Configure interface lpfn1 [ yes ]? n …

10.3.5.5 Specifying the CNS number You must specify the CNS number (0 in this example) during the configuration.
… Each CNS must have a unique number. Please indicate the CNS number from 0-19 [ 0 ]? 0 …

The CNS number determines the (static) IP address (10.0.109.N+1) of the CNS on the Cray X1 mainframe private Ethernet network. For example, for cns0 , where the CNS number (N) is 0, the CNS name is cns0 and the IP address is 10.0.109.1. 10.3.6 Initial Installation Configuration Prompts You must provide information for the initial installation configuration prompts. Here are some samples: Note: FSS is File System Server. 1. At the configuration prompt, Please answer the following questions about your configuration, you see the following text:
… (skipping eth0 - reserved private ethernet interface) eth1 is normally used to connect a File System Server If you have an FSS, say yes. If you don't know what this is, answer no. 76 S–2366–16

Advanced Configuration Topics [10]

Configure interface eth1 [ no ]? … Configure interface eth2 [ yes ]? yes …

Use the information you recorded for C in Figure 2, page 30 to supply the requested information. Note: This is just an example. The values displayed on your system for IP addresses and NETMASK may be different.
How is the IP acquired (none (static), dhcp) [ static ]? static What is the IP address on this interface ? 10.0.1.57 What is the NETMASK on this interface ? 255.255.255.0 What is the IPV6 address ? 2001:408:4000:804::5/64 What is the MTU on this interface [ 1500 ]? 1500 eth2 configuration complete.

where 10.0.5.57 is the IP address and 255.255.255.0 is the NETMASK that you must change to be specific to your site in nnn.nnn.nnn.nnn decimal format. 2001:408:4000:804::5/64 is the IPv6 address if you answered yes to configuring IPv6. 2. At the prompt to configure the next interface, choose the instructions below for either Fibre Channel (lpfn#) or HIPPI (hip#) depending on your specific site setup: Fibre Channel interface:
Configure interface lpfn0 [ yes ]? yes

Use the information you recorded for B in Figure 2, page 30 to supply the requested information: Note: This is just an example. The values displayed on your system for IP addresses and NETMASK will be different.
Is this interface directly connected to the Cray [ yes ]? y Configure interface lpfn0 [ yes ]? IP address? 10.0.1.57 What is the IP address on this interface ? 10.0.0.57 What is the NETMASK on this interface ? 255.255.255.0 What is the IPV6 address ? 2001:408:4000:1408::5/64 The X1 IPV6 address is configured from the CNS using stateless autoconfiguration. Stateless S–2366–16 77

Cray Network Subsystem (CNS) Software Installation and Administration

autoconfiguration specifies ONLY the PREFIX portion of the IPV6 address. What is the IPV6 prefix for the X1 ? 2001:408:4000:1408::/64 lpfn0 configuration complete.

where 10.0.5.57 is the IP address and 255.255.255.0 is the NETMASK that you must change to be specific to your site in nnn.nnn.nnn.nnn decimal format. If you are using IPv6, 2001:408:4000:1408::5/64 is the IPv6 address and 2001:408:4000:1408::/64 is the prefix address for autoconfiguring the Cray X1 series system. HIPPI interface: Note: This is just an example. The values displayed on your system for IP address and NETMASK may be different.
Configure interface hip0 [ yes ]? yes Is this interface directly connected to the Cray [ yes ]? How is the IP acquired (none (static), dhcp) [ none ]? static What is the IP address on this interface [ 192.168.242.58 ]? What is the NETMASK on this interface [ 255.255.255.0 ]? hip0 configuration complete.

where 192.168.242.58 is the IP address and 255.255.255.0 is the NETMASK that you must change to be specific to your site in nnn.nnn.nnn.nnn decimal format, respectively. 3. At the unique CNS number prompt, enter a number between 0 and 19. Note: The [ 0 ] in this example only appears if a CNS number was assigned during a previous installation, and the number inside the bracket may differ according to your CNS number.
Each CNS must have a unique number. Please indicate the CNS number from 0-19 [ 0 ]? nn

where nn is number you assign to your CNS. This will also be part of the CNS name (for example, cns0 for CNS0, that you will use to access the CNS from the CWS or SWS. 4. Configure the gated daemon on or off.
Do you want to configure the gated daemon on [ no ]? no The Zebra daemon is used to configure dynamic routing for IPV6 78 S–2366–16

Advanced Configuration Topics [10]

Do you want to configure the zebra daemon on [ no ]? yes Copying existing static-routes file to config Copying existing static-arp file to config

5. If you are performing an initial software installation on a CNS in a Cray GigaRing system, use Section 10.5, page 85 to configure the private Ethernet. 10.3.7 Installing the Configuration After cns_gen_config completes, you must install the new configuration by entering (using cns0 for this example):
cns0# cns_config install

This is Step 4 in Section 8.2.1, page 33. 10.3.8 Backing Up and Restoring the CNS Configuration The cns_config command can be used to back up or restore a CNS configuration. 10.3.8.1 Backing Up the CNS Configuration Cray recommends backing up the CNS configuration prior to running the cns_gen_config command to change the configuration. To backup the CNS configuration, enter (using cns0 for this example):
cns0# cns_config backup file_name

The cns_config backup file_name command backs up the CNS configuration to the file you specify (for example, cns.backfile) in the current directory. Refer to the cns_config(8) man page for more information. 10.3.8.2 Restoring the CNS Configuration To restore the CNS configuration from a backup file, enter (using cns0 for this example):
cns0# cns_config restore file_name

This command distributes the contents of file_name to the various supported and accommodated configuration files. In addition, the command generates
S–2366–16 79

Cray Network Subsystem (CNS) Software Installation and Administration

a new /cray/etc/cns.conf configuration file based on the contents of file_name backup file. Refer to the cns_config(8) man page for more information. Under some circumstances, after backing up a CNS configuration and then restoring that configuration to a CNS running CNS 1.2 or later release software, the configuration might be restored incorrectly. This problem is caused by incorrect handling of the /etc/sysconfig/network-script/ifcfg-bond0 configuration file. The following command restores a CNS configuration from the OLDCNS file (using cns0 for this example):
cns0# cns_config restore OLDCNS

To ensure correct configuration, you must always run the following two commands after running the cns_config restore command:
cns0# cns_gen_config (answer the questions posed and use defaults as appropriate) cns0# cns_config install

10.4 Remote Access Controller (RAC) Maintenance Port Setup and Configuration
This section describes the setup and configuration of the Remote Access Controller (RAC), the standard CNS maintenance port. 10.4.1 Setting Up the CNS RAC Maintenance Port Follow these steps to set up the CNS RAC maintenance port: Note: This example uses cns0 to match the output for the RAC IP address, but you need to use the CNS number that matches the CNS on which you are working. 1. For Cray X1 series systems, physically connect the CNS RAC maintenance port via an Ethernet cable to the multicabinet switch located in the PC-20 cabinet, which is connected to the CWS port znb5 (Cray Private Administration Network). Refer to Field Change Order (FCO) 4043 for more information. For Cray GigaRing systems, physically connect the CNS RAC maintenance port via the Ethernet cable to a concentrator in a PC-10 cabinet, which is
80 S–2366–16

Advanced Configuration Topics [10]

connected to the SWS's qe0 or qfe0 port. Refer to FCO 4043 for more information. 2. Run the cns_rac firmware command. For more information about the RAC firmware upgrade command, refer to Section 9.6.1, page 54 and the cns_rac(8) man page.
cns0# cns_rac firmware cns_rac: Firmware version of RAC 2.10 cns_rac: Is different than supported version 3.12 Do you want to update the RAC firmware [ yes ]? yes ... cns_rac: racreset complete ... cns_rac: Firmware update complete ... Stopping pppd (RAC) services: [ OK ] cns0#

3. Set the RAC IP address for a CNS in a Cray X1 series environment or Cray GigaRing system environment by entering: Note: For visual help in determining the RAC IP address, refer to Section 10.4.2, page 82.
cns0# cns_rac setip

Output from this command used in a Cray X1 series system environment, for example, with cns0 is similar to:
cns0# cns_rac setip cns_rac: RAC IP address 10.0.109.101 for CNS0 will be set … cns_rac: RAC IP successfully set to 10.0.109.101 cns0#

Output from this command used in a Cray GigaRing system environment, for example, with cns0 is similar to:
cns0# cns_rac setip cns_rac: RAC IP address 10.1.124.240 for CNS0 will be set … cns_rac: RAC IP successfully set to 10.1.124.240 cns0#

S–2366–16

81

Cray Network Subsystem (CNS) Software Installation and Administration

4. Set the RAC password by entering:
cns0# cns_rac password [password] … cns_rac: setting RAC root login password complete cns0#

where password is your desired RAC password. 10.4.2 Configuring the CNS RAC Maintenance Port For Cray X1 series systems: The network portion of the CNS IP address (10.0.109.) and CNS number N determine the (static) IP address of the CNS on the private Ethernet network as 10.0.109.N+1. For example with CNS0, the CNS number is 0; the CNS name is cns0; and the IP address is 10.0.109.1. A CNS configured this way would be accessible from the CWS as 10.0.109.1 or cns0. The RAC IP address is based on the network portion of the CNS IP address (10.0.109.) and the CNS number (N) plus 101. For example, CNS0 (cns0) would have an IP address of 10.0.109.1, a RAC name of cns0-rac, and a RAC IP address of 10.0.109.101 (see Figure 8). The cns0 RAC would be accessible from the CWS as 10.0.109.101 or cns0-rac. Edit the /etc/hosts file on the CWS and manually enter the RAC name and its IP address to configure the RAC. It should look something like this:
# # CNS region # 10.0.109.101 10.0.109.102 10.0.109.103 10.0.109.104 10.0.109.105 10.0.109.106 10.0.109.107 10.0.109.108 10.0.109.109 10.0.109.110

cns0-rac cns1-rac cns2-rac cns3-rac cns4-rac cns5-rac cns6-rac cns7-rac cns8-rac cns9-rac

cns0-rac.cns cns1-rac.cns cns2-rac.cns cns3-rac.cns cns4-rac.cns cns5-rac.cns cns6-rac.cns cns7-rac.cns cns8-rac.cns cns9-rac.cns

82

S–2366–16

Advanced Configuration Topics [10]

Example: cnsN IP address.(N+101) = RAC IP address (where N = CNS number) For cnsN where N = 0, N+101 = 101 cns0 RAC IP address = 10.0.109.101

Your cnsN RAC IP address:

cns0-rac

10.0.104.1 Private Ethernet
CWS

to RAC CNS0 to CNS
RAC cns0-rac 10.0.109.101

telnet

Switch

[root]# connect video RAC to CNS

Serial line cnsN
N= CNS number

cns0 10.0.109.1

VGA+keyboard

Figure 8. Connecting to the RAC from the CWS

For Cray GigaRing systems: The network portion of the CNS IP address (10.1.124.) of the CNS and CNS number N, which is determined by bootp (see Figure 9 and Section 10.5, page 85), determines the CNS IP address on the private Ethernet eth0 network. Most private IP addresses on a SWS will be 10.1.124.N+200. Where N is a unique value in the bootp configuration. For example, for CNS0, where the CNS number is 0, the CNS name is cns0, CNS IP address might be 10.1.124.206 (from the bootp configuration). A CNS configured this way would be accessible from the SWS as 10.1.124.206 or cns0. The RAC IP address in a Cray GigaRing system environment is based on
S–2366–16 83

Cray Network Subsystem (CNS) Software Installation and Administration

the network portion of the CNS IP address and the CNS number plus 240 (10.1.124.N+240), where N is the actual CNS number, not the number in the bootp configuration used for the host portion of the CNS IP address. For example, CNS0 (0), which might be assigned an IP address of 10.1.124.206 (in the bootp configuration), would have a RAC name of cns0-rac and RAC IP address of 10.1.124.240 (see Figure 9). Edit the /etc/hosts file on the SWS and manually enter the RAC name and its IP address to configure the RAC. It should look something like this:
# # CNS region # 10.1.124.240 10.1.124.241 10.1.124.242 10.1.124.243 10.1.124.244 10.1.124.245 10.1.124.246 10.1.124.247 10.1.124.248 10.1.124.259

cns0-rac cns1-rac cns2-rac cns3-rac cns4-rac cns5-rac cns6-rac cns7-rac cns8-rac cns9-rac

cns0-rac.cns cns1-rac.cns cns2-rac.cns cns3-rac.cns cns4-rac.cns cns5-rac.cns cns6-rac.cns cns7-rac.cns cns8-rac.cns cns9-rac.cns

84

S–2366–16

Advanced Configuration Topics [10]

Example: cnsN IP address = usually 10.1.124.N + 200 (where N is determined by bootp configuration)

cnsN might be 10.1.124.206
cnsN RAC IP address = 10.1.124.N + 240 (where N is the CNS number you assign) For cnsN where N=0, CNS0 = cns0 cns0 RAC IP address = 10.1.124.240

Your cnsN RAC IP address:

Usually 10.0.124.200

to RAC CNS0 to CNS
Switch

Private Ethernet
SWS

telnet

Serial line cnsN
N=

RAC cns0-rac 10.1.124.240 cns0 10.1.124.206 (bootp)

[root]# connect video RAC to CNS

CNS number

VGA+keyboard

Figure 9. Connecting to the RAC from the SWS

At this point, you should be able to access the RAC from the CWS or SWS. For more information about logging in to the RAC, refer to Section 9.6.2, page 55.

10.5 Configuring Private Ethernet for Cray GigaRing Systems
Cray systems configured with Cray GigaRing I/O utilize bootp protocol to configure the private Ethernet. Use the following information to configure the private Ethernet connection.

S–2366–16

85

Cray Network Subsystem (CNS) Software Installation and Administration

Set the SWS to act as the bootp server for the CNS by executing this command:
sws# /opt/CYRIpkg/bin/swsbp

Note: This command leads you through the steps necessary to set up the SWS to act as a bootp server for the CNS. You can use Example 12 as a guideline to configure your private Ethernet connection. Example 12: Setting Up the SWS to Act as a bootp Server for the CNS A dialog similar to this takes place when using the swsbp command: Note: Substitute CNS and cns for L7R and l7r, respectively, throughout this example.
root = etcdir = /etc swsbp may change the contents of the following files: /etc/bootptab /etc/ethers /etc/hosts /etc/hosts.equiv /etc/hostname.qfe0 Do you wish to continue? Answer y or n [default is y]:

Press Return.
Enter private network name [default is "taccon-private"]: note: default is the "SWS's private ethernet host name"

Press Return if the default is correct, or enter the name of the SWS on the correct private Ethernet interface.
Enter the taccon-private IP address [default is 10.1.124.200]: note: default is IP address of the SWS on the private network

Press Return.
Enter the number of the task you would like to perform: 1. Make a new SPN entry in bootptab 2. Make a new MPN entry in bootptab 3. Make a new L7R entry in bootptab 4. Delete an existing bootptab entry 5. Modify an existing bootptab entry 6. Display the bootptab file 86 S–2366–16

Advanced Configuration Topics [10]

7. Additional setup for test bay 8. Exit # 3

Enter the L7R node name, example cray-l7r0 [no default]:

Note: Change the cns0 in cray-cns0 to be cnsN where N is the number of the CNS.
# cray-cns0 Enter the Cray L7R hardware address, for example, 00501c04c0ee [no default]: 000d56b8e32a

Note: This means the CNS hardware address, which is also referred to as the CNS MAC address. The hardware address needs to be entered without the colons and using hex numbers in lower case. You can display the hardware address using the output of the ifconfig eth0 command when logged into the CNS over the serial connection using the command:
tip -57600 /dev/ttya or tip -57600 /dev/ttyb

You can also use the RAC console; see Section 9.6.2, page 55 for more information. The hardware address is displayed in colon-delimited notation following the HWaddr term shown in the ifconfig eth0 output in this example:
Example: [root@cray-cns0 /root]# ifconfig eth0 Link encap:Ethernet HWaddr 00:0D:56:B8:E3:2A inet addr:10.1.124.206 Bcast:10.1.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:504 errors:0 dropped:0 overruns:0 frame:0 TX packets:259 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:37513 (36.6 Kb) TX bytes:18349 (17.9 Kb) Interrupt:28 Memory:fca10000-fca20000

eth0

The HWaddr output denotes the eth0 hardware address.

S–2366–16

87

Cray Network Subsystem (CNS) Software Installation and Administration

Enter the L7R IP address [default is 10.1.124.206 ]:

Press Return and choose to exit:
ADDING cns0 to /etc/bootptab ADDING 10.1.124.206 Cray-CNS-name to /etc/hosts ADDING cns3 to /etc/hosts.equiv Enter the number of the task you would like to perform: 1. Make a new SPN entry in bootptab 2. Make a new MPN entry in bootptab 3. Make a new L7R entry in bootptab 4. Delete an existing bootptab entry 5. Modify an existing bootptab entry 6. Display the bootptab file 7. Additional setup for test bay 8. Exit # 8

The private Ethernet connection should be functional now. To log in to the CNS from the SWS, you can telnet to the chosen host name, for example cns0.

88

S–2366–16

CNS Troubleshooting [11]

This chapter describes the basics needed to diagnose and resolve problems relating to the initial installation and configuration of the CNS. The troubleshooting approach presented here is broken down by different sections of the CNS network environments as shown in Figure 10, page 90 and Figure 11, page 91 and are related as follows: 1. What to do when the CNS is unable to boot (Section 11.1, page 91) 2. Accessing the CNS for troubleshooting (Section 11.2, page 92) 3. Troubleshooting CNS access to Cray mainframes (Section 11.3, page 93) 4. Troubleshooting CNS access to the site network (Section 11.4, page 95) 5. Troubleshooting remote host access to and from Cray mainframes (Section 11.5, page 97) 6. Controlling the tcp_assistd daemon for troubleshooting (Section 11.6, page 98) For situations where it is necessary to disable the tcp_assistd daemon, see Section 11.6, page 98.

S–2366–16

89

Cray Network Subsystem (CNS) Software Installation and Administration

Cray X1 series mainframe

CWS

Cray mainframe network

CNS

host 1

host 2

host ...

Key External Ethernet Private Ethernet Serial line (RS232) Fibre Channel

Figure 10. The CNS with Ethernet Connections in a Cray X1 Series Environment

90

S–2366–16

CNS Troubleshooting [11]

SWS GigaRing mainframe Cray mainframe network

CNS

host 1

host 2

host ...

Key External Ethernet Private Ethernet Serial line (RS232) HIPPI

Figure 11. The CNS with HIPPI Connections in a Cray GigaRing System Environment

11.1 What to Do When the CNS is Unable to Boot
If you are unable to boot the CNS: 1. Gain access to the CNS serial console or VGA monitor (refer to Section 11.2, page 92 for details). 2. Power-cycle the CNS using the CNS Power button. 3. Observe the screen output for clues to the problem.
S–2366–16 91

Cray Network Subsystem (CNS) Software Installation and Administration

4. Contact Cray Support if you are still unable to boot the CNS. An alternate method is to access the RAC (Section 9.6.2, page 55) and the CNS console as described in Section 9.6.3, page 55.

11.2 Accessing the CNS For Troubleshooting
Access to the CNS from the Cray Workstation (CWS) or Cray system workstation (SWS) is with both serial line and over the private Ethernet network, including access through the RAC. The CNS ships with a serial cable. Use this cable to connect the CWS or SWS to the CNS. On the CNS use port 1 for the connector marked with the |O|O| symbol. Use the connector marked: • serial on the back of the CWS • B on the back of the SWS To establish a serial connection from the CWS to the CNS:
cws$ tip -57600 /dev/ttya

To establish a serial connection from the SWS to the CNS:
sws$ tip -57600 /dev/ttyb

The CNS eth0 interface connects to the CWS or SWS over a private Ethernet network as follows: • For the CWS, the Private Administration subnetwork uses the CWS znb5 interface. Refer to the section I/O Networks in Cray X1 Series System Configuration and CWS Administration for details about the Private Administration subnetwork. • For the SWS, the private Ethernet interface is qex or qfex, where x is a number 0 through 4. If the private Ethernet on the CNS is connected properly, the following message logs in /var/log/messages on the CNS and the /opt/craylog/ops.log file on the CWS (for earlier CWS releases they were kept in /opt/craylog/cns.log):
eth0 NIC link is up, 100 Mbps full duplex

Note: You will not see the message on the SWS.
92 S–2366–16

CNS Troubleshooting [11]

If the private Ethernet on the CNS is not connected properly, the following message displays; use the serial connection from the CWS or SWS to the CNS to further diagnose the problem:
eth0 NIC link is down

The CNS RAC maintenance port interface connects to the CWS or SWS over a private Ethernet network (Figure 8, page 83 or Figure 9, page 85) and the RAC can be used to access the CNS with the connnect video or connect com1 command Section 9.6.3, page 55). A RAC Ethernet cable is shipped with the CNS or provided as described in Field Change Order (FCO) 4043.

11.3 Troubleshooting CNS Access to Cray Mainframes
This section describes how to diagnose problems with the connection that the CNS uses to access and communicate with the Cray mainframe. For Cray X1 series systems, which use a Fibre Channel connection to the Cray network, see Section 11.3.1, page 93. For other supported Cray systems, see Section 11.3.2, page 95.

Cray mainframe Cray mainframe network

CNS

Figure 12. CNS Access to the Cray Mainframe

11.3.1 Troubleshooting CNS Fibre Channel Connections When troubleshooting the Fibre Channel link to the Cray X1 series mainframe, examine the /var/log/messages file on the Cray X1 series system. If the link

S–2366–16

93

Cray Network Subsystem (CNS) Software Installation and Administration

is established and the loop is up, you will find messages similar to those shown in Example 13. Example 13: Fibre Channel Link Messages in /var/log/messages on the Cray X1 Series System
Feb 17 00:12:03 6A:sn702 unix: CPU 0VN0S4 (0x4) : qfa0: qlogic 2312, port name 0x100000e08b070ebd Feb 17 00:12:03 6A:sn702 unix: CPU 0VN0S6 (0x6) : qfa0: LIP detected Feb 17 00:12:04 6A:sn702 unix: CPU 0VN0S6 (0x6) : qfa0: LOOP UP detected Feb 17 00:12:04 6A:sn702 unix: CPU 0VN0S6 (0xb) : qfa0: Data rate is 2Gbps

To test the link, execute these commands from the CNS: Note: This example uses cns0; your CNS number may differ.
cns0# cns0# cns0# cns0# cns0# cns0# cns0# cns0# ifdown lpfn0 ifdown lpfn1 rmmod lpfndd rmmod ipfcdd insmod lpfcdd insmod lpfndd ifup lpfn0 ifup lpfn1

To interpret the results, execute this command on the /var/log/messages file on the Cray X1 series system:
$ tail -f /var/log/messages

If messages similar to those shown in Example 13, page 94 do not occur, there is a problem with the Fibre Channel connection between the Cray X1 series mainframe and the CNS. If you do find messages similar to those shown in Example 13, page 94, the Fibre Channel connection is up. Once the IP addresses have been chosen, you must establish IP communication. The IP network for the Fibre Channel link will have to be on a separate subnetwork than the Gigabit Ethernet or HIPPI networks, which are connected to the site networks. See Section 8.2, page 33 for information about configuring the Fibre Channel network. When the Fibre Channel network is configured and the connection is up, you can use the ping command to verify IP communication. You can use the netstat -ia command to check for errors of the Fibre Channel interface (LPNsx), where x is the ordinal number of the interface.
94 S–2366–16

CNS Troubleshooting [11]

You can use the ifconfig command to check the status of the Fibre Channel interface. 11.3.2 Troubleshooting CNS HIPPI Connections In order for the CNS to function properly, the HIPPI connection must be operational such that the CNS can communicate using TCP/IP to and from the Cray mainframe. For details about HIPPI configurations, see Appendix B, page 105. Executing one of the following commands on the CNS terminates or initializes the HIPPI interface, respectively:
ifdown hip0 (terminates) or ifup hip0 (initializes)

To re-initialize the HIPPI interface, execute the following sequence of commands:
ifdown hip0 ifup hip0

When the HIPPI interface is configured, you can use the ping command to verify IP communication occurs between the CNS and the Cray mainframe. You can use the netstat -ia command to check for errors on the HIPPI interface. You can use the ifconfig command to check the status of the HIPPI interface.

11.4 Troubleshooting CNS Access to Site Networks
This section describes how to diagnose problems with the connection that the CNS uses to access and communicate with the site network via one of its Ethernet connections.

S–2366–16

95

Cray Network Subsystem (CNS) Software Installation and Administration

CNS

host 1

host 2

host ...

Figure 13. CNS Access to Hosts

From the CNS perspective, this involves the following configuration information for the ethN interface (see Figure 2, page 30):
ethN

where N identifies the interface number. To terminate the specified Ethernet interface, execute the following command on the CNS:
ifdown ethN

where N is the desired interface number. The Gigabit Ethernet interface is usually eth2. To initialize the specified Ethernet interface, execute the following command on the CNS:
ifup ethN

where N is the desired interface number. To re-initialize the interface, execute the following sequence of commands:
ifdown ethN ifup ethN

where N is the desired interface number.
96 S–2366–16

CNS Troubleshooting [11]

When the Ethernet interface is configured, you can use the ping command to verify IP communication occurs between the CNS and the host on a site network. When the Ethernet interface is configured, you can use the ping command to verify IPv4 communication occurs between the CNS and the host on a site network. For IPv6, use the ping6 command to verify communication occurs between the CNS and the host on the site network. You can use the netstat -ia command to check for errors on the host interface. You can use the ifconfig command to check the status of the host interface.

11.5 Troubleshooting Remote Host Access to and from Cray Mainframes
This section describes how to diagnose problems with the routing necessary for remote host systems to utilize the CNS to access or communicate with Cray mainframe(s).

Cray mainframe

CNS

host 1

host 2

host ...

Figure 14. Routing to Utilize the CNS

S–2366–16

97

Cray Network Subsystem (CNS) Software Installation and Administration

The CNS functions as a router, so the mainframe site must choose how the routing is configured on the CNS. The choices are static, dynamic, or a combination of static and dynamic routing. For more information on configurations for routing, refer to Section 10.3, page 60. If you suspect a routing problem, you can use the route command to check the routing table of the CNS. Use route -A inet6 -n to check the IPv6 routes. You can use the traceroute command to verify that routes are configured properly for IPv4. Use the traceroute6 command to verify IPv6 routes.

11.6 Controlling the tcp_assistd Daemon
The tcp_assistd daemon is a key feature of the CNS and improves TCP performance of the Cray mainframe. However, it is sometimes necessary to disable the tcp_assistd daemon, for example, when debugging and troubleshooting. Note: This example uses cns0; your CNS number may differ. To stop the tcp_assistd daemon:
cns0# tcp_assist stop

To start the tcp_assistd daemon:
cns0# tcp_assist start

98

S–2366–16

CNS Hardware Connections [A]

This appendix describes the CNS hardware. The CNS hardware you receive from Cray depends on the type of Cray (host) interface and the outbound (network) interface your environment uses. See Section A.1, page 99 for the CNS (Dell 1650), Section A.2, page 101 for the CNS (Dell 2650), and Section A.3, page 102 for the CNS (Dell 2850).

A.1 CNS (Dell 1650)
Figure 15 and Figure 16 are front and rear views of the CNS (Dell 1650) showing connections, controls, and indicators.

Power indicator eth0 eth1 Disk activity indicator

On-board Ethernet activity indicators

Cover latch Keyboard/ mouse Power button/light CD tray Diskette drive Hard disk drive Video monitor

Figure 15. CNS Front View (Dell 1650)

S–2366–16

99

Cray Network Subsystem (CNS) Software Installation and Administration

eth2
SERIAL HPPI 850mm SCR. OUT DATA LINK

Server management port Remote Access Controller (RAC) Ethernet
DEST. IN

eth1 eth0 Serial port

Ipfn0

Keyboard Mouse
Figure 16. CNS Rear View (Dell 1650)

Video monitor

Power

Controls and indicators on the CNS (Dell 1650) have the following characteristics: Power Button/Light The power button is used to turn power on and off. The power light, which is contained within the button, has the following meanings when on: Blinking Solid Power Indicator The power indicator has the following meanings when on: Amber and Blinking The CNS (Dell 1650) is either booting or running. Blue There is power to the CNS (Dell 1650), but the unit is not powered up. There is power to the CNS (Dell 1650), but the unit is not powered up. The CNS (Dell 1650) is either booting or running.

Ethernet Indicators (eth0, eth1) Indicate the state of the onboard Ethernet port when on. Amber Green Blinking Green There is Ethernet activity on this port.
100 S–2366–16

There is no Ethernet connectivity on this port. There is Ethernet connectivity on this port.

CNS Hardware Connections [A]

Disk Activity Indicator If this indicator is blinking, there is activity on the CNS disk drive.

A.2 CNS (Dell 2650)
Figure 17 and Figure 18 are front and rear views of the CNS (Dell 2650) showing connections, controls, and indicators.

Hard disk drive

Video monitor Keyboard/mouse

Display

Diskette drive

CD tray

On-board Ethernet eth0 activity indicators eth1

Power button

Figure 17. CNS Front View (Dell 2650)

S–2366–16

101

Cray Network Subsystem (CNS) Software Installation and Administration

Gigabit Ethernet interface (eth2)

Fibre Channel (dual port)

Power

eth1 eth0

Serial line

ard Video Mouse

Remote Access Controller (RAC) Ethernet port
Figure 18. CNS Rear View (Dell 2650)

Controls and indicators on the CNS (Dell 2650) have the following characteristics: Ethernet Indicators (eth0, eth1) Indicates the state of the onboard Ethernet port. Green Off Power Button The power button is used to turn power on and off. Display Displays internal information. There is Ethernet connectivity on the port. The Ethernet is not connected.

A.3 CNS (Dell 2850)
Figure 19 and Figure 20 are front and rear views of the CNS (Dell 2850) showing connections, controls, and indicators.

102

S–2366–16

CNS Hardware Connections [A]

USB ports

Video

On-board Ethernet activityindicators eth0 Diskette drive eth1

CD tray

Empty

Display Power button
Figure 19. CNS Front View (Dell 2850)

Power

Remote Access Controller (RAC) ethernet port Serial port Video port Keyboard

Mouse eth1 eth0

USB ports

Figure 20. CNS Rear View (Dell 2850)

Controls and indicators on the CNS (Dell 2850) have the following characteristics: Ethernet Indicators (eth0, eth1) Indicates the state of the onboard Ethernet port. Green Off There is Ethernet connectivity on the port. The Ethernet is not connected.

S–2366–16

103

Cray Network Subsystem (CNS) Software Installation and Administration

Power Button The power button is used to turn power on and off. Display Displays internal information.

104

S–2366–16

HIPPI Network Configuration [B]

The following information is required to complete the configuration of the HIPPI network for the CNS on GigaRing systems: • Host name or IP address and I-fields for all Cray mainframes on the HIPPI network. This is the IP address that the CNS will use for the HIPPI interface (for example, hip0). Obtain this information from the site network administrator. Note: If the CNS and Cray mainframe are connected through a HIPPI modem, the I-fields can be set to 00:00:00:00:00:00. • Network Mask This is the mask that defines the bits of the IP address that identifies the network portion of this address. Obtain this information from the site network administrator.

B.1 Requirements
To configure the CNS HIPPI network, the following must be available: • The physical HIPPI hardware must be installed and operational. • An IP network must be defined and all hosts that exist on this network must have assigned, unique IP addresses. • All I-fields must be identified for each IP address and host name for each host on the HIPPI network.

B.2 Instructions for Configuring the CNS HIPPI Interface
The following instructions explain how to configure the CNS HIPPI interface (hip0). 1. Perform a telnet to the CNS and log in as root. 2. Cray does not support dynamic Address Resolution Protocol (ARP) for HIPPI connections. After running the cns_gen_config command, you must run the

S–2366–16

105

Cray Network Subsystem (CNS) Software Installation and Administration

cns_config install command to write the configuration. To activate the configuration, you must reboot the CNS. Note: (Deferred implementation) Currently, the cns_gen_config command does not have the ability to generate static ARP for HIPPI, so the prompts for this do not occur. Until cns_gen_config is capable of generating static ARP for HIPPI, edit the /cray/etc/cns.conf file to set the static ARP entries for HIPPI by adding the ARP entries just prior to the line that contains:
END%ARP

Each line in the file defines one host on the HIPPI network and must have the following format:
hip0 hippi IP address or hostname 00:00:I-field

The first token, hip0, defines the HIPPI interface to which the ARP entry will be applied. The second token, hippi, is a constant used for association of this entry to the CNS HIPPI interface. The third token is the IP address or host name associated with the Cray mainframe's HIPPI interface or site HIPPI-connected host. The fourth token is the I-field that the CNS is to use to route packets through the HIPPI hardware to the indicated Cray mainframe (00:00: prefix is required). An example for the site HIPPI-connected host is:
hip0 hippi crayhost1-hippi hip0 hippi crayhost2-hippi 00:00:01:00:00:04 00:00:01:00:00:42

An example for the CNS HIPPI connected to the Cray mainframe via a HIPPI modem is:
hip0 hippi crayhost-1 00:00:00:00:00:00

The host names (crayhost1-hippi and crayhost2-hippi) identify Cray mainframes attached via HIPPI. The I-fields (01:00:00:04, 01:00:00:42, and 00:00:00:00) are used to route packets through the HIPPI hardware when sending packets to the respective Cray mainframe.

106

S–2366–16

HIPPI Network Configuration [B]

The components that get configured by the cns_gen_config command are described below:
DEVICE=hip0 IPADDR=ip address NETMASK=netmask MTU=30000 ONBOOT=yes BOOTPROTO=none

DEVICE=hip0 Identifies the name of the physical card in the CNS that will be used to access the HIPPI network. IPADDR=IP address Identifies the CNS's IP address on the HIPPI network. NETMASK=netmask Identifies the HIPPI network's netmask. MTU=30000 Identifies the maximum transmission unit (MTU) of the HIPPI network, as viewed from the CNS. Note: MTU values larger than 30000 are not supported. ONBOOT=yes Indicates that this interface is to be initialized during CNS initialization (during the boot process).

BOOTPROTO=none Indicates that this interface is defined statically. Note: The 30000 byte maximum transmission unit (MTU) does not require changes to the MTU chosen for use by the Cray mainframe. However, all UDP applications (for example, NFS) that will use the CNS as their gateway must be configured to use a message size of 30000 bytes or less. These applications can still use up to the Cray mainframe's HIPPI MTU value when going Cray mainframe to Cray mainframe over HIPPI. The MTU on the hip0 interface is set to 30000, as this appears to provide the best throughput to the Cray mainframe going through the CNS. The 30000 byte MTU is the only supported value. It was chosen based on the results obtained during performance testing where the MTU was varied between 16K and 64K.
S–2366–16 107

Cray Network Subsystem (CNS) Software Installation and Administration

3. Initialize the CNS HIPPI interface. Execute the following command:
ifup hip0

Note: The definition of the CNS's own I-field in the static-arp file might produce an error message, because loopback over the physical HIPPI interface is not available. The error message indicates that the I-field definition is not a valid MAC address for the loopback interface. B.2.1 Configuration Requirements for the Cray Mainframe for GigaRing Systems The Cray GigaRing mainframe HIPPI arp file (/etc/ghippi.arp) must be updated to include the mapping of the CNS host name to its associated I-field. This is required for the Cray mainframe to correctly route IP packets, which are destined for the CNS HIPPI IP address through the HIPPI network. For information about how to update the Cray mainframe's HIPPI arp file, see UNICOS Networking Facilities Administrator's Guide or UNICOS/mk Networking Facilities Administration, depending on your operating system.

B.3 Verification
At this point, the CNS should be able to communicate with the Cray mainframe using TCP/IP. You can use the ping command, from the Cray mainframe or the CNS, to verify that IP access exists between them.

108

S–2366–16

Index

A Accessing Cray documentation, 15 Account crayadm, 52 root, 50 Address resolution protocol (ARP) configuring a proxy, 66 arp file, 108 Automatic failover configuration, 71 description, 48 B Backup, 79 bfc command, 17 bfc man page, 17 bondedfibrechannel man page, 17 Bonding configuration, 71 Fibre Channel IP description, 48 Books, 16 accessing, 15 BOOTPROTO, defining, 107 C CERT advisories, 54 CIT documentation, 16 CNS comparison to Cray L7R, 7 compatibility, system, 2 configuration backup, 79 configuration, restore, 79 configuring HIPPI interface, 105 definition, 3 description, 2 features, 45 firmware, 21
S–2366–16

hardware, 99 host interface, 19 network configuration requirements, 60 networking interface compatibility, 19 platform description Dell 1650, 99 Dell 2650, 101 Dell 2850, 102 purpose, 45 security, 49–50, 53 software, 21 specialized function, 2, 45 terminology, 3 cns.conf file, 11, 60 cns_config backup command backup, 79 cns_config restore command restore, 79 cns_gen_config command, 33 static routes, 60, 106 cns_rac command, 52, 54–55 Compatibilities, 7 networking interface, 19 system, 2 Configuration automatic failover, 71 backup, 79 bonded interface, 71 CNS HIPPI interface, 105 CNS interface for FSS, 73 cns_gen_config, 33 default route to Cray X1 series mainframe, 64 default route to GigaRing mainframe, 65 dynamic routing, 66 gated.conf default, 61 generating, 33 new CNS, 33 NTP, 59

109

Cray Network Subsystem (CNS) Software Installation and Administration

proxy ARP, 66 restore, 79 reverting to prior, 42 routing, 35, 60 site network routers, 66 static default route to CNS, 60 static-route.options default, connect command, 56 console, 57 Contact information Customer Support Center, 25 Software Distribution Center, 17 Training, 26 Cray L7R, 7 Cray Network Subsystem (CNS) See CNS Cray Service Bulletin, 26–27 Cray websites, 27 CRInform, 25 publications, 15 support, 25 training, 26 /cray/etc/cns.conf file, 11, 60 CRInform, 25 CRSB, 27 Customer services, 25 Customer Support Center, 25 Customs, 22 cwsroute command, 43 D Device for HIPPI access, defining, Differences, 7 Distribution Center, 17, 23 Documentation, 15–16 accessing, 15 Dynamic routes, 61 E Enhancements, 5 Errata, 16 /etc/gated.conf file,
110

Ethernet interface initialize, 96 re-initialize, 96 terminate, 96 ethN interface, 96 Export license, 22 64 F Failover configuration, 71 description, 48 FCO See Field Change Order Features, 45 Fibre Channel configuring a bonded interface, 71 IP bonding description, 48 mainframe connection, 45 Field Change Order 4043, 21, 32, 80–81, 93 Field notices (FNs), 26 File system server (FSS), 73 G gated daemon,

61

107

H Hard copy, 16 Hardware, 99 HIPPI interface, terminate or initialize, mainframe connection, 45 HTML, 16 I I-field, 106 ifconfig command, 95, 97 ifdown command, 94–96 ifup command, 94, 96 Installation software initial procedure, 39 software upgrade procedure, 35 IP address, defining, 107

95

61

S–2366–16

Index

iptables, 53 L Letter of assurance, 22 Licensing CNS for Cray X1 series systems, 22 CNS for GigaRing systems, 22 Limitations, 11 after power failure, 13 cns_config, 11 cns_gen_config, 11 PCI slot Ethernet card naming, 12 Proxy ARP not supported for IPv6, 14 tcp_assist daemon not supported in IPv6, 14 tcp_assistd, 12–13 temporary directory cleanup, 14 Logging, 54 M Maintenance port, 54 setup and configuration, 80 Maintenance port, CNS configuration for Cray X1 series systems, 82 configuration for GigaRing systems, 83 setup, 80 Man page collections, 16 Man pages accessing, 15 Maximum transmission unit (MTU), setting, 107 N netstat command, 94–95, 97 Network defining MTU, 107 defining subnet mask, 107 restrictions for CNS, 60 Network Time Protocol (NTP), 59 NTP See Network Time Protocol (NTP) ntpd daemon, 59

O ONBOOT, defining, 107 Ordering documentation, 17 software, 22 P password RAC root, 52 Password changing RAC root, 55 crayadm, 52 initial, 34–35, 41 resetting, 51 root, 50 PDF, 16 ping command, 94–95, 97 Problems, 25 Proxy ARP, configuring, 66 Publications, 15–16 accessing, 15 R racadm command, 54 Release package, 19, 21 Remote Access Controller (RAC), 54 setup and configuration, 80 subcommands, 56 Remote Access Controller (RAC), CNS configuration for Cray X1 series systems, 82 configuration for GigaRing systems, 83 setup, 80 Request for Technical Assistance (RTA), 25–26 Requirements for CNS network configuration restrictions, 60 mainframe, 19 Requirements for hardware and software, 19 Restore, 79 Reverting to prior configuration, 42 route command, 98 Routing, 60
111

S–2366–16

Cray Network Subsystem (CNS) Software Installation and Administration

configuring, 35 S Security, 49–50, 53 CERT advisories, 54 firewall, packet filtering (iptables or ip6tables), 53 firewall, packet filtering (iptables), 53 Linux kernel, 54 private Ethernet network, 53 serveraction command, 58 Shell, using, 43 Shipping, 23 Software enhancements, 5 Software Problem Report (SPR), 25–26 Static IP addressing, 60 static-route.options file, 64 Subscriber, CRInform, 25 Support, 22 Support agreement, 21–22, 25 Support Center, 25 Synchronization, 59 T tcp_assist command, 12

tcp_assistd daemon, 98 dead connections, 13 side effects, 12 Technical support, 25 Timekeeping, 59 traceoptions parameter, 61 traceroute command, 98 traceroute6 command, 98 Training, 26 Troubleshooting, 89 access to mainframes, 93 access to site networks, 95 Ethernet connections, 95 routing, 97 U Upgrade software, procedure, 35 support, 7 Upgrades, 21 V /var/log/messages file,

93

112

S–2366–16

Sign up to vote on this title
UsefulNot useful