ES400 - Ultra Enterprise 10000 Administration - Oh - 0698

Sun Educational Services
Ultra Enterprise 10000

Administration
ES-400
Sun Cluster High Availability Administration June 1998

About This Course

This course is designed introduce students to the Ultra™
Enterprise™ 10000 system. It will explain the capabilities and
configuration of the system; show how to load the software,
discuss the operation and management of the system, the
configuration, and use of its special capabilities; and how to
troubleshoot failures.
The course is intended for experienced system administrators
and hardware specialists with a thorough background in
Sun™ SPARC™ system administration or Sun hardware
maintenance.
Ultra Enterprise 10000 Administration Preface 1 of 10

Copyright 1998 Sun Microsystems, Inc. All Rights Reserved. SunService June 1998
Course Map

Module-by-Module
Overview
• Ultra Enterprise 10000 Capabilities
and Features
• Architecture Overview
• SSP Software Installation
• System Operation
• Domains
• Installing Solaris in a Host Domain
• System Boot Process
• Alternate Pathing
• Dynamic Reconfiguration
• Diagnostic Information

Appendices
• Configuring NTP
• OBP Device Aliases

Course Objectives
• Describe the differences between the Enterprise 10000
and other Sun server systems.
• Describe the features of the Enterprise 10000 system.

• Install and configure the SSP.
• Obtain status and configuration data about the
Enterprise 10000 from the command line and by using
Hostview.
• Control the Enterprise 10000 system from the command
line and from Hostview.
• Load Solaris™ into an Enterprise 10000 domain.

Course Objectives
• Describe the boot flow for an Enterprise 10000 domain.
• Configure, test, and boot an Enterprise 10000 domain.
• Describe the system configuration requirements for
Dynamic Reconfiguration and Alternate Pathing.
• Perform initial troubleshooting and error condition
identification for the Enterprise 10000 and its domains.
• Perform Dynamic Reconfiguration operations from the
command line and from Hostview, and understand
their restrictions and constraints.
• Perform disk and network Alternate Pathing operations
and understand their restrictions and constraints.
Topics Not Covered

• Network adminstration
• General Solaris problem resolution
• Data center operations
• System tuning
• Disk storage management

Course Prerequisites
• Basic Sun system administration as demonstrated in
Solaris 2.X System Administration Essentials (SA-135), and
Solaris 2.X System Administration (SA-285)
• Basic Sun network administration as demonstrated in

Solaris 2.X Network Administration (SA-380)
• Basic Sun troubleshooting and boot process knowledge

as demonstrated in Solaris Fault Analysis Workshop
(ST-350)
• Basic organization and operation of the SPARCcenter™

2000, SPARCserver™ 1000, Ultra Enterprise 3000, 4000,
5000, and 6000, and storage arrays

Introductions
1. Who are you?
2. What do you do?
3. What is your background and experience as it relates
to this course?
4. What do you hope to accomplish this week?

How to Use the Course Materials

• Course Map
• Relevance
• Additional Resources
• Overhead Image
• Lecture
• Exercise
• Check Your Progress
• Think Beyond

Module 1

Capabilities and Features
Ultra Enterprise 10000 Administration June 1998

Module Overview
• Course map
• Relevance
• Objectives
Ultra Enterprise 10000 Administration Module 1, slide 2 of 25

Course Map

Objectives
• Explain the capabilities and features of the Enterprise
10000 system.
• List the hardware and software components of the

Enterprise 10000 system.
• Describe the Enterprise 10000 system packaging.
• Explain the special features of the Enterprise 10000
system, such as domains, Alternate Pathing and
Dynamic Reconfiguration.
• Describe the concepts of system operation.

Ultra Enterprise 1000 Capabilities

• Solaris 2.5.1 and 2.6 compatible
• Bus clock of 83.3 MHz, with 250 and 333 MHz

processors
• Data and address buses protected by ECC
• I/O flexibility and high I/O bandwidth; SBus and PCI
• No single points of hardware failure
• Can run multiple OS copies simultaneously – domains
• Hot swap boards while the OS is running – DR
• Dual I/O patch support – AP

Ultra Enterprise 1000 Features

• System Service Processor (SSP)
• Network Virtual Console (netcon)

• Hostview Graphical User Interface (GUI)
• POST (Power On Self Test)
• Blacklisting

System Cabinet
System service
processor (SSP)
Access panel
Styling panel
(I/O expansion cabinet not shown) Processor cabinet

(includes disk trays)

System Domains
SB3 SB10
SB2
SB1 SB6 SB9
SB0 SB5 SB8
SB4 SB7
domain_a domain_b domain_c
Enterprise 10000 platform

system disk system disk system disk

Alternate Pathing
Domain1
System Board 1 System Board 3
Alternate paths
Active paths
Ethernet
A5000 or SSA

Dynamic Reconfiguration
• Allows the addition or removal of a system board from
a running domain
• You can use Dynamic Reconfiguration to:
• Upgrade a system board
• Modify the domain hardware configuration

• Repair a faulty system board
• Run test or temporary systems
• Controlled by the SSP

Operating System Support

• Solaris 2.6 HW 5/98 is pre-installed in one domain
• A domain has kernel architecture sun4u1
• Supported like any other architecture
• The virtual console daemon cvcd is the only boot

change
• Standard Solaris 2.5.1 4/97 or higher is installed on the
SSP
• SSP software will not run on 2.6

Operating System Enhancements

• Resource management – Processor groups, allowing
grouped processors to be given specific attributes
• Parallel processing – HPC support, providing extended
parallel programming execution support
• Memory management – Support for 64 Gbytes of
physical memory
• Added reliability, availability and serviceability (RAS)

The SSP
• SPARC 5 with 64 Mbytes of main storage and a 1-Gbyte
disk
• Responsible for controlling and managing the

Enterprise 10000
• Supports booting, error logging, environmental
monitoring
• Line mode and graphical command interfaces
• Configures and controls domains
• Extensive status displays and error notification
• Dedicated exclusively to the Enterprise 10000
SSP Logical Connectivity
Cu
sto
me
rE
the
rne
t
et SP
ern d ard S
s Eth boar rol bo To
u l t
SB ontro l con
To c a
To option
To
t SP
up por nal S
te s optio
mo To
Re
Public-switched System console (SSP)

network
Transceiver
(optional)
Telephone cable
DTE
Optional second SSP
Modem
DTE

The SSP User Environment

SSP Window
SSP or Other Workstation Display
SSP or Other Workstation Display
SSP Window
% rlogin ssp \
-l ssp
SSP window SSP
SSP
% rlogin ssp \ system
-1 ssp
• Used to manage the SSP itself

• Just normal X-Windows windows

Network Console Window
Logical Connection
Enterprise 10000
Logical Connection Platform
netcon (1M) Windows
Domain 1
% setenv SUNW_HOSTNAME domain1 SSP
% netcon system Network
% setenv SUNW_HOSTNAME domain2 Domain 2

% netcon
• netcon windows act as system consoles for domains

• Only one at a time per domain can enter commands


Hostview
Logical Connection Enterprise 10000 Platform
Logical Connection
Enterprise 10000
Logical Connection Platform
Logical Connection Domain 1
Hostview
Domain 1
Hostview
SSP SSP
Network Network
SSP Window
SSP Window Domain 2
% hostview
% hostview
Domain 2
• Hostview provides a graphical control enviroment for

the entire Enterprise 10000
• It can perform almost all administration tasks
Hardware Configuration Control

• The blacklist controls what hardware is available to a
domain
• Components are not configured as available

regardless of their state
• Figure of merit processing allows optimum

configuration with missing or disabled components
• Chooses best configuration from 45 possible

Diagnostic and Monitoring Tools

• bringup – Catches repeatable hard errors, logging all
failures to a system or domain log file on the SSP
• POST (Power On Self Test) – Exercises system logic on a
subcomponent level, ensuring the domain is ready to
operate
• OBP (OpenBoot™ PROM) – Records the domain
configuration and boots the operating system like a
regular Sun server OBP

Status Monitoring and Display

• SSP
• The SSP constantly actively monitors the state of the

system and all internal components
• SunVTS™
• SunVTS stress tests the system from the OS level
• redx
• An interactive hardware debugger for internal use

only

Resiliency Features
• DC power – Converters on every system board
• System boards – Can be added and removed while the
OS is running
• Processors – Bad CPUs can be configured out
• Memory – Each system board controls its own memory
• I/O interface subsystem – Interface slots can be
configured out
• AP can help you work around these failures

Redundant Components
All components can be redundantly configured:
• System boards
• Control boards
• Centerplane support boards
• Disk storage
• Bulk power subsystems and supplies
• Peripheral controllers and channels
• SSP and its interfaces

Concurrent Serviceability
• Most components can be replaced on-line
• Control boards require domain shutdowns
• Thorough logging helps isolate failures quickly
• System boards can be removed from a running system
• System boards can be powered off independently
• Boards can be functionally tested independently

Error Logging
• The SSP is constantly monitoring the Enterprise 10000
system
• Both transient and hard errors are logged to the SSP

• The SSP is constantly polling the control boards for
errors
• Domain failures will be logged and the domain
rebooted
• Serious hardware errors will cause a logout to the SSP of
the hardware’s internal state

Check Your Progress

• Explain the capabilities and features of the Enterprise
10000 system.
• List the hardware and software components of the

Enterprise 10000 system.
• Describe the Enterprise 10000 system packaging.
• Explain the special features of the Enterprise 10000
system, such as domains, Alternate Pathing, and
Dynamic Reconfiguration.
• Describe the concepts of system operation.

Module 2
Architecture Overview

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Describe the construction of the Enterprise 10000
• Describe the system board interface
• Explain the system board structure
• Recount centerplane operation
• Explain the control board structure
• Explain component packaging
• Understand the function of JTAG
• Describe the SSP interaction with the Enterprise 10000

Enterprise 10000 Packaging

• Sixteen system boards, with four CPUs, one I/O
module and one memory module
• One or two control boards
• Dual centerplane support boards
• An active centerplane with a 144-bit-wide, point-to-
point data interconnect and four address buses
• Eight bulk power supplies to provide 48-volt DC
• Sixteen hot swappable fan trays holding 32 fans

Enterprise 10000
Component List
Quantity per
Component Function
System
Centerplane Contains address and data interconnect to all 1 (2 logical
system boards halves)
Centerplane Provides the centerplane JTAG, clock, and control 2
support board functions
System board Contains processors, memory, I/O subsystem, Up to 16
SBus and PCI boards, and power converters
Processor Mezzanine boards that contain the UltraSPARC Up to 64
modules processor and support chips
Memory Removable DIMMs (Dual In-line Memory Up to 16
Module)
I/O Removable SBus boards Up to 64
Control board Controls the system JTAG, clock, fan, power and Up to 2
Ethernet interface functions
48-volt power system
AC input Receives 220-volt AC, monitors it, and passes it to Up to 4
module the power supplies
48-volt power Converts AC power to 48-volt DC 5 or 8
supply
Circuit breaker Interrupts power to various components within 1
panel the system
AC power Receives 220-volt AC, monitors it, and passes it to 1 or more
sequencer the peripherals
Peripheral Converts AC power to DC for peripherals (In peripheral
power supply cabinet)
Remote power Connects the remote control line between two 1
control module control boards and passes it to a master AC
sequencer
Fan centerplane Provides power to the pluggable fan trays 2
Fan trays Each fan tray contains two fans for system cooling 5 to 8 pairs

Component Locations
The back has the lower-numbered
components
The front has the higher numbered
components
• AC – AC input control modules
• CB – Control Board
• CSB – Centerplane Support Board
• FT– Fan tray
• PDU – Input power sequencing and
distribution unit
• PS – AC power supply
• RPC – Remote power control
connectors
• SB – System board

Component Locations
PS4
PS5
AC2
PS6
PS7
AC3
FT8
FT9
FT10
FT11
CSB1
SB8
SB9
SB10
SB11
SB12
SB13
SB14
SB15
CB1
FT12
FT13
FT14
FT15

Component Locations
AC0
PS0
PS1
AC1
PS2
PS3
PDU
RPC0
RPC1
RPC2
RPC3
RPC4
FT0
FT1
FT2
FT3
CSB0
CB0
SB0
SB1
SB2
SB3
SB4
SB5
SB6
SB7
FT4
FT5
FT6
FT7

Data Interconnects
Multiple data and address buses connect the system boards
through the centerplane.
• Every board is directly connected to every other board
• The data paths are 144-bit-wide data paths to and from

each system board slot
• Four distinct address transfer requests can occur on the

same or different system board simultaneously
• Four address buses carry up to 167,000,000 transactions

per second at the 83.3 MHz centerplane clock speed

Data Interconnects
1
3
0 1 2 3 4
rd 1
oard 1
boa
10
14
0
mb
d
tem
rd
r
te
oa a
System board 12
bo
b
Sys
Sys
m
st
e
s t em 1
Sy
Sy
d9 5
oar b
oa rd 1
tem
Sys te mb
Sys 2
System board 8
System board 0
3
Sys
tem Sys
tem
rd 7
boa
boa
rd 1
4
Sy
Sy
st
st
em
em
Sys
bo
bo
Sys
ar
ar
d
d
tem
tem
2
6
Crossbar
boa
oar
Configuration
rd 5
d3
System board 4
Domains are 0,2
and 1,3,4

Centerplane Configurability
• Centerplane is a single 26-layer board with active logic
• The system can run without all buses functional
• Operates with one, two or three address buses
• Operates with one data bus

• System will determine the best combination
• Performance degradation is application-dependent
• Boards can be isolated for independent testing

The System Board

System board hosts all other components:
• Four 333 or 250 MHz UltraSPARC™-II CPUs
• Up to 4 Gbytes of memory over four banks
• Two I/O buses per system board, with either
• Two slots for two SBus cards
• One slot for one PCI card

• SBus and PCI can be mixed in a domain but not on a
system board

The System Board
Logical View
Global address arbiter (GAARB)
Global address router (GAMUX)
Global address arbiter
Global address router
Coherency Coherency Coherency Coherency

arbiter (LAARB)
interface interface interface interface

Local address
cntrl (CIC) cntrl (CIC) cntrl (CIC) cntrl (CIC)
Memory Port Port Port

controller controller controller controller
(MC) (PC) (PC) (PC)
U P A a d d r e s s b u s e s
UltraSPARC
UltraSPARC
UltraSPARC
UltraSPARC
Memory
I/O bridge
I/O bridge
Pack/
unpack
arbiter (LDARB)
U P A d a t a b u s e s
Local data
Data Data Data Data

buffer buffer buffer buffer
(XDB) (XDB) (XDB) (XDB)
Local data router (LDMUX)
Global data Global data

arbiter (GDARB) router (GDMUX)

The System Board
Physical View (SBus I/O)
I/O module
Centerplane
SBus card
Four banks of Pack/
eight DIMMs unpack
each
Pack/
unpack
SBus card
Memory
Module
SBus card
~21.1”
Pack/
unpack
Pack/
unpack
SBus card
UltraSPARC UltraSPARC UltraSPARC UltraSPARC

processor processor processor processor
module module module module
~16.0”

Mezzanine (Daughter) Board

Packaging
• System boards are also modular
• Processor modules connect directly to the system board
• Mezzanine (daughter) boards connect DIMMs and I/O
cards
• SBus mezzanine can be replaced with a PCI

mezzanine
• The system can be upgraded as technology changes

SBus Mezzanine Packaging
Memory
mezzanine
I/O
mezzanine
SBus
cards
Processors
Front

PCI Mezzanine Packaging

System board
Personality plate
PCI filler panel

PCI card
PCI front bracket
PCI front cover

PCI I/O module
Top PCI riser card
Bottom PCI riser card

Memory Subsystem
• Each system board has a daughter card that can hold 32
128-Mbyte DIMMs
• Can use 32-Mbyte or 128-Mbyte DIMMs
• Fully configured system is 64 Gbytes

• System board has zero to four banks of 8 DIMMS
• ECC protected
• A solid bad chip only causes single-bit errors
• Interleaving across 2, 4 or 8 banks for performance

Ultra Port Architecture

• Separate address and data interconnect for I/O
• Can plug any UPA module into a UPA port
• Can be I/O interfaces or CPUs
• Uses separate packet-switched address and data buses

• Separates data from addressing and control
• Permits use of 100% of the UPA bus bandwidth
• Enables different optimization of each function

I/O Subsystem
Logical View
I/O Module
Data side of UPA Address side of UPA
SYSIO
Port
Controller
(PC)
SBus SBus
card card
SYSIO
Enterprise 10000 data buffer

SBus SBus
card card

JTAG
• The Joint Test Action Group (JTAG)
• IEEE standard used for testing VLSI electronics

• JTAG+ used test, configure, and monitor the Enterprise
10000
• SSP sends JTAG commands to the control board
• Control board monitors and controls the system
• JTAG supports monitoring and setting various system

hardware component properties

Support Boards
System board
Centerplane
support board
Control board

Control Board
Control board JTAG Peripheral power
(Intra-CB communication) and fan control
SPARClite™
System clocks processor 10-BaseT SSP
to all boards and interface

memory
JTAG
to all subsystems

Enterprise 10000
Client-Server Architecture
Enterprise 10000
Domain Control
(cvcd)
Board
(cbe)
SSP
cbs
netcon- snmpd obp_helper

server
netcon edd hostview post

System Failure Isolation Capabilities

• Multiple point-to-point buses
• Degraded interconnect operation
• Operable on four, three, two, or one address buses or

half the data bus
• ECC on all centerplane address and data paths

• History buffers record where an error occurred
• Two control boards and two centerplane support
boards
• Redundant modular power supplies with hot swapping

Check Your Progress

• Describe the construction of the Enterprise 10000.
• Describe the system board interface.

• Explain the system board structure.
• Recount centerplane operation.
• Explain the control board structure.
• Explain component packaging.
• Understand the function of JTAG.
• Describe the SSP interaction with the Enterprise 10000.

Module 3
SSP Software Installation

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Plan the network for an Enterprise 10000 and its SSP
• Describe the software used on the SSP.
• Describe the SSP configuration options.
• Understand the restrictions of the SSP.
• Completely install the SSP software.
• Perform basic SSP commands and procedures.
• Describe how to change the control board
configuration.

Enterprise 10000 Network Planning

• Numerous network connections needed are:
• Primary and backup SSP
• One or two control boards
• Up to eight domains
• There are some restrictions
• The control boards must have constant attention
• They should be on their own subnet(s)

Enterprise 10000
Network Configurations
• There are several choices
• Will the control boards be on an individual or a

shared network?
• Will the domain to SSP interfaces be on a private or

public network?
• Will a backup SSP be configured?
• Once these decisions are made, you can configure the
network
• The more isolation the better

Control Board Configurations

CB 0 CB 0
E10000 E10000
CB 1 CB 1
Hub Hub 0 Hub 1
Backup Backup
SSP SSP
SSP SSP
Single subnet Dual subnet

Domain Network Configurations

E10000 E10000
CB 0 CB 0
D1 D2 D3 D4 D1 D2 D3 D4
CB 1 CB 1
Hub 0 Hub 1
Hub
Backup Backup
SSP SSP
SSP SSP
Public Network Private Network

The SSP Software

• SSP software comes pre-installed on the factory SSP
• Configuration is still required
• You may need to reinstall it if you have problems

• The SSP should never be used for anything other than
monitoring the Enterprise 10000
• You may delay response to Enterprise 10000

requests, possibly causing it to fail
• You must run Solaris 2.5.1 on the SSP

• The SSP has only two accounts, root and ssp

The SSP Packages

• All 12 packages must be installed, in the specified order
• They are shipped separately from Solaris
• System Service Processor (SSP) 3.1 for the Ultra

Enterprise 10000 CD-ROM
• Make sure you apply the appropriate patches

• An AnswerBook™ is provided as one of the packages

SSP Software Environment Variables

• $SUNW_HOSTNAME
• Specifies which domain you are interacting with
• $SSPETC – Where the startup scripts are

• $SSPVAR – Where the data files are
• $SSPLOGGER – The log file directory
• $SSPOPT – The SSP executables and man pages
All are set when you log into the SSP account
• SUNW_HOSTNAME is prompted for each time

Saving the SSP Configuration Files

• The SSP contains important configuration files
• Much of the Enterprise 10000’s configuration

information comes from the SSP
• These need to be regularly backed up

• Save all the directories that are modified after
installation
• Make sure that all the eeprom.image files are saved

Installing and Configuring

the SSP Solaris Software
• Installs like normal Solaris on a workstation
• Must be Solaris 2.5.1, at Hardware 4/97 or higher
• Should have at least a 1-Gbyte drive
• Package selection is minimal and edited
• Reboot when you are finished

Installing the xntp Software

• A version of the public domain NTP software
• From the SMCC Updates CD-ROM, install SUNWxntp
• The default configuration file is OK for the SSP
• If you want to connect to another system or the

Internet for exact time, see Appendix A
• You can use another system or an atomic clock for

synchronization
• You do not need to reboot yet

Preparing the System Files

• Update your name service for your configuration
• Put files in front of nis or nis+ in
/etc/nsswitch.conf for:
• ethers, netmasks, bootparams, and services
• Put the control board MAC addresses in /etc/ethers

on the SSP

Installing the SSP Software Packages

• You must install them in the specified order
• Install from the SSP 3.1 CD-ROM
ssp# pkgadd -d . SUNWsspue SUNWsspop SUNWsspmn \
SUNWsspdf SUNWsspstSUNWsspr SUNWsspdo SUNWsspob \
SUNWssppo SUNWsspid SUNWsspdr SUNWuessp
• Reboot after installation is complete

Configuring the SSP Environment

The system will request the following information the first
time it boots:
• Enterprise 10000 platform name
• Control Board 0 host name and IP address

• Control Board 1 host name and IP address (if present)
• Whether this is the main or a spare SSP
The same information is required whether you installed the
SSP software or the factory did.

Connecting to the Enterprise 10000

Host System
1. Log in as user ssp.
• ssp is the default password (change it!)
2. When the shell prompts for SUNW_HOSTNAME, use the

platform name.
3. Use the power command to ensure that the system is
powered on.
ssp% power -on -all

Reconfiguring the SSP

Used to change the SSP’s platform configuration information
• Prompts for the same information as the initial SSP

install
• Run it as root
# ssp_config
• Reboot the SSP after running it

• Does not change any OS information like
sys-unconfig does
• ssp_config also changes control board characteristics

Changing the SSP Type

• To change the main SSP to spare status, enter
# ssp_config
• To change an SSP a spare to the main SSP, use:
# ssp_config spare
• Remember to change/etc/ssphostname in all

domains
• The daemons will be started immediately
In either case, make sure you never have two active SSPs
configured as main at the same time.
Dual Control Boards

• Dual control boards provide a good level of backup.
• Only one board is active; the other is silent.
• The backup board waits for the SSP to connect.
• The active, or primary, control board is specified during

SSP configuration.
• Switching between control boards requires the domains
to be reset.
• The switch is performed on the SSP.

The Control Board Configuration File

• The file name is:
$SSPVAR/.ssp_private/cb_config
• It contains only one line.
• The format of this line is:
platform_name:platform_type:cb0:stat0:cb1:stat1
• For example:
presidents:Ultra-Enterprise-10000:jefferson:P:madison:

Switching the Active Control Board

1. Power off the platform or delete all domains.
2. Update the name service if necessary.
• Don’t forget the MAC addresses.
3. Run ssp_config cb on the main and all spare SSPs.

4. Reboot all of the SSPs.
5. Make sure the control boards become active.
6. Power on the platform or re-create your domains.
7. Bring up your domains.

Control Board Executive Image

and Port Specification Files
• The SSP is the boot server for the control boards.
• The control boards load by tftp, like diskless clients.

• Both control boards are loaded at the same time.
• The cbe image is kept in /tftpboot.
• /tftpboot file names are the board’s IP address.
• Easiest way to change this is by using ssp_unconfig.

Changing the Control Board

Configuration
To add a new control board, change CB host name or change
IP addresses:
1. Shut down all domains.
2. Update your name service(s).
3. Run ssp_config cb on the SSP as root.
4. Run cb_reset to reload the control boards.
5. Restart the domains.
Remember to change the primary and backup SSPs.

Check Your Progress

• Plan the network for an Enterprise 10000 and its SSP.
• Describe the software used on the SSP.
• Describe the SSP configuration options.
• Understand the restrictions of the SSP.
• Completely install the SSP software.
• Perform basic SSP commands and procedures.
• Describe how to change the control board

configuration.

Module 4
System Operation

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Describe some of the Enterprise 10000 security issues.
• Explain the functions of the SSP and host daemons.
• Describe how the SSP and host daemons interact.
• Understand Enterprise 10000 error reporting.
• List the Enterprise 10000 SNMP interfaces.
• Use all of the features of Hostview.
• Perform most of the Hostview functions from the
command line.

Security Considerations
• The Enterprise 10000 provides new security challenges.
• The corporate security policy may require revision.
• There are physical, system and network issues.
• General security policy is even more important.
• If you aren’t security conscious, now is a good time.

SSP and Control Board
SoftwareBlock Diagram
Enterprise 10000 domains Enterprise 10000 hardware
Solaris Solaris
Control board JTAG
cvcd cvcd cbe
slave to cbs
SSP
netcon server
Relays messages between
netcon sessions and cbs JTAG scan database:
cvcd or OBP. $SSPVAR/data/Ultra-\
Controls all JTAG Enterprise-10000
operations. Passes
client requests to cb_config
cbe. cb_port
Monitors cbe. domain_config
ssp_resource
straps
Listens for SNMP traps.
Forwards messages to
all connected SNMP
clients.
Other clients
snmpd edd
Uploads monitor scripts. edd.emc
Monitors Enterprise platform edd.erc
SNMP proxy agent: 10000 events. per domain edd.erc
Manages Enterprise Executes response action ssp_resource
.scripts.
10000 database for SNMP
clients. Allows
SNMP clients to
monitor and control
the database. hostview
fad
file locking fad_files
MIB configuration services
and data:
$SSPETC/snmp/
ssp_resource
machine_server ssp_startup ssp_startup.main

Services port registration requests Started by init. ssp_startup.restart_main
from netcom.server and snmpd. Starts daemons.
Monitors daemons for
Services port lookup requests from restart.
various clients.
Routes error messages to correct
“messages” file.
RPC
CBMP
SNMP

Instances of Client Programs

and Daemons
SSP are divided into three categories
• Platform clients - Two types, but seem the same
• One instance per SSP
• One instance per platform

• Domain clients
• One instance per domain
• SUNW_HOSTNAME designates the domain

SSP Platform Client Reference

fan board_id
sys_clock cb_reset
ssp_config cb_prom
ssp_unconfig domain_create,
domain_remove,
domain_status,
domain_rename,
domain_history,
domain_switch
edd_cmd hostview
autoconfig redx

SSP Domain Client Reference

hpost bringup
check_host netcon
netcontool hostint
power hostreset
sigbcmd drview
sys_clock netcon_server
sys_id hostinfo
sys_reset dr and dr shell applications,
including complete_attach,
drain, and so on
Make sure SUNW_HOSTNAME is correct before running these.

SSP Daemon Summary
cbs The control board server provides central access to the
Enterprise 10000 control board for client programs running on
the SSP.
edd The event detector daemon uploads event detection scripts to
control boards. When one of these scripts detects an event, edd
executes a response action script.
fad The file access daemon provides distributed file access services
to SSP clients that need to monitor, read, and write to the SSP
configuration files.
machine_server Provides machine services for netcon and routes host
messages to the proper messages file.
netcon_server The connection point for all netcon clients. netcon_server
communicates with OBP through a control board protocol and
the OS using a TCP/IP protocol.
obp_helper Begins execution of OpenBoot™. obp_helper terminates
when OBP does. During execution, obp_helper provides
services to OBP such as nvram simulation, IDprom simulation,
and time of day. netcon_server is part of obp_helper.
snmpd The SNMP proxy agent daemon listens to a UDP (User Datagram
protocol) port for incoming requests, and services the objects
specified in Ultra-Enterprise-10000.mib.
straps The SNMP trap sink server listens to the SNMP trap port for
incoming trap messages and forwards received messages to all
connected clients.
xntpd The network time protocol (NTP) daemon provides time
synchronization services. This service is used to automatically
synchronize SSP and domain times.
Never run these manually unless directed to

do so by Sun Service personnel.

SSP Daemons
Control Board Server (cbs)
• cbs is responsible for all control board communication
• Normal domain network traffic does not go through it
TCP/IP Network
TCP/IP Network
SSP
Client SSP Enterprise 10000

CBS Enterprise 10000
e.g. Hostview Client Platform
Platform
CBS
e.g. Hostview CBE Control
Control
CBE Board
Board 0 0
CBE Control
Board 1
Control
CBE Board 1

SSP Daemons
Event Detector Daemon (edd)
• Monitors traffic from the control boards
• Handles host-initiated requests
• Runs incoming control board data through rule files
• Detected event will cause the rule to run and execute the
proper response
• Is controlled by edd_cmd
• edd must be turned off during some SSP procedures

SSP Daemons
Event Detector Daemon (edd) .
SNMP-aware Help! Board 7 is

Event Detector Agent over temperature!
Detection
Host view and other Control Board

SNMP aware Control Board Executive
applications
SNMP Raising Board 7

Event Detector Agent fan speed.
Resolution
Over temperature Control Board
response action Control Board Executive
(fan command) Server

SSP Daemons
File Access Daemon (fad)
• Provides file access services to other SSP daemons
• File locking
• Update serialization
• Serializes some daemon’s execution

SSP Daemons
Network Time Protocol Daemon
(xntpd)
• xntpd provides time of day clock synchronization with
the domains
• Is necessary to avoid improper timeouts
• Communicates with both the 2.5.1 and 2.6 domains

• It makes regular small adjustments
• The SSP usually provides the time to the domains

SSP Daemons
The SNMP Daemon (snmpd)
• snmpd acts as the proxy agent for the Enterprise 10000
platform itself
• The domains have their own agents
• Supports SNMP Version 1 requests

• Traps are received from cbs and sent to the network
• Most of the traps are used by other SSP daemons as

well

SSP Daemons
straps and machine_server
• straps listens for Enterprise 10000 SNMP traps
• They are passed on to requesting daemons
• machine_server performs utility functions for other

daemons
• Mostly network configuration processing
• Also forwards error messages

Domain Support Executables

• Run in the domains during initialization, before OBP
• Reside in
$SSPOPT/release/Ultra-Enterprise-10000/os_version
• Normally called by other programs
• For example, these include the hpost tests
• Don’t try to run them yourself

System Operation
Control and management of the Enterprise 10000 system
relies on:
• Line-mode commands
• hostview – A GUI interface to the commands
• netcontool – A GUI interface to netcon

The hostinfo Command

• hostinfo gives detailed system status information
• Its syntax is
hostinfo -F | -S | -h | -p | -t
• The -F, -h, -p and -t information is displayed
graphically by hostview
• The -S operand is not generally useful
• Provides detailed internal CPU status information

hostview
• hostview allows you to perform most Enterprise 10000
platform and domain management tasks
• Most line-mode commands are available
• Provides graphical system status indicators

• Runs only on the main SSP
• May set X Windows to display remotely
• No OS domain commands
• No Alternate Pathing support (a domain function)

hostview Performance
Considerations
• You only need one copy of hostview per platform.
• You can run as many copies as you want.

• You can set the display to any X Windows system.
• Each copy requires 5 to 10 Mbytes of SSP virtual
memory.
• Make sure the SSP can support the processing load.

hostview Main Window
Power
Temperature
Fans
Failure
Support Board
Control Board
System Board
Selected board
Busses
Domain 1
(colored border)
Domain 2
(colored border)

Main Window Processor Symbols
The symbol indicates the last known state of the processor

◆ Operating system
● hpost
■ download_helper
▲ OBP
? Unknown

Main Window Processor Symbols

The background color of the symbol indicates the state of the
processor.
Green Running
Maroon Exiting
Yellow Prerun
Blue Unknown
Black Blacklisted
Red Redlisted
White Unconfigured

Selecting Boards in the Main Window

• Some commands use a selected board to designate the
target domain.
• You can select boards instead of entering their numbers.

• You can select one or more system boards.
• For a single system board, left click on it.
• For additional boards, middle click on them.

• Selected boards are outlined in black.

Help Window

Main Window Buttons
Power Temperature
Fan Failure
The Failure Window

SSP Log Files

• All domain messages are logged to:
$SSPLOGGER/domain_name/messages
• This is a copy of the domain’s /var/adm/messages
• Non-domain platform messages are logged to:

$SSPLOGGER/messages

Viewing a Messages File

With hostview

Administering Power
You can control power to:
• The entire Enterprise 10000 platform
• Individual system boards
• Individual fan trays
• Individual power supplies
• Remote peripheral cabinets

The power Command

• The power command directs the control board to
change the system power state
• Components can be individually controlled
• Groups of components can be controlled

• Power down and trip the input power breakers
• Power supply adjustments can be made
• The system will try to recover automatically from a
power outage

Power Control From Hostview

Monitoring Power Levels in Hostview

Monitoring Power Levels in Hostview

Monitoring Temperature in Hostview .

Monitoring Temperature in Hostview

The fan Command

• It controls the power and speed settings for fans.
• You can look at sides or trays individually.

• Some operands work only on all fans.
• You can speed up or slow down the fans.
• You can power the fans on or off.

Controlling Fans in Hostview

Monitoring Fans in hostview

Monitoring Fans in Hostview
Good tray Bad tray

Both fans are green Upper fan is red

Check Your Progress

• Describe some of the Enterprise 10000 security issues.
• Explain the functions of the SSP and host daemons.
• Describe how the SSP and host daemons interact.
• Understand Enterprise 10000 error reporting.
• List the Enterprise 10000 SNMP interfaces.
• Use all of the features of Hostview.
• Perform most of the Hostview functions from the
command line.

Module 5
Domains

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Describe a domain.
• List the requirements for a domain.
• Describe the function of and create an eeprom.image
file.
• Create, destroy, and rename a domain.
• Describe domain planning issues.
• Identify the SSP domain files.
• Describe blacklisting and how to manage a blacklist.
• Describe how to work with dual control boards

Introduction
The Enterprise 10000 has the ability to divide itself into as
many as eight separate systems.
• Each domain appears as a standalone, self-contained

SPARC system
• Domains are configured and controlled from the SSP
• Domains can be created and destroyed
• System boards can be placed in any desired domain
• Only eight may be active at once

Domain Configurations
SB3 SB10
SB2
SB1 SB6 SB9
SB0 SB5 SB8
SB4 SB7
Enterprise 10000 platform

system disk system disk system disk

Domain Configuration Requirements

You can have active from one to eight domains, with 1 to 16
system boards per domain.
To create a domain:
• The designated boards must be present and not in use
by another domain.
• At least one board has a network interface.
• At least one board has a processor.
• The boards have sufficient memory to support the OS.
• The domain name and its host name must match.

Domain Planning
• You will need to pre-plan your domains before creating
them.
• The control boards should be on dedicated subnets.
• The control boards only support 10Base-T.
• You may want to have a dedicated domain

management subnet.
• Make sure your IP addresses and netmasks are properly

configured.

The eeprom.image Files

• Act as the ID PROM (NVRAM) for the E10000 domains
• Must have one for each domain, up to eight
• May be ’reused,’ but not simultaneously
• The original copies are located in
$SSPVAR/.ssp_private/eeprom_save/eeprom.image.x
• You will get key and other information with the

domains
• Save this; you’ll need it to re-create these files

Creating eeprom.image Files

From the ssp account:
1. Use the platform name or an existing domain’s name
for SUNW_HOSTNAME.
2. cd to $SSPVAR/.ssp_private/eeprom_save.
3. Supply a serial number or hostid for each domain to
be created to the sys_id command.
• Use the serial number for the first domain
• Use the hostid for all others
4. Check the result.

Creating eeprom.image Files

ssp% sys_id -f eeprom.image.jackson -k 49933C54C64C858CD4CF -h 0x80a66e05
ssp% sys_id -d -f eeprom.image.jackson
IDPROM in eeprom.image.jackson
Format = 0x01
Machine Type = 0x80
Ethernet Address = 0:0:be:a6:6e:5
Manufacturing Date = Wed Dec 31 16:00:00 1969
Serial number (machine ID) = 0xa66e05
Checksum = 0x3f
ssp%

hostid Information
Serial Number hostid Ethernet MAC Address
First domain 65XXX 80a65XXX 0:0:be:a6:5X:XX
Example 65014 80a65014 0:0:be:a6:50:14
Additional domains 66XXX 80a66XXX 0:0:be:a6:6X:XX
Example 66053 80a66053 0:0:be:a6:60:53
Control boards NA NA 0:0:be:1:XX:XX

Obtaining Domain Status

From the Command Line
• The domain_status command displays the contents of
the domain_config file.
franklin:presidents% domain_status
DOMAIN TYPE PLATFORM OS SYSBDS
grant Ultra-Enterprise-10000 presidents 2.5.1 4 5
jackson Ultra-Enterprise-10000 presidents 2.5.1 2 3
new26 Ultra-Enterprise-10000 presidents 2.6 0 1
franklin:presidents%
• The domain_history command displays the contents

of the domain_history file.


From hostview

Switching Domains
• Many SSP commands are domain specific.
• The value of SUNW_HOSTNAME is important.
• A command could execute against the wrong

domain if SUNW_HOSTNAME is wrong.
• domain_switch is a C shell alias to set SUNW_HOSTNAME.
• Is installed by the SSP packages in the ssp account.
• Changes the value of the SUNW_HOSTNAME and the

shell prompt.

Creating Domains
• Use the domain_create to create a new domain
ssp% domain_create -d domain_name

-b system_board_list -o os_version -p platform
• SUNW_HOSTNAME must be set to a valid domain or the

platform name
• Deleted domains can easily be re-created
ssp% domain_create -d domain_name

Creating Domains
From hostview

Removing Domains From the

Command Line
• Use the domain_remove to delete a new domain
ssp% domain_remove -d domain_name
• SUNW_HOSTNAME must be set to the domain
• Deleted domains can easily be re-created
• domain_status information is moved to

domain_history
• Save the logs and files if you will re-create the domain

Removing Domains
From hostview

Renaming Domains
• Use the domain_rename command to rename an
existing domain.
ssp% domain_rename -d old_name -n new_name
• The new name must not be an existing domain.
• SUNW_HOSTNAME must specify the old domain.
• All the proper SSP directories and files are renamed.

• Remember to change SUNW_HOSTNAME to access the
renamed domain.

Renaming Domains
From hostview

Creating a netcon Window for a

Domain
To start netcon in an SSP window:
1. Start a window.
• It helps to label and color the window.

ssp% cmdtool -title domain_name -bg domain_color -fg black &
2. Use domain_switch to set SUNW_HOSTNAME for the

window.
3. Start netcon in the window.

Bringing Up a Domain
1. Ensure that the SUNW_HOSTNAME variable is set
properly.
• Use domain_switch if necessary
2. Make sure all domain system boards are powered on.

ssp% power -on
3. Bring up the domain.
ssp% bringup
4. When bringup finishes, start netcon.

The bringup Command

• Activates a domain
bringup [-f] [-A [on|off]] [-C] [boot_arguments]

• bringup does the following for the domain:
• Sets the hardware configuration

• May configure the centerplane
• Runs POST
• Loads and starts the OBP

• Starts the netcon_server

From hostview

Overview of netcon
• netcon:
• Provides the console interface for the Enterprise

10000
• Uses tip to make its connection
• Runs in several different modes

• Manages sessions with tip ~ commands
• netcontool is a graphical version of netcon

Using netcontool
netcontool can be started from a window or from hostview.
• You can control its operation from the tool bar.
• Many buttons correspond to netcon ~ commands.
• You have no control over the window colors.

netcon Session Types

There are four types of netcon sessions that you can request,
either from netcon directly or through netcontool.
• Read Only Session – Default
• Unlocked Write – May enter commands; write privilege

may be taken by anyone
• Locked Write – May enter commands; write privilege
preempted by Exclusive mode
• Exclusive Session – May enter commands; all other
netcon sessions for the domain are terminated
You can change your session type at any time.

netcontool Window Configuration

netcontool Buttons
• Disconnect – Disconnects and closes the console
window
• OBP/kadb – Sends a break (Stop-A) to the domain

• JTAG – Switches consoles modes to and from JTAG
• Lock Write – Requests locked write mode
• Unlock Write – Requests unlocked write mode
• Excl. Write – Requests exclusive write mode
• Rel. Write – Releases write access
• Status – Displays domain netcon sessions

Blacklisting Components
Almost any Enterprise 10000 hardware component can be
configured out of use.
• The blacklist file is read only at bringup time.
• There is one blacklist file per platform, not per
domain.
• blacklisted components are not visible to the domain.
• It is good for temporary purposes such as
benchmarking and problem isolation.
• The blacklist is never updated by the system.

Using the blacklist

• blacklist can be used to isolate intermittent failures.
• hpost will eliminate bad components it finds without
the blacklist.
• You can edit the file with a text editor or use hostview.
• Changes take effect on the next bringup for the affected
domain(s).
• The default location is
$SSPVAR/etc/platform_name/blacklist

Blacklisting Boards and Buses

With hostview

Blacklisting Processors
With hostview

Clearing the Blacklist File

Delete the file or replace it with an empty file.
• From hostview:
1. Choose Edit ➤ Blacklist File.

2. Choose File ➤ New.
3. Choose File ➤ Close.
Changes take effect after bringup.

Processor Sets
• Allows exclusive use of groups of processors by certain
processes
• Very different from pbind(1M)
• Managed by the psrset(1M) command
• Controlled by root
• System-defined processor sets may be used by any user
• DR will release bindings if necessary

Check Your Progress

• Describe a domain.
• List the requirements for a domain.
• Describe the function of and create an eeprom.image
file.
• Create, destroy, and rename a domain.
• Describe domain planning issues.
• Identify the SSP domain files.
• Describe blacklisting and how to manage a blacklist.

Module 6
Installing Solaris in a Host Domain

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Describe the Enterprise 10000 Solaris environment.
• Install Solaris on an empty disk in an Enterprise 10000

domain.
• Install the appropriate SMCC Updates CD-ROM
packages.
• Configure factory pre-installed Solaris in a domain.
• Perform the default configuration for NTP in a domain.

The Enterprise 10000 Environment

• Each domain has its own unshared copy of Solaris.
• The minimum level of Solaris required for a domain is

2.5.1 Hardware 4/97 or 2.6 Hardware 5/98.
• Solaris 2.5.1 installs separate patches.
• Patches are integrated in the 2.6 release.

• The platform architecture type is sun4u1.
• Domain name and hostname must be the same.
• The OS is loaded over the network.

Configuring the SSP as a Boot Server

1. Log in as root.
2. Add the domain hostname and IP address to
/etc/inet/hosts.
3. Add the Ethernet address to /etc/ethers.
4. Insert the Solaris CD-ROM in the CD-ROM drive.
5. Run add_install_client to prepare the network
boot information.
Consider leaving this server configured for
maintenance purposes.

Preparing the Domain

1. Bring up the domain with bringup -A off.
2. Start netcon or netcontool.
3. Create a devalias for the boot network interface.
• Use show-nets to help
• Create the boot-device alias if you want
4. Use boot net to begin the install process.

Installing Solaris
• Install Solaris as you would normally
• Choose the Entire Distribution plus OEM Support
option
• For the disk partitions, remember to:
• Allocate plenty of swap
• Reserve a slice for AP

• Remember to update your boot-device parameter
• Boot the domain and respond to the suninstall
prompts

Booting the Domain for the First Time

• Unshare and eject the CD-ROM
• Bring up the domain and start netcon
• Respond to the Solaris configuration prompts as normal
• Respond to the SSP hostname prompts
• Make sure you specify the proper SSP hostname for

the subnet
• Install any appropriate additional patches

Installing Packages From the 2.6

SMCC Server Supplement CD-ROM
• Insert and share the CD-ROM from the SSP
• Change to the Product directory
• Install:
• SUNWabhdw – Hardware AnswerBook
• SUNWehea – Enterprise header files
• SUNWeman – Enterprise man pages
• Apply any appropriate patches, then reboot

Installing Packages From the 2.5.1

SMCC Hardware Updates CD-ROM
• Insert and share the CD-ROM from the SSP
• Install:
Package Description
SUNWabdr Dynamic Reconfiguration AnswerBook
SUNWehea Header file extensions
SUNWprtnu Processor partition utilities
SUNWxntp Network time protocol
SUNWabhdw SMCC Hardware AnswerBook
SUNWeman Enterprise Solaris man pages
• Apply any appropriate patches, then reboot

Finishing the Installation – Solaris 2.6

After the domain has rebooted:
1. Copy /etc/inet/ntp.client to
/etc/inet/ntp.conf
• Insert the line server ssp_domain_hostname prefer
• Add the other configuration lines
2. Configure the Alternate Pathing software.
3. Reboot.

Finishing the Installation – Solaris 2.5.1

After the domain has rebooted:
1. Adjust xntp configuration.
• Update /etc/opt/SUNWxntp/ntp.conf
• Change the line server 127.127.1.7 to
server 127.127.1.9
• Insert the line peer ssp_domain_hostname
2. Install and configure the Alternate Pathing software.
3. Reboot.

Preinstalled Domain Software

• Solaris may be preinstalled in a domain at the factory.
• SMCC supplements or updates will be installed.
• Boot from the identified drive (usually c0t0d0s0).
• Respond to the normal suninstall prompts.

• Respond to the ssp-config prompts.
• Configure NTP in the domain.
• Make sure that your domain name service is updated.
• Install any appropriate patches.

Check Your Progress

• Describe the Enterprise 10000 Solaris environment.
• Install Solaris on an empty disk in an Enterprise 10000

domain.
• Install the appropriate SMCC Updates CD-ROM
packages.
• Configure factory pre-installed Solaris in a domain.
• Perform the default configuration for xntp in a domain.

Module 7
System Boot Process

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Describe steps of the SSP boot process.
• List the SSP daemons and their

functions.
• Explain how to boot a domain.
• List steps of the domain boot process.
• Identify the files used in the domain
boot process.
• Explain the domain hardware
configuration process.
• Describe the purpose of the
eeprom.image files.
• Decode SBus slot and processor
physical locations.
• Describe the OBP environment
variables specific to the Enterprise
10000.
The SSP Boot Process

• The SSP boots like a normal workstation
• Make sure the SSP hardware is ready
• SSP runs POST, then loads the kernel

• /sbin/init runs, which runs /etc/inittab
• inittab runs ssp_startup only on the main SSP
• Starts the SSP daemons
• edd loads event detection scripts into the active

Control Board

The ssp_startup Script
• Starts the daemons named in the
ssp_startup.main file.
• machine_server
• fad
• cb_reset
• cbs
• straps
• snmpd
• edd
• obp_helper
• netcon_server
• Each of these daemons will be
restarted if it dies.
• It is checked every 30 seconds.
Domain Bringup Flow
SSP Domain
On the SSP as user ssp

Set SUNW_HOSTNAME to proper domain
Issue power -on to domain, if necessary
bringup
Kill old obp_helper and netcon_server daemons

Run hpost, using .postrc and blacklist
Configure centerplane, if necessary
POST tests
*.elf
Start obp_helper
download_helper
OBP
eeprom.image
Start netcon_server TOD value
netcon
ok prompt
Communication path
cbs cbe domain

RPC JTAG+

The bringup Command

• Runs from the SSP to configure and boot the domain
• Target domain is set by SUNW_HOSTNAME
• Coordinates all commands and processes required to

configure, test, and boot the domain
• Requires separate bringup for each domain
• Will take several minutes

The hpost Command

• Runs on the SSP
• Directs domain configuration and initialization through
the control board.
• hpost controls:
• Creating normal system POST for the domain
• Configuring out blacklisted components
• Building the device tree

hpost Control Files

• .postrc contains configuration and execution
directives for hpost.
• There is no default .postrc file; none is required.
• hpost looks for .postrc in these places (in this order):
• The current working directory
• $SSPVAR/etc/platform_name/domain_name
• The user’s home directory (default is ~ssp/.postrc)

blacklist and redlist Files

• Tell hpost not to configure specific domain hardware
components
• There are no default files
• blacklist - Don’t use the component
• redlist - Don’t see the component
• Using a redlisting may cause domain problems

The obp_helper Command

• The Enterprise 10000 OpenBoot PROM (OBP) is not a
hardware PROM.
• It is a file loaded from the SSP during domain bring

up.
• bringup starts obp_helper.

• obp_helper executes download_helper in the domain.
• It then downloads and starts the OBP itself in the host
domain.

The download_helper Command

• download_helper helps download programs into
domain memor.y
• download_helper and obp_helper support the OBP.
• download_helper runs in the domain.
• obp_helper runs on the SSP.

• download_helper is the first code into the domain after
hpost finishes.
• It remains resident for the life of the bringup.

Other Boot-Time Software

• netcon_server coordinates the domain netcon
session(s)
• Uses TCP/IP when the domain is running
• Uses JTAG when it is not

• Solaris must be 2.5.1 Hardware 4/97 or higher, or 2.6
Hardware 5/98 or higher
• Solaris 2.6 hardware 3/98 is supported, but without

AP and DR

Console Communication Paths
Control board
cbe
cbs
JTAG
download_helper
netcon_server
cvcd OBP
netcon netcon
Domain

The OpenBoot PROM

• OBP is loaded by obp_helper and download_helper
• Performs normal OBP functions
• Also provides:
• AP booting support
• dr-max-mem management
• Specific Enterprise 10000 support

obp
• obp is the OpenBoot PROM image for the Enterprise
10000.
• All domains running the same OS level use the same

copy.
• obp is OS release level dependent
• New updates are installed by SSP patches

• obp is located in
$SSPOPT/release/Ultra-Enterprise-10000/2/6/hostobjs/
obp

eeprom.image
• Replaces the normal SPARC hardware ID PROM
• Performs the same functions
• Loads from the SSP after the OBP

• Has a unique copy for each domain
• Sends changes (setenv, devalias) back to the SSP

Managing the eeprom.image Files

• The template for an eeprom.image is in:
$SSPVAR/.ssp_private/eeprom_save/
eeprom.image.domain_name
• The domain’s specific eeprom.image file is in:

$SSPVAR/etc/platform_name/domain_name/eeprom.image
• obp_helper keeps the five previous versions.

• sys_id is used to see the hostid information.

OBP Environment Variables

Specific to the Enterprise 10000
Variable Name Default Value
dr-max-mem 0
sir-sync? false
xir-sync? false
redmode-sync? false
redmode-reboot? true
watchdog-sync? false
watchdog-reboot? true

The OBP Device Tree

Decoding an interface card location
/sbus@41 = system board 0, sbus 1
4 1
0 1 0 0 0 0 0 1
1 = IO sbus number
0 = proc
0-1
sys board number
0-F
/qec@1 = SBus slot 1
SBus slot number

0-1
PCI location is the same as SBus, but it is always slot 0.

The OBP Device Tree
Decoding an SBus Number
SystemBboard SBus 0 SBus 1
0 /sbus@40 /sbus@41
1 /sbus@44 /sbus@45
2 /sbus@48 /sbus@49
3 /sbus@4c /sbus@4d
4 /sbus@50 /sbus@51
5 /sbus@54 /sbus@55
6 /sbus@58 /sbus@59
7 /sbus@5c /sbus@5d
8 /sbus@60 /sbus@61
9 /sbus@64 /sbus@65
10 /sbus@68 /sbus@69
11 /sbus@6c /sbus@6d
15 /sbus@7c /sbus@7d

The OBP Device Tree

Decoding a Processor Location
/SUNW,UltraSPARC@1f,0 = system board 7, proc 3

1 f
0 0 0 1 1 1 1 1
1 = IO processor number
0 = proc
0-3
sys board number
0-F

The OBP Device Tree
Decoding a Processor Location
System
Processor 0 Processor 1 Processor 2 Processor 3
Board
0 /SUNW,UltraSPARC@0,0 /SUNW,UltraSPARC@1,0 /SUNW,UltraSPARC@2,0 /SUNW,UltraSPARC@3,0
2 /SUNW,UltraSPARC@8,0 /SUNW,UltraSPARC@9,0 /SUNW,UltraSPARC@a,0 /SUNW,UltraSPARC@b,0
3 /SUNW,UltraSPARC@c,0 /SUNW,UltraSPARC@d,0 /SUNW,UltraSPARC@e,0 /SUNW,UltraSPARC@f,0
6 /SUNW,UltraSPARC@18,0 /SUNW,UltraSPARC@19,0 /SUNW,UltraSPARC@1a,0 /SUNW,UltraSPARC@1b,0
7 /SUNW,UltraSPARC@1c,0 /SUNW,UltraSPARC@1d,0 /SUNW,UltraSPARC@1e,0 /SUNW,UltraSPARC@1f,0

Check Your Progress
• Describe steps of the SSP boot process.
• List the SSP daemons and their

functions.
• Explain how to boot a domain.
• List steps of the domain boot process.
• Identify the files used in the domain
boot process.
• Explain the domain hardware
configuration process.
• Describe the purpose of the
eeprom.image files.
• Decode SBus slot and processor
physical locations.
• Describe the OBP environment
variables specific to the Enterprise
10000.
Module 8
Alternate Pathing

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Describe the concepts of Alternate Pathing (AP).
• Discuss the supported device types.
• List the AP device restrictions.
• Install and set up the AP software.
• Use the AP command-line commands.
• Configure alternate network paths.
• Configure alternate disk array paths.
• Configure alternate paths for the boot drive.

AP Concepts
Domain1
System Board 1 System Board 3
Alternate paths
Active paths
Ethernet
A5000 or SSA

AP Implementation
User process
Disk
/dev/mxxx
Network
User process
Stream head
/dev/ap/dsk/meta-device
AP Disk Meta Driver

(ap_dmd)
Read Write
proc- proc-
Two alternates essing essing
Disk driver
(e.g., ssd for SSA)
AP network Meta Driver
Read Write (mxx)
proc- proc-
Nexus driver essing essing
(e.g., pln/SOC for SSA)
Stream end
Device Read Write (xx driver)
(e.g., SSA disk array)
Driver routines
Once
per
interface
Physical network
interface

AP Implementation
• AP provides a second physical path
• Only one is active at a time
• AP works with Dynamic Reconfiguration
• Applications use only the meta-device names
• Only AP "knows" which paths go to the same device
• Path switching is usually manual
• Pathgroup definitions are stored in AP state databases
• AP can be used with SEVM or SDS

AP Requirements
To use Alternate Pathing:
• AP software must be installed in the domain and on the
SSP
• You must have at least one AP database
• For disk alternate paths, two ports of the storage device
must be connected to the same domain
• For network alternate paths, you must have two
interfaces of the same device type (such as qfe and qfe)
on the same physical subnet
You cannot create pathgroups across domains

Supported Devices
• Disk devices: Only A5000 (Solaris 2.6 only) and SSA
• SDS 3.0 and SEVM 2.3, 2.4 and 2.5
• Network devices:
• SunFastEthernet™ 2.0 (hme)
• FDDI 3.0 (bf) (Solaris 2.5.1 only) and 5.0 (nf) SAS and
DAS
• LE Ethernet (le)
• QE Ethernet (qe) and QFE Ethernet (qfe)
• netcon running over a supported adapter

Installing AP
• Solaris 2.6: Install AP 2.1 from the SMCC Server
Supplement CD-ROM
• Solaris 2.5.1: Install AP 2.0 from the AP 2.0 CD-ROM

• On the SSP:
• Install SUNWapssp from the CD-ROM
• 2.1 supports domains with both 2.6 and 2.5.1
• Install the latest patches

• AP configuration is the same for 2.6 and 2.5.1

Physical Paths
Domain
Physical path
System board
and adapters
Reference a device
using a device node (such as
/dev/dsk/c0t1d1s0)
I/O device

Meta-Disk
Domain
System board
Reference a meta-disk SOC or SOC+ adapters

using a device node, like with one or two
/dev/ap/dsk/mc0t1d1s0 pln or sf ports.
(Black ports are con-
nected to the array.)
Array
I/O
Array port

Disk Pathgroup
• A disk pathgroup consists of two physical paths leading
to the same device
• A physical path that is part of a pathgroup is an alternate

path
• Alternate paths are identified by the pln or sf port
number
• Alternate means either possible path
• The primary path provides the pathgroup name
• It is also tried first

Meta-Network
Domain
System modules
and adapters
le1
le2
le5
le6
le0
le3
le4
le7
I/O
Network

Network Pathgroup
Domain
Alternate path Alternate path

(Primary path) (Active alternate)
System Modules
and adapters
le1
le2
le5
le6
le0
le3
le4
le7
Switch I/O
Network

Sample AP Configurations
Gigaplane XB
Board 1 Board 2
SBus I/F SBus I/F

SBus SBus
SBus adapters Ethernet
Fibre Channel
A5000 storage array

AP With Mirroring
Backplane
Board 1 Board 2
SBus I/F SBus I/F

SBus SBus
SBus adapters Fibre Channels
Mirrored
A5000 A5000

Device Paths
SEVM/SDS driver
AP driver AP driver
sf driver sf driver sf driver sf driver
Mirrored
A5000 A5000

The AP State Database

• Holds all AP configuration information
• Each domain has its own
• Contains all devices, paths, pathgroups and status

• Must be a raw disk slice of about 300 Kbytes
• The slice cannot be shared with other products
• You should have multiple copies of the database
• Copies are automatically synchronized

AP Database Configuration
Considerations
• You should have at least three to five copies
• The copies should have no common I/O adapters
• Can be on any disk device
• Don’t need to have alternate paths
• The database copies should be on adapters on different

system boards
• SSP keeps some AP information for boot support

Creating the AP Database

• There must be a database before you can configure AP
• Use apdb to create the original AP database or copies

# apdb -c /dev/rdsk/c0t3d0s4 -f
• Database locations are kept in /etc/system
• The list is automatically maintained
• Without a database, you’ll see boot and DR messages

• Use apdb -Z to refresh disk copies from memory

AP Databases on
Alternate Pathed Disks
• Create two copies of the database, one on each physical
path to the drive
• The system "thinks" there are two separate copies
• Updates are made twice
• There are no conflicts

• Two entries are made in /etc/system
• The whole process works outside of AP itself

Viewing AP Database Status

• Use apconfig -D to see the state of all copies
# apconfig -D
path: /dev/rdsk/c0t1d0s4
major: 32
minor: 12
timestamp: Sat May 16 16:24:27 1998
checksum: 687681819
corrupt: No
inaccessible: No

Deleting a Copy of the AP Database

• Use apdb -d to delete a copy of the AP database
# apdb -d /dev/rdsk/c0t1d0s4 -f
• You must use -f to delete the last or next-to-last copy
• If you delete the last copy, AP is no longer available
• If you try to use meta-devices you will get failures

• Check /etc/vfstab before rebooting

Viewing Pathgroup Information

• The AP database holds information about SCSI and
network device pathgroups
• When created, a pathgroup definition is uncommitted

• The pathgroup cannot be used until it is committed
• Commit entries with apdb -C
• This command commits both disk and network

pathgroups
• All uncommited entries will be commited

• There is no way to be selective

Viewing Network Entries

• Use apconfig -N -u to see uncommitted network
entries:
# apconfig -N -u
metanetwork: mle0 U
physical devices:
le2
le0 P A
• Use apconfig -N to see committed entries

# apconfig -N
• No way to see both types from one command

Planning Network Pathgroups

and Meta-Devices
• Both network adapters in a pathgroup must the same
physical type
• Both must be physically connected to the same network
• Only one adapter at a time is active
• Adapters should be on different system boards

• The meta-devices are cloneable

Meta-Network Interfaces
• Meta-network interfaces are named after the primary
path
• For a pathgroup with qfe7 as primary, the meta-device
name is mqfe7
• Use mqfe7 anywhere you would use qfe7
• ifconfig, snoop, /etc/hostname.xxx and so on
• FDDI nf networks can be SAS or DAS
• AP 2.0 supports bf SAS devices; AP 2.1 doesn’t

Creating a Network Pathgroup

1. Use apnet to create the uncommitted pathgroup.
# apnet -c -p le0 -a le2
2. Verify the result with apconfig -N -u.
3. Use apdb -C to commit the new database entry.
4. Use apconfig -N to check the result.
5. Stop all direct usage of the pathgroup interfaces.
6. Reboot or manually configure the interface using the
meta-device interface name.

FDDI Setup Considerations

• FDDI interfaces have unique MACID
• The meta-device needs a unique MACID as well
• Do not use one from an existing adapter
• You can invent one, or ask the IEEE for an official one
• Without a separate MACID for the meta-device, path
switching will not work
• Place an ifconfig command for the meta-device in
/etc/rcS.d/S30rootusr.sh to set its MACID

Switching a Network Pathgroup

• You can switch while the meta-device is active
• To switch, use the apconfig -P command
# apconfig -P mle0 -a le2
• -P – Pathgroup name
• -a – New active physical path

• The switch is immediate; there are no commits
• Check the result with apconfig -N; the P will have
moved to the new interface
• Be careful – AP makes no sanity checks; it just switches

Deleting a Network Pathgroup

1. Make sure that the meta-interface is inactive
# ifconfig mle0 down unplumb
2. Delete the meta-interface using the pathgroup name

# apnet -d mle0
3. Commit the AP database

# apdb -C
4. The pathgroup is removed from the system
• apnet -z will reverse an uncommitted delete

Alternately Pathing
the Primary Network Interface
There are three ways to activate a pathgroup on an active
interface:
1. Create the pathgroup, create a new
/etc/hostname.mxxx file, and reboot the domain.
2. Set up a script file to activate without rebooting.
3. Log in from another network interface.
This can be done to the interface you are using.

Viewing Disk Entries

• Works just like network pathgroups, except you use -S.
# apconfig -S
c1 pln0 P A
c3 pln1
metadiskname(s):
mc1t5d0 R
mc1t4d0
mc1t3d0
mc1t2d0
• Uncommitted entries have U next to each device name.

Disk Path Components

Request Request
socal soc
sf sf pln pln
IB ssd IB (ses) ssd
(drives) (drives)
A5000 SSA

Planning a Disk Pathgroup

and Meta-Disks
Examples will show SSAs; A5000s work the same way.
You must understand your disk connections.
1. You will need to map sf and pln numbers to
controller numbers.
• Use the apinst command
2. Then map the controller numbers to arrays.
• Use the ssaadm or luxadm command

Meta-Disk Configuration
Example
Running the apinst command creates the
following output:
# apinst
pln3
/dev/dsk/c7t0d0
/dev/dsk/c7t1d0
/dev/dsk/c7t2d0
/dev/dsk/c7t3d0
/dev/dsk/c7t4d0
/dev/dsk/c7t5d0
pln2
/dev/dsk/c5t0d0
/dev/dsk/c5t1d0
/dev/dsk/c5t2d0
/dev/dsk/c5t3d0
/dev/dsk/c5t4d0
/dev/dsk/c5t5d0
pln0
/dev/dsk/c1t0d0
/dev/dsk/c1t1d0
/dev/dsk/c1t2d0
/dev/dsk/c1t3d0
/dev/dsk/c1t4d0
/dev/dsk/c1t5d0
pln1
/dev/dsk/c3t0d0
/dev/dsk/c3t1d0
/dev/dsk/c3t2d0
/dev/dsk/c3t3d0
/dev/dsk/c3t4d0
/dev/dsk/c3t5d0
...
Meta-Disk Configuration Example

• The apinst command identifies each pln and sf port
and corresponding controller.
• Use the ssadm or luxadm command to obtain the World

Wide Name (WWN) for the array attached to each
controller.
• The WWN is a unique ID that identifies every Fibre
Channel device.
• You will find the same SSA is on c1 (pln0) and c3
(pln1), and the other on c5 (pln2) and c7 (pln3).
• The two pathgroups must be pln0 and pln1, and pln2
and pln3.

Creating a Disk Pathgroup

and Meta-Disks
1. Use apdisk to create the uncommitted pathgroup.
# apdisk -c -p pln0 -a pln1
2. Verify the result with apconfig -S -u.
3. Use apdb -C to commit the new database entry.
4. Use apconfig -S to check the result.
5. Run drvconfig -i ap_dmd to create /devices entries.
6. Run apconfig -R to create /dev/ap entries.
7. Use ls to check the results.
Using the Meta-Devices

• Replace physical device references with /dev/ap/dsk
or /dev/ap/rdsk.
• Change /etc/vfstab and remount or reboot
• Do not continue to use the physical paths
• Do not change boot device partitions this way
• Use apboot instead (See Appendix B)
• The SSP will direct OBP to try the alternate path

• AP 2.1 supports mirrored boot drives

Disk Managers and AP

Sun Enterprise Volume Manager
• SEVM 2.3, 2.4, and 2.5 are supported
• DMP must be disabled in Version 2.5
Solstice Disk Suite
• Only Version 3.0 will work with AP
• Later versions will work only without AP

Manually Switching the Active Path

• You can switch while the meta-device is active
• To switch, use the apconfig -P command
# apconfig -P pln0 -a pln1
• -P – Pathgroup name
• -a – New active physical path

• The switch is immediate; there are no commits
• Check the result with apconfig -S; the P will have
moved to the new interface
• Be careful – AP makes no sanity checks; it just switches

Switching Back to the Primary Path

• Use apconfig , specifying the primary path
# apconfig -P pln0 -a pln0
• The other physical path is transparently activated

• Do not confuse the path with the pathgroup name

Automatic Disk Pathgroup Switching

(AP 2.1)
• AP 2.1 will automatically switch pathgroups if:
• A failure is detected on one path
• DR requests it (Enterprise 10000)

• Known bad paths are marked T (tried)
• AP will not use the path automatically
• Reset the T by reboot, DR, or manually

• You can still switch to it manually

Deleting a Disk Pathgroup

1. Make sure that the pathgroup is inactive
• Partitions unmounted, disk groups deported, and so
forth.
2. Delete the meta-interface using the pathgroup name.
# apdisk -d pln0
3. Commit the AP database.

# apdb -C
• apdisk -z will reverse an uncommitted delete

AP and the Boot Disk

• You can define an alternate path for the boot disk
• AP will retry the boot if the OBP path fails
• AP retry will occur before looking at a mirror drive
• AP 2.1 will retry using the mirror path
• Procedure is the same for AP 2.0 and 2.1

Placing the Boot Disk

Under AP Control
1. Create an AP pathgroup the includes the boot disk
2. Run apboot, specifying the boot disk meta-disk
# apboot mc2t0d0
3. Set the OBP boot-device to one of the physical paths
4. Create a backup devalias but do not add it to the
boot-device value
5. Reboot the system
Do not try to update /etc/vfstab manually

Removing AP Support
From the Boot Disk
1. Run apboot, specifying a physical device name
# apboot c2t0d0
2. Reboot the system
• Do not remove the AP packages without first removing
the AP boot drive support
• Do not try to edit /etc/vfstab manually

Using apboot With a

Mirrored Boot Disk (Solaris 2.6 Only)
• AP 2.1 will retry a failed boot to a boot device mirror
• The devices should be in separate pathgoups
• Use apboot -m to specify the mirror meta-disk
# apboot -m mc2t3d0
• Do not specify the mirror in the OBP boot-device
value
• Remove mirror support with apboot -u
# apboot -u mc2t3d0

The AP Recovery Boot Sequence

• When a boot path fails, AP tries the other
• The SSP has recorded the other path, and sends it to the
domain
• It may take a minute or two to try the other path
• Wait for the retry process to complete
• Manual invervention may cause new problems
• Remember to rerun apboot if the disk configuration

changes
• The SSP might have old information

Using AP in Single-User Mode

There are copies of the AP commands in /sbin
• These do not need to AP daemon to run
Two types of AP disk problems can cause a boot to fail

• Paths do not go to the same device
• Correct the AP configuration
• A device, other than the boot disk, needed at boot time

has a bad primary path
• Switch paths manually and retry

Check Your Progress

• Describe the concepts of Alternate Pathing (AP).
• Discuss the supported device types.
• List the AP device restrictions.
• Install and set up the AP software.
• Use the AP command-line commands.
• Configure alternate network paths.
• Configure alternate disk array paths.
• Configure alternate paths for the boot drive.

Module 9

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Describe the requirements for dynamic configuration.
• List the DR process steps for attach and detach.

• Discuss the restrictions and problems that can occur
with DR.
• Display DR information from both dr and hostview.
• Perform a DR attach from both dr and hostview.
• Perform a DR detach from both dr and hostview.
• Solve problems that prevent DR from succeeding.
• Manage AP and DR interaction.

Dynamic Reconfiguration Capabilities

• Dynamic Reconfiguration (DR) enables you to:
• Add (DR attach) and
• Remove (DR detach)

an entire system board from a running domain for
• Board addition
• Board deletion
• Board replacement
• New domain creation

• DR attach is easier than DR detach
• When adding a board, the OS adds the resources

• When removing a board, the OS must give up the
resources
• The OS may not be able to do this
• In this case, the board cannot be removed
• DR is controlled from the SSP, but performed by the

hardware and domain OS

dr-max-mem (Solaris 2.6)

• dr-max-mem is a domain OBP environment variable
• Set to zero, no DR is allowed in the domain
• Greater than zero, DR is allowed in the domain
• The value is not important
• Set by setenv requires a bringup

• Set by setenv-dr only needs a reset

dr-max-mem (Solaris 2.5.1)

• Solaris 2.5.1 has static kernel memory management
structures
• Attached memory requires pre-allocation of these

• dr-max-mem specifies the maximum domain memory
size
• Add the normal amount to the most you will add
• Any memory attached over this amount is not usable
• Can still use CPUs and interface cards
• Set on a domain basis in the OBP

dr-max-mem (Solaris 2.5.1)

• Set too high, you waste memory
• 8+ Mbytes is used per Gbyte specified
• Cannot be used for anything else
• Capped in small domains to prevent performance

problems.
Current Physical dr-max-mem
Memory Maximum Value
512 Mbytes 16 Gbytes

dr-max-mem Usage by Solaris 2.5.1

• Set before booting the domain
• Use setenv-dr if possible
ok setenv-dr dr-max-mem NNNNN
• Solaris gives you memory status messages
• At boot time and after DR operations
• Tells how much memory is configurable

DR Attach
• To be able to attach a system board to a domain:
• The target system board must be installed.

• The system board must be powered up.
• The system board must not belong to a domain.
• The system board must contain at least one
processor.
• dr-max-mem must nonzero before booting.
• If the requirements are not met, you cannot do the
attach.

DR Attach
dr
init_attach
abort_attach
complete_attach
reconfig
exit

Attaching a Board With DR

dr> drshow 6 IO
SBus Controllers and Devices for Board 6
------------------------ Sbus 0 : Slot 0 : SUNW,pln0 --------------
device opens name usage

------ ----- ---- -----
ssd0 0 /dev/dsk/c1t0d0s0
------------------------ Sbus 0 : Slot 1 : SUNW,pln2 --------------

------ ----- ---- -----

Attaching a System Board

With hostview


With hostview .


With hostview

I/O Device Reconfiguration After a

DR Operation
• Performs the same commands as boot -r
• After attach, adds new devices into the domain
• Not needed if the devices were in the domain before
• After detach, removes the devices from the domain
• Do if the devices will never return to the domain
• Done when not needed could cause disk controller

renumbering

Viewing System Information

• dr and hostview provide system status information
• Use drshow in dr, specific buttons in hostview
• You can see the following categories:
• cpu – Activity on a board’s CPUs
• memory – Memory configuration and drain status

• device – Board I/O device status
• obp – Summary of all board hardware
• unsafe – Domain’s suspend unsafe devices

System Information Using drshow

dr> drshow 3 IO
SBus Controllers and Devices for Board 3
------------------------ Sbus 0 : Slot 0 : QLGC,isp1 ------------------------

------ ----- ---- -----
sd15 0 /dev/dsk/c1t0d0s0
--------------------------- Sbus 1 : Slot 1 : qec1 ---------------------------

------ ----- ---- -----
qe4 qe4
qe5 qe5
qe6 qe6
qe7 qe7

System Information Using hostview

System Information Using
hostview .

System Information Using hostview .


System Information Using
hostview .


DR Detach
To be able to detach a system board from a domain:
1. dr-max-mem must a proper nonzero value
2. Active file systems and network interfaces must be
configured off the board (usually with AP)
3. Devices on the board must be closed
4. There must be sufficient swap space
5. Detach-unsafe devices must be closed or the drivers
unloaded
6. Suspend-unsafe devices may have to be managed

Detaching a System Board

Like DR attach, there are three steps
1. drain – Free up the memory on the board
2. complete_detach – Free the CPUs and I/O devices
• Also disconnect the board from the domain
3. reconfig – Just like DR attach
• The operation can be stopped with abort_detach

Finishing the Complete Detach

Operation
• Can complete after board memory is completely free
• Operation fails if all devices are not closed
• Can retry as often as you need to
• You can abort the detach operation at any time
• Downed network interfaces will remain down
• The board is now free for other uses

Configuring for DR Detach

• System must be ready for DR Detach
• Many configurations will not work
• dr-max-mem must be nonzero

• Items to consider include:
• I/O device configuration
• Swap space
• Memory interleaving
• Memory usage

Configuring for DR Detach

I/O Devices
• Plan ahead – all I/O devices must be removable
• Critical devices must be mirrored or alternate pathed

• You must be able to:
• Unmount all file systems
• Remove any active swap devices
• Stop all raw partition usage
• These are all manual actions except disk AP in 2.6

Detaching Network Devices

• DR automatically deconfigures the board’s network
interfaces
• The detach will fail if a board interface is:
• The primary network interface for the domain
• On the same subnet as the SSP

• Any unconfigured interfaces will not be reconfigured
on detach failure

Detaching Non-Network Devices

• All devices must be closed before they can be detached
• hostview and drshow show device usage
• To find device users, use fuser
• To detach the board, all devices must be closed
• This includes raw partition users like SEVM, SDS, or

swap
• A3000 controllers must be taken offline
• You may have to do much of this work manually

DR Detach-Safe Devices
• When a driver opens a device, it does not plan to close it
• For DR detach, a driver needs to give up devices

• Drivers that can release their devices are called
detach-safe
• Drivers that can not are detach-unsafe
• Detach-unsafe devices must be manually freed
• The driver has to be removed from memory
• You can tell the system about drivers it does not "know"

Unloading a Loaded
Detach-Unsafe Driver
To detach a system board with active detach-unsafe device(s):
1. Stop all usage of all controllers of the same type in
the entire domain.
2. Manually close the device(s) and use modunload on
the driver.
3. Detach the system board.
4. Start using the remaining devices.
If the driver does not support modunload, you can not remove
the board.

Swap Space
• You need lots of swap space in multiple locations
• Separate controllers on different boards
• Must be enough swap space left after detach for:

• The memory freed by drain.
• A full panic dump in the primary partition
• Running work and performance
• The system will not check to make sure there is enough
memory

Memory Interleaving
• DR does not support removing eight-way interleaved
boards
• Would use twice the swap space
• Other hardware issues
• Eight-way interleaving is never a default
• Must be explicitly enabled in .postrc
• Do not do it if you want to use DR

Memory Usage
• To free a board, its memory must be emptied
• Permanent memory makes this difficult
• Kernel memory has fixed real addresses
• Memory must be moved to another board
• This board is emptied first
• There must be an acceptable target board

• Hardware memory errors may cause problems

Detaching a Board With dr

dr
drain
abort_detach
complete_detach
reconfig
exit

Detaching a Board With hostview

Detaching a Board With hostview

Pageable and Permanent Memory

• Permanent memory can not be moved from its physical
address
• It must be moved to remove a board containing it
• The memory and physical address are copied to another
board
• The system must be inactive during this process
• This process is called quiesce
• The entire domain is suspended for a short time

Operation:Permanent Memory
on Target Board
• dr-max-mem not zero puts permanent memory only on
the lowest numbered board
• Usually the boot devices and critical network
interfaces are there as well
• A second board’s memory is freed (to swap)

• The first board’s contents and address are swapped
• The OS is stopped (quiesce) during this process

Operating System Quiesce

• The domain freezes while permanent memory is
relocated
• Takes about a minute or so

• Lots of console messages
• Quiesce is required only if the board contains permanent
memory
• The domain centerplane must be idle during the copy
• If the quiesce fails, the complete_detach operation
fails
• You can retry complete_detach as often as necessary

Operating System Quiesce Failures

• You can manually suspend suspend-unsafe devices
• You can decide to stall real-time threads
• You can then try to force a quiesce
• If you are wrong, you could crash the domain
• Test this before you try it in production

• Any centerplane activity could crash the domain

Suspend-Safe and Suspend-Unsafe

Devices
• Suspend-safe supports the quiesce process
• Almost all Sun drivers are suspend-safe
• All pseudo-devices are suspend safe
• SCSI tape driver st is the only known unsafe driver

• You can tell the OS about non-Sun drivers
• Suspend-safe list – supports the proper calls
• Suspend-bypass list – no interrupts (pseudo-devices)

Adding New Suspend-Safe Drivers

• You can define additional drivers as suspend-safe
• Intended for new or third-party drivers
• Add the driver to a list in /etc/system

• Do not add the driver unless you are sure the driver is
safe
• Do not add the driver to the list just to bypass a

forced quiesce
• Ask your vendor if you do not know if the driver is

suspend-safe

Adding New Suspend-Bypass Drivers

• You can define additional drivers to be ignored
• Intended for new or third-party drivers
• These are usually for pseudo-devices
• Add the driver to the list in /etc/system

• Do not add the driver unless you are sure the driver is
safe to add
• Ask your vendor if you do not know if the driver is safe

DR and AP Interaction
DR Attach
• Run apconfig -F to clear the AP disk pathgroup
detached (DE) status
• AP automatically synchronizes returning AP database

copies

DR Detach
• DR notifies AP as it proceeds
• Disk and network pathgroups must be manually

switched (Solaris 2.5.1)
• Disk and network pathgroups will be automatically
switched (Solaris 2.6)
• AP will mark pathgroups detached (DE) or drained (DR)
in the database (Solaris 2.6)
• You must use apconfig -F to update the status of
detached devices (Solaris 2.6)

Check Your Progress

• Describe the requirements for dynamic configuration.
• List the DR process steps for attach and detach.

• Discuss the restrictions and problems that can occur
with DR.
• Display DR information from both dr and hostview.
• Perform a DR attach from both dr and hostview.
• Perform a DR detach from both dr and hostview.
• Solve problems that prevent DR from succeeding.
• Manage AP and DR interaction.

Module 10
Troubleshooting

Module Overview
• Course map
• Relevance
• Objectives

Course Map

Objectives
• Understand how an Enterprise 10000 fails and recovers.
• Understand the role of the SSP in failure logging.
• Discuss the different types of failures.
• Find where failure information is recorded.
• Interpret failure information.
• Understand support information requirements.

Standard Domain Message Logs

• Each domain has its own messages file on the SSP
$SSPLOGGER/domain_name/messages
• This is a copy of the domain’s /var/adm/messages

file
• The message file for the Enterprise 10000 platform is

$SSPVARLOGGER/messages
• Contains messages not related to a domain
• Remember that the SSP’s messages are in its

/var/adm/messages
Bus Configurations and

the Figure of Merit
• The Enterprise 10000 system has four address buses and
two data buses
• It can continue to run if some of these have failed

• To support this, bringup calculates a Figure of Merit
(FOM)
• All 45 possible bus configurations are checked
• Fifteen different address bus combinations, with

• Three different data bus combinations


the Figure of Merit
• The system checks each possible configuration
• They are numbered with a bit pattern reflecting the

configuration
• There may be conflicting ways to configure the system

• bringup must determine which one gives you the most
components
• The FOM does this by calculating what’s accessible in
each configuration


the Figure of Merit
Board 6 Board 12
Board 8 Board 14


the Figure of Merit
Wp Wm Ws Wa Wd
FOM = P ×M ×S ×A ×D
P = Number of processors
M = Memory in units of number of equivalent 64-Mbyte SIMM
banks
S = Number of interface cards
A = Number of address buses
D = Number of data buses
Wx = A weight value for each of the above
• The default weights are all 1

• The FOM is calculated for each configuration
Redlist and Blacklist Files

hpost checks for component configuration information while
bringup runs.
• The blacklist file describes components that cannot

be seen by the domain
• The redlist file describes components that cannot be

seen by the platform
• Changes take effect on domain bringup
• There are default locations or .postrc can specify

The autoconfig Command

• autoconfig must be run when adding a new revision of
a system board to the system.
• It may also be required when moving a system board

to a new slot.
• It is not necessary if all system boards are the same

revision level.
• If you are not sure, run it.
Never run autoconfig against a system board that is running

the OS or on the centerplane if any domain is running the OS.

Diagnostic Tools
• bringup invokes hpost for error checking
• hpost can be run from the SSP command line for

diagnostic purposes
• It will crash a running domain
• Be careful with SUNW_HOSTNAME
• SunVTS is a system exerciser that tests from the OS level

• It is good for stressing the system

Diagnostic Tools
• prtdiag displays:
• System configuration
• System clock frequency

• Processor speeds and cache size
• The size of each memory bank
• A listing of all I/O cards
• It runs from a domain
• It is very helpful with configuration questions

Correctable Memory Errors

• Usually an early indicator of a failing memory DIMM
• Can cause performance degradation if extensive
• No data can be lost or damaged

• Not reported by default
• Failing location reported by OBP or OS
• Not an emergency condition
• Can enable OS reporting in /etc/system

System Failures
The Enterprise 10000 can encounter many types of system
failure:
• Reboot request
• Panic
• Watchdog/Redmode/XIR
• Heartbeat failure
• arbstop

Reboot Request
The OS in a domain asks to be rebooted.
The process is:
• The OS shuts down cleanly and tells the OBP.
• The OBP passes the request to the SSP.
• edd detects the request which is sent by the control

board.
• edd logs the event and locates the rule.

• edd reboots the domain with bringup -L -F.

Panic
A panic means the OS can not recover from a problem.
When a panic is detected:
• Parts of the kernel are saved to the primary swap
partition.
• edd detects the panic (notified by the control board).
• edd logs the panic and locates the proper rule.
• edd reboots the domain with bringup -L -F.
• The OS saves the dump when it reboots.

Watchdog, Redmode, and XIR Resets

These are SPARC chip exception states
• A watchdog reset occurs when a recursive trap happens

in the trap-handler.
• A redmode occurs when the system has received too

many concurrent traps.
• A XIR is an outside request for the CPU to reset.

Heartbeat Failure (Hung Host)

• hung host implies that a processor has stalled
• Processors update a special heartbeat register
• The control board saw this register was not being

updated
• It is often related to I/O problems
• For some devices, the CPU is held until the device

responds

Heartbeat Failure (Hung Host)

Some software problems can look like a hung host.
Intervene manually, in this order:
1. Try another login.
2. Try Stop-A (~#) and sync.
3. Force a panic with hostint.
4. Force an xir with hostreset.
5. As a last resort, bring up the domain with
bringup -f.

Arbstop
• An Arbstop is a centerplane interconnect error
• recordstops are similar
• They may occur on a board or on the centerplane itself.
• Owning domain fails if it is on a board
• All domains fail if on the centerplane

• The system takes a hardware "panic dump."

Creating a Hardware State Dump File

• Do not reboot or run bringup immediately if a domain
fails.
• Wait for edd to finish logging out the domain.
• If the domain was running, the filename will be:
$SSPLOGGER/domain_name/Edd_Arbstop_Dump-
mm.dd.hh.mm:ss
• If hpost was running, the dump file will be:

$SSPLOGGER/domain_name/xfstate.mmdd.hhmm.ss
• The files are in the same directory as the SSP’s domain

/var/adm/messages copy.
redx
• redx is hardware debugger like adb is for software.
• You should never have a need to use redx.

*redx is intended for use by trained Sun support personnel only*
• You can take down running domains just looking at
system components.
• It runs in on-line and off-line modes.
• On-line reads and writes directly to the hardware
• Local or off-line (-l) examines hardware dump files

Technical Information for Escalation

• Software configuration information
• Hardware configuration information
• If possible, identify the type of problem
• Indicate the severity of the problem

• Try to have any problem-specific information available

Check Your Progress

• Understand how an Enterprise 10000 fails and recovers.
• Understand the role of the SSP in failure logging.

• Discuss the different types of failures.
• Find where failure information is recorded.
• Interpret failure information.
• Understand support information requirements.

Course Contents
About This Course ....................................................................................................................... Preface-1

Course Map ................................................................................................................................... Preface-2
Module-by-Module Overview ................................................................................................... Preface-3
Appendices ................................................................................................................................... Preface-4
Course Objectives ......................................................................................................................... Preface-5
Topics Not Covered ..................................................................................................................... Preface-7
Course Prerequisites .................................................................................................................... Preface-8
Introductions ................................................................................................................................ Preface-9
How to Use the Course Materials ............................................................................................ Preface-10

Capabilities and Features ..........................................................................................................1-1
Module Overview ................................................................................................................................... 1-2
Course Map .............................................................................................................................................. 1-3
Objectives ................................................................................................................................................. 1-4
Ultra Enterprise 1000 Capabilities ........................................................................................................ 1-5
Ultra Enterprise 1000 Features .............................................................................................................. 1-6
Ultra Enterprise 1000 System Cabinet ................................................................................................. 1-7
System Domains ...................................................................................................................................... 1-8
Alternate Pathing .................................................................................................................................... 1-9
Dynamic Reconfiguration .................................................................................................................... 1-10
Operating System Support .................................................................................................................. 1-11
Operating System Enhancements ....................................................................................................... 1-12
The SSP ................................................................................................................................................... 1-13
SSP Logical Connectivity ..................................................................................................................... 1-14
UltraEnterprise 10000 Administration i

SSP Window ....................................................................................................................................... 1-15

Network Console Window ............................................................................................................... 1-16
Hostview ............................................................................................................................................. 1-17
Hardware Configuration Control ....................................................................................................... 1-18
Diagnostic and Monitoring Tools ....................................................................................................... 1-19
Status Monitoring and Display ........................................................................................................... 1-20
Resiliency Features ............................................................................................................................... 1-21
Redundant Components ...................................................................................................................... 1-22
Concurrent Serviceability .................................................................................................................... 1-23
Error Logging ........................................................................................................................................ 1-24
Check Your Progress ............................................................................................................................ 1-25
Architecture Overview ..............................................................................................................2-1

Module Overview ................................................................................................................................... 2-2
Course Map .............................................................................................................................................. 2-3
Objectives ................................................................................................................................................. 2-4
Enterprise 10000 Packaging ................................................................................................................... 2-5
Enterprise 10000 Component List ......................................................................................................... 2-6
Component Locations ............................................................................................................................ 2-7
Data Interconnects ................................................................................................................................ 2-10
Data Interconnects ................................................................................................................................ 2-11
Centerplane Configurability ................................................................................................................ 2-12
The System Board ................................................................................................................................. 2-13
Mezzanine (Daughter) Board
Packaging ............................................................................................................................................ 2-16
SBus Mezzanine Packaging ................................................................................................................. 2-17
UltraEnterprise 10000 Administration ii

PCI Mezzanine Packaging ................................................................................................................... 2-18

Memory Subsystem .............................................................................................................................. 2-19
Ultra Port Architecture ......................................................................................................................... 2-20
I/O Subsystem
Logical View ....................................................................................................................................... 2-21
JTAG ........................................................................................................................................................ 2-22
Support Boards ...................................................................................................................................... 2-23
Control Board ........................................................................................................................................ 2-24
Enterprise 10000
Client-Server Architecture ................................................................................................................ 2-25
System Failure Isolation Capabilities ................................................................................................. 2-26
SSP Software Installation .........................................................................................................3-1

Module Overview ................................................................................................................................... 3-2
Course Map .............................................................................................................................................. 3-3
Objectives ................................................................................................................................................. 3-4
Enterprise 10000 Network Planning .................................................................................................... 3-5
Enterprise 10000
Network Configurations ..................................................................................................................... 3-6
Domain Network Configurations ......................................................................................................... 3-8
The SSP Software .................................................................................................................................... 3-9
The SSP Packages .................................................................................................................................. 3-10
SSP Software Environment Variables ................................................................................................ 3-11
Saving the SSP Configuration Files .................................................................................................... 3-12
Installing and Configuring
the SSP Solaris Software .................................................................................................................... 3-13
Installing the xntp Software ................................................................................................................ 3-14
Preparing the System Files .................................................................................................................. 3-15
Installing the SSP Software Packages ................................................................................................. 3-16
UltraEnterprise 10000 Administration iii

Configuring the SSP Environment ..................................................................................................... 3-17

Connecting to the Enterprise 10000
Host System ........................................................................................................................................ 3-18
Reconfiguring the SSP .......................................................................................................................... 3-19
Changing the SSP Type ........................................................................................................................ 3-20
Dual Control Boards ............................................................................................................................. 3-21
The Control Board Configuration File ............................................................................................... 3-22
Switching the Active Control Board .................................................................................................. 3-23
Control Board Executive Image
and Port Specification Files .............................................................................................................. 3-24
Changing the Control Board
Configuration ..................................................................................................................................... 3-25
System Operation ......................................................................................................................4-1

Module Overview ................................................................................................................................... 4-2
Course Map .............................................................................................................................................. 4-3
Objectives ................................................................................................................................................. 4-4
Security Considerations ......................................................................................................................... 4-5
SSP and Control Board SoftwareBlock Diagram ................................................................................ 4-6
Instances of Client Programs
and Daemons ........................................................................................................................................ 4-7
SSP Platform Client Reference .............................................................................................................. 4-8
SSP Domain Client Reference ................................................................................................................ 4-9
SSP Daemon Summary ........................................................................................................................ 4-10
SSP Daemons
Control Board Server (cbs) ............................................................................................................... 4-11
SSP Daemons
Event Detector Daemon (edd) .......................................................................................................... 4-12
SSP Daemons
UltraEnterprise 10000 Administration iv

File Access Daemon (fad) ................................................................................................................. 4-14

SSP Daemons
Network Time Protocol Daemon (xntpd) ...................................................................................... 4-15
SSP Daemons
The SNMP Daemon (snmpd) ............................................................................................................ 4-16
SSP Daemons
straps and machine_server ...............................................4-17
Domain Support Executables .............................................................................................................. 4-18
System Operation .................................................................................................................................. 4-19
The hostinfo Command .................................................................................................................... 4-20
hostview ................................................................4-21
hostview Performance Considerations ............................................................................................. 4-22
hostview Main Window ..................................................................................................................... 4-23
Main Window Processor Symbols ...................................................................................................... 4-24
Selecting Boards in the Main Window .............................................................................................. 4-26
Help Window ........................................................................................................................................ 4-27
The Failure Window ............................................................................................................................. 4-29
SSP Log Files .......................................................................................................................................... 4-30
Viewing a Messages File
With hostview ..........................................................4-31
Administering Power ........................................................................................................................... 4-32
The power Command ........................................................................................................................... 4-33
Power Control From Hostview ........................................................................................................... 4-34
Monitoring Power Levels in Hostview .............................................................................................. 4-35
Monitoring Power Levels in Hostview .............................................................................................. 4-36
Monitoring Temperature in Hostview. .............................................................................................. 4-37
Monitoring Temperature in Hostview ............................................................................................... 4-38
The fan Command ............................................................................................................................... 4-39
Controlling Fans in Hostview ............................................................................................................. 4-40
Monitoring Fans in hostview ................................................4-41
Monitoring Fans in Hostview ............................................................................................................. 4-42
UltraEnterprise 10000 Administration v

Domains ....................................................................................................................................5-1
Module Overview ................................................................................................................................... 5-2
Course Map .............................................................................................................................................. 5-3
Objectives ................................................................................................................................................. 5-4
Introduction ............................................................................................................................................. 5-5
Domain Configurations ......................................................................................................................... 5-6
Domain Configuration Requirements ................................................................................................. 5-7
Domain Planning .................................................................................................................................... 5-8
The eeprom.image Files ........................................................................................................................ 5-9
Creating eeprom.image Files .............................................................................................................. 5-11
hostid Information .............................................................................................................................. 5-12
From the Command Line .................................................................................................................. 5-13
From hostview ..........................................................5-14
Switching Domains ............................................................................................................................... 5-15
Creating Domains
Creating Domains
From hostview ..........................................................5-17
Removing Domains From the Command Line ................................................................................ 5-18
Removing Domains
From hostview ..........................................................5-19
Renaming Domains
Renaming Domains
From hostview ..........................................................5-21
Creating a netcon Window for a Domain ........................................................................................ 5-22
UltraEnterprise 10000 Administration vi

The bringup Command ....................................................................................................................... 5-24
From hostview ..........................................................5-25
Overview of netcon ........................................................5-26
Using netcontool .........................................................5-27
netcon Session Types ........................................................................................................................... 5-28
netcontool Window Configuration ................................................................................................. 5-29
netcontool Buttons ............................................................................................................................. 5-30
Blacklisting Components ..................................................................................................................... 5-31
Blacklisting Boards and Buses
With hostview ..........................................................5-33
Blacklisting Processors
With hostview ..........................................................5-34
Clearing the Blacklist File .................................................................................................................... 5-35
Processor Sets ........................................................................................................................................ 5-36
Installing Solaris in a Host Domain .........................................................................................6-1

Module Overview ................................................................................................................................... 6-2
Course Map .............................................................................................................................................. 6-3
Objectives ................................................................................................................................................. 6-4
The Enterprise 10000 Environment ...................................................................................................... 6-5
Configuring the SSP as a Boot Server .................................................................................................. 6-6
Preparing the Domain ............................................................................................................................ 6-7
Installing Solaris ...................................................................................................................................... 6-8
Booting the Domain for the First Time ................................................................................................ 6-9
Installing Packages From the 2.6 SMCC Server Supplement CD-ROM ....................................... 6-10
Finishing the Installation – Solaris 2.6 ............................................................................................... 6-12
UltraEnterprise 10000 Administration vii

Finishing the Installation – Solaris 2.5.1 ............................................................................................ 6-13

Preinstalled Domain Software ............................................................................................................ 6-14
System Boot Process ..................................................................................................................7-1

Module Overview ................................................................................................................................... 7-2
Course Map .............................................................................................................................................. 7-3
Objectives ................................................................................................................................................. 7-4
The SSP Boot Process .............................................................................................................................. 7-5
The ssp_startup Script ........................................................................................................................ 7-6
Domain Bringup Flow ............................................................................................................................ 7-7
The bringup Command ......................................................................................................................... 7-8
The hpost Command ............................................................................................................................. 7-9
hpost Control Files ............................................................................................................................... 7-10
blacklist and redlist Files ........................................................................................................... 7-11
The obp_helper Command ................................................................................................................ 7-12
The download_helper Command ..................................................................................................... 7-13
Other Boot-Time Software ................................................................................................................... 7-14
Console Communication Paths ........................................................................................................... 7-15
The OpenBoot PROM ........................................................................................................................... 7-16
obp .....................................................................7-17
eeprom.image ............................................................7-18
Managing the eeprom.image Files .................................................................................................... 7-19
OBP Environment Variables
Specific to the Enterprise 10000 ........................................................................................................ 7-20
The OBP Device Tree ............................................................................................................................ 7-21
The OBP Device Tree ............................................................................................................................ 7-24
UltraEnterprise 10000 Administration viii

Alternate Pathing ......................................................................................................................8-1

Module Overview ................................................................................................................................... 8-2
Course Map .............................................................................................................................................. 8-3
Objectives ................................................................................................................................................. 8-4
AP Concepts ............................................................................................................................................. 8-5
AP Implementation ................................................................................................................................. 8-6
AP Implementation ................................................................................................................................. 8-7
AP Requirements .................................................................................................................................... 8-8
Supported Devices .................................................................................................................................. 8-9
Installing AP .......................................................................................................................................... 8-10
Physical Paths ........................................................................................................................................ 8-11
Disk Pathgroup ..................................................................................................................................... 8-13
Meta-Network ....................................................................................................................................... 8-14
Network Pathgroup .............................................................................................................................. 8-15
Sample AP Configurations .................................................................................................................. 8-16
Device Paths ........................................................................................................................................... 8-18
The AP State Database ......................................................................................................................... 8-19
AP Database Configuration Considerations ..................................................................................... 8-20
Creating the AP Database .................................................................................................................... 8-21
AP Databases on
Alternate Pathed Disks ...................................................................................................................... 8-22
Viewing AP Database Status ............................................................................................................... 8-23
Deleting a Copy of the AP Database .................................................................................................. 8-24
Viewing Pathgroup Information ........................................................................................................ 8-25
Viewing Network Entries .................................................................................................................... 8-26
Planning Network Pathgroups
and Meta-Devices ............................................................................................................................... 8-27
Meta-Network Interfaces ..................................................................................................................... 8-28
Creating a Network Pathgroup .......................................................................................................... 8-29
FDDI Setup Considerations ................................................................................................................. 8-30
UltraEnterprise 10000 Administration ix

Switching a Network Pathgroup ........................................................................................................ 8-31

Deleting a Network Pathgroup ........................................................................................................... 8-32
Viewing Disk Entries ............................................................................................................................ 8-34
Disk Path Components ......................................................................................................................... 8-35
Planning a Disk Pathgroup
and Meta-Disks ................................................................................................................................... 8-36
Meta-Disk Configuration Example .................................................................................................... 8-37
Meta-Disk Configuration Example .................................................................................................... 8-38
Creating a Disk Pathgroup
and Meta-Disks ................................................................................................................................... 8-39
Using the Meta-Devices ....................................................................................................................... 8-40
Disk Managers and AP ........................................................................................................................ 8-41
Manually Switching the Active Path .................................................................................................. 8-42
Switching Back to the Primary Path ................................................................................................... 8-43
Automatic Disk Pathgroup Switching (AP 2.1) ................................................................................ 8-44
Deleting a Disk Pathgroup .................................................................................................................. 8-45
AP and the Boot Disk ........................................................................................................................... 8-46
Placing the Boot Disk
Under AP Control .............................................................................................................................. 8-47
Removing AP Support
From the Boot Disk ............................................................................................................................ 8-48
Using apboot With a
Mirrored Boot Disk (Solaris 2.6 Only) ............................................................................................. 8-49
The AP Recovery Boot Sequence ........................................................................................................ 8-50
Using AP in Single-User Mode ........................................................................................................... 8-51
Dynamic Reconfiguration .........................................................................................................9-1

Module Overview ................................................................................................................................... 9-2
Course Map .............................................................................................................................................. 9-3
UltraEnterprise 10000 Administration x

Objectives ................................................................................................................................................. 9-4

Dynamic Reconfiguration Capabilities ................................................................................................ 9-5
Dynamic Reconfiguration ...................................................................................................................... 9-6
dr-max-mem (Solaris 2.6) ........................................................................................................................ 9-7
dr-max-mem (Solaris 2.5.1) ..................................................................................................................... 9-8
DR Attach ............................................................................................................................................... 9-11
Attaching a Board With DR ................................................................................................................. 9-13
With hostview ..........................................................9-14
With hostview. .................................................................................................................................. 9-15
With hostview ..........................................................9-16
I/O Device Reconfiguration After a DR Operation ......................................................................... 9-17
Viewing System Information .............................................................................................................. 9-18
System Information Using drshow .............................................9-19
System Information Using hostview. ............................................................................................... 9-21
System Information Using hostview. ............................................................................................... 9-22
DR Detach .............................................................................................................................................. 9-26
Detaching a System Board ................................................................................................................... 9-27
Finishing the Complete Detach Operation ........................................................................................ 9-28
Configuring for DR Detach ................................................................................................................. 9-29
Detaching Network Devices ................................................................................................................ 9-31
Detaching Non-Network Devices ....................................................................................................... 9-32
DR Detach-Safe Devices ....................................................................................................................... 9-33
Unloading a Loaded
Detach-Unsafe Driver ........................................................................................................................ 9-34
Detaching a Board With dr ..................................................9-38
Detaching a Board With hostview ............................................9-39
Detaching a Board With hostview ............................................9-40
Pageable and Permanent Memory ..................................................................................................... 9-41
UltraEnterprise 10000 Administration xi

Operation:Permanent Memory
on Target Board .................................................................................................................................. 9-42
Operating System Quiesce ................................................................................................................... 9-43
Suspend-Safe and Suspend-Unsafe
Devices ................................................................................................................................................. 9-45
Adding New Suspend-Safe Drivers ................................................................................................... 9-46
Adding New Suspend-Bypass Drivers .............................................................................................. 9-47
DR Attach ............................................................................................................................................ 9-48
DR Detach ........................................................................................................................................... 9-49
Troubleshooting .......................................................................................................................10-1
Module Overview ................................................................................................................................. 10-2
Course Map ............................................................................................................................................ 10-3
Objectives ............................................................................................................................................... 10-4
Standard Domain Message Logs ........................................................................................................ 10-5
the Figure of Merit ............................................................................................................................. 10-6
Redlist and Blacklist Files .................................................................................................................. 10-10
The autoconfig Command .............................................................................................................. 10-11
Diagnostic Tools .................................................................................................................................. 10-12
Correctable Memory Errors ............................................................................................................... 10-14
System Failures .................................................................................................................................... 10-15
Reboot Request .................................................................................................................................... 10-16
Panic ...................................................................................................................................................... 10-17
Watchdog, Redmode, and XIR Resets ............................................................................................. 10-18
Heartbeat Failure (Hung Host) ......................................................................................................... 10-19
Arbstop ................................................................................................................................................. 10-21
UltraEnterprise 10000 Administration xii

Creating a Hardware State Dump File ............................................................................................ 10-22

redx ...................................................................10-23
Technical Information for Escalation ............................................................................................... 10-24
Check Your Progress .......................................................................................................................... 10-25
UltraEnterprise 10000 Administration xiii


ES400 - Ultra Enterprise 10000 Administration - Oh - 0698

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ES400 - Ultra Enterprise 10000 Administration - Oh - 0698

Uploaded by

Copyright:

Available Formats

Sun Educational Services

Ultra Enterprise 10000

Sun Educational Services

Sun Cluster High Availability Administration June 1998

About This Course

Ultra Enterprise 10000 Administration Preface 1 of 10

Ultra Enterprise 10000 Administration Preface 2 of 10

Ultra Enterprise 10000 Administration Preface 3 of 10

Ultra Enterprise 10000 Administration Preface 4 of 10

• Describe the features of the Enterprise 10000 system.

Ultra Enterprise 10000 Administration Preface 5 of 10

Topics Not Covered

Ultra Enterprise 10000 Administration Preface 7 of 10

• Basic Sun network administration as demonstrated in

• Basic Sun troubleshooting and boot process knowledge

• Basic organization and operation of the SPARCcenter™

Ultra Enterprise 10000 Administration Preface 8 of 10

Ultra Enterprise 10000 Administration Preface 9 of 10

How to Use the Course Materials

Ultra Enterprise 10000 Administration Preface 10 of 10

Ultra Enterprise 10000

Ultra Enterprise 10000 Administration June 1998

Ultra Enterprise 10000 Administration Module 1, slide 2 of 25

Ultra Enterprise 10000 Administration Module 1, slide 3 of 25

• List the hardware and software components of the

Ultra Enterprise 10000 Administration Module 1, slide 4 of 25

Ultra Enterprise 1000 Capabilities

• Bus clock of 83.3 MHz, with 250 and 333 MHz

Ultra Enterprise 10000 Administration Module 1, slide 5 of 25

Ultra Enterprise 1000 Features

• Network Virtual Console (netcon)

Ultra Enterprise 10000 Administration Module 1, slide 6 of 25

(I/O expansion cabinet not shown) Processor cabinet

Ultra Enterprise 10000 Administration Module 1, slide 7 of 25

domain_a domain_b domain_c

Enterprise 10000 platform

domain_a domain_b domain_c

Ultra Enterprise 10000 Administration Module 1, slide 8 of 25

System Board 1 System Board 3

Ultra Enterprise 10000 Administration Module 1, slide 9 of 25

• You can use Dynamic Reconfiguration to:

• Upgrade a system board

• Modify the domain hardware configuration

Ultra Enterprise 10000 Administration Module 1, slide 10 of 25

Operating System Support

• A domain has kernel architecture sun4u1

• Supported like any other architecture

• The virtual console daemon cvcd is the only boot

• SSP software will not run on 2.6

Ultra Enterprise 10000 Administration Module 1, slide 11 of 25

Operating System Enhancements

Ultra Enterprise 10000 Administration Module 1, slide 12 of 25

• Responsible for controlling and managing the

Public-switched System console (SSP)

Ultra Enterprise 10000 Administration Module 1, slide 14 of 25

The SSP User Environment

• Used to manage the SSP itself

The SSP User Environment

% setenv SUNW_HOSTNAME domain2 Domain 2

• netcon windows act as system consoles for domains

Ultra Enterprise 10000 Administration Module 1, slide 16 of 25

The SSP User Environment

• Hostview provides a graphical control enviroment for