Test cases Description

Corner Case
kill , termial close of while hpnicrecovery running script
stop and start of auto recovery deamon while running manual recovery

In fact, the user retry "/sbin/init.d/hpnicrecovery dio_recover" after

stopping after "bus reset success", it ends up with:

ERROR: Interface not in DEAD state!! Hardware Path:0/0/0/3/0/0/2

PPA:lan18

WARNING!!
Don't Try to run recovery operation for Non-Dead card....

You can list up required lanadmin commands for the interface which was not
resumed in the function.

You may also consider another function like print_warn_before_reset()

which
can simply print "Retry /sbin/init.d/hpnicrecovery dio_recover" or something
like that.

Control parallel script execution.

Script should provide proper warning message, when user press “n” and
“ctrl+c” option.
Script should not display standup cards and other vendor cards

If hpvm services are stopped, while running a script, script should exit with
hpvm services are not running. As of now, it is not implemented.
Services are not running but some ports are exported in DIO pool. VSP ports
gone to dead. Are we going to support manual recovery or documentation. If
documentation, script should provide proper warning message.

If hpvm bundle is not available in the machine then dio recover script exit
with hpvm bundle is not available.
HPVM bundle is available but guest is not configured in the machine
 Manual recovery script should accept only “y” or “n” . As of now it is
accepting after “y” any letter.
Logs:
Above table shows the information of all the ports belonging to the dead
adapter distributed across VMs/host.
Kindly confirm if this is correct? (y/n) :yn

Export all the ports to guest and make it dead and run "ioscan -kfnNH 0/1/"
in VM/VPAR and "ioscan -fknNC hpvmdio" in while loop. And run
dio_recover script to recover the deas ports.
Change iocxgbe state to best "kcmodule -B iocxgbe=best", then run
dio_recover script
Try manual recovery with invalid vm name option
Try manual recovery with invalid hardware path

While running manual recovery try "n" option for below operaions,
1) while selecting the configuration is correct
2) Do you really want to proceed

while doing manual recovery in any stage , press ctrl+c should exit with
without any harm
Run the manual recovery script, where the hpvm bundle is not installed
machine
Run the manual recovery script where the hpvm services is stopped
Run the manual recovery script, where the standup cards available
Run the manual recovery script where guest is not configured

Generic test cases

VSP
Keep it all the ports in VSP and make it dead and check auto recovery works
fine or not

Make it dead and initiate manual recovery during auto recovery

Make it dead and do auto recovery and check the lun status of the port
(FCOE)
Make it dead , reset the IC module during auto recovery
Run a manual recovery on non-dead ports
Create vswitch and run traffic in hpvm and make dead
Stop the nicrecd services only in VSP and run the manual recovery

Stop the nicrecd services in VSP and Guest and run the manual recovery
Check the syslog file for dead port and after manual recovery
Check the nettl log file for dead port and after manual recovery
 While doing manual recovery, make another HBA port dead in VSP host,
and check whether auto recovery working starting after manual recovery
completes
Make VSP HBA ports dead, and guest HBA (differnant) port dead parellely,
while running a auto recovery , try to run manual recovery and check the
behavior
Check after removing the manual recovery depot, whether dio log file is
deleted or not
Check the hprecovery log file after the manual recovery

DIO Pool
Add all the ports to DIO pool and run manual recovery
Keep primary ports in VSP and add secondary ports to DIO pool and make it
dead in VSP ports and run manual recovery
Keep primary ports in DIO pool and add secondary ports in VSP and make it
dead in VSP ports and run manual recovery
Keep primary ports in HPVM/Vpar and add secondary ports to DIO pool and
make it dead in guest ports and run manual recovery
Keep primary ports in DIO pool and add secondary ports in HPVM/Vpar and
make it dead in guest ports and run manual recovery
Keep some ports to DIO and VSP , make it VSP ports to be dead and remove
the ports from the DIO pool then try manual recovery and check status of the
ports
Run a manual recovery on non-dead ports
Keep some port in DIO and VSP, make it dead and add dead dio ports to
guest and run manual recovery

HPVM/Vpar
Export all the ports to single VM/Vpar and make it dead and try manual
recovery
Export all the ports to different VM/Vpar and make it dead and try manual
recovery
Export different HBA ports to single VM/Vpar and make dead on all the
HBA ports and try manual recovery
Export a different HBA ports to different VM/Vpar and make dead on all the
HBA ports and try manual recovery
Keep primary ports in VSP and export secondary ports in HPVM/Vpar and
make dead and do manual recovery
Export a primary ports to HPVM/Vpar and keep secondary ports in VSP and
make dead and try manual recovery
Add primary ports in hpvm/vpar which is in OFF state, then make dead and
try manual recovery
After complete recovery, remove the ports from HPVM/Vpar
After complete recovery, add the VSP ports to HPVM/Vpar
 Keep some ports in VSP, DIO Pool and export to HPVM/Vpar and make
dead and try manual recovery
Run netperf traffic and make dead and try manual recovery
Try recovery in non-dead ports
Keep single cpu in HPVM/Vpar run traffic and make dead and try manual
recovery

Create a LM with same HBA ports and make dead and try manual recovery
Create a LM with different HBA ports and make it dead in primary port and
try recovery
Create a LACP with different HBA ports and make dead any one of the port
and try manual recovery

Create a tagged vlan and run traffic and make dead and try manual recovery
Create guest based vlan and run traffic and make dead and try manual
recovery
After make it dead , unload the driver after the suspend operation then try
resume operation
After complete recovery, try unload and load driver
Keep it some ports in VSP and some ports in guest/Vpar and make dead ,
after dead check the auto nicrecd log file , whether the DEAD message
logging continuously or once
Keep some ports in VSP and some ports in guest, and make the port to be
dead and stop the guest and try manual recovery
Check the syslog file for dead port and after manual recovery
Check the nettl log file for dead port and after manual recovery
Keep hpvm is in EFI shell , then make VSP ports to dead then try manual
recovery
Run netperf traffic with ioscan parellely and make dead and do DIO manual
recovery
Run netperf traffic with msscaf discovery test suite parellely and make dead
and do DIO manual recovery

hprecovery file
Check -a file created , after detecting the dead NIC in all the folder
Try manual recovery with vm name and hrdware path options

Installation

Install the bundle as below procedure,

1) Driver depot
2) DLPI patch
3) Nwmgr patch
Only install DLPI and check the dependency

Only install nwmgr patch and check the dependency

Verify all the bundle
Remove the DLPI patch and check old DLPI patch is available or not

Remove the nwmgr patch and check the old nwmgr patch is available or not
AT111 AT049A AT118

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS
PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

Pass PASS PASS

NA NA NA NA

PASS PASS PASS

PASS PASS PASS

Progress Progress Progress

PASS PASS PASS
PASS PASS PASS
Progress Progress Progress

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

PASS PASS PASS
PASS PASS PASS
PASS PASS PASS

PASS PASS PASS

PASS PASS PASS

 p

Verified in Superdome 2
Verified in Superdome 2

# /sbin/init.d/hpnicrecovery dio_recover ERROR: The script may be running in background for a while.

In nettl log file card after getting recovered messages are not coming
Nwmgr patch dependent on DLPI patch, It worked as expected
If we remove nwmgar patch, we can not run dio_recover. That is expected. To get the DIO recover solution,
latest DLPI and nwmgr patch should be installed.
S.No

5
 Title of the issue

While executing dio_recover command displays incorrect nwmgr commad. 'nwmgr -g -q info lan<ppa>' needs to
change 'nwmgr -g -q info -c lan<ppa>'
Verifying selected card is in dead state or not :

Warning !!
All the LAN ports are in Guest
So manually verify the ports are in dead state or not from the guest
running 'nwmgr -g -q info lan<ppa>' on each guest.
You can go ahead even if they are not in DEAD but it will make the interface
on each guest go DOWN during this operation and it can make network access
unavailable through the interface on the guest till the recovery operation ends.

After DIO manual recovery changes, recovery time is not getting updated in the automatic recovery log file. Previously
it used to update all the time details in “last_five”, “first” and “last”.
Automatic recovery log(/var/opt/hprecovery/hprecoverylog):
last_five:
first:
last:
Recovering Interface: 0/0/0/7/0/0

Observed below ERROR message during manual dio_recover if dio ports already in suspended state.
Verifying selected card is in dead state or not :
hpvmhwmgmt: ERROR - path: '0/0/0/5/0/0/0' cannot be deleted as it is SUSPENDED.
hpvmhwmgmt: Unable to manage dio pool resource.
hpvmhwmgmt: ERROR - '0/0/0/5/0/0/0' is already in the device database for guest use.
hpvmhwmgmt: Unable to manage dio pool resource.

After DIO manual recovery changes, recovery time is printing "reset" in the automatic recovery log file. Previously it
used to update all the time details in “last_five”, “first” and “last”.
Automatic recovery log(/var/opt/hprecovery/hprecoverylog):
Fri Nov 19 00:42:27 IST 2021
Lan14 in dead state!

0/0/0/5/0/0
last_five: reset reset reset
first: reset
last:
Recovering Interface: 0/0/0/5/0/0

During DIO manual recovery, if any of the VSP HBA ports gone to DEAD and if dio manual recovery script is not
completed/ if we exit early, the auto recovery is not scanning the system wide for any dead interface. Due to this the
VSP DEAD port not recovering automatically.
Severity Comments

Medium

Medium

Medium

Medium

Serious