===========================================================================

= CLASSROOM TROUBLESHOOTING =
= for foundation-based RED HAT ENTERPRISE LINUX 9 =
= Version: 9.x-7 20-May-2022 (ChangeLog at end of document.) =
= North American INSTRUCTOR HOTLINE +1 (855) 236-0912 =
= (Check with NIIT for hotline numbers for other regions) =
===========================================================================

This file contains the old "Appendix" from ClassroomSetup.txt.

INSTALLATION OF FOUNDATION SYSTEMS

----------------------------------

There are several log files that might provide additional information on
the progress of installations.

If the problems occur "early" in the boot process, our first thought, as
mentioned above, is that we are conflicting on existing content on the
hard drive. Try running "destroy", then try installing again. If this is
foundation0, then you can see the destroy option on the boot menu. If you
are doing this from the PXE menu, try hitting Escape and just typing
"destroy" at the boot: prompt.

If "destroy" does not help, we have seen reports from the field of aborted
rsync data on the USB causing problems, particularly in the isos/ directory.
Look to see if there is a hidden "dot" file in the isos/ directory and
delete it.

If "destroy" does not help and the media has no hidden file(s), then look to
confirm that RHEL 9 is fully compatible with the particular hardware by
performing a default installation using burned media of
rhel-baseos-9.x-x86_64-dvd.iso. Note that it is a dual-layer DVD now.

There have also been reports of "leftover" data from the manufacturer of
the USB that confuses anaconda's ability to mount the USB and loop mount
the .iso file on the media. Try using dd to truly wipe the USB stick
and then re-populate with rht-usb-* as described in ClassPrep.txt.

# dd if=/dev/zero of=/dev/sdd bs=1M

# sync
(remove, reinsert)
# ./rht-usb-* usbmkpart /dev/sdd msdos
# ./rht-usb-* usbformat /dev/sdd1
# ./rht-usb-* usbadd RHCIfoundation-<Tab>

There have also been reports of foundationX installs universally failing

part way through installation of packages. This would potentially be due
to corrupt artifacts on foundation0. You can check the artifacts on
foundation0 in one of two ways:

f0# rht-verify-f0 --validate

f0# rht-usb f0validate

There have been occasional reports of foundationX installs prompting for

language and keyboard. This is most likely due to the rht-locale file not
being properly created on foundation0 during its first boot. foundation0
is configured to scrape the needed information from /root/anaconda-ks.cfg
populating the /content/ks/rht-locale file that is retrieved by the
foundationX kickstart. If the file is empty on foundation0, use the
/etc/rc.d/rc.local-foundation0 file to determine what needs to be scraped
into it (typically the keyboard, lang, and timezone lines).

If it is not related to the above efforts, then if you can get to

console tty2, the files in /tmp might provide some insight as to which
disk is which and which NIC is which, etc:
- /tmp/rht-ksconfig - additional header line to kickstart
- /tmp/rht-packages - additional packages to include (normally for f0)
- /tmp/rht - resulting /etc/rht variables
- /tmp/rht-ksvars - specific kickstart variables (from pre to post)
- /tmp/rht-locale - downloaded timezone, keyboard, language from f0
- /tmp/rht-ks-pre.log - output from %pre

If the problem occurs later on, or after a successful reboot, if something

happened during the kickstart, reviewing the files on the system:
- /etc/rht - general variables used by lots of scripts
- /etc/rht-ksvars - specific kickstart variables (from pre to post)
- /root/rht-ks-pre.log - output from %pre
- /root/rht-ks-post.log - output from %post
- /tmp/rht-usb-*.log - debug output from rht-usb tool

If the problem occurs even later with rht-verify-* scripts, consider copying
them also:
- /root/rht-verify-f0.log - captured output of tool
- classroom:/root/rht-verify-classroom.log - captured output of tool

If you are submitting an error report to Jira, one or more of the above
files might be helpful in troubleshooting your problem. To collect all of
them, consider using rht-sosreport as described in the below section
"REPORTING ISSUES TO RED HAT"

MISMATCHED HARDWARE IN ROOM

---------------------------

There have been reports from the field that some classrooms have hardware
that is not identical throughout the room. While it is relatively simple
to address this when teaching a course, it can become problematic when
preparing to proctor an exam.

A utility was added to each and every foundationX system (including f0)
called rht-check-fX. This utility can be used in a number of ways:

A) Determine if hardware meets a certain hardware requirements level:

fX$ rht-check-fX --level 3 --verbose --novms

B) Check that VMs are running/functional (and qcow2 files are clean):

fX$ rht-check-fX --verbose

The second option can be rather time consuming as it will perform an

md5sum on each VM qcow2 file on the local machine. It will also test
connectivity to the expected workstation VM.

RHT-USB VERIFICATION REVIEW

---------------------------
rht-usb provides output in the various verbs of verify, usbverify,
udbvalidate, f0validate, and verifyquick. The following table endeavors
to list the purpose of each response.

GOODSUM - md5sum checked out OK - happy

CORRUPT - md5sum came back "bad" - not happy
MISSING - file is missing - should be there - not happy
EXISTS - file exists - we weren't asked to check md5sum - probably OK
NOCHECK - not checking for "some" reason - probably OK
ABORTED - md5sum aborted for some reason - not good
EXEFAIL - md5sum has *really bad failure* - not good
NOFSTAB - iso content not in fstab - not good
NOMOUNT - iso content not loop mounted - not good
WRNGMNT - iso content loop mounted to wrong place - not good
NOLOOPS - iso loop mount check failed - not good
RPMFAIL - rpm verification failed - probably not good
NOLINKS - hardlinks missing - not good
ALREADY - skipped md5sum because previously done - should be fine

FOUNDATION SYSTEM HARDWARE ISSUES

----------------------------------

Sometimes RHEL9 foundation machines have issues with certain hardware

combinations. If you suspect this is the case and you believe kernel
parameters will solve the problem, test your solution on one machine to
confirm that it fixes the problem.

If the hardware issue affects all of the student machines and you
have confirmed that extra kernel arguments solve the problem, use the
rht-pushkernelargs command to push your solution to their machines.
This will copy the default GRUB configuration on foundationX and create
a new boot stanza that includes the additional kernel options you specify
at the end of the "linux16" line.

There are two ways to run rht-pushkernelargs:

rht-pushkernelargs <station_number>
rht-pushkernelargs all

The program will prompt you for the kernel arguments you want added.
Double check your entry before you confirm your selection and push them
to the foundation machines specified.

NETWORK CONNECTIVITY ISSUES

---------------------------

The installation of foundation0 will associate the classroom network with

the *first* NIC in the system (which, depending on BIOS, might be the one
on the motherboard or might be the add-on card). The *second* NIC is
associated with the external facility network. If you are seeing a lack
of connectivity from *all* student foundationX systems, try swapping
cables on foundation0.

The installation of foundationX will attempt to associate the classroom

network with the NIC used for PXE booting. If the foundationX system has
more than one NIC, make sure that only one NIC is plugged in to the
classroom network. If the particular NIC dies part way through installation,
try using the "other" NIC connected to the classroom, or you may have to
disable one of the NICs as our attempts at link detection are being stymied
by the particular NIC chipset and/or switch.

If your foundation0 or foundationX system has three or more NICs, be

aware that the kickstart scripts will only examine the first two NICs as
discovered by the kernel, so be sure to plug your cable in to whatever
your system deems one of the first two.

A utility named rht-movebr0 can be used post-installation to swap the

underlying NIC.

fX$ rht-movebr0 --status

- will display the current networking configuration
fX$ rht-movebr0 --interface IFACE
- will disassociate br0 from the current interface then
associate br0 with the specified IFACE

STANDALONE/VIRTUAL INSTALLATION
-------------------------------

To support instructor practice with a nested virtualization foundation0

on robust hardware, a standalone configuration has been available. Rather
than choosing or typing "f0" to install, choose or type "standalone". It
takes all the same arguments and the basic process is the same as
described in ClassroomSetup.txt.

The fundamental difference is that a "standalone" foundation0 cannot

support any student foundationX systems. Also, the demonstration virtual
machines believe that the RHT_ENROLLMENT is student 1. In essence, this
is a hybrid instructor/student1 system that runs the instructor-only VMs
(like the classroom VM) along with student1 VMs.

Technically, this is accomplished with two changes:

- RHT_ENROLLMENT=513
- br0 (classroom bridge) has no physical NIC backing

When defining a virtual instance of "standalone", only a single NIC should

be created that is attached to a NAT'd virtual NIC that supports DHCP
(like virbr0 on RHEL). On the resulting foundation0, you would use the
rht-external utility to configure upstream access through the virtual NIC.

The virtual instance must also be configured to support host-passthrough.

This is to support the running of virtual machines inside the "standalone"
virtual instance. For example in a libvirt configuration:
<cpu mode='host-passthrough'/>
Confirm the proper virtualization support by running rht-verify-f0 and/or
virt-host-validate.

FOUNDATION SYSTEM LOCKUP ISSUES

--------------------------------

Sometimes RHEL9 foundation machines have issues with certain hardware

memory configurations that have translated in to periodic system lockup.
We believe that some of these are related to mismatches with upgraded
RAM. One way to test and verify this would be to run MemTest86+ on the
system.

Option A)
Boot from RHCI Foundation media (either CD or USB). At the resulting
boot: prompt, type "memtest" and hit return.

Option B)
Boot the system from the network showing the PXE menu served up by
foundation0. Hit Escape, and at the resulting boot: prompt, type "memtest"
and hit return.

**************************** WARNING ********************************

* Since the distributed memtest is a 16-bit program, it is *
* only available if you boot the system in Legacy mode. If *
* that is not an option, look to download a memory test *
* utility perhaps from memtest86.com to confirm that the *
* memory modules are fully functional. *
*********************************************************************

DNS CONNECTIVITY ISSUES

-----------------------

The ILT classroom is running two DNS servers, one on foundation0 and one
on classroom. The foundation0 DNS server is providing answers for the
foundation layer (physical systems) and domain .ilt.example.com. The
classroom VM is providing answers for the virtual layer (virtual machines)
and domain .example.com. Each forwards to the other for the defined domains
but each forward to root name servers for everything else.

Since both nameservers are sending external inquiries to root nameservers

on the Internet, be aware that some facilities block direct access to those
root nameservers.

To get around DNS restrictions by the facility, edit /etc/named.conf on

classroom and foundation0 to configure a forwarder to the facility's DNS
server (there is a commented example in the file). Depending on the
facility's DNS server, dnssec options may need adjustment too.

NETWORK CONFIGURATION EXTREME DETAIL (<= v8.x)

----------------------------------------------

Experimentation in the various toolchains on either foundation0 or

classroom have been known to "break" network configuration. Alternatively,
we have noticed that some will attempt to add a second NIC to foundation0
post install. So, here we will attempt to identify what the end result
configuration files should look like.

Let's look at foundation0 first. In /etc/sysconfig/network-scripts/, there

should be five ifcfg files for lo, br0, br1, {NIC1}, and {NIC2}.
The name of {NIC1} and {NIC2} will vary depending on the physical hardware.
The variable representations ${} below should be replaced with appropriate
values. {NIC1} and its parent br0 are physically connected to the
classroom network. If it exists, {NIC2} is connected to the external
facility network.

[root@foundation0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-br0

# Bridge Networking Interface
DEVICE=br0
NAME="Bridge ${NIC1}"
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
IPADDR0=172.25.254.250
PREFIX0=24
GATEWAY0=172.25.254.254
DEFROUTE=yes
IPADDR1=172.25.0.250
PREFIX1=24
DNS1=172.25.254.250
DOMAIN="ilt.example.com example.com"
IPV6INIT=no
PEERNTP=no
PEERDNS=yes
TYPE=Bridge
DELAY=0
STP=no
[root@foundation0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-${NIC1}
DEVICE=${NIC1}
NM_CONTROLLED=yes
BOOTPROTO=none
HWADDR=${NIC1_HWADDR}
UUID=${NIC1_UUID}
BRIDGE=br0
ONBOOT=yes
TYPE=Ethernet
NAME="System ${NIC1}"
[root@foundation0 ~]# cat /etc/sysconfig/network-scripts/ifcfg-${NIC2}
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=no
PEERROUTES=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=no
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=${NIC2}
DEVICE=${NIC2}
UUID=${NIC2_UUID}
ONBOOT=yes
NM_CONTROLLED=yes
ZONE=external

Note from the above configuration that br1 does not have an ifcfg file as
it is configured via libvirt (/usr/share/foundation0-config/br1.xml). Also,
note that the relationship between the physical NIC and its parent bridge
is defined in the physical NIC's ifcfg file as the line BRIDGE=. You could
manually reverse your NICs by modifying the two files and adjusting those
values.

Networking is managed by NetworkManager, so "nmcli con reload" can be used

to expedite activation of network file changes.
To enable external access, the rht-external utility can be used to update
the above ifcfg-${NIC2} file. The use of "rht-external --status" will show
you which NIC is externally configured or which NICs may be available for
configuration.

Note, when desired, external access can be turned off by running

"rht-external --reverse" on foundation0, then run "rht-external --configure
${NIC2}" to re-enable access.

Next, let's look at classroom. In /etc/sysconfig/network-scripts/, there

should be numerous ifcfg files for lo, ${NIC1}, and ${NIC2}. Depending on
the version of RHEL for classroom there may be a number of ${NIC1}:${subnet}
files too. Those ${subnet} files were used to correspond to each student
"X" (typically 0-20) along with two infrastructure values of 251 and 253.
Those were eliminated in more recently designed courses.

[root@classroom ~]# cat /etc/sysconfig/network-scripts/ifcfg-${NIC1}

DEVICE=${NIC1}
BOOTPROTO=static
ONBOOT=yes
TYPE=Ethernet
USERCTL=yes
PEERDNS=no
IPV6INIT=no
PERSISTENT_DHCLIENT=1
NETMASK=255.255.255.0
IPADDR=172.25.254.254
DNS1=172.25.254.254
ZONE=trusted
[root@classroom ~]# cat /etc/sysconfig/network-scripts/ifcfg-${NIC1}:${subnet}
DEVICE=${NIC1}:${subnet}
IPADDR=172.25.${subnet}.254
NETMASK=255.255.255.0
ONPARENT=yes
[root@classroom ~]# cat /etc/sysconfig/network-scripts/ifcfg-${NIC2}
DEVICE=${NIC2}
BOOTPROTO=dhcp
ONBOOT=no
TYPE=Ethernet
USERCTL=yes
PEERDNS=no
IPV6INIT=no
ZONE=external

To enable external access for the virtual layer, the rht-config-nat script
on classroom simply brings up the ${NIC2} interface. It will *always* use DHCP
now to get an address from br1's dnsmasq on foundation0.

PHYSICAL REGISTRATIONS
----------------------

The student (and additional instructor) physical systems will register

with foundation0 as part of the kickstart scripting.

These registrations are located on foundation0 under /var/www/register/

and amount to a content file named for the physical system's fully-
qualified domain name (FQDN). For example, we will find a number of files
named foundationX.ilt.example.com in that directory.
While management of the directory is relatively automatic (physical system
installations will add to the directory and rht-wipeclassroom will remove
entries), the instructor may run in to situations where manual management
is required.

Use "touch" and "rm" to add and remove registrations manually to/from this
directory.

**************************** WARNING ********************************

* Do not add foundation0.ilt.example.com as that system is *
* used/managed uniquely. Every other physical system should *
* appear in this directory. *
*********************************************************************

A pair of new utilites has been introduced to help with managing the
/var/www/register/ directory: rht-discover and rht-unregister.

f0$ rht-unregister all

- will remove all registrations from the directory.

f0$ rht-unregister 14
- will remove registrations with 14 as the number.

f0$ rht-discover all

- will loop through all IPs 0-200 and ping/register them.

f0$ rht-discover all 15

- will loop through all IPs 0-15 and ping/register them.

f0$ rht-discover 14
- will ping/register with 14 as the number.

The registrations (list of files) are used by the following utilities:

/usr/local/sbin/rht-wipeclassroom
/usr/local/bin/rht-clearcourse
/usr/local/bin/rht-pushcourse
/usr/local/bin/rht-pushkernelargs
/usr/local/bin/rht-showstate
/usr/local/bin/rht-each

COURSE MANIFEST DEPLOYMENT MANAGEMENT

-------------------------------------

With the release of 8.x-7 of rht-usb, we have introduced a concept of

quiescing deployed manifests. This means that foundation0 (and classroom)
can only have *one* active course manifest at a time. While we can copy
additional manifests onto foundation0, they will not be activated and
their manifest filename in /content/manifests/ will end in _quiesced.
When you now list the manifests on foundation0, you will see:

f0$ sudo rht-usb f0list

INFO Configuration file: /root/.icrm/config.yml
INFO Listing deployed manifests: /content/manifests
Active course : RH423-IDM4.5-0.r2018022615-ILT+VT+ROLE+RAV-7-en_US.icmf
Infrastructure : RHCIdevel-RHEL7-0.r2018032817-ILT-7-en_US.icmf
Infrastructure : RHCIfoundation-RHEL74-6.r2018032820.beta-ILT-7-en_US.icmf
Quiesced course: RH124-RHEL70-3.r52870-ILT-7-en_US.icmf_quiesced
 INFO Found 4 manifests deployed.

Note that there are now three "states" of manifests: Infrastructure,

Active course (there should be only one), and Quiesced course (there may
be several of these).

This now identifies that we have course manifests that are deployed or
copied onto foundation0, but we have not mounted the isos nor installed
the rpms. Only the active manifest has its isos mounted and rpms installed.
The very first course manifest deployed onto a foundation0 or classroom is
automatically activated. It is subsequent manifests that will only be
deployed (in a quiesced state).

This has introduced a new verb called f0activate that will swap which
course manifest is active. It starts by quiescing the currently active
manifest by unmounting the isos, uninstalling the rpms, and renaming the
manifest file in /content/manifests. It then mounts the isos, installs the
rpms, and renames the manifest file in /content/manifests for the
requested course manifest.

f0$ sudo rht-usb f0activate RH124-RHEL70-3.r52870-ILT-7-en_US.icmf_quiesced

Be aware, you should look at ClassroomReset.txt to properly clear the

original course prior to activating a different course manifest. That
document also identifies that after swapping course manifests, you will
still need to run rht-setcourse to then fully use the now active course.
What is now different is that we should no longer need to use f0remove
to fully delete the manifest from foundation0 before deploying additional
course manifests.

CLASSES WITH MORE THAN TWENTY STUDENTS

--------------------------------------

While we strongly discourage scheduling classes with a single instructor

and more than twenty students, there is tooling to support up to 200
students in a single classroom provided there is available hardware and
network bandwidth.

The first twenty foundation systems are selectable from the menu. To
install one beyond 20, at the menu hit Escape. At the resulting
boot: prompt enter the system you want to install ("f25").

boot: f25

Ignore the RHT_MAXSTATIONS variable in /etc/rht on foundation0. It was

used before we added PHYSICAL REGISTRATIONS (see above). The utilities
listed there now use the /var/www/register/ directory instead of looping
from 1 to RHT_MAXSTATIONS.

While the physical layer is prepared for up to 200 out of the box, the
virtual layer, in particular the classroom virtual machine, was capped at
twenty. To modify the capacity of the virtual layer, you will need to run
a utility called rht-config-capacity.

classroom$ rht-config-capacity 50

**************************** WARNING ********************************

* Several versions of rht-config-capacity have been plagued *
 * with bugs so there is some risk here. For example, in one *
* version there is a variable case mismatch preventing the *
* new value being properly passed to the various rht-setup-* *
* utilities. In another version, certain rht-setup-* scripts *
* fail to properly clean up and recreate items. The risk *
* factor here is directly proportional to the "age" of the *
* classroom VM (newer is generally better). *
* *
* We are shifting to all classes using a generic classroom *
* image which will allow us to better update/fix released *
* bits. *
*********************************************************************

Recent courses have the classroom virtual machine set to 200 by default.

CLASSES WITH OTHER OPERATING SYSTEMS

------------------------------------

Historically the Red Hat classroom assumed an ability to entirely wipe

and replace anything that may reside on the local system hard drives.
Coexisting with other operating systems in the classroom introduces a
large number of potential complications.

If you want to support a "dual boot" environment in the classroom, you

should install the "other" operating system first and shrink the disk
usage to provide for at least 100GB of free/unpartitioned space on the
local hard drives. Some classes will require even more free disk space
for the number of isos/VM images. Refer to the Hardware Requirements doc
to identify the amount of disk space that should be available to the
installation of Red Hat Enterprise Linux for the Red Hat classroom.

Once the "other" operating system is properly configured, there are two
avenues to installing a dual boot scenario depending on whether you are
looking at foundation0 or foundationX.

When installing foundation0, use the appropriate keystrokes to add an

argument to the installation called "dualboot" and the system will no
longer wipe the existing non-Linux partitions. This can also be used to
support installation of foundationX when being done via CD/USB.

When installing foundationX via PXE, you will want to first use the
utility named rht-dualboot on foundation0 to adjust the arguments of the
PXE menu to default to dualboot or not.

f0$ rht-dualboot on

will turn on dualboot by default for all foundationX systems installed

via PXE. The argument "off" will disable this default while the argument
"status" will report the current default.

The destroy option via GRUB is also adjusted with the use of dualboot
during install by adding dualboot to the default GRUB destroy menu entry.
When the dualboot option is passed to destroy it will attempt to avoid
removing *all* partitions from the local hard drive if it detects a
Microsoft-based bootable partition (tested with DOS and Windows 10). If
you want to wipe a classroom (see ClassroomRest.txt) and remove all
partitions, pass a new option to rht-wipeclassroom called --nodualboot
which will remove the protections for all foundationX systems. If you
want to manually remove the protections, simply edit the GRUB entry and
delete the "dualboot" argument from Destroy.

Be aware that running destroy from booting from CD or USB will always
remove all partitions.

VM MANAGEMENT
-------------

The virtual machines used in these classes are based on images that are
downloaded from foundation0.ilt.example.com's /content/${RHT_VMTREE}/vms/
directory. The naming of the image files is based on
${RHT_COURSE}-VMNAME-DISKNAME.qcow2, i.e. rh124-desktop-vda.qcow2 is the
first disk of the desktop VM for course rh124. The resulting image files
are stored locally in /var/lib/libvirt/images/. They should *never* be
accessed directly. The rht-vmctl tool not only downloads the image files
for us, but creates an overlay file (in the same images directory) and
adjusts the libvirt xml definition to point to the overlay.

The rht-vmctl will rely on /etc/rht on the foundationX system (updated by

rht-setcourse on f0 and rht-pushcourse for fX). In particular, it retrieves:
RHT_COURSE - Course code (i.e. "rh124")
RHT_VMTREE - where images are kept (i.e. "rhel7.0/x86_64")
RHT_VMS - list of VMNAMEs to deploy (i.e. "desktop server")
- used for validation (or looped when we use "all")

The rht-vmctl tool has a number of verbs. Quoted items are the external
virsh commands. We
point students to the following verbs:

fX$ rht-vmctl start VMNAME

- will download if missing qcow2, create if missing overlay, and
"start" the VM - end result should be a running VM.
fX$ rht-vmview view VMNAME
- if running, will launch virt-viewer to see console of VM.
fX$ rht-vmctl stop VMNAME
- will attempt to "shutdown" VM, and after a timeout "destroy" VM -
end result is the VM no longer running.
fX$ rht-vmctl reset VMNAME
- will "destroy" VM, remove the overlay, restore if exists last
saved checkpoint as overlay, and calls above start - end result
is a running VM reverted to previous state.

**************************** WARNING ********************************

* The "shutdown" operation typically times out due to a bug *
* in Gnome3 in early RHEL 7.x which prevents the VM from *
* recognizing and acting on the shutdown request if at the GDM *
* login prompt. The shutdown operates correctly if the student *
* has logged in to the VM. Alternatively, students can do a *
* manual shutdown of the VM prior to running rht-vmctl. *
*********************************************************************

While we point students to use the particular VMNAME, the instructor may
want to take advantage of using "all" as the VMNAME. The rht-vmctl tool has
a number of additional verbs that may be of use to you as the instructor:

fX$ rht-vmctl status VMNAME

- will "list" the VM
 fX$ rht-vmctl poweroff VMNAME
- will "destroy" VM.
fX$ rht-vmctl save VMNAME
- will call above stop, copy the overlay to time stamped backup,
call above start - end result is the same running VM.
fX$ rht-vmctl fullreset VMNAME
- will "destroy" VM, delete timed backups, delete overlays, delete
qcow2 image files, "undefine" VM, call above start - end result
is a running VM from virgin f0 images.
fX$ rht-vmctl get VMNAME
- will download if missing qcow2, create if missing overlay, and
update the libvirt XML, but will *NOT* start the VM.
fX$ rht-vmctl restore VMNAME
- if a saved checkpoint exists, will "destroy" VM, remove the
overlay, restore last saved checkpoint as overlay, and calls
above start. If no saved checkpoint, aborts.

GRADING SCRIPT ISSUES

---------------------

Grading scripts are physically found on foundation0 and classroom in

/content/courses/<coursecode>/<release>/grading-scripts/.

The scripts are dynamically downloaded as the student runs the *lab*
wrapper script. If you find any logic or code errors in the scripts,
correct them in the above directory, but you will also have to remove
the script from each student's virtual machine (or it happens automatically
when the student resets them) in /usr/local/{bin or lib}/ to trigger lab to
download the updated script. Also, please forward any patches to
curriculum-support@redhat.com and attach to a Jira report.

You will find that the lab scripts only work on the virtual machines.
This is by design.

Earlier courses downloaded the lab-<scriptname> file to /usr/local/bin/

but now downloads to /usr/local/lib/. Earlier courses also had you run
the lab script on whichever virtual machine the exercise was geared for
but now is only run from the workstation virtual machine.

A newer lab wrapper script also supports TAB completion. To view all
script names, on workstation:
workstation$ lab <TAB><TAB>

To download a particular lab script without it performing any actions

workstation$ lab <scriptname> help
You can now review the lab script by looking at the contents of the
/usr/local/lib/lab-<scriptname>.

If you want to update a grading script and distribute to your students,

on foundation0, navigate to the appropriate grading-scripts/ directory
and modify the lab-<scriptname> file. Then on each student machine, wipe
out the current grading-scripts.
f0$ rht-each "ssh root@workstation lab --refresh"
- or -
f0$ rht-each "ssh root@workstation rm -f /usr/local/lib/lab-<scriptname>"
You may need to substitute other VM names for workstation if doing this
on earlier courses.
In more recently developed courses, the lab script will log its activity
in /var/tmp/labs/ capturing both stdout and stderr in two files named
<scriptname> and <scriptname>.err.

WAYLAND VS. XORG

----------------

The installation of physical systems will default to trying to use

Wayland as the Gnome display server. Some video hardware that does not
support Wayland will fallback to use Xorg.

Wayland, at least initially in RHEL 8.0, has been known to not work in
the ways many instructors are used to with Xorg. If desired, a utility
named rht-switchgnome can be used post-installation to swap the desired
display server.

fX$ rht-switchgnome --status

- will display the current usage and default configuration
fX$ rht-switchgnome --configure xorg
- will change the default configuration to use Xorg. A reboot is
necessary to activate.
fX$ rht-switchgnome --configure wayland
- will change the default configuration to use Wayland. A reboot
is necessary to activate.

MONITOR/PROJECTOR ISSUES
------------------------

Projector and monitor blink about 3 seconds apart.

--
May see this message on the TTYs:
[drm] nouveau 0000:03:00.0: no native mode, forcing panel scaling.

Possible workarounds (add to GRUB):

* drm_kms_helper.poll=0 (thanks, Michael J. Clarkson)
* nomodeset (thanks, RS Pete Davis)
* acpi=off (thanks, Scott McBrien)

Machine will not boot with monitor connected

--
Try booting the system, then connecting the projector.
There may be function keys (on a laptop) that will change display output.

The following was submitted by Petros Tselios (thanks)

--
QA at the UK uses 2 monitors in their classroom. One of them is connected
to the VGA port of the instructor PC and the second one on the DP of it.
The VGA is connected to an ATEN KVM which is connected to the beamer as
well. And here is the problem. The monitors have a native resolution of
1280x1024 and the beamer support this resolution as well. However, the
EDID (probably because of the ATEN) is probably broken and Wayland
ignores it. Thus, the VGA monitor and the beamer will have only 1024x768
as a resolution.

Since xrandr isn't working under wayland, the solution is to force this
resolution for the specific interface. Thus, you need to add the following
line to the kernel options:

drm_kms_helper.edid_firmware=card0-VGA-1:edid/1280x1024.bin

You can verify the card information from the command:

ls -l /sys/class/drm

The following was submitted by Janez Trenz (thanks)

--
On some graphics cards it is not advisable to disable kernel mode setting,
because it makes X very, very, very sluggish. Some drivers don't work
without KMS anymore.

So, before you disable KMS with the nomodeset kernel parameter,
because your screen turns black or monitor goes to sleep, try a couple
of parameters. The problem might be that there is no correct EDID data
available. There are a few generic EDID profiles in the kernel that you
can use.

drm_kms_helper.edid_firmware=VGA-1:edid/1024x768.bin

will set the resolution for *both* console *and* X.Org server. There are
only 1024x768, 1280x1024, 1600x1050 and 1920x1080 resolutions available
with 60Hz refresh rate. Of course you should first find out which output
is being used on graphics card, you can boot into emergency or rescue
targets and look at /sys/class/drm/card*. For the above parameter there
should be a /sys/class/drm/card0-VGA-1.

If you have more than one output you can check which one is connected
by looking into /sys/class/drm/*/status files.

There is a way to set only the console resolution, so you can see
if KMS actually does work and produces a picture on your screen and
projector with

video=VGA-1:1280x800

This kernel parameter unfortunately will *not* setup the X.Org resolution
and if you need such a wide resolution which is not available in kernel
EDID data, you will have to create a /etc/X11/xorg.conf.d/50-screen.conf
file with a content like this.

Section "Screen"
DefaultDepth 24
Identifier "Builtin Default Screen 0"
Subsection "Display"
Depth 24
Modes "1280x800" "1024x768"
ViewPort 0 0
EndSubsection
Subsection "Display"
Depth 16
Modes "1280x800" "1024x768"
ViewPort 0 0
EndSubsection
EndSection

SLOW MACHINE
------------

Slow Machine after boot (Thanks, Wander Boessenkool)

--
If your students workstations (or foundation0 machine for that matter)
become unbearably laggy at some points it might be ksm.
An easy fix for this is to run the following commands on foundation0 and
foundationX:

systemctl disable ksmtuned

systemctl stop ksmtuned
systemctl disable ksm
systemctl stop ksm

Slow Machine during installation (Thanks, Steve Bonneville)

--
The installation may be slow if you are using USB 1.1 ports. Some machines
still have USB 1.1 ports on the system. There may be no visible distinction
on the case between these ports and USB 2 or USB 3 ports. You may want to
try to use a different port on the system for the USB installation stick.

GRUB ISSUES
-----------

There is a new feature of RHEL that uses USB devices before hard drives:
https://bugzilla.redhat.com/show_bug.cgi?id=497019

This means that using a USB device to install is usually seen as /dev/sda,
and grub may be written to this device. The kickstart file attempts to find
the real hard drive and install grub to it. If this does not succeed, leave
your USB device in foundation0 to boot it. Once it has booted, use
'grub2-install /dev/sda' to install grub2 to the MBR. You may also need to
remove entries pointing to the USB device found in /etc/fstab.

BOOT ISSUES (Thanks, Brent Langston)

-----------

If you have an instructor system that will not boot after installation, one
possible fix is to avoid plugging in the USB device until after the machine
has booted from the rhci-foundation CD. Plug in the USB device and type
f0.

OTHER BOOT ISSUES (Thanks Randy Hamilton and Petros Tselios)

-----------------

We are making a fundamental shift to the "Boot from local drive" in the
Legacy PXE menu. Historically, we were using "localboot 0xffff" which has
a number of problems with many Dell and HP equipment. We have changed to
using a pair of lines: kernel chain.c32 and append hd0. This new setting
is expected to work more universally across vendors, but are anxious to
hear feedback from the field with specific hardware this may be having a
regression due to this change.

As a workaround if you are experiencing a regression, you can edit the

/var/lib/tftpboot/pxelinux.cfg/default file, comment the two new lines
and uncomment the localboot line.

SETTING KEYBOARD ISSUES (Thanks, Wander Boessenkool)

-----------------------

The straight-forward, and completely keyboard based method to set

keyboards on a VM:

1) SSH in (you can do this in a for loop if you want)

2) localectl set-x11-keymap <something>
Where <something> is the correct keyboard layout. If you need variants
and options specified, those go on that line as well, "see man
localectl". locatectl list-x11-keymap-layouts can show you the entire
available list.
N.B. This also sets the console keymap, in /etc/vconsole.conf
3) Even though the console keymap is set, your systems won't use it.
Why? Because we have vconsole.keymap entries in grub.cfg to make the
keyboard behave during early boot as well. Let's fix that:

. /etc/vconsole.conf; sed -i "s/vconsole.keymap=[^ ]*/vconsole.keymap=${KEYMAP}/"

/etc/default/grub
grub2-mkconfig > /boot/grub2/grub.cfg

4) shutdown, rht-vmctl save from foundationX, profit

After this, students logging in to Gnome will have the choice between
keyboards, using the keyboard selector in the top-right corner of the
screen, with the recently set keyboard as the default. Removing the US
keyboard from that list can also be done, but it requires scripting it
with dconftool, or by forcing a system dconf database on users using
/etc/dconf/db/site.d/. Both are not worth the effort, especially since
some of your students will want to use a US layout, no matter what the
actual symbols on the physical keyboard say.

REPORTING ISSUES TO RED HAT

---------------------------

Red Hat has added a utility to foundation0 to aid in the collection of

data about your environment called rht-sosreport. This utiliy requires
one argument, your RHNID, or perhaps known as your redhat.com login
username which allows you to access Instructor Central. This RHNID will
be embedded in the filename of the gzip tarball. The expectation is that
you would attach this file to a Jira report or possibly send to the
author of RHCIfoundation, rlocke@redhat.com, with details of the issues
that you faced.

=============================================================================

CHANGELOG
=========
* Fri May 20 2022 Robert Locke <rlocke@redhat.com> 9.x-7
- prep for release
* Fri Sep 17 2021 Robert Locke <rlocke@redhat.com> 8.x-7
- add section on STANDALONE/VIRTUAL INSTALLATION
* Fri Jul 16 2021 Robert Locke <rlocke@redhat.com> 8.x-7
- add section on MISMATCHED HARDWARE
* Thu Jan 27 2019 Robert Locke <rlocke@redhat.com> 8.x-7
- add references to local boot pxe configuration
- add tip for managing resolution in Wayland
* Thu May 30 2019 Robert Locke <rlocke@redhat.com> 8.x-7
- prep for release
* Thu Apr 11 2019 Robert Locke <rlocke@redhat.com> 8.x-7
- Review for changes based on RHEL8
* Sun Mar 17 2019 Robert Locke <rlocke@redhat.com> 8.x-7
- add references to rht-usb usbmkpart
* Thu Jul 26 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add references to dualboot support
* Wed Apr 4 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add references to rht-usb f0activate
* Tue Mar 20 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add references to rht-sosreport
* Fri Mar 2 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- remove references to second physical system
* Tue Feb 6 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add RHT-USB VERIFICATION REVIEW
* Mon Jan 22 2018 Robert Locke <rlocke@redhat.com> 7.x-5
- add mention of /var/tmp/labs/
* Mon Nov 27 2017 Robert Locke <rlocke@redhat.com> 7.x-5
- add a cap to rht-discover all
* Thu Jul 6 2017 Robert Locke <rlocke@redhat.com> 7.x-5
- add references to rht-movebr0
- add references to rht-discover/rht-unregister
* Mon Sep 19 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- expand grading script issues - thanks Phil
* Wed Jul 13 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- add references to empty/missing /content/ks/rht-locale
* Fri Jun 10 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- shift uplink references to rht-external
- shift to new unbacked br1 in documentation
* Fri Apr 22 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- add references to managing oversized classes
* Tue Feb 2 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- change hostX reference to galaxyX
* Mon Oct 12 2015 Robert Locke <rlocke@redhat.com> 7.x-4
- prep for release
* Wed Aug 5 2015 Robert Locke <rlocke@redhat.com> 7.1-4
- add information about memtest availability
* Mon Jul 6 2015 Robert Locke <rlocke@redhat.com> 7.1-4
- bump release
* Mon Jun 1 2015 Robert Locke <rlocke@redhat.com> 7.1-3
- add reference to rht-verify-f0 --validate
* Wed May 20 2015 Robert Locke <rlocke@redhat.com> 7.1-3
- bump release
* Mon Apr 27 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- minor versioning fixes
* Mon Mar 16 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- add documentation on /var/www/register/
* Tue Mar 10 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- add detail about no wireless support currently
* Thu Mar 5 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- minor edits
* Fri Jan 16 2015 Robert Locke <rlocke@redhat.com> 7.0-2
- add references to "dot" files on USB media
* Fri Dec 5 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- add references to rht-verify-*.log files
* Sun Sep 28 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Add sync to dd of USB stick
* Tue Sep 16 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Add section on NETWORK CONFIGURATION EXTREME DETAIL
* Thu Aug 14 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Minor cleanups
- Add references to /tmp/rht-usb-<date>.log
* Fri Jul 18 2014 Robert Locke <rlocke@redhat.com>
- Additional foundation installation possibilities
* Wed Jul 2 2014 George Hacker <ghacker@redhat.com>
- Add more detailed X/video troubleshooting tip by Janez Trenz
* Mon Jun 30 2014 Robert Locke <rlocke@redhat.com>
- Add keyboard setting recommendation (Thanks Wander!)
* Fri Jun 27 2014 Robert Locke <rlocke@redhat.com>
- Add some DHCP and more multi-NIC information
* Wed Jun 25 2014 George Hacker <ghacker@redhat.com>
- Added instructions on how to use rht-pushkernelargs script
* Tue Jun 24 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Split out to ClassroomReset.txt and ClassroomTroubleshooting.txt