This is the RH342 Practice Exam.
Be sure to watch the introduction video for details regarding testing your progress.
Environment
LDAP/Kerberos authentication host:
ip-10-0-1-5.ec2.internal | 10.0.1.5
LDAP Base: dc=example,dc=com
Kerberos Realm: example.com
HostA:
ip-10-0-1-10.ec2.internal | 10.0.1.10
HostB:
ip-10-0-1-11.ec2.internal | 10.0.1.11
HostC:
ip-10-0-1-12.ec2.internal | 10.0.1.12
HostD:
ip-10-0-1-13.ec2.internal | 10.0.1.13
Changes requested in the tasks should persist through a restart of the system.
Solution
Start by logging in to the lab servers using the credentials provided on the hands-on lab
page:
ssh cloud_user@PUBLIC_IP_ADDRESS
Become the root user:
sudo su -
I recommend logging in to all four of the lab servers in separate tabs or windows to help
ensure we are running the right commands on the right hosts.
Also, keep in mind that there is a grading script that can be run on each host at any time
to help check your work. Be sure to replace the hostname with the name of the host you
are running the script on. Example:
score_HostA.sh
Task 1: Debug an application
On HostA, there is an application /root/example_app - troubleshoot and resolve this
application so that it will run successfully.
1. Install strace:
yum install -y strace
2. Run strace on the example_app:
strace example_app
3. Resolve the "No such file or directory" error by creating the necessary file:
touch /etc/example.conf
Task 2: Test an application for a memory leak
On HostA, test the application /root/memleak_test_app for a memory leak. If a
memory leak is detected, remove the executable bit from the application.
1. Install valgrind:
yum install -y valgrind
2. Run valgrind against memleak_test_app:
valgrind ./memleak_test_app
3. valgrind tells us there is indeed a memory leak. Remove the ability to execute
the application:
chmod -x memleak_test_app
Task 3: Manage a kernel module
On HostA, configure the module nf_conntrack to enable timestamping of network
flows. This change should persist through a reboot. Any services stopped to make this
change should be restarted.
1. Stop the firewall:
systemctl stop firewalld
2. Unload the module:
modprobe -r nf_conntrack
3. Load it w/ parameter:
modprobe nf_conntrack tstamp=1
4. Make it persist:
echo 'options nf_conntrack tstamp=1' >>
/etc/modprobe.d/nf_conntrack.conf
Task 4: Upgrade a package
On HostA, upgrade Apache (httpd) to the latest version and apply version control to
limit future upgrades.
1. Attempt to upgrade:
yum upgrade -y httpd
2. Edit /etc/yum.conf:
vim /etc/yum.conf
At the bottom of the file, remove the exclusion:
exclude=httpd
Save and close the file:
:wq
3. Upgrade Apache:
yum upgrade -y httpd
4. Install versionlock:
yum install -y yum-plugin-versionlock
5. Use versionlock:
yum versionlock httpd
Task 5: Resolve SELinux issues
On HostA, resolve any issues starting the Apache (httpd) service, or issues viewing the
content. The existing web content and configuration should not be modified.
1. Start Apache:
systemctl start httpd
2. Enable the service to start on system boot:
systemctl enable httpd
3. Attempt to display the default webpage:
curl localhost
4. Use yum to determine which package provides sealert to help us troubleshoot
the "Permission Denied" error:
yum provides */sealert
5. Install the necessary packages:
yum install -y setroubleshoot stroubleshoot-server
6. Load the audit log into sealert:
sealert -a /var/log/audit/audit.log
7. Find the error and utilize the suggested fix, enabling the boolean to allow httpd
to read user content:
setsebool -P httpd_read_user_content 1
8. Let's go ahead and enable the httpd service to start on system boot:
systemctl enable httpd
Task 6: Resolve authentication issues
On HostA, the LDAP/Kerberos user testuser01 should be able to log in and obtain a
Kerberos ticket using the password welcome1.
1. View the contents of authconfig to ensure the necessary configuration is set:
cat /etc/sysconfig/authconfig
2. Launch authconfig-tui:
authconfig-tui
o For the LDAP Settings:
Server: ldap://10.0.1.5
o For Kerberos Settings:
Realm: EXAMPLE.COM
KDC: 10.0.1.5
Admin Server: 10.0.1.5
3. Restart LDAP name scheduler:
systemctl restart nslcd
4. Log in to localhost as the testuser01 user:
ssh testuser01@localhost
5. Issue a Kerberos ticket:
kinit
6. List the Kerberos ticket:
klist
7. Log out as the testuser01 user:
exit
Task 7: Monitor system for vital characteristics
On HostA, there is a Performance Co-Pilot archive file. What was the 1 min kernel load
at 17:19:46?
Enter the value as a number into /root/load.txt. For example, if the load was 7.700,
the command would be:
echo "7.700" > /root/load.txt
1. Extract the archive:
tar xvzf archive.tgz
2. Install Performance Co-Pilot:
yum install -y pcp
3. Start Performance Co-Pilot:
systemctl start pmcd
4. Use pmval to determine the kernel load at the specified time:
pmval -t1m kernel.all.load -a ip-10-0-1-
10.ec2.internal/20190324.17.08.0
5. Insert the requested value into load.txt:
echo 9.090 >> /root/load.txt
6. We are finished with the tasks for HostA. Let's run the grading script to ensure
all of the steps were completed successfully:
score_HostA
Reboot HostA:
shutdown -r now
When the server has finished rebooting, log back in:
ssh cloud_user@PUBLIC_IP_ADDRESS
Become the root user:
sudo su -
Run the grading script again to ensure all tasks are able to persist through a
reboot:
score_HostA
Task 8: Recover data from a corrupted file system
On HostB, recover the data from the filesystem mounted at /storage.
The integrity of the files on the filesystem must be preserved.
1. View the contents of /etc/fstab:
cat /etc/fstab
2. Recover the ext4 filesystem:
fsck.ext4 -y /dev/mapper/vg_2-lv_2
3. Attempt to mount /storage:
mount /storage
4. View the contents of /storage:
ll /storage
Task 9: Recover data from a broken LVM configuration
On HostB, troubleshoot and resolve the LVM issues preventing the data on /lvm_mount
from being accessed.
The integrity of the files on the filesystem must be preserved.
1. View the contents of /etc/fstab:
cat /etc/fstab
2. Attempt to mount /lvm_mount:
mount /lvm_mount/
3. Use vgcfrestore to determine what broke the LVM:
vgcfgrestore -l vg_1
4. Use the config file from before the volume group was resized:
vgcfgrestore -f /etc/lvm/archive/vg_1_00002-1291080026.vg vg_1
5. Disable the logical volume:
lvchange -an /dev/vg_1/lv_1
6. Active the logical volume:
lvchange -ay /dev/vg_1/lv_1
7. Mount the logical volume:
ll /lvm_mount
Task 10: Recover data from an encrypted file system
On HostB, troubleshoot and resolve the issues preventing the data located at /luks from
being accessed. The integrity of the files on the filesystem must be preserved.
1. View the contents of /etc/fstab:
cat /etc/fstab
2. View the contents of /etc/crypttab:
cat /etc/crypttab
3. View slots:
cryptsetup luksDump /dev/mapper/vg_2-lv_1
4. Restore the header:
cryptsetup luksHeaderRestore --header-backup-file /root/vg_2-
lv_1.header /dev/mapper/vg_2-lv_1
5. View the contents of /etc/crypttab again:
cat /etc/crypttab
6. Manually unlock:
cryptsetup luksOpen /dev/mapper/vg_2-lv_1 luks-vg_2-lv_1 --key-
file /root/passphrase.key
7. View the contents of /luks:
ll /luks
Task 11: Troubleshoot issues affecting service start
On HostB, start the Performance Co-Pilot service and configure it to start automatically
upon reboot.
1. Attempt to start the pmcd service:
systemctl start pmcd
2. View the status of the service to determine what is causing the issue:
systemctl status pmcd
3. Resolve the cyclic dependency issue:
vi /etc/systemd/system/pmcd.service.d/dependency.conf
Be sure to change After to Before.
4. Start the pmcd service:
systemctl start pmcd
5. Enable the pmcd service to start automatically on server boot:
systemctl enable pmcd
6. We are finished with the tasks for HostB. Let's run the grading script to ensure
all of the steps were completed successfully:
score_HostB
Reboot HostB:
shutdown -r now
When the server has finished rebooting, log back in:
ssh cloud_user@PUBLIC_IP_ADDRESS
Become the root user:
sudo su -
Run the grading script again to ensure all tasks are able to persist through a
reboot:
score_HostB
Task 12: Troubleshoot system resource consumption
On HostC, something is consuming an excessive amount of CPU load. Prevent
whatever it is from being executable in the future.
1. View the load average:
uptime
2. Attempt to use pcp atop:
pcp atop
3. Start the pmcd service:
systemctl start pmcd
4. Use pcp atop to determine what is causing the high CPU load:
pcp atop
5. View more information about the process, using the PID of the problematic
process identified in the previous step:
ps aux | grep <PID>;
6. Remove the ability to execute the problematic script:
chmod -x /usr/bin/polling.sh
7. Kill the process:
kill -9 <PID>;
Task 13: Troubleshoot an application
On HostC, troubleshoot and resolve the issue affecting the use of the dig command.
1. Attempt to use dig on google.com:
dig google.com
2. Determine which shared libraries are being used:
ldd $(which dig)
3. Determine which packages provide the missing shared library:
yum provides */libdns.so.100
4. Reinstall bind-libs:
yum reinstall -y bind-libs
5. Use dig on google.com to verify the application is working properly:
dig google.com
Task 14: Troubleshoot service authentication
On HostC, troubleshoot and resolve the issue preventing cloud_user from listing the
local Samba shares. Do not modify the existing Samba configuration file
/etc/samba/smb.conf.
Username: cloud_user
Password: welcome1
1. Use smbclient to attempt to list the local Samba shares:
smbclient -U cloud_user -L localhost
2. View the most recent logs:
tail /var/log/samba/log/smbd
3. Verify the samba package
rpm -V samba
4. Move the changed file to `/tmp/:
mv /etc/pam.d/samba /tmp/
5. Reinstall samba:
yum reinstall -y samba
6. Verify that this has resolved the issue:
smbclient -U cloud_user -L localhost
Task 15: Troubleshoot iSCSI
HostC is an iSCSI target. Troubleshoot issues preventing HostD from accessing the
target. HostD should successfully be able negotiate the iSCSI connection and attach to
the block device.
On HostC
1. Start and enable the service to start automatically on server boot:
systemctl start target && systemctl enable target
2. Add a firewall rule to allow port 3260/tcp:
firewall-cmd --permanent --add-port=3260/tcp
3. Reload the firewall:
firewall-cmd --reload
4. View the current configuration:
echo "ls" | targetcli
On HostD
1. Verify the target from HostC matches the target on HostD:
vim /etc/iscsi/initiatorname.iscsi
Replace the initiator name in this file to match HostC.
2. Verify the authentication matches what is configured on HostC:
vim /etc/iscsi/iscsid.conf
Comment the following line:
node.session.auth.authmethod = CHAP
3. View any current records:
iscsiadm -m node
4. Discover the target:
iscsiadm -m discovery -t sendtargets -p 10.0.1.12
5. Log in using the target shown from the previous step:
iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.ip-10-0-1-
10.x8664:sn.a3776832068c -l
6. Verify:
lsblk
Task 16: Configure remote logging
Configure HostC as a remote log server for HostD. The connection should be over the
standard TCP port.
On HostC
1. Edit the rsyslog.conf file:
vim /etc/rsyslog.conf
Uncomment the following lines:
$ModLoad imtcp
$InputTCPServerRun 514
2. Restart the rsyslog service:
systemctl restart rsyslog
3. Add a firewall rule to allow port 514/tcp:
firewall-cmd --permanent --add-port=514/tcp
4. Reload the firewall:
firewall-cmd --reload
On HostD:
1. Edit rsyslog.conf:
vim /etc/rsyslog.conf
Uncomment the following line:
*.* @@remote-host:514
Edit that line to look like this:
*.* @@10.0.1.12:514
2. Restart the rsyslog service:
systemctl restart rsyslog
3. Send a log message:
logger "This is a test"
4. Run the grading script to ensure Task 15 and Task 16 are correctly configured:
score_HostD
Task 17: Troubleshoot RPM issues
HostD has a corrupt RPM database. Troubleshoot and resolve this issue.
1. Change to the rpm directory:
cd /var/lib/rpm
2. Attempt to verify the Packages file:
/usr/lib/rpm/rpmdb_verify Packages
3. Dump Packages and use it to load in the new Packages.new:
/usr/lib/rpm/rpmdb_dump Packages | /usr/lib/rpm/rpmdb_load
Packages.new
4. Verify the new Packages.new:
/usr/lib/rpm/rpmdb_verify Packages.new
5. Rename Packages to Packages.bad:
mv Packages Packages.bad
6. Rename Packages.new to Packages:
mv Packages.new Packages
7. Rebuild the RPM database:
rpm --rebuilddb
8. Clean yum:
yum clean all
Task 18: Troubleshoot networking
Step 1
Troubleshoot any networking issues preventing HostD from accessing the Apache
content on HostA at http://ip-10-0-1-10/index.html
On HostD
1. Attempt to view the Apache content on HostA:
curl http://ip-10-0-1-10/index.html
2. Attempt to view the Apache content directly:
curl 10.0.1.10
On HostA
1. Add a firewall rule to allow HTTP traffic:
firewall-cmd --permanent --add-service=http
2. Reload the firewall:
firewall-cmd --reload
On HostD
1. View the Apache content directly:
curl 10.0.1.10
2. Attempt to view the Apache content at the requested URL:
curl http://ip-10-0-1-10/index.html
3. View the contents of /etc/resolv.conf:
cat /etc/resolv.conf
4. Edit nsswitch.conf:
vim /etc/nsswitch.conf
Edit the hosts line to look like this:
hosts: files dns myhostname
5. View the Apache content at the requested URL:
curl http://ip-10-0-1-10/index.html
Step 2
There is a packet capture file located at /root/packet.cap. Examine this file and
discern the password used by testuser01. Enter this password as a single line into the
file /root/passwd.txt.
Example: echo "password" >> /root/passwd.txt
1. Install tcpdump:
yum -y install tcpdump
2. Use tcpdump to search the packet capture:
tcpdump -Ar packet.cap | less
Search the tcpdump output for testuser01:
/testuser01
3. Copy the password and save it in the file passwd.txt:
echo mypaSSword | /root/passwd.txt
Task 19: Examine and retrieve information from an sosreport
On HostD, examine the sosreport located at /root/sosreport-ip-10-0-1-11-2019-
03-22-wxoxhnk.tar.xz and determine the LDAP hostname the host attempted to bind
to.
Enter the name of this hostname as a single line in /root/test.txt.
Example: echo "server.example.com" >> /root/test.txt
1. Extract the sosreport:
tar xvf sosreport-ip-10-0-1-11-2019-03-22-wxoxhnk.tar.xz
2. Search the sosreport for LDAP:
cat sosreport-ip-10-0-1-11-2019-03-22-wxoxhnk/var/log/messages
| grep LDAP
3. Copy the LDAP server name and save it in the file /root/test:
echo ldap.example.com >> /root/test
Task 20: Assist in providing information for third-party investigation
On HostD, permit cloud_user to run example.stp as a module named cloud_tap.
1. Use SystemTap to run the file example.stp:
stap example.stp
2. Compile the module:
stap -p4 -m cloud_tap example.stp
3. Make a directory for our module:
mkdir /lib/modules/$(uname -r)/systemtap
4. Move the module to the new directory:
mv cloud_tap.ko /lib/modules/$(uname -r)/systemtap/
5. Run the module:
staprun cloud_tap
6. Add cloud_user to the stapusr group:
usermod -aG stapusr cloud_user
7. Ensure the cloud_user user can run the cloud_tap module:
runuser -l cloud_user -c 'staprun cloud_tap'
8. Just as before, reboot the host:
shutdown -r now
Once the server has rebooted, log in again:
ssh cloud_user@PUBLIC_IP_ADDRESS
Become the root user:
sudo su -
Run the grading script to ensure all steps were completed correctly:
score_HostD.sh
Conclusion
Congratulations, you've completed this hands-on lab!
