Professional Documents
Culture Documents
ClassroomTroubleshooting-8 x-7 R2019061318gite4a6727
ClassroomTroubleshooting-8 x-7 R2019061318gite4a6727
= CLASSROOM TROUBLESHOOTING =
= for foundation-based RED HAT ENTERPRISE LINUX 8 =
= Version: 8.x-7 30-May-2019 (ChangeLog at end of document.) =
= North American INSTRUCTOR HOTLINE +1 (855) 236-0912 =
= (Check with NIIT for hotline numbers for other regions) =
===========================================================================
There are several log files that might provide additional information on
the progress of installations.
If the problems occur "early" in the boot process, our first thought, as
mentioned above, is that we are conflicting on existing content on the
hard drive. Try running "destroy", then try installing again. If this is
foundation0, then you can see the destroy option on the boot menu. If you
are doing this from the PXE menu, try hitting Escape and just typing
"destroy" at the boot: prompt.
If "destroy" does not help, we have seen reports from the field of aborted
rsync data on the USB causing problems, particularly in the isos/ directory.
Look to see if there is a hidden "dot" file in the isos/ directory and
delete it.
If "destroy" does not help and the media has no hidden file(s), then look to
confirm that RHEL 8 is fully compatible with the particular hardware by
performing a default installation using burned media of
rhel-8.x-x86_64-dvd.iso. Note that it is a dual-layer DVD now.
There have also been reports of "leftover" data from the manufacturer of
the USB that confuses anaconda's ability to mount the USB and loop mount
the .iso file on the media. Try using dd to truly wipe the USB stick
and then re-populate with rht-usb-* as described in ClassPrep.txt.
If the problem occurs even later with rht-verify-* scripts, consider copying
them also:
- /root/rht-verify-f0.log - captured output of tool
- classroom:/root/rht-verify-classroom.log - captured output of tool
If you are submitting an error report to Jira, one or more of the above
files might be helpful in troubleshooting your problem. To collect all of
them, consider using rht-sosreport as described in the below section
"REPORTING ISSUES TO RED HAT"
If the hardware issue affects all of the student machines and you
have confirmed that extra kernel arguments solve the problem, use the
rht-pushkernelargs command to push your solution to their machines.
This will copy the default GRUB configuration on foundationX and create
a new boot stanza that includes the additional kernel options you specify
at the end of the "linux16" line.
rht-pushkernelargs <station_number>
rht-pushkernelargs all
The program will prompt you for the kernel arguments you want added.
Double check your entry before you confirm your selection and push them
to the foundation machines specified.
Option A)
Boot from RHCI Foundation media (either CD or USB). At the resulting
boot: prompt, type "memtest" and hit return.
Option B)
Boot the system from the network showing the PXE menu served up by
foundation0. Hit Escape, and at the resulting boot: prompt, type "memtest"
and hit return.
The ILT classroom is running two DNS servers, one on foundation0 and one
on classroom. The foundation0 DNS server is providing answers for the
foundation layer (physical systems) and domain .ilt.example.com. The
classroom VM is providing answers for the virtual layer (virtual machines)
and domain .example.com. Each forwards to the other for the defined domains
but each forward to root name servers for everything else.
Note from the above configuration that br1 does not have an ifcfg file as
it is configured via libvirt (/usr/share/foundation0-config/br1.xml). Also,
note that the relationship between the physical NIC and its parent bridge
is defined in the physical NIC's ifcfg file as the line BRIDGE=. You could
manually reverse your NICs by modifying the two files and adjusting those
values.
Networking is managed by NetworkManager, so "nmcli con reload" can be used
to expedite activation of network file changes.
To enable external access for the virtual layer, the rht-config-nat script
on classroom simply brings up the ${NIC2} interface. It will *always* use DHCP
now to get an address from br1's dnsmasq on foundation0.
PHYSICAL REGISTRATIONS
----------------------
Use "touch" and "rm" to add and remove registrations manually to/from this
directory.
A pair of new utilites has been introduced to help with managing the
/var/www/register/ directory: rht-discover and rht-unregister.
f0$ rht-unregister 14
- will remove registrations with 14 as the number.
f0$ rht-discover 14
- will ping/register with 14 as the number.
This now identifies that we have course manifests that are deployed or
copied onto foundation0, but we have not mounted the isos nor installed
the rpms. Only the active manifest has its isos mounted and rpms installed.
The very first course manifest deployed onto a foundation0 or classroom is
automatically activated. It is subsequent manifests that will only be
deployed (in a quiesced state).
This has introduced a new verb called f0activate that will swap which
course manifest is active. It starts by quiescing the currently active
manifest by unmounting the isos, uninstalling the rpms, and renaming the
manifest file in /content/manifests. It then mounts the isos, installs the
rpms, and renames the manifest file in /content/manifests for the
requested course manifest.
The first twenty foundation systems are selectable from the menu. To
install one beyond 20, at the menu hit Escape. At the resulting
boot: prompt enter the system you want to install ("f25").
boot: f25
While the physical layer is prepared for up to 200 out of the box, the
virtual layer, in particular the classroom virtual machine, was capped at
twenty. To modify the capacity of the virtual layer, you will need to run
a utility called rht-config-capacity.
classroom$ rht-config-capacity 50
**************************** WARNING ********************************
* Several versions of rht-config-capacity have been plagued *
* with bugs so there is some risk here. For example, in one *
* version there is a variable case mismatch preventing the *
* new value being properly passed to the various rht-setup-* *
* utilities. In another version, certain rht-setup-* scripts *
* fail to properly clean up and recreate items. The risk *
* factor here is directly proportional to the "age" of the *
* classroom VM (newer is generally better). *
* *
* We are shifting to all classes using a generic classroom *
* image which will allow us to better update/fix released *
* bits. *
*********************************************************************
Recent courses have the classroom virtual machine set to 200 by default.
Once the "other" operating system is properly configured, there are two
avenues to installing a dual boot scenario depending on whether you are
looking at foundation0 or foundationX.
When installing foundationX via PXE, you will want to first use the
utility named rht-dualboot on foundation0 to adjust the arguments of the
PXE menu to default to dualboot or not.
f0$ rht-dualboot on
The destroy option via GRUB is also adjusted with the use of dualboot
during install by adding dualboot to the default GRUB destroy menu entry.
When the dualboot option is passed to destroy it will attempt to avoid
removing *all* partitions from the local hard drive if it detects a
Microsoft-based bootable partition (tested with DOS and Windows 10). If
you want to wipe a classroom (see ClassroomRest.txt) and remove all
partitions, pass a new option to rht-wipeclassroom called --nodualboot
which will remove the protections for all foundationX systems. If you
want to manually remove the protections, simply edit the GRUB entry and
delete the "dualboot" argument from Destroy.
Be aware that running destroy from booting from CD or USB will always
remove all partitions.
VM MANAGEMENT
-------------
The virtual machines used in these classes are based on images that are
downloaded from foundation0.ilt.example.com's /content/${RHT_VMTREE}/vms/
directory. The naming of the image files is based on
${RHT_COURSE}-VMNAME-DISKNAME.qcow2, i.e. rh124-desktop-vda.qcow2 is the
first disk of the desktop VM for course rh124. The resulting image files
are stored locally in /var/lib/libvirt/images/. They should *never* be
accessed directly. The rht-vmctl tool not only downloads the image files
for us, but creates an overlay file (in the same images directory) and
adjusts the libvirt xml definition to point to the overlay.
The rht-vmctl tool has a number of verbs. Quoted items are the external
virsh commands. We
point students to the following verbs:
While we point students to use the particular VMNAME, the instructor may
want to take advantage of using "all" as the VMNAME. The rht-vmctl tool has
a number of additional verbs that may be of use to you as the instructor:
fX$ rht-vmctl status VMNAME
- will "list" the VM
fX$ rht-vmctl poweroff VMNAME
- will "destroy" VM.
fX$ rht-vmctl save VMNAME
- will call above stop, copy the overlay to time stamped backup,
call above start - end result is the same running VM.
fX$ rht-vmctl fullreset VMNAME
- will "destroy" VM, delete timed backups, delete overlays, delete
qcow2 image files, "undefine" VM, call above start - end result
is a running VM from virgin f0 images.
fX$ rht-vmctl get VMNAME
- will download if missing qcow2, create if missing overlay, and
update the libvirt XML, but will *NOT* start the VM.
fX$ rht-vmctl restore VMNAME
- if a saved checkpoint exists, will "destroy" VM, remove the
overlay, restore last saved checkpoint as overlay, and calls
above start. If no saved checkpoint, aborts.
The scripts are dynamically downloaded as the student runs the *lab*
wrapper script. If you find any logic or code errors in the scripts,
correct them in the above directory, but you will also have to remove
the script from each student's virtual machine (or it happens automatically
when the student resets them) in /usr/local/{bin or lib}/ to trigger lab to
download the updated script. Also, please forward any patches to
curriculum-support@redhat.com and attach to a Jira report.
You will find that the lab scripts only work on the virtual machines.
This is by design.
A newer lab wrapper script also supports TAB completion. To view all
script names, on workstation:
workstation$ lab <TAB><TAB>
In more recently developed courses, the lab script will log its activity
in /var/tmp/labs/ capturing both stdout and stderr in two files named
<scriptname> and <scriptname>.err.
Wayland, at least initially in RHEL 8.0, has been known to not work in
the ways many instructors are used to with Xorg. If desired, a utility
named rht-switchgnome can be used post-installation to swap the desired
display server.
MONITOR/PROJECTOR ISSUES
------------------------
So, before you disable KMS with the nomodeset kernel parameter,
because your screen turns black or monitor goes to sleep, try a couple
of parameters. The problem might be that there is no correct EDID data
available. There are a few generic EDID profiles in the kernel that you
can use.
drm_kms_helper.edid_firmware=VGA-1:edid/1024x768.bin
will set the resolution for *both* console *and* X.Org server. There are
only 1024x768, 1280x1024, 1600x1050 and 1920x1080 resolutions available
with 60Hz refresh rate. Of course you should first find out which output
is being used on graphics card, you can boot into emergency or rescue
targets and look at /sys/class/drm/card*. For the above parameter there
should be a /sys/class/drm/card0-VGA-1.
If you have more than one output you can check which one is connected
by looking into /sys/class/drm/*/status files.
There is a way to set only the console resolution, so you can see
if KMS actually does work and produces a picture on your screen and
projector with
video=VGA-1:1280x800
This kernel parameter unfortunately will *not* setup the X.Org resolution
and if you need such a wide resolution which is not available in kernel
EDID data, you will have to create a /etc/X11/xorg.conf.d/50-screen.conf
file with a content like this.
Section "Screen"
DefaultDepth 24
Identifier "Builtin Default Screen 0"
Subsection "Display"
Depth 24
Modes "1280x800" "1024x768"
ViewPort 0 0
EndSubsection
Subsection "Display"
Depth 16
Modes "1280x800" "1024x768"
ViewPort 0 0
EndSubsection
EndSection
SLOW MACHINE
------------
GRUB ISSUES
-----------
There is a new feature of RHEL that uses USB devices before hard drives:
https://bugzilla.redhat.com/show_bug.cgi?id=497019
This means that using a USB device to install is usually seen as /dev/sda,
and grub may be written to this device. The kickstart file attempts to find
the real hard drive and install grub to it. If this does not succeed, leave
your USB device in foundation0 to boot it. Once it has booted, use
'grub2-install /dev/sda' to install grub2 to the MBR. You may also need to
remove entries pointing to the USB device found in /etc/fstab.
After this, students logging in to Gnome will have the choice between
keyboards, using the keyboard selector in the top-right corner of the
screen, with the recently set keyboard as the default. Removing the US
keyboard from that list can also be done, but it requires scripting it
with dconftool, or by forcing a system dconf database on users using
/etc/dconf/db/site.d/. Both are not worth the effort, especially since
some of your students will want to use a US layout, no matter what the
actual symbols on the physical keyboard say.
REPORTING ISSUES TO RED HAT
---------------------------
=============================================================================
CHANGELOG
=========
* Thu May 30 2019 Robert Locke <rlocke@redhat.com> 8.x-7
- prep for release
* Thu Apr 11 2019 Robert Locke <rlocke@redhat.com> 8.x-7
- Review for changes based on RHEL8
* Sun Mar 17 2019 Robert Locke <rlocke@redhat.com> 8.x-7
- add references to rht-usb usbmkpart
* Thu Jul 26 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add references to dualboot support
* Wed Apr 4 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add references to rht-usb f0activate
* Tue Mar 20 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add references to rht-sosreport
* Fri Mar 2 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- remove references to second physical system
* Tue Feb 6 2018 Robert Locke <rlocke@redhat.com> 7.x-6
- add RHT-USB VERIFICATION REVIEW
* Mon Jan 22 2018 Robert Locke <rlocke@redhat.com> 7.x-5
- add mention of /var/tmp/labs/
* Mon Nov 27 2017 Robert Locke <rlocke@redhat.com> 7.x-5
- add a cap to rht-discover all
* Thu Jul 6 2017 Robert Locke <rlocke@redhat.com> 7.x-5
- add references to rht-movebr0
- add references to rht-discover/rht-unregister
* Mon Sep 19 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- expand grading script issues - thanks Phil
* Wed Jul 13 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- add references to empty/missing /content/ks/rht-locale
* Fri Jun 10 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- shift uplink references to rht-external
- shift to new unbacked br1 in documentation
* Fri Apr 22 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- add references to managing oversized classes
* Tue Feb 2 2016 Robert Locke <rlocke@redhat.com> 7.x-5
- change hostX reference to galaxyX
* Mon Oct 12 2015 Robert Locke <rlocke@redhat.com> 7.x-4
- prep for release
* Wed Aug 5 2015 Robert Locke <rlocke@redhat.com> 7.1-4
- add information about memtest availability
* Mon Jul 6 2015 Robert Locke <rlocke@redhat.com> 7.1-4
- bump release
* Mon Jun 1 2015 Robert Locke <rlocke@redhat.com> 7.1-3
- add reference to rht-verify-f0 --validate
* Wed May 20 2015 Robert Locke <rlocke@redhat.com> 7.1-3
- bump release
* Mon Apr 27 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- minor versioning fixes
* Mon Mar 16 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- add documentation on /var/www/register/
* Tue Mar 10 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- add detail about no wireless support currently
* Thu Mar 5 2015 Robert Locke <rlocke@redhat.com> 7.1-2
- minor edits
* Fri Jan 16 2015 Robert Locke <rlocke@redhat.com> 7.0-2
- add references to "dot" files on USB media
* Fri Dec 5 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- add references to rht-verify-*.log files
* Sun Sep 28 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Add sync to dd of USB stick
* Tue Sep 16 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Add section on NETWORK CONFIGURATION EXTREME DETAIL
* Thu Aug 14 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Minor cleanups
- Add references to /tmp/rht-usb-<date>.log
* Fri Jul 18 2014 Robert Locke <rlocke@redhat.com>
- Additional foundation installation possibilities
* Wed Jul 2 2014 George Hacker <ghacker@redhat.com>
- Add more detailed X/video troubleshooting tip by Janez Trenz
* Mon Jun 30 2014 Robert Locke <rlocke@redhat.com>
- Add keyboard setting recommendation (Thanks Wander!)
* Fri Jun 27 2014 Robert Locke <rlocke@redhat.com>
- Add some DHCP and more multi-NIC information
* Wed Jun 25 2014 George Hacker <ghacker@redhat.com>
- Added instructions on how to use rht-pushkernelargs script
* Tue Jun 24 2014 Robert Locke <rlocke@redhat.com> 7.0-1
- Split out to ClassroomReset.txt and ClassroomTroubleshooting.txt