Linux Troubleshooting Tips

Hi everyone. Linux is the most commonly used OS in servers. There are many distributions
of Linux which are used for different purposes. While configuring, running services, taking
backups etc we face many issues and have to resolve them. Here in this article, I have
shown some tips to resolve those issues. So let's begin.

There are many commands and tools useful for detecting the problem and solve them. Also,
there may be a hardware-level issue.

In case of a network-related issue, we should do a ping to the server or if any automated

ping-script configured, we should see it’s result. Then check the cable's connectivity to the
hardware and make sure that there are no loose connections or faults in the
connectors.  Here we can use ethtool to check if the ethernet is detectable or not. In case
of power failure, we should also check the power cables and the supply.

Using dmidecode command we can view the server’s SMBIOS report about it’s hardware.
To check RAM, we can use a flag --type memory with dmidecode. We can also install
custom programs to check RAM.

cat /etc/redhat-release -view RHEL release

free -m -view memory details (total, used, free, shared, cache, available)

w -view logged users

uptime -view system uptime

cat /proc/cpuinfo -view CPU details

cat /proc/loadavg -view average load of server

If there are no server problem then we have to check the service or application running on
it. To check if an application is running or not, we can use multiple commands-

ps -ef | grep APPLICATION_NAME,

netstat -plunt | grep APPLICATION_NAME,

service status APPLICATION_NAME.

To view the status of all available services at once use service --status-all. Here we can
additionally use more or less for page-wise viewing.

To view the services that are enabled on boot, use chkconfig --list.

We can also use pstree to view running services in a hierarchical manner.

We can also check the logs from the subdirectories of /var/log or /etc/systemd to know
more details.

To view the kernel messages, use dmesg. Then use tail to view the last 10 messages.

We can monitor the Linux process using top command. It shows Process ID (PID) with its
user, priority, nice level, virtual, resident and shareable memories used by the process,
percentage of CPU and memory usage, the process running time, command.

Here we can sort the processes based on attributes (memory, CPU usage, running time
etc) by pressing M, P, T. Press o and then enter the attribute and value (PID=Value or
COMMAND=apache) to see details.

To displays the processes in a parent-child hierarchy press V (capital). Press h for help.

Htop is a more advanced and interactive tool to monitor the Linux process with better UI,
shortcut keys etc. By default, it is not installed in RHEL or CentOS or Ubuntu systems but
we can install it using yum (Yellowdog Updater Modified) or apt. Here we can scroll the
process list vertically and horizontally to see all processes and complete command lines.

After identifying the process, we can kill it or change it’s priority using renice -n
PRIORITY_VALUE -p PID to reduce its memory and CPU usage.

To view the disk space usage, we can use many commands. Some of them are-

Disk filesystem is a very useful command. Use df -h to view the storage in the hard drive.
To know the filesystem type use df -hT. To view the used inodes, use df -i.

To view the delivered reads, writes, read KB/s, write KB/s and the average time for I/O
(await number) for a device, use iostat -xz 1. It is a utility of sysstat package. If the drive is
saturated with data requests or has a hardware problem, then it’s await number will be
bigger. Also, %util shows device utilization.

By default, sysstat is not installed in systems, we can install it using yum or apt.

Learn the networking basics every sysadmin needs to know

One of the sysadmin's most important domains is the network.

While understanding everything there is to know about networking is a

big topic, there's much to learn from your own humble Linux
computer's networking stack.
Learning basic networking commands can help you understand how a
device knows what network to connect to, how to find a shared printer
or a file share—or the biggest network of all, the internet.

This article covers the basics of network management using open

source.

[ You might also enjoy reading: 5 Linux network troubleshooting

commands ]

What is a network?

In computing, a network is a collection of two or more computers that

can communicate.

In order for networking to facilitate communication between devices,

the machines on the network must be able to find each other.

The systems responsible for making this possible are TCP and IP.

Transmission Control Protocol (TCP)

Communication requires a means of transport for messages between

them, and computers communicate using digital signals carried over
Ethernet cables or radio waves or microwaves.

The specifications for this are formally defined as the TCP protocol.

Internet protocol (IP)

Computers on a network identify themselves and each other with IP

addresses, such as 10.0.0.1 or 192.168.0.8.

These are also generally mapped to hostnames, such

as laptop and desktop or darkstar or penguin or whatever name you
give each machine.
The specifications for this are formally defined as the IP protocol.

Minimal networking

The most simple network possible is a single-node network.

This might seem like it's cheating, but in fact, it's a valid network in the
sense that a computer needs to know how to address itself.

Each computer considers itself as the localhost node, with an internal-

only IP address of 127.0.0.1.

You can verify this with the ping command:

$ ping -c 1 localhost
PING localhost (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.031 ms
The localhost designation is defined in the /etc/hosts file:

$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4
localhost4.localdomain4
::1         localhost localhost.localdomain localhost6
localhost6.localdomain6
Having an internal IP address is significant because important services,
such as the CUPS print server and the server management
system Cockpit, provide interfaces over TCP/IP connections. To access
them from the machine they're running on, you can use localhost plus
the port number (for instance localhost:631 or 127.0.0.1:631) in a web
browser.

Creating a basic network

While a single-node network is useful for some tasks, a

network usually refers to more than one computer.
Linux and the TCP/IP stack do a lot of work to make networking simple,
but when automated settings aren't good enough to get you the setup
your organization needs, it's up to you as the sysadmin to understand
how to create network configurations.

To begin with, start simple and try creating a two-computer network.

To eliminate automatic settings so that you can get used to building a

network yourself, try using a specially wired Ethernet cable called
a crossover cable.

A crossover cable connects transmit signals coming from one computer

to the appropriate receptors on another computer.

Image
Crossover cable 
With no router between the computers, all network management must
be done manually on each machine, making this a good introductory
exercise for networking basics.

Using a crossover cable to connect two computers together, you

eliminate any external network controller to offer guidance, so neither
computer does anything to create or join a network.

In this simple setup, you are the ultimate networking authority.

To create a network, you first must assign an IP address to each
computer.

The block reserved for self-assigned IP addresses is 169.254.x.x.

View your network interfaces on Linux

To create a network, you need network interfaces.

The Ethernet port is usually designated with the term eth plus a

number starting with 0, but some devices get reported with different
terms.

You can discover the interfaces on a computer with the ip command:

$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 ...
    link/loopback 00:00:00:00:00:00 brd ...
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> ...
    link/ether dc:a6:32:be:a3:e1 brd ...
3: wlan0: <BROADCAST,MULTICAST> ...
    link/ether dc:a6:32:be:a3:e2 brd ...
In this case, eth0 turns out to be the correct interface name.

However, you'll see en0 or enp0s1 or similar in some cases, so it's

important to always verify a device name before using it.

Assign a static IP address on Linux

Normally, an IP address is assigned dynamically from a dedicated DHCP

server or a router running an embedded DHCP server.
It's the job of the DHCP server to broadcast offers for addresses over its
network.

When a computer gets connected to a network, it requests an address.

The DHCP server assigns it one and registers which device on the

network, identified by its media access control (MAC) address, has been
assigned which address.

That's how computers know how to find one another across a network.

In the case of this simple network, however, there is no DHCP server

handing out IP addresses or registering devices, so you must create an
IP address yourself.

To assign an IP address to a computer, use the ip command:

$ sudo ip address add 169.254.0.1 dev eth0

And again on the other computer, this time incrementing the IP address
by 1:

$ sudo ip address add 169.254.0.2 dev eth0

Now each computer has a means of transport (the crossover cable) and
a way to be found on the network (a unique IP address).

The problem is, neither computer knows it's a member of a network.

Setting up a network route on Linux

Normally, an external router defines the paths that network traffic

must take in order to get from point A to point B.

This is called a routing table, and it's essentially a "city map" for your
network.

For the simple network you've created, no routing table yet exists.
You can verify this with the route command on either or both
computers:

$ route
Kernel IP routing table
Destination | Gateway | Genmask | Flags|Metric|Ref | Use | Iface
$ 
Alternatively, you can use the ip command:

$ ip route
$ 
Either way, there's no meaningful output because there is currently no
route defined.

But you can add a route with the ip command:

$ sudo ip route \
add 169.254.0.0/24 \
dev eth0 \
proto static
This command adds a route to an address range starting from
169.254.0.0 and ending at 169.254.0.255 over the eth0 interface.

It sets the routing protocol to static to indicate that the route was
created by you, the administrator, as an intentional override for any
dynamic routing.

Verify your routing table with the route command:

$ route
Kernel IP routing table
Destination | Gateway | Genmask       | ... | Iface
link-local  | 0.0.0.0 | 255.255.255.0 | ... | eth0
Or use the ip command for a different view:
$ ip route
169.254.0.0/24 dev eth0 proto static scope link 
Ping your neighbor

Your network now has:

 A means of transport
 A means of addressing
 A network route

With those components in place, each computer can reach hosts

beyond just localhost.

Test it out with ping.

For example, from the computer assigned the address 169.254.0.1:

$ ping -c1 169.254.0.2

64 bytes from 169.254.0.2: icmp_seq=1 ttl=64 time=0.233 ms

--- 169.254.0.2 ping statistics ---

1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.244/0.244/0.244/0.000 ms
You can also view the neighbors you've interacted with:

$ ip neighbour
169.254.0.2 dev eth0 lladdr e8:6a:64:ac:ef:7c STALE
The network switch

While it's great for surprisingly fast file transfers, there aren't many
other needs for two-node hard-wired networks.

That's where hardware like a network switch comes in.

A switch allows several Ethernet cables to be attached to it, and it
distributes messages from the computer sending them to the intended
destination, identified by its IP address.

On many modern general-purpose networks, a physical switch for

physical cables isn't practical or desired, so a WiFi access point is used
instead, but the same principles apply.

Router

Local networks connect many devices in practice, and the number is

growing as more devices become network-aware.

When you connect a network to the internet (a network of inter-

connected networks), that number goes up by orders of magnitude.

It's impractical to configure a network manually. So everyday tasks are

assigned to specific nodes on the network, and each computer runs a
daemon to populate network settings received from authoritative
servers on the network.

Each task is usually assigned to a separate dedicated server in a large

network to ensure focus and resiliency.

These tasks include:

 DHCP server to assign and track IP addresses to devices joining

the network
 DNS server to convert registered domain names like redhat.com
to IP addresses like 209.132.183.105
 Firewall to protect your network from unwanted incoming traffic
or forbidden outgoing traffic
 Router to efficiently direct traffic on the network, serve as a
gateway to other networks (such as the Internet), and perform
network address translation (NAT) 
Thanks to open source projects like VyOS, you can even run your own
open source router built out of commodity hardware.

[ Free cheat sheet: Get a list of Linux utilities and commands for

managing servers and networks. ] 

Wrap up

The more networking schemes you implement in your lab, the better
you understand all the issues that can arise in the real world.

Sometimes the issues are borne of individual computers and

devices; other times it's hardware and infrastructure, and still other
times it's non-optimal design.

Practice setting up different network topographies, either with virtual

machines or real hardware (or both), and get to know the open source
tools of the trade.

Networking is a big topic, but networks are everywhere.

Invest in learning it now, and it'll pay dividends, one way or another,
sooner than later.