Linux Networking Mastery Series Part 7: Monitoring and Troubleshooting Tools

Welcome back to Linux Networking Mastery!
Up to this point we have covered:

• Part 1 – network stack basics and inspection tools
• Part 2 – interface and IP configuration (temporary + persistent via Netplan, nmcli, systemd-networkd)
• Part 3 – routing tables, static/policy routing, namespaces, simple router setup
• Part 4 – name resolution, systemd-resolved, per-link/global DNS, troubleshooting
• Part 5 – firewalls with nftables, firewalld, ufw, stateful rules
• Part 6 – services (hardened SSH, Nginx basics, NFS/Samba shares, DHCP with dnsmasq)

Now we shift focus from configuration to visibility and debugging.

A production Linux system rarely works perfectly on the first try. You'll need tools to see what's happening in real time, capture problematic packets, measure performance, identify bottlenecks, and correlate events across logs.

This post introduces the most useful modern monitoring and troubleshooting utilities, shows typical workflows, and ties them back to issues you might encounter from earlier parts (DNS failures, firewall blocks, routing loops, slow services, etc.).

1. Socket and Connection Statistics – ss

ss (socket statistics) is the modern replacement for netstat. It's faster and shows more detail.

Most common usages:

# All listening sockets
ss -ltnp

# All established TCP connections with process name
ss -tnp

# Connections to/from a specific port
ss -tn 'sport = :80 or dport = :80'

# UDP sockets
ss -unap

# Summary statistics
ss -s

# Filter by state (very useful for troubleshooting)
ss -tn state established
ss -tn state syn-sent   # stuck handshakes

Compare with old netstat (still installed on many systems but slower):

netstat -tulnpe    # legacy equivalent

2. Packet Capture – tcpdump

tcpdump remains the gold standard CLI packet analyzer.

Basic examples:

# Capture everything on interface (quick look)
sudo tcpdump -i enp0s3 -n

# Capture HTTP traffic (port 80 or 443)
sudo tcpdump -i any -n tcp port 80 or tcp port 443

# Capture DNS queries/responses
sudo tcpdump -i any -n udp port 53 or tcp port 53

# Show ASCII payload (-A) for debugging HTTP
sudo tcpdump -i any -A -s0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

# Save to file for later analysis
sudo tcpdump -i enp0s3 -w capture.pcap 'host 8.8.8.8'

Then read later:

tcpdump -nnr capture.pcap

Pro tip: Combine with Wireshark (GUI) by saving .pcap files and opening them on your workstation.

3. Performance & Bandwidth Testing – iperf3

Install: sudo apt install iperf3 or sudo dnf install iperf3

Server mode (on remote machine):

iperf3 -s

Client mode (measure throughput):

iperf3 -c server-ip -t 30 -P 4          # 30s, 4 parallel streams
iperf3 -c server-ip -u -b 0             # UDP unlimited bandwidth test
iperf3 -c server-ip --bidir             # bidirectional test

Useful for diagnosing slow links, MTU issues, firewall rate limiting, etc.

4. Real-time Bandwidth & Connection Monitoring

• iftop – top-like interface for bandwidth per connection

  sudo iftop -i enp0s3
  

• nload – simple per-interface graphs

  sudo nload enp0s3
  

• bmon – another lightweight bandwidth monitor

  sudo bmon
  

5. Log Analysis for Network Events

Most network-related messages appear in journalctl or /var/log/syslog / /var/log/messages.

Key queries:

# Kernel network messages (interface up/down, DHCP, etc.)
journalctl -k -u NetworkManager
journalctl -k | grep -i "enp0s3"

# SSH login attempts / failures
journalctl -u ssh

# Firewall drops (nftables logging – if enabled)
journalctl | grep -i "nft\|drop"

# systemd-resolved DNS issues
journalctl -u systemd-resolved

# dnsmasq DHCP logs
journalctl -u dnsmasq

Typical Troubleshooting Workflow

Example scenario: "Clients can't reach the web server on port 443"

1. Check listening socket
   ss -ltn '( sport = :443 )'

2. Check firewall
   sudo nft list ruleset | grep 443
   or firewall-cmd --list-all / ufw status

3. Check routing & ARP
   ip route get 192.168.100.55
arp -n

4. Capture traffic
   sudo tcpdump -i enp0s3 -nn 'host 192.168.100.55 and port 443'

5. Test connectivity layers
   ping 192.168.100.55 → curl -v https://192.168.100.55 → curl -v https://example.com

6. Check logs for denials / errors

Hands-On Exercises

1. Use ss to find which process is listening on port 80/443 and all established connections to it.
2. Run tcpdump while you curl a website → identify the DNS query and TLS handshake.
3. Set up iperf3 server on one machine, client on another → measure throughput and look for packet loss.
4. Intentionally break something (e.g., block port 22 in firewall), then use the workflow above to diagnose and fix.

Tip: Install these tools in your lab VM now — they'll be invaluable for the container networking lab in Part 10.

What's Next?

In Part 8 we explore advanced link-layer and tunneling features: interface bonding for redundancy/load-balancing, VLAN tagging (802.1Q), software bridges, and modern VPN solutions with a focus on WireGuard (now the clear favorite in 2026).