Linux Networking, SSH, Logs, and Troubleshooting

A Linux server can be running normally and still be unreachable. An application may be healthy locally but inaccessible from another machine. A domain name may resolve to the wrong address, a service may listen on the wrong port, or a firewall may block otherwise valid traffic.

These problems are common in cloud environments, where applications depend on multiple layers of networking and configuration.

For cloud engineers, knowing how to investigate these failures is as important as knowing how to deploy a server. This requires understanding how to connect to Linux machines remotely, inspect network activity, interpret logs, and isolate the source of a problem.

This article covers the Linux tools and troubleshooting techniques used to investigate these situations.

Connecting to Linux Servers With SSH

Secure Shell (SSH) is a protocol for securely accessing and administering remote machines over a network.

For example, suppose you have an Ubuntu EC2 instance and need to manage it from your local computer. Instead of interacting with the server through a physical terminal, you can establish an SSH connection.

The general command is:

ssh username@server_ip

For an Ubuntu EC2 instance, it might look like:

ssh ubuntu@203.0.113.10

The IP address above is an example address, not a real server to connect to.

When connecting to a cloud instance, you need the correct username, reachable server address, network access, and valid authentication credentials.

Connecting With an SSH Key

Cloud servers commonly use SSH key-based authentication.

If your private key is stored in my-server-key.pem, connect with:

ssh -i my-server-key.pem ubuntu@203.0.113.10

On Linux and macOS, SSH may reject a private key file if its permissions are too open. You can restrict access with:

chmod 400 my-server-key.pem

This allows the file's owner to read the key while preventing other users from accessing it through ordinary file permissions.

Keep private keys secure. Do not upload them to public repositories, send them through unsecured channels, or copy them onto servers unnecessarily.

Understanding the SSH Connection

An SSH connection requires several things to work together:

1. The destination server must be reachable over the network.

2. An SSH server, commonly sshd, must be running and listening.

3. Network controls must permit traffic to the SSH port, usually TCP port 22.

4. The client must use an accepted username and authentication method.

5. The server must accept the client's connection and authentication attempt.

In a cloud environment, network access may also depend on security groups, network ACLs, routing, and whether the instance has a reachable address.

A failure at any one of these layers can prevent access.

Troubleshooting SSH Connections

When an SSH connection fails, start by identifying the error.

For more detailed diagnostic output, use:

ssh -v -i my-server-key.pem ubuntu@203.0.113.10

The -v option provides verbose output that helps reveal where the connection is failing. Additional verbosity is available with -vv or -vvv.

Common errors include:

Error	Possible explanation
Connection timed out	Network path, routing, firewall, or security group issue
Connection refused	The destination is reachable, but the port is not accepting connections
Permission denied (publickey)	Incorrect username, key, or authentication configuration
Could not resolve hostname	Invalid hostname or DNS resolution problem
Host key verification warning	The server identity cannot be verified against the saved host key

These errors are clues, not definitive diagnoses. For example, a timeout does not prove that a security group is responsible; the problem could also be routing, an address issue, or a host that is unavailable.

Never bypass host key verification simply to make a connection succeed. If a host key changes unexpectedly, verify the server's identity through a trusted channel before updating the saved key.

---

Inspecting Network Configuration on Linux

Before troubleshooting connectivity, establish which network configuration the server is using.

Linux provides several commands for inspecting network interfaces, addresses, and routes.

Viewing Network Interfaces and IP Addresses

Run:

ip addr

Or, for a more concise view:

ip -br addr

Example output:

lo       UNKNOWN  127.0.0.1/8 ::1/128
eth0     UP       10.0.1.25/24

Here, eth0 has the private IPv4 address 10.0.1.25.

The lo interface is the loopback interface, used for communication within the local machine. Its IPv4 address is typically 127.0.0.1.

This distinction matters when testing applications. A service accessible through localhost may not be accessible through the server's network interface.

Understanding the Routing Table

An IP address tells you how an interface is addressed. A routing table helps determine where network traffic should go.

Inspect it with:

ip route

Example:

default via 10.0.1.1 dev eth0
10.0.1.0/24 dev eth0 proto kernel scope link

The default route tells Linux where to send traffic when no more specific route matches the destination.

If a server cannot reach an external destination, an incorrect or missing route may be part of the problem.

In cloud environments, the host's routing table is only one part of the picture. Cloud route tables and other network controls may also determine whether traffic can leave a subnet or reach another network.

Testing Basic Reachability With ping

The ping command sends ICMP echo requests to test whether a destination responds.

For example:

ping -c 4 1.1.1.1

The -c 4 option sends four requests and then stops.

If replies arrive, the destination is responding to those requests through the current network path.

If no replies arrive, however, that does not necessarily mean the destination is down. ICMP traffic may be blocked or ignored by the destination or an intermediate network device.

Use ping as one diagnostic signal rather than the final test of whether an application is reachable.

---

Checking Listening Ports and Network Connections

A service can be running but still fail to accept traffic on the expected port.

For example, an application might start successfully but listen only on 127.0.0.1:8000 rather than on an address that allows access from other machines.

The ss command helps inspect listening sockets and active network connections.

Finding Listening Ports

Run:

sudo ss -lntup

The options mean:

• -l — show listening sockets

• -n — display numeric addresses and ports

• -t — show TCP sockets

• -u — show UDP sockets

• -p — show associated processes, where available

Example:

Netid State  Local Address:Port
tcp   LISTEN 0.0.0.0:22
tcp   LISTEN 127.0.0.1:8000
tcp   LISTEN 0.0.0.0:80

This example suggests:

• SSH is listening on port 22 on all local IPv4 interfaces.

• The application on port 8000 is listening only on the loopback interface.

• A service is listening on port 80 on all local IPv4 interfaces.

The exact output varies by server and configuration.

Why the Listening Address Matters

Consider a Python application started with:

python3 -m http.server 8000 --bind 127.0.0.1

It listens on the loopback interface. Requests made from the same machine can reach it, but requests directed to the server's private or public IP address generally cannot reach that listener.

For a controlled test where remote access is intended, the application could instead bind to an appropriate network interface or to all IPv4 interfaces:

python3 -m http.server 8000 --bind 0.0.0.0

Binding to 0.0.0.0 does not, by itself, make the application publicly accessible. It allows the service to listen on local IPv4 interfaces. Firewalls, cloud security groups, routing, and other controls still determine whether remote traffic can reach it.

Also, do not expose a development server directly to the public internet without appropriate security controls.

Testing a Service Locally

Use curl to send an HTTP request:

curl -I http://127.0.0.1:8000

The -I option requests response headers.

If you need to inspect the full response, run:

curl -v http://127.0.0.1:8000

A successful local request confirms that the service responds through that local address. It does not establish that the service is reachable from another machine.

That distinction is essential when diagnosing cloud networking issues.

---

DNS: Resolving Hostnames to IP Addresses

Applications often communicate using hostnames rather than hard-coded IP addresses.

For example:

api.example.com

DNS translates a hostname into records that clients can use to locate services.

When a hostname fails to resolve, an application may be unable to connect even when the destination server itself is healthy.

Testing Name Resolution

A quick way to test name resolution on Linux is:

getent hosts example.com

This uses the system's configured name-service resolution mechanism.

If the dig utility is installed, you can inspect DNS responses more directly:

dig example.com

To request only the returned IPv4 addresses:

dig +short A example.com

For IPv6 address records:

dig +short AAAA example.com

These commands help distinguish name-resolution issues from connection issues.

For example, if a hostname resolves to an unexpected address, investigate its DNS records, resolver configuration, or caching behavior before assuming the application itself is broken.

A successful DNS lookup also does not prove that the destination is reachable. It only confirms that a DNS response was obtained.

---

Inspecting Logs During Troubleshooting

Network symptoms often originate in application or service configuration. Logs provide evidence about what happened when a request failed.

As covered in the previous article, systemd-managed services can be inspected through journalctl.

For example, to inspect SSH service logs on a system using systemd:

sudo journalctl -u ssh -n 50

On some distributions, the unit is named sshd instead:

sudo journalctl -u sshd -n 50

To follow new entries:

sudo journalctl -u ssh -f

Use the unit name appropriate to your distribution.

Depending on the service and system configuration, logs may also be stored in files under /var/log.

For example:

sudo less /var/log/syslog

This file is common on Ubuntu and Debian systems, but it may not exist or be used in the same way on every distribution.

Search Logs for Relevant Errors

If an application writes logs to a file, you can search for errors with:

grep -i "error" application.log

To follow new log entries:

tail -f application.log

These commands build on the file-reading and searching techniques covered earlier in the series.

Rather than reading every log entry, narrow the investigation by looking at the time the problem occurred, the affected service, and the specific error.

---

A Practical Troubleshooting Example

Suppose you deploy a Python web application on an Ubuntu EC2 instance. The application appears to start, but you cannot reach it from your browser.

Assume the application is intended to listen on port 8000.

Instead of immediately changing cloud security groups or restarting the instance, work through the problem in layers.

Step 1: Confirm the Application Is Running

Inspect its process or, if it is managed by systemd, check its service status:

sudo systemctl status myapp

Replace myapp with the actual service name.

If the application is not managed by systemd, inspect the relevant process using tools such as ps or pgrep.

Step 2: Inspect the Application Logs

sudo journalctl -u myapp -n 50

Look for startup failures, configuration errors, missing dependencies, or messages indicating that the application could not bind to its port.

Step 3: Confirm the Listening Port

sudo ss -lntp

Check whether the application is listening on port 8000 and which address it uses.

If it listens only on 127.0.0.1:8000, requests directed to the instance's private or public IP will not reach that listener.

Step 4: Test the Application Locally

curl -v http://127.0.0.1:8000

If this request fails, focus on the application, its configuration, or its local listener before investigating the external network path.

If it succeeds, continue testing the path from outside the server.

Step 5: Inspect the Network Configuration

Check the server's address and routes:

ip -br addr
ip route

Confirm that the instance has the expected network configuration and that its routes are appropriate for the intended traffic.

Step 6: Check Cloud and Host Network Controls

If the application responds locally but remains unreachable remotely, verify:

• The application is listening on an appropriate interface.

• The host firewall permits the intended traffic.

• The EC2 security group allows the required inbound traffic from the appropriate source.

• Network ACLs and subnet routing permit the traffic and its return path.

• The instance has the appropriate addressing and routing for the intended access method.

For public access, an internet gateway route and a suitable public IPv4 address or other valid public addressing arrangement may be required. A public address alone does not guarantee reachability.

If the application is meant to be accessed through a load balancer or reverse proxy, validate that architecture's intended path instead of opening port 8000 to the entire internet.

Step 7: Test From the Client

From a machine that should be able to access the application, run:

curl -v http://SERVER_ADDRESS:8000

Replace SERVER_ADDRESS with the appropriate reachable address or hostname.

Interpret the result alongside your other observations:

• Connection refused: the destination was reached, but the connection was not accepted at that endpoint.

• Connection timed out: packets or responses may be blocked, routed incorrectly, or lost along the path.

• HTTP error response: an HTTP server responded, but the request was not handled successfully.

• Successful response: the tested endpoint responded; further checks may still be necessary if the full application is not working.

This process helps isolate the failure rather than changing several unrelated settings at once.

---

Common Linux Networking Mistakes

A few recurring mistakes make server troubleshooting harder than it needs to be.

Assuming a Running Process Is Reachable

A process can be running without listening on the expected port or network interface. Confirm the listener and test the endpoint.

Assuming ping Proves Application Availability

ICMP reachability is not the same as TCP connectivity or a successful HTTP response. Test the actual protocol and port used by the application.

Opening Every Port to the Internet

Allowing unrestricted access may hide a connectivity problem while creating a security risk. Permit only the traffic required by the intended architecture and restrict source addresses where appropriate.

Changing Cloud Network Rules Before Testing Locally

If an application cannot respond to a request from the same machine, changing an external security group is unlikely to fix the underlying application problem. Start with the local listener and work outward.

Ignoring Return Traffic

A connection requires a viable path in both directions. Routing, firewalls, network ACLs, and other network controls can prevent responses from returning even when requests reach their destination.

Treating Every Failure as a DNS Problem

If a hostname resolves correctly but the connection fails, investigate the listener, destination address, port, route, and network controls as appropriate.

Ignoring the Time of an Error

Logs are most useful when correlated with the time of the failed request. Compare application logs, service logs, and connection tests from the same period.

---

A Repeatable Troubleshooting Method

When a Linux server or application becomes unreachable, follow a structured process:

1. Define the symptom. Identify the exact hostname, address, port, protocol, and error.

2. Check the application. Confirm that it is running and inspect relevant logs.

3. Check the listener. Use ss to verify the expected port and binding address.

4. Test locally. Use curl or another protocol-appropriate client.

5. Check name resolution and routing. Inspect DNS responses, interface addresses, and routes as needed.

6. Investigate network controls. Check host firewalls, cloud security groups, network ACLs, and routing.

7. Test from the intended client. Verify the complete path, not just the server itself.

8. Make one controlled change at a time. Repeat the relevant tests to establish whether the change fixed the problem.

This approach makes troubleshooting more efficient because each test eliminates or narrows down possible causes.

Conclusion

Linux networking troubleshooting starts with understanding how to access a server, inspect its network configuration, verify listening ports, test services, and investigate logs.

Tools such as ssh, ip, ss, ping, curl, dig, and journalctl help you gather evidence at different layers of the system. Used together, they make it easier to distinguish an application failure from a name-resolution, routing, or connectivity problem.

The objective is not to memorize every networking command. It is to identify where communication fails, gather evidence, and make the smallest safe change needed to restore the intended behavior.

With these Linux administration skills in place, the final article in this series will cover Bash Scripting and Linux Server Automation, showing how to turn repeated command-line tasks into reliable, reusable scripts.