# Linux Networking, SSH, Logs, and Troubleshooting A Linux server can be running normally and still be unreachable. An application may be healthy locally but inaccessible from another machine. A domain name may resolve to the wrong address, a service may listen on the wrong port, or a firewall may block otherwise valid traffic. These problems are common in cloud environments, where applications depend on multiple layers of networking and configuration. For cloud engineers, knowing how to investigate these failures is as important as knowing how to deploy a server. This requires understanding how to connect to Linux machines remotely, inspect network activity, interpret logs, and isolate the source of a problem. This article covers the Linux tools and troubleshooting techniques used to investigate these situations. ## Connecting to Linux Servers With SSH **Secure Shell (SSH)** is a protocol for securely accessing and administering remote machines over a network. For example, suppose you have an Ubuntu EC2 instance and need to manage it from your local computer. Instead of interacting with the server through a physical terminal, you can establish an SSH connection. The general command is: ```bash ssh username@server_ip ``` For an Ubuntu EC2 instance, it might look like: ```bash ssh ubuntu@203.0.113.10 ``` The IP address above is an example address, not a real server to connect to. When connecting to a cloud instance, you need the correct username, reachable server address, network access, and valid authentication credentials. ### Connecting With an SSH Key Cloud servers commonly use SSH key-based authentication. If your private key is stored in `my-server-key.pem`, connect with: ```bash ssh -i my-server-key.pem ubuntu@203.0.113.10 ``` On Linux and macOS, SSH may reject a private key file if its permissions are too open. You can restrict access with: ```bash chmod 400 my-server-key.pem ``` This allows the file's owner to read the key while preventing other users from accessing it through ordinary file permissions. Keep private keys secure. Do not upload them to public repositories, send them through unsecured channels, or copy them onto servers unnecessarily. ### Understanding the SSH Connection An SSH connection requires several things to work together: 1. The destination server must be reachable over the network. 2. An SSH server, commonly `sshd`, must be running and listening. 3. Network controls must permit traffic to the SSH port, usually TCP port 22. 4. The client must use an accepted username and authentication method. 5. The server must accept the client's connection and authentication attempt. In a cloud environment, network access may also depend on security groups, network ACLs, routing, and whether the instance has a reachable address. A failure at any one of these layers can prevent access. ### Troubleshooting SSH Connections When an SSH connection fails, start by identifying the error. For more detailed diagnostic output, use: ```bash ssh -v -i my-server-key.pem ubuntu@203.0.113.10 ``` The `-v` option provides verbose output that helps reveal where the connection is failing. Additional verbosity is available with `-vv` or `-vvv`. Common errors include: | Error | Possible explanation | | --- | --- | | `Connection timed out` | Network path, routing, firewall, or security group issue | | `Connection refused` | The destination is reachable, but the port is not accepting connections | | `Permission denied (publickey)` | Incorrect username, key, or authentication configuration | | `Could not resolve hostname` | Invalid hostname or DNS resolution problem | | Host key verification warning | The server identity cannot be verified against the saved host key | These errors are clues, not definitive diagnoses. For example, a timeout does not prove that a security group is responsible; the problem could also be routing, an address issue, or a host that is unavailable. **Never bypass host key verification simply to make a connection succeed.** If a host key changes unexpectedly, verify the server's identity through a trusted channel before updating the saved key. * * * ## Inspecting Network Configuration on Linux Before troubleshooting connectivity, establish which network configuration the server is using. Linux provides several commands for inspecting network interfaces, addresses, and routes. ### Viewing Network Interfaces and IP Addresses Run: ```bash ip addr ``` Or, for a more concise view: ```bash ip -br addr ``` Example output: ```text lo UNKNOWN 127.0.0.1/8 ::1/128 eth0 UP 10.0.1.25/24 ``` Here, `eth0` has the private IPv4 address `10.0.1.25`. The `lo` interface is the loopback interface, used for communication within the local machine. Its IPv4 address is typically `127.0.0.1`. This distinction matters when testing applications. A service accessible through `localhost` may not be accessible through the server's network interface. ### Understanding the Routing Table An IP address tells you how an interface is addressed. A routing table helps determine where network traffic should go. Inspect it with: ```bash ip route ``` Example: ```text default via 10.0.1.1 dev eth0 10.0.1.0/24 dev eth0 proto kernel scope link ``` The default route tells Linux where to send traffic when no more specific route matches the destination. If a server cannot reach an external destination, an incorrect or missing route may be part of the problem. In cloud environments, the host's routing table is only one part of the picture. Cloud route tables and other network controls may also determine whether traffic can leave a subnet or reach another network. ### Testing Basic Reachability With ping The `ping` command sends ICMP echo requests to test whether a destination responds. For example: ```bash ping -c 4 1.1.1.1 ``` The `-c 4` option sends four requests and then stops. If replies arrive, the destination is responding to those requests through the current network path. If no replies arrive, however, that does not necessarily mean the destination is down. ICMP traffic may be blocked or ignored by the destination or an intermediate network device. Use `ping` as one diagnostic signal rather than the final test of whether an application is reachable. * * * ## Checking Listening Ports and Network Connections A service can be running but still fail to accept traffic on the expected port. For example, an application might start successfully but listen only on `127.0.0.1:8000` rather than on an address that allows access from other machines. The `ss` command helps inspect listening sockets and active network connections. ### Finding Listening Ports Run: ```bash sudo ss -lntup ``` The options mean: * `-l` — show listening sockets * `-n` — display numeric addresses and ports * `-t` — show TCP sockets * `-u` — show UDP sockets * `-p` — show associated processes, where available Example: ```text Netid State Local Address:Port tcp LISTEN 0.0.0.0:22 tcp LISTEN 127.0.0.1:8000 tcp LISTEN 0.0.0.0:80 ``` This example suggests: * SSH is listening on port 22 on all local IPv4 interfaces. * The application on port 8000 is listening only on the loopback interface. * A service is listening on port 80 on all local IPv4 interfaces. The exact output varies by server and configuration. ### Why the Listening Address Matters Consider a Python application started with: ```bash python3 -m http.server 8000 --bind 127.0.0.1 ``` It listens on the loopback interface. Requests made from the same machine can reach it, but requests directed to the server's private or public IP address generally cannot reach that listener. For a controlled test where remote access is intended, the application could instead bind to an appropriate network interface or to all IPv4 interfaces: ```bash python3 -m http.server 8000 --bind 0.0.0.0 ``` **Binding to** `0.0.0.0` **does not, by itself, make the application publicly accessible.** It allows the service to listen on local IPv4 interfaces. Firewalls, cloud security groups, routing, and other controls still determine whether remote traffic can reach it. Also, do not expose a development server directly to the public internet without appropriate security controls. ### Testing a Service Locally Use `curl` to send an HTTP request: ```bash curl -I http://127.0.0.1:8000 ``` The `-I` option requests response headers. If you need to inspect the full response, run: ```bash curl -v http://127.0.0.1:8000 ``` A successful local request confirms that the service responds through that local address. It does not establish that the service is reachable from another machine. That distinction is essential when diagnosing cloud networking issues. * * * ## DNS: Resolving Hostnames to IP Addresses Applications often communicate using hostnames rather than hard-coded IP addresses. For example: ```text api.example.com ``` DNS translates a hostname into records that clients can use to locate services. When a hostname fails to resolve, an application may be unable to connect even when the destination server itself is healthy. ### Testing Name Resolution A quick way to test name resolution on Linux is: ```bash getent hosts example.com ``` This uses the system's configured name-service resolution mechanism. If the `dig` utility is installed, you can inspect DNS responses more directly: ```bash dig example.com ``` To request only the returned IPv4 addresses: ```bash dig +short A example.com ``` For IPv6 address records: ```bash dig +short AAAA example.com ``` These commands help distinguish name-resolution issues from connection issues. For example, if a hostname resolves to an unexpected address, investigate its DNS records, resolver configuration, or caching behavior before assuming the application itself is broken. A successful DNS lookup also does not prove that the destination is reachable. It only confirms that a DNS response was obtained. * * * ## Inspecting Logs During Troubleshooting Network symptoms often originate in application or service configuration. Logs provide evidence about what happened when a request failed. As covered in the previous article, systemd-managed services can be inspected through `journalctl`. For example, to inspect SSH service logs on a system using systemd: ```bash sudo journalctl -u ssh -n 50 ``` On some distributions, the unit is named `sshd` instead: ```bash sudo journalctl -u sshd -n 50 ``` To follow new entries: ```bash sudo journalctl -u ssh -f ``` Use the unit name appropriate to your distribution. Depending on the service and system configuration, logs may also be stored in files under `/var/log`. For example: ```bash sudo less /var/log/syslog ``` This file is common on Ubuntu and Debian systems, but it may not exist or be used in the same way on every distribution. ### Search Logs for Relevant Errors If an application writes logs to a file, you can search for errors with: ```bash grep -i "error" application.log ``` To follow new log entries: ```bash tail -f application.log ``` These commands build on the file-reading and searching techniques covered earlier in the series. Rather than reading every log entry, narrow the investigation by looking at the time the problem occurred, the affected service, and the specific error. * * * ## A Practical Troubleshooting Example Suppose you deploy a Python web application on an Ubuntu EC2 instance. The application appears to start, but you cannot reach it from your browser. Assume the application is intended to listen on port `8000`. Instead of immediately changing cloud security groups or restarting the instance, work through the problem in layers. ### Step 1: Confirm the Application Is Running Inspect its process or, if it is managed by systemd, check its service status: ```bash sudo systemctl status myapp ``` Replace `myapp` with the actual service name. If the application is not managed by systemd, inspect the relevant process using tools such as `ps` or `pgrep`. ### Step 2: Inspect the Application Logs ```bash sudo journalctl -u myapp -n 50 ``` Look for startup failures, configuration errors, missing dependencies, or messages indicating that the application could not bind to its port. ### Step 3: Confirm the Listening Port ```bash sudo ss -lntp ``` Check whether the application is listening on port `8000` and which address it uses. If it listens only on `127.0.0.1:8000`, requests directed to the instance's private or public IP will not reach that listener. ### Step 4: Test the Application Locally ```bash curl -v http://127.0.0.1:8000 ``` If this request fails, focus on the application, its configuration, or its local listener before investigating the external network path. If it succeeds, continue testing the path from outside the server. ### Step 5: Inspect the Network Configuration Check the server's address and routes: ```bash ip -br addr ip route ``` Confirm that the instance has the expected network configuration and that its routes are appropriate for the intended traffic. ### Step 6: Check Cloud and Host Network Controls If the application responds locally but remains unreachable remotely, verify: * The application is listening on an appropriate interface. * The host firewall permits the intended traffic. * The EC2 security group allows the required inbound traffic from the appropriate source. * Network ACLs and subnet routing permit the traffic and its return path. * The instance has the appropriate addressing and routing for the intended access method. For public access, an internet gateway route and a suitable public IPv4 address or other valid public addressing arrangement may be required. A public address alone does not guarantee reachability. If the application is meant to be accessed through a load balancer or reverse proxy, validate that architecture's intended path instead of opening port `8000` to the entire internet. ### Step 7: Test From the Client From a machine that should be able to access the application, run: ```bash curl -v http://SERVER_ADDRESS:8000 ``` Replace `SERVER_ADDRESS` with the appropriate reachable address or hostname. Interpret the result alongside your other observations: * **Connection refused:** the destination was reached, but the connection was not accepted at that endpoint. * **Connection timed out:** packets or responses may be blocked, routed incorrectly, or lost along the path. * **HTTP error response:** an HTTP server responded, but the request was not handled successfully. * **Successful response:** the tested endpoint responded; further checks may still be necessary if the full application is not working. This process helps isolate the failure rather than changing several unrelated settings at once. * * * ## Common Linux Networking Mistakes A few recurring mistakes make server troubleshooting harder than it needs to be. ### Assuming a Running Process Is Reachable A process can be running without listening on the expected port or network interface. Confirm the listener and test the endpoint. ### Assuming `ping` Proves Application Availability ICMP reachability is not the same as TCP connectivity or a successful HTTP response. Test the actual protocol and port used by the application. ### Opening Every Port to the Internet Allowing unrestricted access may hide a connectivity problem while creating a security risk. Permit only the traffic required by the intended architecture and restrict source addresses where appropriate. ### Changing Cloud Network Rules Before Testing Locally If an application cannot respond to a request from the same machine, changing an external security group is unlikely to fix the underlying application problem. Start with the local listener and work outward. ### Ignoring Return Traffic A connection requires a viable path in both directions. Routing, firewalls, network ACLs, and other network controls can prevent responses from returning even when requests reach their destination. ### Treating Every Failure as a DNS Problem If a hostname resolves correctly but the connection fails, investigate the listener, destination address, port, route, and network controls as appropriate. ### Ignoring the Time of an Error Logs are most useful when correlated with the time of the failed request. Compare application logs, service logs, and connection tests from the same period. * * * ## A Repeatable Troubleshooting Method When a Linux server or application becomes unreachable, follow a structured process: 1. **Define the symptom.** Identify the exact hostname, address, port, protocol, and error. 2. **Check the application.** Confirm that it is running and inspect relevant logs. 3. **Check the listener.** Use `ss` to verify the expected port and binding address. 4. **Test locally.** Use `curl` or another protocol-appropriate client. 5. **Check name resolution and routing.** Inspect DNS responses, interface addresses, and routes as needed. 6. **Investigate network controls.** Check host firewalls, cloud security groups, network ACLs, and routing. 7. **Test from the intended client.** Verify the complete path, not just the server itself. 8. **Make one controlled change at a time.** Repeat the relevant tests to establish whether the change fixed the problem. This approach makes troubleshooting more efficient because each test eliminates or narrows down possible causes. ## Conclusion Linux networking troubleshooting starts with understanding how to access a server, inspect its network configuration, verify listening ports, test services, and investigate logs. Tools such as `ssh`, `ip`, `ss`, `ping`, `curl`, `dig`, and `journalctl` help you gather evidence at different layers of the system. Used together, they make it easier to distinguish an application failure from a name-resolution, routing, or connectivity problem. The objective is not to memorize every networking command. It is to identify where communication fails, gather evidence, and make the smallest safe change needed to restore the intended behavior. With these Linux administration skills in place, the final article in this series will cover **Bash Scripting and Linux Server Automation**, showing how to turn repeated command-line tasks into reliable, reusable scripts.