Network connectivity troubleshooting step by step with commands


This post is an enhanced version of my post on common practices for troubleshooting firewall rules that have been implemented to allow a server to connect with another server.

To help users in troubleshooting problems with network connectivity and firewall rule implementations, included below are step by step commands one can execute on a Windows or Unix system.

Step 1: Confirm network connectivity

On the source server, perform a operating system level connectivity test on established ports from source server/firewall to destination server/firewall. The test should verify routing, network address translation, ports, and URLs where applicable.

Telnet

Telnet is a good command to start your network connectivity tests. Open a command prompt on your server (e.g. cmd.exe or Powershell on Windows and a console or SSH session in Unix) and execute the following:

Note for Windows 7 users, you may have enable the Telnet client/command. To enable telnet on Windows 7, search in the start menu for “Turn Windows features on or off”. Click into that menu item and turn on the “Telnet client”.

<source_server> command prompt>> telnet <destination> <destination_port>

$ telnet justintung.wordpress.com 80

If there is connectivity from the source server to the destination, the telnet session should begin. On Windows and Unix, a blank screen or area should appear with a prompt. If there is no connectivity, it will just hang after the telnet command like the following:

Connecting To <destination_server>…

and then finally show:

Connecting To <destination_server>…Could not open connection to the host, on port <destination_port>: Connect failed.

Telnet hang on Windows PowerShell. Same thing happens on the normal cmd.exe window.

Moments later, we know the connectivity failed.

Other Telnet Failures

It is also possible you receive a message:

telnet: Unable to connect to remote host: Network is unreachable

This telnet message could occur from:

  • Routing issues – Try a ping to the destination server (ping <destination>). Hopefully the ping is successful, if not check the netstat -rn to examine gateways used for routing. It could be that a static/alternatve route needs to be created to the destination server using an alternative gateway.
  • Network infrastructure down – It is possible a piece of the network on the path to your destination server is down.
  • Incorrect information in telnet command – Make sure the IPs or fully qualified domains you are using in telnet are correct.

Terminology

<source_server> is the server we are coming from and where the command is being executed on the operating system. The firewall should be opened from the source server and its source ports to the destination server and its destination ports.

<destination> is the destination server IP or fully qualified domain name (FQDN). The IP should work whether you are using an IPv4 or IPv6 address. However, if you are testing IPv6 connectivity, it is likely you will use a FQDN due to the long length of IPv6 addresses.

<destination_port> is the port on the destination server you are trying to get to. The destination server should be listening on this port. The following command  (netstat -a) can be used on the destination server to check if the port is listening and you should be looking for the port after the IP address and whether its state is listening:

$ netstat -a

 Proto  Local Address          Foreign Address        State

 TCP    10.86.32.211:80            fcore:0                LISTENING

In this example, we know the machine where we are executing the command is listening on port 80 on the IP address 10.86.32.211.

If there is no connectivity, follow steps below for troubleshooting procedures.

Step 2: Check other stuff

Check firewall rule and network implementation

Check you have the right IPs, fully qualified domain names (FQDN), and ports for the servers. Sometimes firewall rules are not requested properly and more work needs to be done to make connectivity happen such as:

  • Host file entries
  • Routing: static routes, address resolution protocol (ARP) configurations
  • Network Address Translation (NAT)
  • Security: VPN configuration, If applicable, ensure SSL certificate trust is in place and/or certificate exchanges installed and in place (e.g. 1-way, 2-way SSL certificates installed, and destination network can see successful SSL handshakes).
  • Servers may have several IPs – make sure the correct ones are used and seen on the firewall(s)

Another type of test: Application Interface Test (depending on application availability)

This type of testing occurs from the application running on the source server and is common if you do not have access to the servers themselves and may be easier to perform. In this test, the application on the source server generates a test message/request to be received on the destination server interface and source confirms a valid response. This test confirms URL in place and basic messaging interface is available.

Step 3: Network connectivity troubleshooting on server

1. Test Connectivity on source server(s)

a. Initial test by server operator:
Telnet to destination IP via destination port. If IPs are using network address translation (NAT), choose the appropriate IP from source server – the IP the firewall sees and translates from the source server.

b. If telnet fails:
Check if static routing is done in source server routing table. Make sure routing goes to the correct default gateway and ping default gateway to check if it works.

Netstat – Checking the IP configuration on the server

netstat -rn

to check IP configuration on server and:

ping <destination_server>

Runs in Windows natively and in Unix sometimes under /usr/sbin/ping

c. If server routing table is ok:
Conduct the telnet test while having network resources monitor firewalls between source and destination points.

d. If traffic is not picked up on firewall:
Check points along the network path. If possible trace the network route. Traffic bound for the destination address should be monitored in case source server IPs are not seen on the firewall.

Traceroute – Find network routes

Use:

/usr/sbin/traceroute

to destination server and check for default gateway (assuming devices along the way do not have ICMP blocked).

Step 4: Network protocol/packet analyzers

If you have exhausted the steps outlined above and there is still no connectivity … 😦
you may have to use logging tools and applications traces.

The use of Wireshark, TCPdump, or similar tools for network protocol and packet analysis on the source and/or destination servers and firewalls will aid in the troubleshooting by checking for abnormalities or errors in packets.

Analyzing application logs

If you are working with applications that need the network connectivity, you can enable verbose logging on the application. When testing web services, you can enable logging for protocol level APIs. For example, if a Java application is a web service client and is connecting to the web service endpoint, we could enable logging of the HTTP implementation in Java via JVM arguments or look at the verbose logging of the Java SSL or Java Secure Socket Extension (JSSE) APIs.

Random thoughts:

There is no place like ::1

Advertisements

11 thoughts on “Network connectivity troubleshooting step by step with commands

  1. Chao anh, anh cho em hoi lam cach nao de test duoc IPs are using network address translation (NAT). Lam cach nao biet duoc IP la su dung NAT (behind NAT) bang Javascript hoac dong lenh command line nao cung duoc ma phu hop voi moi he dieu hanh?
    Em cam on anh nhieu.

    • Hi lekimtrang, if you can ask your question in English, that would help me answer.

      I’ve used a translator for your Vietnamese and interpreted your question. It sounded like you are looking for a command line command or JavaScript in relation to NAT IPs that can be used on all operating systems.

      If you are testing connectivity, usually the Telnet command should work on more operating systems like Linux, Unix, Windows, and the Unix shell in Mac OS Terminal application.

      For JavaScript and assuming you are checking for HTTP/S (e.g. port 80 and 443) web resources that you know exist, something similar to the following syntax like [bracket]img src=’http://DestinationServer.com/imageResource.jpg'[bracket]
      can help you.

      Telnet and that webpage with the JavaScript should work on all major operating systems and browsers

  2. Hello Justin, thank you. But what information after telnet that I can know the IP address is behind NAT ? Thank you.

    • You’re welcome lekimtrang.

      If you only have access on the source (client) end to test connectivity, there is not any way to check if the IP address is a NAT or just a regular IP. However, if you are connecting to a server in a large infrastructure (major corporation, cloud, ISPs), you can guess that they are probably using NATs.

      Generally, only the firewall/device doing the translation between the source and destination servers would know or can track the the NAT IPs if there are any.

  3. Hi Justin, I am writing a code to check this. But I do not know how to realize it. For example, is there any command line to know (such as ping, traceroute, netstat, etc ) . Thanks.

    • Hi lekimtrang, What is the computer language you are using to code?

      An operating system script or scripting language should be able to handle the command line execution and you could check the command output.

      Here is a code snippet from Perl using IO::Socket that uses some variables ($iphost is where we are running it, $ipaddr is what we are testing, $port is the port we are testing)

      use strict;
      use IO::Socket;

      my $remote = IO::Socket::INET->new(
      Proto => “tcp”,
      PeerAddr => $ipaddr,
      PeerPort => $port,
      Timeout => 8,
      );
      #
      # Print response
      #
      if ($remote) {
      # print “$host:$port is alive\n”;
      print “$iphost,$ipaddr,$port,alive\n”;
      close $remote;
      # exit 0;
      }
      else {
      # print “$host:$port failed\n”;
      print “$iphost,$ipaddr,$port,failed\n”;
      # exit 1;
      }

      If it is in a program, here an example from Java that utilizes java.net.Socket to test connections:
      http://stackoverflow.com/questions/1273453/test-socket-connection-with-java

      That could be run in your source to test connectivity to a destination server.

  4. hello, i am looking for this information too, that’s great to update my skil both Speaking English and Networking 🙂 .
    Thank you!

  5. dear Justin i neeed your help , i want become a LINUX Develoepr like you cna u please guid eme

    • Hi Shahabz,

      Try doing these steps to get you on your way to development on Linux:
      1. Start using Linux (if you aren’t already) on your computer.
      a. Get familiar with the different distributions and programs. Ubuntu is one of the most popular distributions so if you choose that you are guaranteed help from other people and online documentation. I like to use Oracle’s VirtualBox to try out different distributions of Linux on one computer.
      b. Try installing and running commonly used software on Linux (e.g. Apache, MySQL, PHP/Perl/Python – LAMP stack).
      2. Learn a programming language for software on Linux. C is a good start – lots of resources online for that.
      3. Join a local group and/or online group interested in Linux and talk to other Linux developers/users.

  6. Pingback: Topic 7: Wireless Network Hardware – ClueVxVless

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s