This post is an enhanced version of my post on common practices for troubleshooting firewall rules that have been implemented to allow a server to connect with another server.

To help users in troubleshooting problems with network connectivity and firewall rule implementations, included below are step by step commands one can execute on a Windows or Unix system.

Step 1: Confirm network connectivity

On the source server, perform a operating system level connectivity test on established ports from source server/firewall to destination server/firewall. The test should verify routing, network address translation, ports, and URLs where applicable.

Telnet

Telnet is a good command to start your network connectivity tests. Open a command prompt on your server (e.g. cmd.exe or Powershell on Windows and a console or SSH session in Unix) and execute the following:

Note for Windows 7 users, you may have enable the Telnet client/command. To enable telnet on Windows 7, search in the start menu for “Turn Windows features on or off”. Click into that menu item and turn on the “Telnet client”.

<source_server> command prompt>> telnet <destination> <destination_port>

$ telnet justintung.wordpress.com 80

If there is connectivity from the source server to the destination, the telnet session should begin. On Windows and Unix, a blank screen or area should appear with a prompt. If there is no connectivity, it will just hang after the telnet command like the following:

Connecting To <destination_server>…

and then finally show:

Connecting To <destination_server>…Could not open connection to the host, on port <destination_port>: Connect failed.

Telnet hang on Windows PowerShell. Same thing happens on the normal cmd.exe window.

Moments later, we know the connectivity failed.

Other Telnet Failures

It is also possible you receive a message:

telnet: Unable to connect to remote host: Network is unreachable

This telnet message could occur from:

• Routing issues – Try a ping to the destination server (ping <destination>). Hopefully the ping is successful, if not check the netstat -rn to examine gateways used for routing. It could be that a static/alternatve route needs to be created to the destination server using an alternative gateway.
• Network infrastructure down – It is possible a piece of the network on the path to your destination server is down.
• Incorrect information in telnet command - Make sure the IPs or fully qualified domains you are using in telnet are correct.

Terminology

<source_server> is the server we are coming from and where the command is being executed on the operating system. The firewall should be opened from the source server and its source ports to the destination server and its destination ports.

<destination> is the destination server IP or fully qualified domain name (FQDN). The IP should work whether you are using an IPv4 or IPv6 address. However, if you are testing IPv6 connectivity, it is likely you will use a FQDN due to the long length of IPv6 addresses.

<destination_port> is the port on the destination server you are trying to get to. The destination server should be listening on this port. The following command  (netstat -a) can be used on the destination server to check if the port is listening and you should be looking for the port after the IP address and whether its state is listening:

$ netstat -a

 Proto  Local Address          Foreign Address        State

 TCP    10.86.32.211:80            fcore:0                LISTENING

In this example, we know the machine where we are executing the command is listening on port 80 on the IP address 10.86.32.211.

If there is no connectivity, follow steps below for troubleshooting procedures.

Step 2: Check other stuff

Check firewall rule and network implementation

Check you have the right IPs, fully qualified domain names (FQDN), and ports for the servers. Sometimes firewall rules are not requested properly and more work needs to be done to make connectivity happen such as:

• Host file entries
• Routing: static routes, address resolution protocol (ARP) configurations
• Network Address Translation (NAT)
• Security: VPN configuration, If applicable, ensure SSL certificate trust is in place and/or certificate exchanges installed and in place (e.g. 1-way, 2-way SSL certificates installed, and destination network can see successful SSL handshakes).
• Servers may have several IPs – make sure the correct ones are used and seen on the firewall(s)

Another type of test: Application Interface Test (depending on application availability)

This type of testing occurs from the application running on the source server and is common if you do not have access to the servers themselves and may be easier to perform. In this test, the application on the source server generates a test message/request to be received on the destination server interface and source confirms a valid response. This test confirms URL in place and basic messaging interface is available.

Step 3: Network connectivity troubleshooting on server

1. Test Connectivity on source server(s)

a. Initial test by server operator:
Telnet to destination IP via destination port. If IPs are using network address translation (NAT), choose the appropriate IP from source server – the IP the firewall sees and translates from the source server.

b. If telnet fails:
Check if static routing is done in source server routing table. Make sure routing goes to the correct default gateway and ping default gateway to check if it works.

Netstat – Checking the IP configuration on the server

netstat -rn

to check IP configuration on server and:

ping <destination_server>

Runs in Windows natively and in Unix sometimes under /usr/sbin/ping

c. If server routing table is ok:
Conduct the telnet test while having network resources monitor firewalls between source and destination points.

d. If traffic is not picked up on firewall:
Check points along the network path. If possible trace the network route. Traffic bound for the destination address should be monitored in case source server IPs are not seen on the firewall.

Traceroute – Find network routes

Use:

/usr/sbin/traceroute

to destination server and check for default gateway (assuming devices along the way do not have ICMP blocked).

Step 4: Network protocol/packet analyzers

If you have exhausted the steps outlined above and there is still no connectivity …
you may have to use logging tools and applications traces.

The use of Wireshark, TCPdump, or similar tools for network protocol and packet analysis on the source and/or destination servers and firewalls will aid in the troubleshooting by checking for abnormalities or errors in packets.

Analyzing application logs

If you are working with applications that need the network connectivity, you can enable verbose logging on the application. When testing web services, you can enable logging for protocol level APIs. For example, if a Java application is a web service client and is connecting to the web service endpoint, we could enable logging of the HTTP implementation in Java via JVM arguments or look at the verbose logging of the Java SSL or Java Secure Socket Extension (JSSE) APIs.

Random thoughts:

There is no place like ::1