Updated January 26, 2021
This post is a detailed version of common practices for troubleshooting firewall rules that have been implemented to allow a server to connect with another server.
To help users in troubleshooting problems with network connectivity and firewall rule implementations, included below are step by step commands one can execute on a Windows or Unix system.
Step 1: Confirm network connectivity
On the source server, perform a operating system level connectivity test on established ports from source server/firewall to destination server/firewall. The test should verify routing, network address translation, ports, and URLs where applicable.
Telnet
Telnet is a good command to start your network connectivity tests. Open a command prompt on your server (e.g. cmd.exe or Powershell on Windows and a console or SSH session in Unix) and execute the following:
Note for Windows 7 users, you may have enable the Telnet client/command. To enable telnet on Windows 7, search in the start menu for “Turn Windows features on or off”. Click into that menu item and turn on the “Telnet client”.
<source_server> command prompt>> telnet <destination> <destination_port>
$ telnet justintung.wordpress.com 443
If there is connectivity from the source server to the destination, the telnet session should begin. On Windows and Unix, a blank screen or area should appear with a prompt. If there is no connectivity, it will just hang after the telnet command like the following:
Connecting To <destination_server>…
and then finally show:
Connecting To <destination_server>…Could not open connection to the host, on port <destination_port>: Connect failed.
Telnet hang on Windows PowerShell. Same thing happens on the normal cmd.exe window.
Moments later, we know the connectivity failed.
Other Telnet Failures
It is also possible you receive a message:
telnet: Unable to connect to remote host: Network is unreachable
This telnet message could occur from:
- Routing issues – Try a ping to the destination server (ping <destination>). Hopefully the ping is successful, if not check the netstat -rn to examine gateways used for routing. It could be that a static/alternatve route needs to be created to the destination server using an alternative gateway.
- Network infrastructure down – It is possible a piece of the network on the path to your destination server is down.
- Incorrect information in telnet command – Make sure the IPs or fully qualified domains you are using in telnet are correct.
Terminology
<source_server> is the server we are coming from and where the command is being executed on the operating system. The firewall should be opened from the source server and its source ports to the destination server and its destination ports.
<destination> is the destination server IP or fully qualified domain name (FQDN). The IP should work whether you are using an IPv4 or IPv6 address. However, if you are testing IPv6 connectivity, it is likely you will use a FQDN due to the long length of IPv6 addresses.
<destination_port> is the port on the destination server you are trying to get to. The destination server should be listening on this port. The following command (netstat -a) can be used on the destination server to check if the port is listening and you should be looking for the port after the IP address and whether its state is listening:
$ netstat -a
Proto Local Address Foreign Address State
TCP 10.86.32.211:80 fcore:0 LISTENING
In this example, we know the machine where we are executing the command is listening on port 80 on the IP address 10.86.32.211.
If there is no connectivity, follow steps below for troubleshooting procedures.
Step 2: Check other stuff
Check firewall rule and network implementation
Check you have the right IPs, fully qualified domain names (FQDN), and ports for the servers. Sometimes firewall rules are not requested properly and more work needs to be done to make connectivity happen such as:
- Host file entries
- Routing: static routes, address resolution protocol (ARP) configurations
- Network Address Translation (NAT)
- Security: VPN configuration, If applicable, ensure SSL certificate trust is in place and/or certificate exchanges installed and in place (e.g. 1-way, 2-way SSL certificates installed, and destination network can see successful SSL handshakes).
- Servers may have several IPs – make sure the correct ones are used and seen on the firewall(s)
Another type of test: Application Interface Test (depending on application availability)
This type of testing occurs from the application running on the source server and is common if you do not have access to the servers themselves and may be easier to perform. In this test, the application on the source server generates a test message/request to be received on the destination server interface and source confirms a valid response. This test confirms URL in place and basic messaging interface is available.
Step 3: Network connectivity troubleshooting on server
1. Test Connectivity on source server(s)
a. Initial test by server operator:
Telnet to destination IP via destination port. If IPs are using network address translation (NAT), choose the appropriate IP from source server – the IP the firewall sees and translates from the source server.
b. If telnet fails:
Check if static routing is done in source server routing table. Make sure routing goes to the correct default gateway and ping default gateway to check if it works.
Netstat – Checking the IP configuration on the server
netstat -rn
to check IP configuration on server and:
ping <destination_server>
Runs in Windows natively and in Unix sometimes under /usr/sbin/ping
c. If server routing table is ok:
Conduct the telnet test while having network resources monitor firewalls between source and destination points.
d. If traffic is not picked up on firewall:
Check points along the network path. If possible trace the network route. Traffic bound for the destination address should be monitored in case source server IPs are not seen on the firewall.
Traceroute – Find network routes
Use:
/usr/sbin/traceroute
to destination server and check for default gateway (assuming devices along the way do not have ICMP blocked).
Step 4: Network protocol/packet analyzers
If you have exhausted the steps outlined above and there is still no connectivity … 😦
you may have to use logging tools and applications traces.
The use of Wireshark, TCPdump, or similar tools for network protocol and packet analysis on the source and/or destination servers and firewalls will aid in the troubleshooting by checking for abnormalities or errors in packets.
Analyzing application logs
If you are working with applications that need the network connectivity, you can enable verbose logging on the application. When testing web services, you can enable logging for protocol level APIs. For example, if a Java application is a web service client and is connecting to the web service endpoint, we could enable logging of the HTTP implementation in Java via JVM arguments or look at the verbose logging of the Java SSL or Java Secure Socket Extension (JSSE) APIs.
Random thoughts:
There is no place like ::1
Justin Tung 
Chao anh, anh cho em hoi lam cach nao de test duoc IPs are using network address translation (NAT). Lam cach nao biet duoc IP la su dung NAT (behind NAT) bang Javascript hoac dong lenh command line nao cung duoc ma phu hop voi moi he dieu hanh?
Em cam on anh nhieu.
LikeLike
Hi lekimtrang, if you can ask your question in English, that would help me answer.
I’ve used a translator for your Vietnamese and interpreted your question. It sounded like you are looking for a command line command or JavaScript in relation to NAT IPs that can be used on all operating systems.
If you are testing connectivity, usually the Telnet command should work on more operating systems like Linux, Unix, Windows, and the Unix shell in Mac OS Terminal application.
For JavaScript and assuming you are checking for HTTP/S (e.g. port 80 and 443) web resources that you know exist, something similar to the following syntax like [bracket]img src=’http://DestinationServer.com/imageResource.jpg'[bracket]
can help you.
Telnet and that webpage with the JavaScript should work on all major operating systems and browsers
LikeLike
Hello Justin, thank you. But what information after telnet that I can know the IP address is behind NAT ? Thank you.
LikeLike
You’re welcome lekimtrang.
If you only have access on the source (client) end to test connectivity, there is not any way to check if the IP address is a NAT or just a regular IP. However, if you are connecting to a server in a large infrastructure (major corporation, cloud, ISPs), you can guess that they are probably using NATs.
Generally, only the firewall/device doing the translation between the source and destination servers would know or can track the the NAT IPs if there are any.
LikeLike
Hi Justin, I am writing a code to check this. But I do not know how to realize it. For example, is there any command line to know (such as ping, traceroute, netstat, etc ) . Thanks.
LikeLike
Hi lekimtrang, What is the computer language you are using to code?
An operating system script or scripting language should be able to handle the command line execution and you could check the command output.
Here is a code snippet from Perl using IO::Socket that uses some variables ($iphost is where we are running it, $ipaddr is what we are testing, $port is the port we are testing)
use strict;
use IO::Socket;
my $remote = IO::Socket::INET->new(
Proto => “tcp”,
PeerAddr => $ipaddr,
PeerPort => $port,
Timeout => 8,
);
#
# Print response
#
if ($remote) {
# print “$host:$port is alive\n”;
print “$iphost,$ipaddr,$port,alive\n”;
close $remote;
# exit 0;
}
else {
# print “$host:$port failed\n”;
print “$iphost,$ipaddr,$port,failed\n”;
# exit 1;
}
If it is in a program, here an example from Java that utilizes java.net.Socket to test connections:
http://stackoverflow.com/questions/1273453/test-socket-connection-with-java
That could be run in your source to test connectivity to a destination server.
LikeLike
Hello Justin, thank you!
LikeLike
hello, i am looking for this information too, that’s great to update my skil both Speaking English and Networking 🙂 .
Thank you!
LikeLike
dear Justin i neeed your help , i want become a LINUX Develoepr like you cna u please guid eme
LikeLike
Hi Shahabz,
Try doing these steps to get you on your way to development on Linux:
1. Start using Linux (if you aren’t already) on your computer.
a. Get familiar with the different distributions and programs. Ubuntu is one of the most popular distributions so if you choose that you are guaranteed help from other people and online documentation. I like to use Oracle’s VirtualBox to try out different distributions of Linux on one computer.
b. Try installing and running commonly used software on Linux (e.g. Apache, MySQL, PHP/Perl/Python – LAMP stack).
2. Learn a programming language for software on Linux. C is a good start – lots of resources online for that.
3. Join a local group and/or online group interested in Linux and talk to other Linux developers/users.
LikeLike
my client is not able to telnet in daily routine to server IP 10.0.249.14 and port but intermittently he able to access to server . I have check all uplink are working stable .
Ping response , trace route is also correct but i wont understand why sometimes telnet to server is not happening on other site client is able to telnet 10.0.249.23 and 10.0.249.6.
LikeLike
Hi Pramod, based on your description, it sounds like we can rule out firewalls as a problem.
I have seen a problem similar to yours recently. When there is intermittent connectivity problems, some causes could be:
1. An issue on the client, like something blocking the connection or preventing the connection successfully on the client side.
2. The server has intermittent problems and is not available all the time so the IP is sometimes not listening on the port, so the telnet fails.
3. A network issue is occurring like network dropping the connection or maybe there is too much traffic.
To address 1, you can try tests from a different client. Because you said telnet works for other IPs from the client, may be there are no client issues.
For number 2, you can try tests from a different client again or set up a monitor that checks if the server is always up for the port and IP you have. If there is intermittent failures, then we know it is a server problem.
Number 3 requires monitoring logs of the network for any odd things happening.
LikeLike
Dear Justin, There is no firewall configured at client end. Client machine is directly connected to router through LAN.
i have also checked the NAT translation on router .
Below are network connectivity in this case.
Client is connected through MPLS tunnel where link is terminated on Cisco router on both side i.e client as well as my side. Natting is done on client router where bank is able to access all other servers in this range but for this particular server connection is intermittently drops means bank don’t able to telnet on that time.
one thing bring to notice is that when bank not able to telnet then also translation on router and hit-count on my firewall to particular access list is increases.
I have also check with antivirus , utilisation report but it is found correct.
i have also check Uplinks with the help of ISP but no issue found .
LikeLike
Dear Justin,
we thought that it could be due to cross-connect was not upgraded. But now, the cross-connect for link has been upgraded and Airtel link bandwidth is 16 Mbps.
Still same issue recurring.
LikeLike
Hi Pramod, pardon the delay in replying.
Thanks for the details of the issue.
The issue could be a network, hardware, or application issue (just a guess). I referenced work my team have done on a similar problem. For troubleshooting that issue, we did monitoring at every part of the connection. So during a telnet or client test – monitor the client, the router/firewall, LAN, and the endpoint server. For client and server monitoring, we used a tool called Wireshark. In our case, the problem turned out to be a hardware encryption device between the client and server. None of us could have predicted that and it was not related to router, nating, or the applicaiton.
If you run some tests, the network can detect the network error so we know what’s happening, Then investigation is needed to determine what might be causing the error.
From what you describe the problem will not be a common issue since you’ve already checked the obvious things like uplink, routing, antivirus and bandwidth.
LikeLike