Network & DNS Troubleshooting Tools

From SonicWiki
Jump to: navigation, search

Sonic.net provides a number of tools to assist in troubleshooting a variety of network-related problems. This document deals specifically with troubleshooting tools located at http://www.sonic.net/stats/.

Ping and Traceroute Gateways

Ping is a means of sending an echo request to another host on the Internet. This can be used to determine whether a particular system is online and responding to traffic. It can also be used to measure latency, or turn-around-time for requests. Please note that many routers and firewalls do not respond to ping requests. To ping another host from Sonic.net, load the Ping & Traceroute Gateway (http://www.sonic.net/cgi/named/icmp.cgi) and type in the name or IP address of the host you would like to test. The results will look something like this:

Executing /bin/ping -c  5 198.189.195.58 2>&1
PING 198.189.195.58 (198.189.195.58) from 208.201.224.115 : 56(84) bytes of data.
64 bytes from 198.189.195.58: icmp_seq=0 ttl=54 time=7.565 msec
64 bytes from 198.189.195.58: icmp_seq=1 ttl=54 time=9.276 msec
64 bytes from 198.189.195.58: icmp_seq=2 ttl=54 time=7.860 msec
64 bytes from 198.189.195.58: icmp_seq=3 ttl=54 time=7.603 msec
64 bytes from 198.189.195.58: icmp_seq=4 ttl=54 time=7.873 msec

--- 198.189.195.58 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/mdev = 7.565/8.035/9.276/0.638 ms

This output shows information for each individual attempt to ping the remote host (by default, five attempts will be made), followed by a summary of all attempts. In this example, five packets were transmitted, of which all five were returned. Traceroute is similar to ping, but instead of simply sending requests to the target host, each intermediate host along the way is tested. This can be useful for locating problems between yourself and some other host on the Internet. To trace a route to another host from Sonic.net, load the Ping & Traceroute Gateway (http://www.sonic.net/cgi/named/icmp.cgi), type in the name or IP address of the host you would like traced to, and select "traceroute" from the pull-down menu. The results will look something like this:

Executing /usr/sbin/traceroute -n -w2 -q  5
www.google.com 2>&1
traceroute: Warning: www.google.com has multiple addresses; using 66.102.7.104
traceroute to www.l.google.com (66.102.7.104), 30 hops max, 38 byte packets
 1  208.201.224.36  0.394 ms  0.280 ms  0.282 ms  0.284 ms  0.279 ms
 2  64.142.0.198  1.586 ms  1.623 ms  1.612 ms  1.602 ms  1.580 ms
 3  64.142.0.181  1.694 ms  1.920 ms  1.660 ms  1.842 ms  1.695 ms
 4  209.237.229.25  1.885 ms  1.813 ms  1.993 ms  1.582 ms  1.658 ms
 5  209.237.224.18  1.851 ms  2.001 ms  1.868 ms  1.921 ms  1.930 ms
 6  209.237.224.22  3.299 ms  3.349 ms  3.428 ms  3.300 ms  3.799 ms
 7  206.223.116.21  4.046 ms  4.206 ms  3.938 ms  3.744 ms  3.882 ms
 8  72.14.236.3  5.082 ms  5.427 ms  5.150 ms  4.186 ms  5.301 ms
 9  72.14.236.11  4.712 ms  4.812 ms  4.357 ms  4.817 ms  4.381 ms
10  64.233.174.54  85.770 ms  80.384 ms 66.249.94.31  84.533 ms  88.621 ms 76.815 ms
11  216.239.49.150  6.665 ms  7.079 ms  7.484 ms  7.319 ms  7.245 ms
12  66.102.7.104  4.786 ms  5.284 ms  5.133 ms  5.237 ms  4.828 ms


In this case, there were 12 hops. For each hop along the route, five pings were send, and the round-trip time for each ping was recorded. Note that on the 10th hop in this trace, the round-trip time was far higher than any others sampled. This is commonly the case with busy routers that give ping and traceroutes a low priority. Matt's Traceroute is a traceroute that simply displays differently. To use Matt's Traceroute on another host from Sonic.net, load the Ping & Traceroute Gateway (http://www.sonic.net/cgi/named/icmp.cgi) and type in the name or IP address of the host you would like traced to, and select "Matt's Traceroute" from the pull-down menu. The results will look something like this (times listed are in milliseconds):

Executing /usr/sbin/mtr --report --report-cycles=5 www.google.com 2>&1
HOST                                    LOSS  RCVD SENT    BEST     AVG   WORST
2.ge-2-1-0.gw.sr.sonic.net                0%     5    5    0.30    0.36    0.43
0.ge-0-1-0.gw4.200p-sf.sonic.net          0%     5    5    1.62    2.78    7.27
0.at-0-0-0.gw3.200p-sf.sonic.net          0%     5    5    1.81    2.15    2.87
pos-2-2.sfo1.unitedlayer.com              0%     5    5    1.64    1.83    2.14
ge1-br01-200p-sfo.unitedlayer.com         0%     5    5    1.71    1.88    2.19
GE2-br01-eqx-sjc.unitedlayer.com          0%     5    5    3.24    3.68    4.96
eqixsj-google-gige.google.com             0%     5    5    3.74   17.38   69.68
72.14.236.3                               0%     5    5    5.06    5.17    5.35
72.14.236.11                              0%     5    5    4.45   14.09   50.52
66.249.94.29                              0%     5    5    4.51    5.04    5.92
216.239.49.150                           20%     4    5    6.23    6.52    7.12
66.102.7.99                               0%     5    5    4.65    4.87    5.19

Note that the individual round-trip times for the individual pings are not displayed, replaced with a simple "best / average / worst" listing, which can nicely summarize the health of a route. Also note that on the twelfth hop, 20% loss is recorded. This simply means that one of the five pings sent during this test did not return. When errors of this sort appear, it is prudent to test again to see if the problem is consistent. For each of these three tools, it is possible to specify whether you would like to have each host pinged from five to 100 times. Below are the results of running Matt's Traceroute on "www.google.com" with the number of pings set to 50:

Executing /usr/sbin/mtr --report --report-cycles=50 www.google.com 2>&1
HOST                                    LOSS  RCVD SENT    BEST     AVG   WORST
2.ge-2-1-0.gw.sr.sonic.net                0%    50   50    0.28    0.90   15.38
0.ge-0-1-0.gw4.200p-sf.sonic.net          0%    50   50    1.59    1.94    6.90
0.at-0-0-0.gw3.200p-sf.sonic.net          0%    50   50    1.67    2.04    7.96
pos-2-2.sfo1.unitedlayer.com              0%    50   50    1.57    1.83    2.25
ge1-br01-200p-sfo.unitedlayer.com         0%    50   50    1.64    1.91    2.64
GE2-br01-eqx-sjc.unitedlayer.com          0%    50   50    3.10    3.43    4.64
eqixsj-google-gige.google.com             0%    50   50    3.54    8.49   90.14
72.14.236.3                               0%    50   50    3.91    5.43   23.97
72.14.236.11                              0%    50   50    4.15    5.11   21.91
66.249.94.29                              0%    50   50    4.48    5.12   14.79
216.239.49.146                            8%    46   50    5.82    6.84    8.20
66.102.7.99                               0%    50   50    4.55    4.93    6.33

nslookup, host, dig, and doc DNS tools

nslookup simulates a name server query. The basic purpose of this is to either look up an IP address based on a domain name (e.g. www.sonic.net) or to look up a domain name based upon an IP address (e.g. 209.204.190.64). To use nslookup, open the DNS Troubleshooting Gateway (http://www.sonic.net/cgi/named/dns.cgi) and type in the name or IP of the host you would like to look up. The results will look something like this:

Executing /usr/bin/nslookup -silent  www.sonic.net 2>&1
Server:		208.201.224.11
Address:	208.201.224.1153

Name:	www.sonic.net
Address: 209.204.190.64

This shows the name and IP address of the name server being used (both are 208.201.224.11), as well as the name and IP address of the host you were looking up. host is very similar to nslookup, and returns the IP of a particular domain name or the name associated with a particular IP address. To use host, open the DNS Troubleshooting Gateway (http://www.sonic.net/cgi/named/dns.cgi), type in the name or IP of the host you would like to look up, and select "host" from the pull-down menu. The results will look something like this:

Executing /usr/bin/host www.sonic.net 2>&1
www.sonic.net has address 209.204.190.64

dig is another tool used to determine the IP address of a given name, or the name of a given IP. To use dig, open the DNS Troubleshooting Gateway (http://www.sonic.net/cgi/named/dns.cgi), type in the name or IP of the host you would like to look up, and select "dig" from the pull-down menu. The results will look something like this:

Executing /usr/bin/dig www.sonic.net 2>&1

; <<>> DiG 9.2.1 <<>> www.sonic.net
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5557
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3

;; QUESTION SECTION:
;www.sonic.net.			IN	A

;; ANSWER SECTION:
www.sonic.net.		14400	IN	A	209.204.190.64

;; AUTHORITY SECTION:
sonic.net.		86400	IN	NS	c.auth-ns.sonic.net.
sonic.net.		86400	IN	NS	a.auth-ns.sonic.net.
sonic.net.		86400	IN	NS	b.auth-ns.sonic.net.

;; ADDITIONAL SECTION:
a.auth-ns.sonic.net.	86400	IN	A	209.204.159.20
b.auth-ns.sonic.net.	86400	IN	A	64.142.88.72
c.auth-ns.sonic.net.	86400	IN	A	69.9.186.104

;; Query time: 37 msec
;; SERVER: 208.201.224.1153(208.201.224.11)
;; WHEN: Wed Nov 30 20:04:27 2005
;; MSG SIZE  rcvd: 151

Dig provides a prolific amount of information, including the question, answer, additonal information regarding the source of the answers, and diagnostic information about how long the transaction took to perform. doc is a heavy-duty tool that is not for the faint of heart. Doc will query each of the root servers, gtld servers, and any systems that these servers have delegated to, down the chain of authority until it arrives at the legitimate, authoritative servers for the name queried. To use doc, open the DNS Troubleshooting Gateway (http://www.sonic.net/cgi/named/dns.cgi), type in the name of the host you would like to look up, and select "doc" from the pull-down menu. The results will look something like this:

Executing /nfs/httpd/cgi-bin/named/doc/doc -v privateerpress.com 2>&1
Doc-2.2.3: doc -v privateerpress.com
Doc-2.2.3: Starting test of privateerpress.com.   parent is com.
Doc-2.2.3: Test date - Wed Nov 30 20:17:26 PST 2005
DIGERR (NOT_AUTHORIZED): dig @a.gtld-servers.net. for SOA of parent (com.) failed
DIGERR (NOT_AUTHORIZED): dig @b.gtld-servers.net. for SOA of parent (com.) failed
soa @c.gtld-servers.net. for com. has serial: 1133410643
soa @d.gtld-servers.net. for com. has serial: 1133410643
soa @e.gtld-servers.net. for com. has serial: 1133410643
soa @f.gtld-servers.net. for com. has serial: 1133410643
soa @g.gtld-servers.net. for com. has serial: 1133410643
soa @h.gtld-servers.net. for com. has serial: 1133410643
soa @i.gtld-servers.net. for com. has serial: 1133410643
soa @j.gtld-servers.net. for com. has serial: 1133410643
soa @k.gtld-servers.net. for com. has serial: 1133410643
soa @l.gtld-servers.net. for com. has serial: 1133410643
soa @m.gtld-servers.net. for com. has serial: 1133410643
SOA serial 's agree for com. domain
Found 2 NS and 2 glue records for privateerpress.com. @c.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @d.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @e.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @f.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @g.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @h.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @i.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @j.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @k.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @l.gtld-servers.net. (non-AUTH)
Found 2 NS and 2 glue records for privateerpress.com. @m.gtld-servers.net. (non-AUTH)
DNServers for com.
   === 0 were also authoritatve for privateerpress.com.
   === 11 were non-authoritative for privateerpress.com.
Servers for com. (not also authoritative for privateerpress.com.)
   === agree on NS records for privateerpress.com.
NS list summary for privateerpress.com. from parent (com.) servers
  == ns1.hardhathosting.net. ns2.hardhathosting.net.
soa @ns1.hardhathosting.net. for privateerpress.com. serial: 503310001
soa @ns2.hardhathosting.net. for privateerpress.com. serial: 503310001
SOA serial 's agree for privateerpress.com.
Authoritative domain (privateerpress.com.) servers agree on NS for privateerpress.com.
ERROR: NS list from privateerpress.com. authoritative servers does not
  === match NS list from parent (com.) servers
NS list summary for privateerpress.com. from authoritative servers
  == ns1.hardhathosting.net. ns1.privateerpress.com. ns2.hardhathosting.net.
  == ns2.privateerpress.com.
Checking 1 potential addresses for hosts at privateerpress.com.
  == 65.77.181.2
in-addr PTR record found for 65.77.181.2
Summary:
   ERRORS found for privateerpress.com. (count: 1)
Done testing privateerpress.com.  Wed Nov 30 20:17:51 PST 2005

The doc tool can be used to track down otherwise-inexplicable cases in which Internet users at different ISPs are getting different results from the same queries. When properly configured, each server involved will agree with the others regarding the serial numbers and authoritative servers.

Zonecheck Comprehensive DNS troubleshooting Tool

Zonecheck (http://www.sonic.net/stats/zonecheck/) is a comprehensive test suite that determines the health of a particule zone on the Internet. Sixteen tests are run to confirm that the hosting for a particular domain name is running properly. Please note that if a fatal error is encountered, no further tests will be performed, though this can be overridden. More information regarding the particulars of the Zonecheck tool can be found here.

DNSreport.com DNS Tools

DNSreport.com DNS Tools (http://dnsreport.com/) is not hosted by Sonic.net. This testing tool allows a 3rd party DNS report. This can be valuable for confirming the accuracy of a [zonecheck Zonecheck] or [ping doc] test from the point of view of a host outside the Sonic.net network.