Check HTTP header on multiple servers

How can we dump the HTTP header for a specified URL?
It’s not difficult since linux has such tools to do that, below is an example:

$ curl -s --head https://www.admon.org/robots.txt
HTTP/1.1 200 OK
Date: Fri, 25 Nov 2011 05:52:11 GMT
Server: Apache/2.2.11 (Unix) PHP/5.3.8 DAV/2 SVN/1.6.4
X-Powered-By: PHP/5.3.8
X-Pingback: https://www.admon.org/xmlrpc.php
Cache-Control: max-age=303445772
Expires: Wed, 07 Jul 2021 08:21:44 GMT
Content-Type: text/plain; charset=utf-8

If the domain name is resolved to multiple IP addresses (e.g. in a CDN environment), how can we check the header on all servers one by one?
It’s not a problem, here’s a script for this task. It resolves the domain names on a list of DNS servers, and collects the resolved IP addresses, then check HTTP header on each of the server:

#!/bin/bash
#
# Check HTTP header on a list of servers.
# The servers are resolved by specified DNS servers.
#            by Joseph Chen <joseph#admon.org>
# GPL v2 licensed.

# Define your DNS servers here!
DNS_LIST=(61.139.2.69 192.168.0.3)

die_msg() {
	echo $@
	exit -1
}

[ $# != 1 ] && die_msg Only accept one URL as parameter.
which  dig >/dev/null || die_msg We need the command dig to do DNS resolving.
which curl >/dev/null || die_msg We need the command curl to send request.

URL=$1
DNAME=$(echo $URL |awk -F// '{print $2}' |awk -F/ '{print $1}')
DFILE=$(echo $URL |awk -F${DNAME} '{print $2}' |awk -F? '{print $1}')

echo Start to check the requested file is $DFILE on domain $DNAME

for ((i=0; i<${#DNS_LIST[@]}; i++)) do echo Resolving $DNAME on DNS server ${DNS_LIST[$i]} dig +time=1 $DNAME @${DNS_LIST[$i]} |awk '!/^;/ && /IN\tA/{print $NF}' >/tmp/.cdn.resolved.${i}
done

echo
echo Starting to check HTTP header for $URL on these IP:
echo $(cat /tmp/.cdn.resolved.* | sort | uniq)
echo

echo Checking Last-Modified for http://${DNAME}${DFILE} 
for RESOLVED_IP in $(cat /tmp/.cdn.resolved.* |sort |uniq)
do
	/bin/echo -n "Resolved IP: $RESOLVED_IP >>>> " 
	curl -s --head --connect-timeout 2 --header "Host: $DNAME" http://${RESOLVED_IP}${DFILE} | grep ^Last-Modified || /bin/echo
done

rm -f /tmp/.cdn.resolved.* 2>/dev/null

A running example can be seen as below:

$ sh dump-CDN-header.sh http://adm.fwmrm.net/p/fox_live/AdManager.swf
Start to check the requested file is /p/fox_live/AdManager.swf on domain adm.fwmrm.net
Resolving adm.fwmrm.net on DNS server 61.139.2.69
Resolving adm.fwmrm.net on DNS server 192.168.0.3

Starting to check HTTP header for http://adm.fwmrm.net/p/fox_live/AdManager.swf on these IP:
124.40.41.30 124.40.41.40 125.141.136.100 125.141.136.101 125.141.136.102 125.141.136.98 193.108.88.192 205.168.236.217 205.168.236.227

Checking Last-Modified for http://adm.fwmrm.net/p/fox_live/AdManager.swf
Resolved IP: 124.40.41.30 >>>> Last-Modified: Tue, 22 Nov 2011 08:21:11 GMT
Resolved IP: 124.40.41.40 >>>> Last-Modified: Tue, 22 Nov 2011 08:21:11 GMT
Resolved IP: 125.141.136.100 >>>>
Resolved IP: 125.141.136.101 >>>>
Resolved IP: 125.141.136.102 >>>>
Resolved IP: 125.141.136.98 >>>>
Resolved IP: 193.108.88.192 >>>>
Resolved IP: 205.168.236.217 >>>> Last-Modified: Tue, 22 Nov 2011 08:21:11 GMT
Resolved IP: 205.168.236.227 >>>> Last-Modified: Tue, 22 Nov 2011 08:21:11 GMT 

Leave a comment

Your email address will not be published. Required fields are marked *