User Agent Differences
I wrote this script for the Netscape 6 development project. We developed automated scripts which reported for each build our key MTBF, code size, and performance metrics. Wherever relevant, we compared 6.0 to Netscape's most recent shipping release and Microsoft's most recent shipping release. In interpreting the results for page loading performance, we wondered: what if web sites were serving up a different page depending on the browser's user agent? If lots of sites did so, then our results could be apples to oranges. To investigate, I wrote uadiff.pl.
uadiff.pl loads a URL twice, once using a Netscape user agent and once using a Microsoft user agent. To a web server, it looks exactly like an end-user using Netscape or IE. uadiff.pl then compares the results. This allows you to quickly assess whether a site is serving up different content depending on the browser being used.
You can enter one or more URLs on the command line, or pipe a long list of them into STDIN. At Netscape we used a list of "Top 100" web sites to check against.
Note: Since most top web sites include ad banners which change frequently, don't assume that just because the results for a given page are different, that the site is differentiating between browsers. Most of the time, but not always, the discrepancy is caused simply by differences in the two ad banners being returned. To check for this, use the -v option.
Feel free to use, copy, or modify this script. I use it on Linux, but can't vouch for it on any "lesser" operating
systems. Please send bug reports, modifications, or feature requests to bob at lisbonne.com.
Sample output
<![CDATA[
[bob@speedy]$ ./uadiff.pl -b www.lisbonne.com www.aol.com www.yahoo.com
Run started at Thu Oct 7 09:59:51 1999
http://www.lisbonne.com Moz 8371 bytes IE 8371 bytes same
http://www.aol.com Moz 38150 bytes IE 38150 bytes same
http://www.yahoo.com Moz 10509 bytes IE 10503 bytes DIFFERENT
Run ended at Thu Oct 7 10:00:11 1999
]]>
uadiff.pl
<![CDATA[
#!/usr/bin/perl -w
#------------------------------------------------------------------------------------
#
# uadiff.pl
#
# User Agent Differences. One or more URLs are loaded via http. Each URL is loaded once
# using a Mozilla 5 user agent, and once using the IE 5 user agent. The results are
# compared and a brief report is output.
#
# For usage run: uadiff.pl --help
#
# by Bob Lisbonne, 1999
#
# revision history:
# 8/28/99 v1.0
#
#------------------------------------------------------------------------------------
# Copyright (C) 1999, Bob Lisbonne
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use URI::Heuristic;
use Getopt::Std;
use vars qw($opt_v $opt_p $opt_h $opt_l $opt_b);
$moz5 = "Mozilla/5.0 [en] (Win95; I)";
$ie5 = "Mozilla/4.0 (compatible; MSIE 5.0; Windows 98)";
#$nav2 = "Mozilla/2.0 (Macintosh; I; PPC)";
$| = 1; #unbuffers output
getopts('vphlb');
print_help() unless not $opt_h;
print "Run started at ",scalar(localtime),"\n";
#print "\tMoz user agent: \"$moz5\"\n";
#print "\tIE user agent: \"$ie5\"\n";
print "\n";
if ($opt_p) {
while (<>) {
chomp;
push(@urls_to_test,$_);
}
} else {
print_help() unless $ARGV[0];
while ($ARGV[0]) {
push(@urls_to_test,shift);
}
}
foreach $rawurl (@urls_to_test) {
my $url = URI::Heuristic::uf_uristr($rawurl);
print "\n" unless not $opt_v;
printf "%-40s",$url;
$page_moz = get_page($url,$moz5);
print "Moz ",length($page_moz)," bytes\t" unless not $opt_b;
$page_ie = get_page($url,$ie5);
print "IE ",length($page_ie)," bytes\t" unless not $opt_b;
if ($opt_l and length($page_moz) == length($page_ie)) {
print "same\n";
next;
} elsif ($page_moz eq $page_ie) {
print "same\n";
next;
} else {
print "DIFFERENT\n";
next unless $opt_v;
print "\tUnique lines in Moz:\n";
foreach $line (split /\n/,$page_moz) {
print "\t$line\n" unless (index($page_ie,$line) != -1);
}
print "\tUnique lines in IE:\n";
foreach $line (split /\n/,$page_ie) {
print "\t$line\n" unless (index($page_moz,$line) != -1);
}
}
}
print "\n";
print "Run ended at ",scalar(localtime),"\n";
exit 1;
#############################################################
sub get_page() {
my $url = shift;
my $ua_text = shift;
my $req;
my $resp;
my $ua;
$ua = LWP::UserAgent->new();
$ua->agent($ua_text);
$req = HTTP::Request->new(GET=>$url);
$resp = $ua->request($req);
return $resp->content();
}
sub print_help() {
print <<ENDOFHELP;
$0 [options] list-of-URLs
-v verbose mode. Prints diffs to show where user agent causes differences.
-l length mode. Considers two pages identical if the lengths are equivalent.
-p piped input mode. Look for list of URLs to come from STDIN (instead of command line).
-b bytes printed. Outputs bytes per page for each page for each user agent.
-h help. Prints these usage instructions.
ENDOFHELP
exit;
}
]]>