IIS Log Files

April 2007

Log files from web servers usually contain a bunch of information, and in fact most statistics packages for log files, expect them to containquite a few more fields that are always collected.

This was the case for a site I recently encountered. For one reason or another, the site is setup to save only the littlest of information about requests: time, client's IP, request method, the URI, and the HTTP status code. Because of this the software that I would normally run to make a report choked, and there happens to be no way to change what fields the software is looking for. Many freeware/shareware tools suffered the same hiccup and there was no time to get the popular free (as in freedom) alternatives, Analog, or AWStats, installed, configured and working correctly.

Luckily, the file format was so simple, it was trivial to create a shell script that counted requests to a given URI and output them sorted. The files were named exYYMMDD.log and contained this:

#Fields: time c-ip cs-method cs-uri-stem sc-status 08:47:08 GET /robots.txt 404

Stringing together a combination of a loop, echo, cat, sed, awk, [e]grep, sort, and uniq is all it takes. I've used similar pipelines to list "live referrers," but I never considered it to be a log analyzer. Of course, I then needed to show how big of a dork I am by making it as concise as possible.

for f in *.log;do echo $f|sed -e 's/ex\([0-9][0-9]\)\([0-9][0-9]\)\([0-9][0-9]\).log/\2-\3-20\1/';echo;cat $f|egrep "cfm|/[[:space:]]+"|awk '{print $4;}'|sort|uniq -c|sort;echo;done

178 bytes. The output is decent, and you can filter out files to search for. Here I only looked for cfm (ColdFusion) files and "/". The report looks like this:

04-04-2007 1 /index.cfm/fuseaction/events.viewall 3 /index.cfm 6 /