Recently, my friend Rob pointed me at a really cool program for doing statistical analysis on web logs. It’s called Funnel Web by Quest Software. After seeing the sample reports I was very impressed, but initially I had some problems setting it up to use the web logs from my hosting provider (1and1) Here’s how I fixed it.
Running Funnel Web on my 1and1 logs produced an incomplete report. It was not able to detect the log format correctly. It did the best it could, but did not produce any errors about detecting the format. After a little research, I came across this article on the 1and1 website that gives you the Apache log format string.
So, since they are using a custom format, it means you have to setup Funnel Web to recognize this format.
Looking through Quest’s website, you won’t find a direct link to the Funnel Web application. They are no longer supporting it. But, through the direct link above, you can still download it for free.
Since they have stopped supporting the product, this also means that they have taken down all of their support forums and articles regarding the product. However, I was able to retrieve this article out of Google’s cache. Do you ever feel like you’re dumpster diving when digging though their cache?
In the article, it spells out how to create a custom log format using the command line. The same parameters can also be saved in the Funnel Web settings file (fwp). They are shown at the end of this entry. Based on that article, I was able to come up with this format for my 1and1 logs:
# Custom log file: Format
usecustomformat
custom_seperator 32 #0x00000020 ( )
custom_dateformat 8
custom_dataIndex 0 4
custom_dataIndex 2 6
custom_dataIndex 3 1
custom_dataIndex 6 10
custom_dataIndex 10 9
custom_dataIndex 12 8
custom_dataIndex 5 5
custom_dataIndex 7 7
To use this, just open up your settings file (fwp) with a text editor and search for “Custom log file” and you will find where to place this. There is one other important thing to note. Whenever you save your settings with the Funnel Web program, it will strip off the first two lines: “usecustomformat” and “custom_seperator” breaking your format. To get around this, just make all your setting changes with Funnel Web first. Then open the settings file with your text editor and paste in the custom format changes. Finally, use Funnel Web to open your settings file and you’re all set. Just don’t use the program to make changes or you’ll need to repeat the process.
-usecustomformat | This command turns on custom settings format for processing log files. | ||||||||||||||||||||||||||||||||||
-custom_seperator X</td | Used to define the separator between fields used in the log file. Separator values (X) must be entered in ASCII format. See the table below for a translation of common separators into ASCII format:
|
||||||||||||||||||||||||||||||||||
-custom_dateformat X | This field is used to define the Data Format (X)used in the log file. For an explanation of the numbers associated with each date format, see the table below
|
||||||||||||||||||||||||||||||||||
-custom_timeformat X | This field is used to define the Time Format (X)used in the log file. For an explanation of the numbers associated with each time format, see the table below
|
||||||||||||||||||||||||||||||||||
-custom_dataIndex X Z | This command is used to define the positions of all the different fields in the log file, where X = the Log field type ansd Z = the position in the log file. In the table below is a translation of the number equivalent of the Log Field Types to be entered in the X position:
|
Cool!
Found this after many hours of head scratching… worked absolutely perfectly.
FunnelWeb rocks, it’s a shame Quest screwed it up 😦
Fantastic, this is the quickest way to parse 1and1 files correctly– I’ve been missing a lot of info by using config-kludged log analyzers against these files. Thanks! 🙂