Funnel Web Analyzer Custom Log Format

Recently, my friend Rob pointed me at a really cool program for doing statistical analysis on web logs. It’s called Funnel Web by Quest Software. After seeing the sample reports I was very impressed, but initially I had some problems setting it up to use the web logs from my hosting provider (1and1) Here’s how I fixed it.

Running Funnel Web on my 1and1 logs produced an incomplete report. It was not able to detect the log format correctly. It did the best it could, but did not produce any errors about detecting the format. After a little research, I came across this article on the 1and1 website that gives you the Apache log format string.

So, since they are using a custom format, it means you have to setup Funnel Web to recognize this format.

Looking through Quest’s website, you won’t find a direct link to the Funnel Web application. They are no longer supporting it. But, through the direct link above, you can still download it for free.

Since they have stopped supporting the product, this also means that they have taken down all of their support forums and articles regarding the product. However, I was able to retrieve this article out of Google’s cache. Do you ever feel like you’re dumpster diving when digging though their cache?

In the article, it spells out how to create a custom log format using the command line. The same parameters can also be saved in the Funnel Web settings file (fwp). They are shown at the end of this entry. Based on that article, I was able to come up with this format for my 1and1 logs:

# Custom log file: Format
usecustomformat
custom_seperator 32 #0x00000020 ( )
custom_dateformat 8
custom_dataIndex 0 4
custom_dataIndex 2 6
custom_dataIndex 3 1
custom_dataIndex 6 10
custom_dataIndex 10 9
custom_dataIndex 12 8
custom_dataIndex 5 5
custom_dataIndex 7 7

To use this, just open up your settings file (fwp) with a text editor and search for “Custom log file” and you will find where to place this. There is one other important thing to note. Whenever you save your settings with the Funnel Web program, it will strip off the first two lines: “usecustomformat” and “custom_seperator” breaking your format. To get around this, just make all your setting changes with Funnel Web first. Then open the settings file with your text editor and paste in the custom format changes. Finally, use Funnel Web to open your settings file and you’re all set. Just don’t use the program to make changes or you’ll need to repeat the process.

-usecustomformat This command turns on custom settings format for processing log files.
-custom_seperator X</td

Used to define the separator between fields used in the log file. Separator values (X) must be entered in ASCII format. See the table below for a translation of common separators into ASCII format:

Field Separator ASCII Equivalent
comma 44
tab 9
space 32
-custom_dateformat X This field is used to define the Data Format (X)used in the log file. For an explanation of the numbers associated with each date format, see the table below

Date Format Numeric Value
Fri 07 Mar 2002 1
DD/MMM/YYYY 2
MM/DD/YY 3
DD/MM/YY 4
YYYY-MM-DD 5
Thu Mar 1 10:32:18 2002 6
Unix Time 7
[06/May/2000:14:59:29 -400] 8
-custom_timeformat X This field is used to define the Time Format (X)used in the log file. For an explanation of the numbers associated with each time format, see the table below

Time Format Numeric Value
HH:MM:SS 1
1:40 PM 2
Unix Time 3
-custom_dataIndex X Z This command is used to define the positions of all the different fields in the log file, where X = the Log field type ansd Z = the position in the log file. In the table below is a translation of the number equivalent of the Log Field Types to be entered in the X position:

Log Field Type Numeric Value
Date 0
Time 1
Status 2
Visitor 3
Source 4
File 5
Agent 6
Bytes 7
BytesIn 8
Duration 9
Referral 10
UserName 11
VirtualHost 12
Method 13
Protocol 14
Port 15
Advertisements
This entry was posted in Website. Bookmark the permalink.

2 Responses to Funnel Web Analyzer Custom Log Format

  1. Adam says:

    Cool!

    Found this after many hours of head scratching… worked absolutely perfectly.

    FunnelWeb rocks, it’s a shame Quest screwed it up 😦

  2. Lisa says:

    Fantastic, this is the quickest way to parse 1and1 files correctly– I’ve been missing a lot of info by using config-kludged log analyzers against these files. Thanks! πŸ™‚

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s