NAME

AWFFull - A Webalizer Fork, Full o' features

SYNOPSIS

awffull
[...] [log-file]

DESCRIPTION

AWFFull is a web server log analysis program based on The Webalizer. AWFFull produces usage statistics in HTML format for viewing with a browser. The results are presented in both columnar and graphical format, which facilitates interpretation. Yearly, monthly, daily and hourly usage statistics are presented, along with the ability to display usage by site, URL, referrer, user agent (browser), user name, search strings, entry/exit pages, and country (some information may not be available if not present in the log file being processed).

AWFFull supports the following log formats shown in the following variable list:

CLF: (common log format) log files
Combined: log formats as defined by NCSA and others, and variations of these which it attempts to handle intelligently
xferlog: wu-ftpd formatted log files allowing analysis of ftp servers, and squid proxy logs.

Note

Logs may also be compressed, via gzip. If a compressed log file is detected, it will be automatically uncompressed while it is read. Compressed logs must have the standard gzip extension of .gz.

This documentation applies to AWFFull Version 3.8.2

CHANGES FROM WEBALIZER

AWFFull is based on The Webalizer code and has a number of large and small changes. These include:

•: Beyond the raw statistics: Making use of published formulae to provide additional insights into site usage
•: GeoIP IP Address look-ups for more accurate country detection
•: Resizable graphs
•: Integration with GNU gettext allowing for ease of translations. Currently 32 languages are supported.
•: Display more than 12 months of the site history on the front page.
•: Additional page count tracking and sort by same.
•: Some minor visual tweaks, including Geolizer's use of Kb, Mb etc for Volumes
•: Additional Pie Charts for URL counts, Entry and Exit Pages, and Sites
•: Horizontal lines on graphs that are more sensible and easier to read
•: User Agent and Referral tracking is now calculated via PAGES not HITS
•: GNU style long command line options are now supported (eg --help)
•: Can choose what is a page by excluding ‘what isn't’ vs the original ‘what is’ method
•: Requests to the site being analysed are displayed with the matching referring URL
•: A Table of 404 Errors, and the referring URL can be generated
•: An external CSS file can be used with the generated html
•: Manual performance optimisation of the config file is now easier with a post analysis summary output
•: Specified IP Addresses can be assigned to a given country
•: Additional Dump options for detailed analysis with other tools
•: Lotus Domino v6 logs are now detected and processed

Additional changes and improvements are planned and undergoing implementation. See the TODO file for details.

NEW REPORT MEASUREMENTS

With version 3.8.1 of AWFFull, several new measured results have been added to the detailed report monthly page.

Single Access: Single Access Pages - the only page seen within a given visit
Stickiness: How useful a given entry page is to draw Visitors deeper into your site
Popularity: The Ratio of Page Entries to Page Exits

These metrics can help towards improving insight in the usage of the processed web site. And hence allow the site owner to make positive change to make the site more useful to site visitors. All three metrics appear in the ‘Entry Pages’ Report. ‘Popularity’ is also on the ‘Exit Pages’ Report.

Single Access

More completely: ‘Single Access Pages’. This is a report on the number of times that a given page was the only page viewed within a Visit. Or in English, Someone came to your website. They only viewed one page. The number is the cumulative count of people who did this for that particular page. Why is this useful? Identifying those entry pages that don't draw visitors deeper into your site. Or seeing entry pages that shouldn't be entry pages. It's also a reality check against the next two values which are calculated from this number. The number generated should be a subset of the ‘Entry Page Views’ and/or ‘Exit Page Views’ metric. If it isn't? Let me know, we have a bug. :-)

Stickiness

Is calculated as 1 - (Single Access / Entry Page Views) expressed as a percentage. In essence Stickiness describes how useful a given entry page is to draw Visitors deeper into your site. The stickier the page, the more folk are caught by it. :-) The closer to 100% the better. Generally. Certain pages within YOUR website may not make sense to have a high stickiness or even > 5%. This measurement is a clue to understanding how your site is used, it is not a rule. How is this useful? How and where are people entering your web site. Does that make sense? Should it be here or there? What can you change to fix this and hence improve their use of your website.

Popularity

Popularity is the Ratio of Page Entries to Page Exits. o If it equals 1.0? Then the number of visitors to your site who started with that page, equals the number who left at that page. o If greater then 1.0, then more people entered here then left. o If less then 0? More people left from here then entered. I personally find this metric one of the more useful "At a Glance: How are Pages Performing" metrics. One of the difficulties with using this particular metric is that certain numbers will NOT make sense for YOUR site. In that a natural exit page would expect to have a very low Popularity. It's an exit page, not an entry page. So if an exit page has a high popularity, then you have a real problem. Likewise, a low Popularity for an entry page is unlikely to be a Good Thing(tm).

"Where & Why?" All three of these metrics are covered very nicely in Hack #58 from "Web Site Measurement Hacks" [1]. Which is where, credit where credit due, the inspiration to add these metrics came from.

RUNNING AWFFULL

AWFFull is designed to be run from a Unix command line prompt or as a crond(8) job. There is no need to run with super-user privleges, and indeed, is preferable NOT to.

Once executed, the general flow of the program is:

A default configuration file is scanned for, /usr/local/etc/awffull.conf and, if found, is used.

Any command line arguments given to the program are parsed. This may include the specification of one or more configuration files, which are processed at the time it is encountered.

It can be useful to have multiple config files. A master used for multiple sites, and individualised config files. Do be aware that last option set wins. So last config file, or if after a config file, command line options. Useful if you desire to send the output to an alternate directory.

If a log file was specified, it is opened and made ready for processing. If no log file was given, STDIN is used for input. If the log filename '-' is specified, STDIN will be forced.

If an output directory was specified, AWFFull changes to that directory in preparation for generating output. If no output directory was given, the current directory is used.

If no hostname was given, the program attempts to get the hostname using a uname(2) system call. If that fails, localhost is used.

A history file is searched for in the current directory (output directory) and read if found. This file keeps totals for previous months, which is used in the main index.html HTML document. Note: The file location can now be specified with the HistoryName configuration option.

If incremental processing was specified, a data file is searched for and loaded if found, containing the 'internal state' data of the program at the end of a previous run. Note: The file location can now be specified with the IncrementalName configuration option.

Main processing begins on the log file. If the log spans multiple months, a separate HTML document is created for each month. After main processing, the main index.html page is created, which has totals by month and links to each months HTML document.

A new history file is saved to disk, which includes totals generated by AWFFull during the current run.

If incremental processing was specified, a data file is written that contains the 'internal state' data at the end of this run.

INCREMENTAL PROCESSING

Version 1.2x of The Webalizer added incremental run capability. Simply put, this allows processing large log files by breaking them up into smaller pieces, and processing these pieces instead. What this means in real terms is that you can now rotate your log files as often as you want, and still be able to produce monthly usage statistics without the loss of any detail. Basically, AWFFull saves and restores all internal data in a file named awffull.current. This allows the program to 'start where it left off' so to speak, and allows the preservation of detail from one run to the next. The data file is placed in the current output directory, and is a plain ASCII text file that can be viewed with any standard text editor. It's location and name may be changed using the IncrementalName configuration keyword.

Some special precautions need to be taken when using the incremental run capability of AWFFull. Configuration options should not be changed between runs, as that could cause corruption of the internal data stored. For example, changing the MangleAgents level will cause different representations of user agents to be stored, producing invalid results in the user agents section of the report. If you need to change configuration options, do it at the end of the month after normal processing of the previous month and before processing the current month. You may also want to delete the awffull.current file as well.

AWFFull also attempts to prevent data duplication by keeping track of the timestamp of the last record processed. This timestamp is then compared to current records being processed, and any records that were logged previous to that timestamp are ignored. This, in theory, should allow you to re-process logs that have already been processed, or process logs that contain a mix of processed/not yet processed records, and not produce duplication of statistics.

The only time this may break is if you have duplicate timestamps in two separate log files. Any records in the second log file that do have the same timestamp as the last record in the previous log file processed, will be discarded as if they had already been processed. There are lots of ways to prevent this however, for example, stopping the web server before rotating logs will prevent this situation, or using a tool such as cronolog (⟨http://cronolog.org/⟩). This setup also necessitates that you always process logs in chronological order, otherwise data loss will occur as a result of the timestamp compare.

REVERSE DNS LOOKUPS

AWFFull no longer supports DNS lookups. Please use an external program such as DNShistory or DNSTran instead.

•: ⟨http://www.summary.net/soft/dnstran.html⟩
•: ⟨http://www.stedee.id.au/dnshistory⟩

With version 3.7.1 of AWFFull, GeoIP capability can be used for improved country detection.

COMMAND LINE OPTIONS

AWFFull supports many different configuration options that will alter the way the program behaves and generates output. Most of these can be specified on the command line, while some can only be specified in a configuration file. The command line options are listed below, with references to the corresponding configuration file keywords. See also awffull.conf(5).

General Options

-h, --help

Display all available command line options and exit program

-V, --version

Display program version and exit program

-v --verbose

Verbosity Display debugging information for errors and warnings. Multiple v's will increase the amount of information displayed.

--match_counts

Display optimisation useful information pertaining to the number of matches against various Group, Hide and Ignore options.

-i --ignore_history

IgnoreHist Ignore history. USE WITH CAUTION. This will cause AWFFull to ignore any previous monthly history file only. Incremental data (if present) is still processed.

-p --preserve_state

Incremental Preserve internal data between runs.

-T --timing

TimeMe. Force display of timing information at end of processing.

-c --config=FILE

Use configuration file FILE

-n NAME

HostName. Use the hostname NAME.

-o --output=DIR

OutputDir. Use output directory DIR.

-t NAME

ReportTitle. Use NAME for report title.

F --logtype=TYPE

LogType. Specify log type to be processed. Value can be one of: auto, clf, combined, domino, ftp or squid format. If not specified, will default to auto format. FTP logs must be in standard wu-ftpd xferlog format. A value of ‘auto’ states that the log format automatically ascertained.

-f --fold

FoldSeqErr. Fold out of sequence log records back into analysis, by treating as if they were the same date/time as the last good record. Normally, out of sequence log records are simply ignored.

-Y

CountryGraph. Suppress country graph.

-G

HourlyGraph. Suppress hourly graph.

-x NAME

HTMLExtension. Defines the HTML file extension to use on the created report files. If not specified, defaults to html. Do not include the leading period.

-H

HourlyStats. Suppress hourly statistics.

-L

GraphLegend. Suppress color coded graph legends.

-l NUM

GraphLines. Use background lines. The number of lines and where to place are automatically calculated. For backwards compatibility, any number > 0 enables. Use zero ('0') to disable the lines.

-P NAME"

PageType. Specify file extensions that are considered pages. Sometimes referred to as pageviews.

-m NUM

VisitTimeout. Specify the Visit timeout period. Specified in number of seconds. Default is 1800 seconds (30 minutes). Sometimes referred to as sessions.

-I NAME

IndexAlias. Use the filename name as an additional alias for index.

-M NUM

MangleAgents. Mangle user agent names according to the mangle level specified by num.

Mangle levels are:

5 - Browser name and major version

4 - Browser name, major and minor version

3 - Browser name, major version, minor version to two decimal places

2 - Browser name, major and minor versions and sub-version

1 - Browser name, version and machine type if possible

0 - All information (left unchanged).

-g NUM

GroupDomains. Automatically group sites by domain. The grouping level specified by num can be thought of as 'the number of dots' to display in the grouping. The default value of 0 disables any domain grouping.

Hide Options

-a NAME: HideAgent. Hide user agents matching name.
-r NAME: HideReferrer. Hide referrer matching name.
-s NAME: HideSite. Hide site matching name.
-X NAME: HideAllSites. Hide all individual sites (only display groups).
-u NAME: HideURL. Hide URL matching name.

Table size options

-A --top_agents=NUM: TopAgents. Display the top num user agents table.
-R --top_refers=NUM: TopReferrers. Display the top num referrers table.
-S --top_sites=NUM: TopSites. Display the top num sites table.
-U --top_urls=NUM: TopURLs. Display the top num URL's table.
-C --top_countries=NUM: TopCountries. Display the top num countries table.
-e --top_entry=NUM: TopEntry. Display the top num entry pages table.
-E --top_exit=NUM: TopExit. Display the top num exit pages table.

Other Options

--use_geoip: Enables the use of the Maxmind GeoIP capability for more accurate detection of countries.
NOTE! Do not enable GeoIP if you analyse files that have had the IP Address translated to a Fully Qualified Host Name. Use either raw IP Addresses and GeoIP, or Names and disable GeoIP. ie. Don't use GeoIP AND DNShistory.
--match_counts: Display the various Group/Hide etc Match Counts. This option is ideal for optimisation of the awffull.conf file. Just be careful with optimising Agents in particular, as the order is typically important.

CONFIGURATION FILES

See the awffull.conf(5) man page for complete details of all configuration options.

Configuration files are standard ASCII(7) text files that may be created or edited using any standard editor.

Blank lines and lines that begin with a pound sign ('#') are ignored.

Any other lines are considered to be configuration lines, and have the form ‘Keyword Value’, where the ‘Keyword’ is one of the currently available configuration keywords (see awffull.conf(5)), and ‘Value’ is the value to assign to that particular option.

Any text found after the keyword up to the end of the line is considered the keyword's value, so you should not include anything after the actual value on the line that is not actually part of the value being assigned. The file sample.conf provided with the distribution contains lots of useful documentation and examples as well.

Certain "Keywords" will accept a 2nd value. In those situations, the first value may be enclosed in double quotes (") to allow for whitespace.

BUGS

None currently known. YMMV....

Report bugs to ⟨https://bugs.launchpad.net/awffull⟩, or use the email discussion list: <awffull@stedee.id.au>

NOTES

In case it is not obvious: AWFFull is a play/pun on the word ‘awful’, and is pronounced the same way. Yes it was deliberate.

REFERENCES

[1] Web Site Measurement Hacks. Eric T. Peterson (and others). O'Reilly. ISBN 0-596-00988-7.