NAME

DESCRIPTION

Statistic reports are oriented to user and bandwidth control, this is not a pure cache statistics generator. SquidAnalyzer use flat files to store data and don't need any SQL, SQL Lite or Berkeley databases.

This analyzer is incremental so it should be run in a daily cron. Take care if you have rotate log enable to run it before rotation is done.

REQUIREMENT

INSTALLATION

    perl Makefile.PL
    make
    make install

        FreeBSD: Registering installation in the package database
        FreeBSD: Cannot determine short module description
        FreeBSD: Cannot determine module description

        perl Makefile.PL INSTALLDIRS=site
        make
        make install

        perl Makefile.PL \
                LOGFILE=/var/log/squid3/access.log \
                BINDIR=/usr/bin \
                CONFDIR=/etc \
                HTMLDIR=/var/www/squidreport \
                BASEURL=/squidreport \
                MANDIR=/usr/man/man3 \
                DOCDIR=/usr/share/doc/squidanalyzer

        # Make Perl and SendmailAnalyzer distrib files
        %{__perl} Makefile.PL \
            INSTALLDIRS=vendor \
            QUIET=1 \
            LOGFILE=/var/log/squid/access.log \
            BINDIR=%{_bindir} \
            CONFDIR=%{_sysconfdir} \
            BASEDIR=%{_localstatedir}/lib/%{uname} \
            HTMLDIR=%{webdir} \
            MANDIR=%{_mandir}/man3 \
            DOCDIR=%{_docdir}/%{uname}-%{version} \
            DESTDIR=%{buildroot} < /dev/null

        Alias /squidreport /var/www/squidanalyzer
        <Directory /var/www/squidanalyzer>
            Options -Indexes FollowSymLinks MultiViews
            AllowOverride None
            Order deny,allow
            Deny from all
            Allow from 127.0.0.1
        </Directory>

        # SquidAnalyzer log reporting daily
        0 2 * * * /usr/local/bin/squid-analyzer > /dev/null 2>&1

        /var/log/proxy/squid-access.log {
            daily
            compress
            rotate 730
            missingok
            nocreate
            sharedscripts
            postrotate
                test ! -e /var/run/squid.pid || /usr/sbin/squid -k rotate
                /usr/bin/squid-analyzer -d -l /var/log/proxy/squid-access.log.1
            endscript
        }

USAGE

Usage: squid-analyzer [ -c squidanalyzer.conf ] [logfile(s)]

    -c | --configfile filename : path to the SquidAnalyzer configuration file.
                                 By default: /etc/squidanalyzer/squidanalyzer.conf
    -b | --build_date date     : set the date to be rebuilt, format: yyyy-mm-dd
                                 or yyyy-mm or yyyy. Used with -r or --rebuild.
    -d | --debug               : show debug informations.
    -h | --help                : show this message and exit.
    -j | --jobs number         : number of jobs to run at same time. Default is 1,
                                 run as single process.
    -p | --preserve number     : used to set the statistic obsolescence in
                                 number of month. Older stats will be removed.
    -P | --pid_dir directory   : set directory where pid file will be stored.
                                 Default /tmp/
    -r | --rebuild             : use this option to rebuild all html and graphs
                                 output from all data files.
    -t, --timezone +/-HH       : set number of hours from GMT of the timezone.
                                 Use this to adjust date/time of SquidAnalyzer
                                 output when it is run on a different timezone
                                 than the squid server.
    -v | version               : show version and exit.
    --no-year-stat             : disable years statistics, reports will start
                                 from month level only.
    --no-week-stat             : disable weekly statistics.

Log files to parse can be given as command line arguments or as a comma separated list of file for the LogFile configuration directive. By default SquidAnalyer will use file: /var/log/squid/access.log

There is special options like --rebuild that force SquidAnalyzer to rebuild all HTML reports, useful after an new feature or a bug fix. If you want to limit the rebuild to a single day, a single month or year, you can use the --build_date option by specifying the date part to rebuild, format: yyyy-mm-dd, yyyy-mm or yyyy.

The --preserve option should be used if you want to rotate your statistics and data. The value is the number of months to keep, older reports and data will be removed from the filesystem. Useful to preserve space, for example:

        squid-analyzer -p 6 -c /etc/squidanalyzer/squidanalyzer.conf

will only preserve six month of statistics from the last run of squidanalyzer.

If you have a SquidGuard log you can add it to the list of file to be parsed, either in the LogFile configuration directive log list, either at command line:

        squid-analyzer /var/log/squid3/access.log /var/log/squid/SquidGuard.log

SquidAnalyzer will automatically detect the log format and report SquidGuard ACL's redirection to the Denied Urls report.

MULTIPROCESS

        squid-analyzer -j 8 -l /var/log/squid3/huge_access.log

Here SquidAnalyzer will use 8 cpus to parse the file and compute all statistics reports. It will also use much more memory at the same time.

LOGFORMAT

    logformat squid %ts.%03tu %6tr %>a %Ss/%03>Hs %<st %rm %ru %un %Sh/%<A %mt
    logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh
    logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh

The common and combined log format can have one more field to add mime-type report like with the native squid log format:

    logformat common %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st %Ss:%Sh %mt
    logformat combined %>a %ui %un [%tl] "%rm %ru HTTP/%rv" %>Hs %<st "%{Referer}>h" "%{User-Agent}>h" %Ss:%Sh %mt

Those are the default format used by squid, you can switch to any of the three log format by giving the name at end of the access_log directive:

        access_log /var/log/squid3/access.log squid

or

        access_log /var/log/squid3/access.log common

CONFIGURATION

Here follow the configuration directives used by Squid Analyzer.

Output output_directory

Where SquidAnalyzer should dump all HTML, data and images files. You should give a path that can be read by a Web browser.

WebUrl

The URL of the SquidAnalyzer javascript, HTML and images files. Default: /squidreport

CustomHeader

This directive allow you to replace the SquidAnalyze logo by your custom logo. The default value is defined as follow:

        <a href="$self->{WebUrl}">
        <img src="$self->{WebUrl}images/logo-squidanalyzer.png" title="SquidAnalyzer $VERSION" border="0">
        </a> SquidAnalyzer
    

Feel free to define your own header but take care to not break current design. For example:

        CustomHeader   <a href="http://my.isp.dom/"><img src="http://my.isp.dom/logo.png" title="My ISP link" border="0" width="100" height="110"></a> My ISP Company
                                                                                           126,1         Bas
    

LogFile squid_access_log_file

Set the path to the Squid log file. This can be a comma separated list of files to process several files at the same time. If the files comes from differents Squid servers, they will be merges in a single reports. You can also add to the list a SquidGuard log file, SquidAnalyzer will atomatically detect the format.

UseClientDNSName 0

If you want to use DNS name instead of client Ip address as username enable this directive. When you don't have authentication, the username is set to the client ip address, this allow you to use the DNS name instead. Note that you must have a working DNS resolution and that it can really slow down the generation of reports.

DNSLookupTimeout 0.0001

If you have enabled UseClientDNSName and have lot of ip addresses that do not resolve you may want to increase the DNS lookup timeout. By default SquidAnalyzer will stop to lookup a DNS name after 0.0001 second (100 ms).

NetworkAlias network-aliases_file

Set path to the file containing network alias name. Network are show as Ip addresses so if you want to display name instead create a file with this format:

    LOCATION_NAME IP_NETWORK_ADDRESS
    

Separator must be a tabulation.

You can use regex to match and group some network addresses. See network-aliases file for examples.

UserAlias user-aliases_file

Set path to the file containing user alias name. If you don't have auth_proxy enable users are seen as ip addresses. So if you want to show username or computer name instead, create a file with this format:

    FULL_USERNAME IP_ADDRESS
    

If you have auth_proxy enable but want to replace login name by full user name for example, create a file with this format:

    FULL_USERNAME LOGIN_NAME
    

Separator for both must be a tabulation.

You can use regex to match and group some user login or ip addresses. See user-aliases file for examples.

You can also replace default ip address by his DNS name by enabling directive 'UseClientDNSName'.

AnonymizeLogin 0

Set this to 1 if you want to anonymize all user login. The username will be replaced by an unique id that change at each squid-analyzer run. Default disable.

OrderNetwork bytes|hits|duration

OrderUser bytes|hits|duration

OrderUrl bytes|hits|duration

Used to set how SquidAnalyzer sort Network, User and User detailed Urls reports screen. Value can be: bytes, hits or duration. Default is bytes. Note that OrderUrl is limited to User detailed Urls reports and does not apply to Top Url and Top domain report where there is three reports each already ordered.

OrderMime bytes|hits

Used to set how SquidAnalyzer sort Mime types report screen Value can be: bytes or hits. Default is bytes.

UrlReport 0|1

Should SquidAnalyzer display user url details. This will show all URL read by user. Take care to have enougth space disk for large user. Default is 0, no url detail report.

UserReport 0|1

Should SquidAnalyzer display user details. This will show statistics about user. Default is 1, show user detail report. Disable it to be able to remove any user related reports, statistics about URL and domains will remain.

UrlHitsOnly 0|1

Enable this directive if you don't want the tree Top URL and Domain tables. You will just have the table of Url/Domain ordered per hits then you can still sort the URL/Domain order by clicking on each column. This is useful when you have set a high value to TopNumber.

QuietMode 0|1

Run in quiet mode for batch processing or print debug information. Default is 0, verbose mode.

CostPrice price/Mb

Used to set a cost of the bandwith per Mb. If you want to generate invoice per Mb for bandwith traffic this can help you. Value 0 mean no cost, this is the default value, the "Cost" column is not displayed

Currency currency_abreviation

Used to set the currency of the bandwith cost. Preferably the html special character. Default is €

TopNumber number

Used to set the number of top url and second level domain to show. Default is top 100.

TopUrlUser Use this directive to show the top N users that look at an URL or a domain. Set it to 0 to disable this feature. Default is top 10.

Exclude exclusion_file

Used to set client ip addresses, network addresses, auth login or uri to exclude from report.

You can define one by line exclusion by specifying first the type of the exclusion (USER, CLIENT or URI) and a space separated list of valid regex.

You can also use the NETWORK type to define network address with netmask using the CIDR notation: xxx.xxx.xxx.xxx/n

See example bellow:

        NETWORK        192.168.1.0/24 10.10.0.0/16
        CLIENT         192\.168\.1\.2 
        CLIENT         10\.169\.1\.\d+ 192\.168\.10\..*
        USER           myloginstr
        USER           guestlogin\d+ guestdemo
        URI            http:\/\/myinternetdomain.dom.*
        URI            .*\.webmail\.com\/.*\/login\.php.*
    

you can have multiple line of the same exclusion type.

Include inclusion_file

Used to set client ip addresses, network addresses or auth login to include into the report. All others will not be included. It works as the opposite of the Include parameter.

You can define one by line inclusion by specifying first the type of the inclusion (USER or CLIENT) and a space separated list of valid regex.

You can also use the NETWORK type to define network address with netmask using the CIDR notation: xxx.xxx.xxx.xxx/n

See example bellow:

        NETWORK        192.168.1.0/24 10.10.0.0/16
        CLIENT         192\.168\.1\.2 
        CLIENT         10\.169\.1\.\d+ 192\.168\.10\..*
        USER           myloginstr
        USER           guestlogin\d+ guestdemo
        URI            http:\/\/myinternetdomain.dom.*
        URI            .*\.webmail\.com\/.*\/login\.php.*
    

you can have multiple line of the same inclusion type.

ExcludedMethods

This directive allow exclusion of some unwanted methods in report statistics like HEAD, POST, CONNECT, etc. Can be a comma separated list of methods.

ExcludedMimes

This directive allow exclusion of some unwanted mimetypes in report statistics like text/html, text/plain, or more generally text/*, etc. Can be a comma separated list of perl regular expression. Ex:

        ExcludedMimes   text/.*,image/.*
    

Lang

Used to set the translation file to be used. Value must be set to a file containing all string translated. See the lang directory for translation files. Default is defined internally in English.

ExcludedCodes

This directive allow exclusion of some unwanted codes in report statistics like TCP_DENIED/403 which are generated when a user accesses a page the first time without authentication. Can be a comma separated list of methods. Default is none, all codes will be parsed.

DateFormat

Date format used to display date (year = %y, month = %m and day = %d) You can also use %M to replace month by its 3 letters abbreviation. Default: %y-%m-%d

SiblingHit

Adds peer cache hit (CD_SIBLING_HIT) to be taken has local cache hit. Enabled by default, you must disabled it if you don't want to report peer cache hit onto your stats.

TransfertUnit

Allow to change the default unit used to display transfert size. Default is BYTES, other possible values are KB, MB and GB.

MinPie

Minimum percentage of data in pie's graphs to not be placed in the others item. Lower values will be summarized into the others item.

Locale

Set this to your locale to display generated date in your language. Default is to use the current locale of the system. If you want date in German for example, set it to de_DE.

        Rapport genere le mardi 11 decembre 2012, 15:13:09 (UTC+0100).
    

with a Locale set to fr_FR.

MaxFormatError

When SquidAnalyzer find a corrupted line in his data file, it exit immedialtly. You can force him to wait for a certain amount of errors before exiting. Of course you might want to remove the corrupted line before the next run. This can be useful if you have special characters in some fields like mime type.

TimeZone

Set timezone to use when SquidAnalyzer is used in a different server than the one running squid and there is a different timezone between these two machines. The value must follow format: +/-HH. Default is to use local time. For example:

        TimeZone        +01
    

for a log file generated on zone Europe/Paris with UTC+0100 and parsed on a computer with different timezone.

SUPPORT

AUTHOR

COPYRIGHT

This package is free software and published under the GPL v3 or above license.