NAME

rwpackchecker - Find unusual patterns that may indicate a corrupt file

SYNOPSIS

  rwpackchecker [--value=TEST=VALUE] [--allowable-count=TEST=ALLOWED]
        [--print-all]
        {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

  rwpackchecker --help

  rwpackchecker --version

DESCRIPTION

rwpackchecker reads SiLK Flow records and checks for "unusual" patterns that may indicate the file has been corrupted.

rwpackchecker has a default series of tests it runs on every flow record in an input file. Each default test has two numbers associated with it: a value threshold and an allowed count threshold. A test compares a value on the flow record to the value threshold, and if the value violates the threshold, a counter for that test is incremented. In addition, if the flow record violates the value threshold for any test, a global counter is incremented to denote a suspect record.

Once rwpackchecker finishes processing a file, it determines whether the file appears to be valid. A file is considered valid if either

the global counter of suspect records is 0, or
no test has a counter that exceeds the test's allowed count threshold

If rwpackchecker determines that all files are valid, it does not print any output by default. If rwpackchecker does find an invalid file, it will print the name of the input file, the global number of suspect records it found, and information for those tests where the counter exceeds the allowed count threshold.

As an example, if there are 10 tests and the count threshold for each is 10, it is possible for the global suspect counter to be 90 and for rwpackchecker to consider the file valid.

To force rwpackchecker to print the results for all tests and for all input files, specify the --print-all switch.

Some of the tests that run by default include checking the number of packets, the bytes per second ratio, the bytes per packet ratio, and the bytes per packet ratio for a particular protocol (TCP, UDP, and ICMP).

The --value and --allowable-count switches modify the value threshold and allowed count threshold for a test, respectively. The argument to the switch is the test name and the threshold, separated by a equals sign ("="). Repeat the switches to set multiple thresholds. For example, to change the value thresholds for the max-bytes test to 20000 and for the max-packets test to 15000, specify the following:

 rwpackchecker --value=max-bytes=20000 --value max-packets=15000 ...

To get the most value from rwpackchecker, one should customize it for the particular site where it is being used, since the default value for a threshold may or may not be unusual for a particular installation. For example, a router that has Ethernet connections should have no more than 1500 bytes per packet, since that is the Ethernet MTU; however, the default value for that ratio is 16384. In some cases the default value is the largest value that a SiLK IPv4 hourly repository file can store, making it impossible for a record to violate the threshold.

rwpackchecker supports additional tests which are not run by default. Representative tests include checking whether an IP is (not) in an IPset or whether a port is (not) in an integer list. To run an additional test, specify the name of the test using the --value switch and provide an argument for the test.

To see all of the tests that rwpackchecker supports as well as the value threshold and allowed count threshold for each test, see the "OPTIONS" section below, or run rwpackchecker with the --help switch.

rwpackchecker reads SiLK Flow records from the files named on the command line or from the standard input when no file names are specified and --xargs is not present. To read the standard input in addition to the named files, use "-" or "stdin" as a file name. If an input file name ends in ".gz", the file is uncompressed as it is read. When the --xargs switch is provided, rwpackchecker reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--value=TEST=VALUE: Set the value of TEST to the specified VALUE; separate the test name from value by "=". The available TESTs are given below; the test name can be shortened to the shortest unique prefix. The form of VALUE depends on the type of TEST:

If TEST expects a minimum or maximum, VALUE should be a number.
If TEST expects a list of IPs, VALUE should the name of a file containing an IPset (see rwsetbuild(1)).
If TEST expects a list of numbers (for example, ports or protocols), VALUE should contain a comma separated list of integers and integer-ranges where a range is two integers separated by a hyphen ("-").

Repeat this switch for each value that you wish to set.

--allowable-count=TEST=ALLOWED: Allow the named TEST to be violated ALLOWED of times before treating it as "unusual". ALLOWED is an integer value. Separate the test name from the allowed count by "=". Repeat this switch for each allowable count you wish to set.
--print-all: Print the result of all tests for all input files. Normally only tests that are deemed "unusual" are printed.
--xargs
--xargs=FILENAME: Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwpackchecker opens each named file in turn and reads records from it as if the filenames had been listed on the command line.
--help: Print the available options and exit.
--version: Print the version number and information about how SiLK was configured, then exit the application.

The following tests are always run:

min-bpp-ratio=NUMBER: Byte-per-packet ratio is less than NUMBER. Default value: 1. Allowed count: 0.
max-bpp-ratio=NUMBER: Byte-per-packet ratio is greater than NUMBER. Default value: 16384. Allowed count: 0.
min-bps-ratio=NUMBER: Byte-per-second ratio is less than NUMBER. Default value: 0. Allowed count: 0.
max-bps-ratio=NUMBER: Byte-per-second ratio is greater than NUMBER. Default value: 4294967295. Allowed count: 0.
min-packets=NUMBER: Packet count is less than NUMBER. Default value: 1. Allowed count: 0.
max-packets=NUMBER: Packet count is greater than NUMBER. Default value: 67108864. Allowed count: 0.
min-bytes=NUMBER: Byte count is less than NUMBER. Default value: 1. Allowed count: 0.
max-bytes=NUMBER: Byte count is greater than NUMBER. Default value: 4294967295. Allowed count: 0.
min-tcp-bpp-ratio=NUMBER: TCP byte-per-packet ratio is less than NUMBER. Default value: 1. Allowed count: 0.
max-tcp-bpp-ratio=NUMBER: TCP byte-per-packet ratio is greater than NUMBER. Default value: 16384. Allowed count: 0.
min-udp-bpp-ratio=NUMBER: UDP byte-per-packet ratio is less than NUMBER. Default value: 1. Allowed count: 0.
max-udp-bpp-ratio=NUMBER: UDP byte-per-packet ratio is greater than NUMBER. Default value: 16384. Allowed count: 0.
min-icmp-bpp-ratio=NUMBER: ICMP byte-per-packet ratio is less than NUMBER. Default value: 1. Allowed count: 0.
max-icmp-bpp-ratio=NUMBER: ICMP byte-per-packet ratio is greater than NUMBER. Default value: 16384. Allowed count: 0.

The following tests are only run when the --value switch is used to specify a value for the test.

match-protocol=LIST: Protocol is present in LIST. No default. Allowed count: 0.
nomatch-protocol=LIST: Protocol is not present in LIST. No default. Allowed count: 0.
match-flags=LIST: TCP Flag Combination is present in LIST. No default. Allowed count: 0.
nomatch-flags=LIST: TCP Flag Combination is not present in LIST. No default. Allowed count: 0.
match-sip=IPSET_FILE: Source IP is present in IPSET_FILE. No default. Allowed count: 0.
nomatch-sip=IPSET_FILE: Source IP is not present in IPSET_FILE. No default. Allowed count: 0.
match-dip=IPSET_FILE: Destination IP is present in IPSET_FILE. No default. Allowed count: 0.
nomatch-dip=IPSET_FILE: Destination IP is not present in IPSET_FILE. No default. Allowed count: 0.
match-sport=LIST: Source Port is present in LIST. No default. Allowed count: 0.
nomatch-sport=LIST: Source Port is not present in LIST. No default. Allowed count: 0.
match-dport=LIST: Destination Port is present in LIST. No default. Allowed count: 0.
nomatch-dport=LIST: Destination Port is not present in LIST. No default. Allowed count: 0.
match-nhip=IPSET_FILE: Next Hop IP is present in IPSET_FILE. No default. Allowed count: 0.
nomatch-nhip=IPSET_FILE: Next Hop IP is not present in IPSET_FILE. No default. Allowed count: 0.
match-input=LIST: SNMP Input is present in LIST. No default. Allowed count: 0.
nomatch-input=LIST: SNMP Input is not present in LIST. No default. Allowed count: 0.
match-output=LIST: SNMP Output is present in LIST. No default. Allowed count: 0.
nomatch-output=LIST: SNMP Output is not present in LIST. No default. Allowed count: 0.

EXAMPLES

In these examples, the dollar sign ("$") represents the shell prompt and a backslash ("\") is used to continue a line for better readability. The examples do not use the optional "=" between the --value switch and the switch's argument for better readability.

Given the SiLK Flow file data.rw where the number of flows with various byte and packet counts are described by this table:

 Number of flows    bytes <= 2000000    bytes > 2000000     TOTAL
 packets <= 500         379303                308          379611
 packets >  500         119586               2679          122265
 TOTAL                  498889               2987          501876

Running rwpackchecker:

 $ rwpackchecker --value max-bytes=2000000            \
        --value max-packets=500  data.rw
 data.rw:
              122573/501876 flows are bad or unusual
              122265 flows where Packet Count > 500
                2987 flows where Byte Count > 2000000

The counts for the individual tests are greater than the overall total since 2679 records fall into both categories.

To see the effect of the --allowable-count switch:

 $ rwpackchecker --value max-packets=500                              \
        --value max-bytes=2000000 --allowable max-bytes=3000 data.rw
 data.rw:
              122573/501876 flows are bad or unusual
              122265 flows where Packet Count > 500

 $ rwpackchecker --value max-bytes=2000000                            \
        --value max-packets=500 --allowable max-packets=150000 data.rw
 data.rw:
              122573/501876 flows are bad or unusual
                2987 flows where Byte Count > 2000000

In each case the total number of unusual flows did not change; the violation of the other limit is still noted, even when the test is not printed since the test's allowed count threshold was not reached.

When the allowed count thresholds for none of the tests are reached, rwpackchecker produces no output by default:

 $ rwpackchecker --value max-bytes=2000000 --allowable max-bytes=3000 \
        --value max-packets=500 --allowable max-packets=150000 data.rw
 $

Specify the --print-all switch to print the results:

 $ rwpackchecker --value max-bytes=2000000 --allowable max-bytes=3000 \
        --value max-packets=500 --allowable max-packets=15000         \
        --print-all data.rw
 data.rw:
              122573/501876 flows are bad or unusual
                   0 flows where BPP Calculation is incorrect
                   0 flows where Elapsed Time > 4096
                   0 flows where Byte/Packet Ratio < 1
                   0 flows where Byte/Packet Ratio > 16384
                   0 flows where Byte/Second Ratio < 0
                   0 flows where Byte/Second Ratio > 4294967295
                   0 flows where Packet Count < 1
              122265 flows where Packet Count > 500
                   0 flows where Byte Count < 1
                2987 flows where Byte Count > 2000000
                   0 flows where TCP Byte/Packet Ratio < 1
                   0 flows where TCP Byte/Packet Ratio > 16384
                   0 flows where UDP Byte/Packet Ratio < 1
                   0 flows where UDP Byte/Packet Ratio > 16384
                   0 flows where ICMP Byte/Packet Ratio < 1
                   0 flows where ICMP Byte/Packet Ratio > 16384

NAME

SYNOPSIS

DESCRIPTION

OPTIONS

EXAMPLES

SEE ALSO