NAME

SYNOPSIS

  rwbagcat [ --network-structure[=STRUCTURE] | --bin-ips[=SCALE]
             | --sort-counters[=ORDER]]
        [--print-statistics[=OUTFILE]]
        [--minkey=VALUE] [--maxkey=VALUE] [--mask-set=PATH]
        [--mincounter=VALUE] [--maxcounter=VALUE] [--zero-counts]
        [{ --pmap-file=PATH | --pmap-file=MAPNAME:PATH }]
        [--key-format=FORMAT] [--integer-keys] [--zero-pad-ips]
        [--no-columns] [--column-separator=C]
        [--no-final-delimiter] [{--delimited | --delimited=C}]
        [--output-path=PATH] [--pager=PAGER_PROG]
        [--site-config-file=FILENAME]
        [BAGFILE [BAGFILE...]]

  rwbagcat --help

  rwbagcat --version

DESCRIPTION

As of SiLK 3.12.0, rwbagcat uses information in the Bag file's header to determine how to display the key column.

• A key that is an IP address is printed in the canonical format. Specifically, IPs are printed in the IPv4 canonical format if the Bag contains only IPv4 addresses; otherwise, in the IPv6 canonical format (with IPv4 mapped into the ::ffff:0:0/96 netblock). May be modified by --key-format.
• A key that is a time is printed as a human-readable timestamp. May be modified by --key-format.
• A sensor key prints the name of the sensor. The "decimal" and "hexadecimal" arguments to --key-format may be used.
• A key holding TCP Flags is printed using the characters "F,S,R,P,A,U,E,C". The "decimal" and "hexadecimal" arguments to --key-format may be used.
• A key holding SiLK attributes is printed using the characters "T,C,F,S". The "decimal" and "hexadecimal" arguments to --key-format may be used.
• A country code key uses the abbreviations defined by ISO 3166-1 (see for example <https://www.iso.org/iso-3166-country-codes.html> or <https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2>) or the following special codes: -- N/A (e.g. private and experimental reserved addresses); a1 anonymous proxy; a2 satellite provider; o1 other.
• A key holding a value from prefix map requires that the --pmap-file switch be specified to display the value.

In addition, rwbagcat exits with an error when asked to use an IP format to display keys that are not IP addresses.

rwbagcat reads the BAGFILEs specified on the command line; if no BAGFILE arguments are given, rwbagcat attempts to read the Bag from the standard input. BAGFILE may be the keyword "stdin" or a hyphen ("-") to allow rwbagcat to print data from both files and piped input. If any input does not contain a Bag, rwbagcat prints an error to the standard error and exits abnormally.

When multiple BAGFILEs are specified on the command line, each is handled individually. To process the files as a single Bag, use rwbagtool(1) to combine the bags and pipe the output of rwbagtool into rwbagcat.

OPTIONS

--network-structure

--network-structure=STRUCTURE

For each numeric value in STRUCTURE, group the IPs in the Bag into a netblock of that size and print the number of hosts, the sum of the counters, and, optionally, print the number of smaller, occupied netblocks that each larger netblock contains. When STRUCTURE begins with "v6:", the IPs in the Bag are treated as IPv6 addresses, and any IPv4 addresses are mapped into the ::ffff:0:0/96 netblock. Otherwise, the IPs are treated as IPv4 addresses, and any IPv6 address outside the ::ffff:0:0/96 netblock is ignored. Aside from the initial "v6:" (or "v4:", for consistency), STRUCTURE has one of following forms:

--bin-ips

--bin-ips=SCALE

Invert the bag and count the total number of unique keys for a given value of the volume bin. For example, turn a Bag {sip:flow} into {flow:count(sip)}. SCALE is a string containing the value "linear", "binary", or "decimal".

--sort-counters

--sort-counters=ORDER

Sort the output so the counters are presented in either decreasing or increasing order. Typically the output is sorted by the keys. If the ORDER argument is not given to the switch, the counters are printed in decreasing order. Valid values for ORDER are

--print-statistics

--print-statistics=OUTFILE

Print a breakdown of the network hosts seen, and print general statistics about the keys and counters. When --print-statistics is specified, no other output is produced unless one of --sort-counters, --network-structure, or --bin-ips is also specified. When the OUTFILE argument is not given, the statistics are written to the standard output or to the pager if output is to a terminal. OUTFILE is a filename, named pipe, the keyword "stderr" to write to the standard error, or the keyword "stdout" or "-" to write to the standard output. If OUTFILE names an existing file, rwbagcat exits with an error unless the SILK_CLOBBER environment variable is set, in which case OUTFILE is overwritten. The output statistics produced by this switch are:

--minkey=VALUE

Output records whose key value is at least VALUE. VALUE may be an IP address or an integer in the range 0 to 4294967295 inclusive. The default is to print all records with a non-zero counter.

--maxkey=VALUE

Output records whose key value is not more than VALUE. VALUE may be an IP address or an integer in the range 0 to 4294967295 inclusive. The default is to print all records with a non-zero counter.

--mask-set=PATH

Output records whose key appears in the binary IPset read from the file PATH. (To build an IPset, use rwset(1) or rwsetbuild(1).) When used with --minkey and/or --maxkey, output records whose key is in the IPset and is also within when the specified range. As of SiLK 3.12.0, rwbagcat exits with an error if the --mask-set switch is used on a Bag file whose key-type is neither "custom" nor an IP address type.

--mincounter=VALUE

Output records whose counter value is at least VALUE. VALUE is an integer in the range 1 to 18446744073709551615. The default is to print all records with a non-zero counter; use --zero-counts to show records whose counter is 0.

--maxcounter=VALUE

Output records whose counter value is not more than VALUE. VALUE is an integer in the range 1 to 18446744073709551615, with the default being the maximum counter value.

--zero-counts

Print keys whose counter is zero. Normally, keys with a counter of zero are suppressed since all keys have a default counter of zero. In order to use this flag, either --mask-set or both --minkey and --maxkey must be specified. When this switch is specified, any counter limit explicitly set by the --maxcounter switch is also applied.

--pmap-file=PATH

--pmap-file=MAPNAME:PATH

Use the prefix map file located at PATH to map the key to a string when the type of the Bag's key is one of "sip-pmap", "dip-pmap", "any-ip-pmap", "sport-pmap", "dport-pmap", or "any-port-pmap". This switch is required for Bag files whose key was derived from a prefix map file. The type of the prefix map file must match the key's type, but a different prefix map file may be used. Specify PATH as "-" or "stdin" to read from the standard input. A map-name may be included in the argument to the switch, but rwbagcat currently does not use the map-name. To create a prefix map file, use rwpmapbuild(1). Since SiLK 3.12.0.

--key-format=FORMAT

Specify the format to use when printing a key, where FORMAT is a comma-separated list of the arguments described below. When this switch is not specified, rwbagcat uses the key's type to determine how to format the key, and a key whose type is unknown or "custom" is assumed to be an IP address. rwbagcat exits with an error if the specified format is incompatible with the key's type (for example, attempting to format a timestamp as an IP address).

--integer-keys

This switch is equivalent to --key-format=decimal, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--zero-pad-ips

This switch is equivalent to --key-format=zero-padded, it is deprecated as of SiLK 3.7.0, and it will be removed in the SiLK 4.0 release.

--no-columns

Disable fixed-width columnar output.

--column-separator=C

Use specified character between columns and after the final column. When this switch is not specified, the default of '|' is used.

--no-final-delimiter

Do not print the column separator after the final column. Normally a delimiter is printed. When the network summary is requested (--network-structure=S), the separator is always printed before the summary column and never after that column.

--delimited

--delimited=C

Run as if --no-columns --no-final-delimiter --column-sep=C had been specified. That is, disable fixed-width columnar output; if character C is provided, it is used as the delimiter between columns instead of the default '|'.

--output-path=PATH

Write the textual output of the --network-structure, --bin-ips, or --sort-counters switch to PATH, where PATH is a filename, a named pipe, the keyword "stderr" to write the output to the standard error, or the keyword "stdout" or "-" to write the output to the standard output (and bypass the paging program). If PATH names an existing file, rwbagcat exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this option is not given, the output is either sent to the pager or written to the standard output.

--pager=PAGER_PROG

When output is to a terminal, invoke the program PAGER_PROG to view the output one screen full at a time. This switch overrides the SILK_PAGER environment variable, which in turn overrides the PAGER variable. If the --output-path switch is given or if the value of the pager is determined to be the empty string, no paging is performed and all output is written to the terminal.

--site-config-file=FILENAME

Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwbagcat searches for the site configuration file in the locations specified in the "FILES" section. Since SiLK 3.15.0.

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

 $ rwbagcat mybag.bag
      172.23.1.1|              5|
      172.23.1.2|            231|
      172.23.1.3|              9|
      172.23.1.4|             19|
   192.168.0.100|              1|
   192.168.0.101|              1|
   192.168.0.160|             15|
  192.168.20.161|              1|
  192.168.20.162|              5|
  192.168.20.163|              5|

 $ rwbagcat --network-structure=TABCHX mybag.bag
           172.23.1.1      |              5|
           172.23.1.2      |            231|
           172.23.1.3      |              9|
           172.23.1.4      |             19|
         172.23.1.0/27     |            264|
       172.23.1.0/24       |            264|
     172.23.0.0/16         |            264|
   172.0.0.0/8             |            264|
           192.168.0.100   |              1|
           192.168.0.101   |              1|
         192.168.0.96/27   |              2|
           192.168.0.160   |             15|
         192.168.0.160/27  |             15|
       192.168.0.0/24      |             17|
           192.168.20.161  |              1|
           192.168.20.162  |              5|
           192.168.20.163  |              5|
         192.168.20.160/27 |             11|
       192.168.20.0/24     |             11|
     192.168.0.0/16        |             28|
   192.0.0.0/8             |             28|
 TOTAL                     |            292|

 $ rwbagcat --network-structure=ACS mybag.bag
     172.23.1.0/24     |            264| 4 hosts in 1 /27
 172.0.0.0/8           |            264| 4 hosts in 1 /16, 1 /24, and 1 /27
     192.168.0.0/24    |             17| 3 hosts in 2 /27s
     192.168.20.0/24   |             11| 3 hosts in 1 /27
 192.0.0.0/8           |             28| 6 hosts in 1 /16, 2 /24s, and 3 /27s

 $ rwfileinfo --field=bag Bag2.bag
 Bag2.bag:
   bag          key: duration @ 4 octets; counter: custom @ 8 octets

 $ rwbagcat --key-format=utc Bag2.bag
 rwbagcat: Invalid key-format 'utc':
        Nonsensical for Bag containing duration keys

 $ rwbagcat --key-format=canonical Bag2.bag
 rwbagcat: Invalid key-format 'canonical':
        Nonsensical for Bag containing duration keys

 $ rwbagbuild --bag-input=/dev/null   \
   | rwbagtool --add Bag2.bag stdin   \
   | rwbagcat --key-format=utc
 1970/01/01T00:00:01|                   1|
 1970/01/01T00:00:04|                   2|
 1970/01/01T00:00:07|                  32|
 1970/01/01T00:00:08|                   2|

 $ rwbagbuild --bag-input=/dev/null   \
   | rwbagtool --add Bag2.bag -       \
   | rwbagcat --key-format=canonical
         0.0.0.1|                   1|
         0.0.0.4|                   2|
         0.0.0.7|                  32|
         0.0.0.8|                   2|

 $ rwbagcat Bag2.bag    \
   | rwbagbuild --bag-input=- --key-type=sipv4

 $ rwbagcat --bin-ips mybag.bag
               1|              3|
               5|              3|
               9|              1|
              15|              1|
              19|              1|
             231|              1|

 $ rwbagcat --bin-ips=binary mybag.bag
    2^0 to 2^1-1|              3|
    2^2 to 2^3-1|              3|
    2^3 to 2^4-1|              2|
    2^4 to 2^5-1|              1|
    2^7 to 2^8-1|              1|

 $ rwbagcat --sort-counter mybag.bag
      172.23.1.2|                 231|
      172.23.1.4|                  19|
   192.168.0.160|                  15|
      172.23.1.3|                   9|
      172.23.1.1|                   5|
  192.168.20.162|                   5|
  192.168.20.163|                   5|
   192.168.0.100|                   1|
   192.168.0.101|                   1|
  192.168.20.161|                   1|

 $ rwbagcat --sort-counter=increasing mybag.bag
   192.168.0.100|                   1|
   192.168.0.101|                   1|
  192.168.20.161|                   1|
      172.23.1.1|                   5|
  192.168.20.162|                   5|
  192.168.20.163|                   5|
      172.23.1.3|                   9|
   192.168.0.160|                  15|
      172.23.1.4|                  19|
      172.23.1.2|                 231|

 $ rwbagcat --delim=, mybag.bag | grep ,5
 172.23.1.1,5
 192.168.20.162,5
 192.168.20.163,5

 $ rwbagcat --delim=, --sort-counter=increasing mybag.bag | grep ,5
 172.23.1.1,5
 192.168.20.162,5
 192.168.20.163,5

 $ rwbagcat --delim=, --sort-counter=decreasing mybag.bag | grep ,5
 172.23.1.1,5
 192.168.20.162,5
 192.168.20.163,5

 $ rwbagcat service.bag
 rwbagcat: The --pmap-file switch is required for \
         Bags containing sport-pmap keys

 $ rwfileinfo --fields=bag service.bag
 service.bag:
   bag          key: sport-pmap @ 4 octets; counter: custom @ 8 octets

 $ rwbagcat --pmap-file=ip-map.pmap service.bag
 rwbagcat: Cannot use IPv4-address prefix map for \
        Bag containing sport-pmap keys

 $ rwbagcat --pmap-file=port-map.pmap service.bag
   TCP/SSH|                   1|
  TCP/SMTP|                 800|
  TCP/HTTP|                5642|

 $ echo 'label 1 none'                                      \
   | rwpmapbuild --mode=proto-port --input-path=-           \
        --output-path=tmp.pmap
 $ rwbagcat --pmap-file=tmp.pmap service.bag
   7|                   1|
   8|                 800|
   9|                5642|

 $ rwbagcat --print-statistics mybag.bag

 Statistics
     number of keys:  10
    sum of counters:  292
        minimum key:  172.23.1.1
        maximum key:  192.168.20.163
    minimum counter:  1
    maximum counter:  231
               mean:  29.2
           variance:  5064
 standard deviation:  71.16
               skew:  2.246
           kurtosis:  8.1
    nodes allocated:  0 (0 bytes)
    counter density:  inf%

ENVIRONMENT

SILK_CLOBBER

The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.

SILK_PAGER

When set to a non-empty string, rwbagcat automatically invokes this program to display its output a screen at a time. If set to an empty string, rwbagcat does not automatically page its output.

PAGER

When set and SILK_PAGER is not set, rwbagcat automatically invokes this program to display its output a screen at a time.

SILK_CONFIG_FILE

This environment variable is used as the value for the --site-config-file when that switch is not provided.

SILK_DATA_ROOTDIR

This environment variable specifies the root directory of data repository. As described in the "FILES" section, rwbagcat may use this environment variable when searching for the SiLK site configuration file.

SILK_PATH

This environment variable gives the root of the install tree. When searching for configuration files, rwbagcat may use this environment variable. See the "FILES" section for details.

TZ

When the argument to the --key-format switch includes "localtime" or when a SiLK installation is built to use the local timezone, the value of the TZ environment variable determines the timezone in which rwbagcat displays timestamps. (If both of those are false, the TZ environment variable is ignored.) If the TZ environment variable is not set, the machine's default timezone is used. Setting TZ to the empty string or 0 causes timestamps to be displayed in UTC. For system information on the TZ variable, see tzset(3) or environ(7). (To determine if SiLK was built with support for the local timezone, check the "Timezone support" value in the output of rwbagcat --version.)

FILES

${SILK_CONFIG_FILE}

${SILK_DATA_ROOTDIR}/silk.conf

/data/silk.conf

${SILK_PATH}/share/silk/silk.conf

${SILK_PATH}/share/silk.conf

/usr/local/share/silk/silk.conf

/usr/local/share/silk.conf

Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided.

SEE ALSO