NAME

rwsilk2ipfix - Convert SiLK Flow records to IPFIX records

SYNOPSIS

  rwsilk2ipfix [--ipfix-output=PATH] [--print-statistics]
        [--single-template] [--site-config-file=FILENAME]
        {[--xargs] | [--xargs=FILENAME] | [FILE [FILE ...]]}

  rwsilk2ipfix --help

  rwsilk2ipfix --version

DESCRIPTION

rwsilk2ipfix reads SiLK Flow records, converts the records to an IPFIX (Internet Protocol Flow Information eXport) format, and writes the IPFIX records to the path specified by --ipfix-output or to the standard output when the --ipfix-output switch is not provided and standard output is not the terminal.

rwsilk2ipfix reads SiLK Flow records from the files named on the command line or from the standard input when no file names are specified and --xargs is not present. To read the standard input in addition to the named files, use "-" or "stdin" as a file name. If an input file name ends in ".gz", the file is uncompressed as it is read. When the --xargs switch is provided, rwsilk2ipfix reads the names of the files to process from the named text file or from the standard input if no file name argument is provided to the switch. The input to --xargs must contain one file name per line.

The IPFIX records generated by rwsilk2ipfix will contain six information elements that are in the Private Enterprise space for CERT (the IPFIX Private Enterprise Number of CERT is 6871). These six information elements fall into two groups:

Elements 30 and 31 contain the packing information that was determined by rwflowpack(8), specifically the flowtype and the sensor. These values correspond to numbers specified in the silk.conf(5) file.
Elements 14, 15, 32, and 33 contain information elements generated by the yaf(1) flow meter (<http://tools.netsa.cert.org/yaf/>). The information elements may be present even if yaf was not used to generate the flow records, but their value will be empty or 0.

For each of the six information elements that rwsilk2ipfix will produce, the following table lists its numeric ID, its length in octets, its name, the field name it corresponds to on rwcut(1), and a brief description.

 30  1  silkFlowType     class & type   How rwflowpack categorized
                                        the flow record
 31  2  silkFlowSensor   sensor         Sensor where the flow was
                                        collected
 14  1  initialTCPFlags  initialFlags   TCP flags on first packet in
                                        the flow record
 15  1  unionTCPFlags    sessionFlags   TCP flags on all packets in
                                        the flow except the first
 32  1  silkTCPState     attributes     Flow continuation attributes
                                        set by generator
 33  2  silkAppLabel     application    Guess by flow generator as
                                        to the content of traffic

Templates

As of SiLK 3.12.0, rwsilk2ipfix uses ten different IPFIX templates for writing SiLK Flow records. The --single-template switch causes rwsilk2ipfix to revert to its previous behavior and use a single template for all records.

1.

Template ID 0x9DD0 (40400), for IPv4 records whose protocol is not ICMP, ICMPv6, UDP, SCTP, or TCP:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 39    sourceIPv4Address (8)             13    sIP
  40- 43    destinationIPv4Address (12)       14    dIP
  44- 47    ipNextHopIPv4Address (15)         15    nhIP

2.

Template ID 0x9DD1 (40401), for ICMP IPv4 records:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 37    paddingOctets (210)               13    -
  38- 39    icmpTypeCodeIPv4                  14    dPort
  40- 43    paddingOctets (210)               15    -
  44- 47    sourceIPv4Address (8)             16    sIP
  48- 51    destinationIPv4Address (12)       17    dIP
  52- 55    ipNextHopIPv4Address (15)         18    nhIP

3.

Template ID 0x9DD2 (40402), for IPv4 records whose protocol is UDP or SCTP:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 37    sourceTransportPort (7)           13    sPort
  38- 39    destinationTransportPort (11)     14    dPort
  40- 43    paddingOctets (210)               15    -
  44- 47    sourceIPv4Address (8)             16    sIP
  48- 51    destinationIPv4Address (12)       17    sIP
  52- 55    ipNextHopIPv4Address (15)         18    nhIP

4.

Template ID 0x9DD3 (40403), for IPv4 records whose protocol is TCP and that do not have the expanded TCP flags fields (initial flags and session flags):

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        tcpControlBits (6)                12    flags
  36- 37    sourceTransportPort (7)           13    sPort
  38- 39    destinationTransportPort (11)     14    dPort
  40- 43    paddingOctets (210)               15    -
  44- 47    sourceIPv4Address (8)             16    sIP
  48- 51    destinationIPv4Address (12)       17    dIP
  52- 55    ipNextHopIPv4Address (15)         18    nhIP

5.

Template ID 0x9DD4 (40404), for IPv4 records whose protocol is TCP and that have have the initial flags and session flags fields:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 37    sourceTransportPort (7)           13    sPort
  38- 39    destinationTransportPort (11)     14    dPort
  40        paddingOctets (210)               15    -
  41        tcpControlBits (6)                16    flags
  42        initialTCPFlags (6871, 14)        17    initialFlags
  43        unionTCPFlags (6871, 15)          18    sessionFlags
  44- 47    sourceIPv4Address (8)             19    sIP
  48- 51    destinationIPv4Address (12)       20    dIP
  52- 55    ipNextHopIPv4Address (15)         21    nhIP

6.

Template ID 0x9ED0 (40656), for IPv6 records whose protocol is not ICMP, ICMPv6, UDP, SCTP, or TCP:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 39    paddingOctets (210)               13    -
  40- 55    sourceIPv6Address (27)            14    sIP
  56- 71    destinationIPv6Address (28)       15    dIP
  72- 87    ipNextHopIPv6Address (62)         16    nhIP

7.

Template ID 0x9ED1 (40657), for ICMPv6 IPv6 records:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 37    paddingOctets (210)               13    -
  38- 39    icmpTypeCodeIPv4                  14    dPort
  40- 55    sourceIPv6Address (27)            15    sIP
  56- 71    destinationIPv6Address (28)       16    dIP
  72- 87    ipNextHopIPv6Address (62)         17    nhIP

8.

Template ID 0x9ED2 (40658), for IPv6 records whose protocol is UDP or SCTP:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 37    sourceTransportPort (7)           13    sPort
  38- 39    destinationTransportPort (11)     14    dPort
  40- 55    sourceIPv6Address (27)            15    sIP
  56- 71    destinationIPv6Address (28)       16    dIP
  72- 87    ipNextHopIPv6Address (62)         17    nhIP

9.

Template ID 0x9ED3 (40659), for IPv6 records whose protocol is TCP and that do not have the expanded TCP flags fields (initial flags and session flags):

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        tcpControlBits (6)                12    flags
  36- 37    sourceTransportPort (7)           13    sPort
  38- 39    destinationTransportPort (11)     14    dPort
  40- 55    sourceIPv6Address (27)            15    sIP
  56- 71    destinationIPv6Address (28)       16    dIP
  72- 87    ipNextHopIPv6Address (62)         17    nhIP

10.

Template ID 0x9ED4 (40660), for IPv6 records whose protocol is TCP and that have have the initial flags and session flags fields:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 19    packetDeltaCount (2)               3    packets
  20- 23    octetDeltaCount (1)                4    bytes
  24- 25    ingressInterface (10)              5    in
  26- 27    egressInterface (14)               6    out
  28- 29    silkAppLabel (6871, 33)            7    application
  30- 31    silkFlowSensor (6871, 31)          8    sensor
  32        silkFlowType (6871, 30)            9    class & type
  33        silkTCPState (6871, 32)           10    attributes
  34        protocolIdentifier (4)            11    protocol
  35        paddingOctets (210)               12    -
  36- 37    sourceTransportPort (7)           13    sPort
  38- 39    destinationTransportPort (11)     14    dPort
  40- 43    paddingOctets (210)               15    -
  44        paddingOctets (210)               16    -
  45        tcpControlBits (6)                17    flags
  46        initialTCPFlags (6871, 14)        18    initialFlags
  47        unionTCPFlags (6871, 15)          19    sessionFlags
  48- 63    sourceIPv6Address (27)            20    sIP
  64- 79    destinationIPv6Address (28)       21    dIP
  80- 95    ipNextHopIPv6Address (62)         22    nhIP

When the --single-template switch is provided, rwipfix2silk uses a single IPFIX template for all records. That template has ID 0xAFEA (45034) and contains the following information elements:

  OCTETS    INFORMATION ELEMENT (PEN, ID)    POS    SILK FIELD
 =======    =============================    ===    =============
   0-  7    flowStartMilliseconds (152)        1    sTime
   8- 15    flowEndMilliseconds (153)          2    sTime + duration
  16- 31    sourceIPv6Address (27)             3    sIP
  32- 47    destinationIPv6Address (28)        4    dIP
  48- 51    sourceIPv4Address (8)              5    sIP
  52- 55    destinationIPv4Address (12)        6    dIP
  56- 57    sourceTransportPort (7)            7    sPort
  58- 59    destinationTransportPort (11)      8    dPort
  60- 63    ipNextHopIPv4Address (15)          9    nhIP
  64- 79    ipNextHopIPv6Address (62)         10    nhIP
  80- 83    ingressInterface (10)             11    in
  84- 87    egressInterface (14)              12    out
  88- 95    packetDeltaCount (2)              13    packets
  96-103    octetDeltaCount (1)               14    bytes
 104        protocolIdentifier (4)            15    protocol
 105        silkFlowType (6871, 30)           16    class & type
 106-107    silkFlowSensor (6871, 31)         17    sensor
 108        tcpControlBits (6)                18    flags
 109        initialTCPFlags (6871, 14)        19    initialFlags
 110        unionTCPFlags (6871, 15)          20    sessionFlags
 111        silkTCPState (6871, 32)           21    attributes
 112-113    silkAppLabel (6871, 33)           22    application
 114-119    paddingOctets (210)               23    -

Note that the template contains both IPv4 and IPv6 addresses. One set of those addresses contains the IP addresses and the other set contains only zeros.

OPTIONS

Option names may be abbreviated if the abbreviation is unique or is an exact match for an option. A parameter to an option may be specified as --arg=param or --arg param, though the first form is required for options that take optional parameters.

--ipfix-output=PATH: Write the IPFIX records to PATH, where PATH is a filename, a named pipe, the keyword "stderr" to write the output to the standard error, or the keyword "stdout" or "-" to write the output to the standard output. If PATH names an existing file, rwsilk2ipfix exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is written to the standard output. Attempting to write the binary output to a terminal causes rwipfix2silk to exit with an error.
--print-statistics: Print, to the standard error, the number of records that were written to the IPFIX output file.
--single-template: Use a single IPFIX template for all records. Using this switch produces output identical to that produced by rwsilk2ipfix from SiLK 3.11.0 and earlier. Since SiLK 3.12.0.
--site-config-file=FILENAME: Read the SiLK site configuration from the named file FILENAME. When this switch is not provided, rwsilk2ipfix searches for the site configuration file in the locations specified in the "FILES" section.
--xargs
--xargs=FILENAME: Read the names of the input files from FILENAME or from the standard input if FILENAME is not provided. The input is expected to have one filename per line. rwsilk2ipfix opens each named file in turn and reads records from it as if the filenames had been listed on the command line.
--help: Print the available options and exit.
--version: Print the version number and information about how SiLK was configured, then exit the application.

EXAMPLES

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line.

To convert the SiLK file silk.rw into an IPFIX format and store the results in ipfix.dat:

 $ rwsilk2ipfix --ipfix-output=ipfix.dat silk.rw

To view the contents of ipfix.dat using the yafscii(1) tool (see <http://tools.netsa.cert.org/yaf/>):

 $ yafscii --in=ipfix.dat --out=-

To view the contents of ipfix.dat using the ipfixDump(1) tool (see <http://tools.netsa.cert.org/yaf/>):

 $ ipfixDump --yaf --in=ipfix.dat --out=-

Use the rwipfix2silk(1) tool to convert the IPFIX file back into SiLK Flow format:

 $ rwipfix2silk --silk-output=silk2.rw ipfix.dat

ENVIRONMENT

SILK_CLOBBER: The SiLK tools normally refuse to overwrite existing files. Setting SILK_CLOBBER to a non-empty value removes this restriction.
SILK_CONFIG_FILE: This environment variable is used as the value for the --site-config-file when that switch is not provided.
SILK_DATA_ROOTDIR: This environment variable specifies the root directory of data repository. As described in the "FILES" section, rwsilk2ipfix may use this environment variable when searching for the SiLK site configuration file.
SILK_PATH: This environment variable gives the root of the install tree. When searching for configuration files, rwsilk2ipfix may use this environment variable. See the "FILES" section for details.

FILES

${SILK_CONFIG_FILE}
${SILK_DATA_ROOTDIR}/silk.conf
/data/silk.conf
${SILK_PATH}/share/silk/silk.conf
${SILK_PATH}/share/silk.conf
/usr/local/share/silk/silk.conf
/usr/local/share/silk.conf: Possible locations for the SiLK site configuration file which are checked when the --site-config-file switch is not provided.