NAME

flowkey - SiLK plug-in providing YAF flow key filter and field

SYNOPSIS

  rwfilter --plugin=flowkey.so [--flow-key=VALUE_LIST]

  rwcut --plugin=flowkey.so --fields=FIELDS ...

  rwgroup --plugin=flowkey.so --fields=FIELDS ...

  rwsort --plugin=flowkey.so --fields=FIELDS ...

  rwstats --plugin=flowkey.so --fields=FIELDS --values=FIELDS ...

  rwuniq --plugin=flowkey.so --fields=FIELDS --values=FIELDS ...

DESCRIPTION

The YAF flow key hash is a numeric value that the yaf(1) IPFIX generator computes for every flow record. The flow key hash is computed from the IP protocol, the source and destination IP addresses, the source and destination ports, and the vlan identifier. The getFlowKeyHash(1) tool in YAF distribution reads IPFIX data and computes the flow key hash for each flow record.

The flowkey plug-in uses the same formula as YAF to compute the flow key hash for a record. The flow key hash may be printed by rwcut(1), may be used as part of the sorting key in rwsort(1), may be used as a grouping key in rwgroup(1), rwstats(1), and rwuniq(1), and may be used as a partitioning criterion in rwfilter(1).

Note that the flow key hash computed by this plug-in may be different than the value computed by YAF:

When SiLK processes a bi-directional IPFIX record (a bi-flow), it splits the record into two uni-directional records and reverses the source and destination fields when it stores the reverse record. The flow key hash for this reverse record is different than that of the forward record. The getFlowKeyHash tool has a --reverse switch to duplicate this behavior.
YAF computes the flow key hash using the vlan identifier, but SiLK ignores the vlan ID unless it is explicitly instructed to use it. When SiLK is told to the use the vlan ID, the vlan ID is stored in the the in field of the SiLK Flow record. That field normally holds the SNMP ingress value.
(Instructing SiLK to use the vlan ID depends on whether one is using rwipfix2silk(1), rwflowpack(8), or flowcap(8). For rwipfix2silk, run the tool with the --interface-values=vlan switch. For rwflowpack and flowcap, edit the sensor.conf(5) file and specify "interface-values vlan" in the probe block where the flow is collected.)
Even when SiLK has been told to store the vlan identifier in the field normally used for the ingress interface, rwflowpack typically does not store that field in the files it creates in the data repository. When reading these files, the in field is set to 0. To tell rwflowpack to store the field, run it with the command line switch --pack-interfaces. To tell getFlowKeyHash to ignore the value, specify the --snmp switch.

The flowkey plug-in must be explicitly loaded into an application via the --plugin switch.

OPTIONS

The flowkey plug-in provides the following options to the indicated applications.

rwfilter Switches

When the flowkey plug-in has been loaded, the following switch is added to rwfilter. To pass the filter, the record must pass the test implied by the switch.

--flowkey=VALUE_LIST: Check whether the flow key hash of the flow record matches one of the values in VALUE_LIST, where VALUE_LIST is a comma-separated list of values expressed as either decimal or hexadecimal numbers. Hexadecimal numbers must be preceded by "0x".

rwcut, rwgroup, rwsort, rwstats, and rwuniq Switch

--fields=FIELDS: FIELDS refers to a list of fields to use for the operation. The flowkey plug-in adds the following field for display, sorting, and grouping using the rwcut(1), rwgroup(1), rwsort(1), rwstats(1), and rwuniq(1) tools:

flowkey: Print, sort by, or group by the flow key hash.

EXAMPLES

In the following examples, the dollar sign ("$") represents the shell prompt. The text after the dollar sign represents the command line. Lines have been wrapped for improved readability, and the back slash ("\") is used to indicate a wrapped line.

The file vlan.pcap is a packet capture file created by tcpdump(1). The packets in the file include vlan identifiers.

In the following command, yaf(1) creates IPFIX flow records from the PCAP file, rwipfix2silk(1)) converts the IPFIX records to SiLK Flow records, and rwcut(1) prints the SiLK records as text. Note the use of the --interface-values=vlan switch on rwipfix2silk, and see how the --plugin switch is used on rwcut. The "flowkey" field contains the flow key hash.

 $ yaf < vlan.pcap                                              \
   | rwipfix2silk --interface-values=vlan                       \
   | rwcut --plugin=flowkey.so --fields=1-5,in,flowkey,stime    \
        --ipv6=ignore --timestamp=epoch --num-rec=9
           sIP|          dIP|sPort|dPort|pro|   in|   flowkey|         sTime|
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270|1252941224.465|
      10.0.0.4| 10.128.87.50|   80|32942|  6|    2|  15775704|1252941224.465|
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270|1252941224.505|
  10.128.34.93|     10.0.0.3|41443|46612|  6|    2|2705585162|1252941224.505|
      10.0.0.3| 10.128.34.93|46612|41443|  6|    2|3065308157|1252941224.505|
  10.128.34.93|     10.0.0.3|41442|   21|  6|    2|2705474059|1252941224.465|
      10.0.0.3| 10.128.34.93|   21|41442|  6|    2|  11920380|1252941224.465|
  10.128.44.78|     10.0.0.4|48081|   80|  6|    2|3144764506|1252941276.278|
      10.0.0.4| 10.128.44.78|   80|48081|  6|    2|  15792091|1252941276.279|

Here is the output from getFlowKeyHash(1) when it is run with no arguments. The "hash" column is the flow key hash and the "ms" column is the flow's time stamp.

 $ yaf < vlan.pcap              \
   | getFlowKeyHash             \
   | head -10
           sIP|          dIP|sPort|dPort|pro| vlan|      hash|            ms
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270| 1252941224465
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270| 1252941224505
  10.128.34.93|     10.0.0.3|41443|46612|  6|    2|2705585162| 1252941224505
  10.128.34.93|     10.0.0.3|41442|   21|  6|    2|2705474059| 1252941224465
  10.128.44.78|     10.0.0.4|48081|   80|  6|    2|3144764506| 1252941276278
  10.128.44.78|     10.0.0.4|48081|   80|  6|    2|3144764506| 1252941276279
  10.128.30.43|     10.0.0.4|20803|   80|  6|    2|1373863487| 1252941276278
  10.128.30.43|     10.0.0.4|20803|   80|  6|    2|1373863487| 1252941276280
  10.128.67.47|     10.0.0.4|10912|   80|  6|    2| 704652091| 1252941276278

The rwcut output has two records for each bi-flow record in the getFlowKeyHash output. The hash values match for every-other record.

Adding the --reverse switch to getFlowKeyHash produces the following:

 $ yaf < vlan.pcap              \
   | getFlowKeyHash --reverse   \
   | head -10
           sIP|          dIP|sPort|dPort|pro| vlan|      hash|            ms
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|  15775704| 1252941224465
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|  15775704| 1252941224505
  10.128.34.93|     10.0.0.3|41443|46612|  6|    2|3065308157| 1252941224505
  10.128.34.93|     10.0.0.3|41442|   21|  6|    2|  11920380| 1252941224465
  10.128.44.78|     10.0.0.4|48081|   80|  6|    2|  15792091| 1252941276278
  10.128.44.78|     10.0.0.4|48081|   80|  6|    2|  15792091| 1252941276279
  10.128.30.43|     10.0.0.4|20803|   80|  6|    2|  15740716| 1252941276278
  10.128.30.43|     10.0.0.4|20803|   80|  6|    2|  15740716| 1252941276280
  10.128.67.47|     10.0.0.4|10912|   80|  6|    2|  15731147| 1252941276278

The values for every-other flow record match nearly match, but things appear to get out of sync.

A different approach is to run yaf with the --uniflow switch:

 $ yaf --uniflow < vlan.pcap    \
   | getFlowKeyHash             \
   | head -10
           sIP|          dIP|sPort|dPort|pro| vlan|      hash|            ms
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270| 1252941224465
      10.0.0.4| 10.128.87.50|   80|32942|  6|    2|  15775704| 1252941224465
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270| 1252941224505
  10.128.34.93|     10.0.0.3|41443|46612|  6|    2|2705585162| 1252941224505
      10.0.0.3| 10.128.34.93|46612|41443|  6|    2|3065308157| 1252941224505
  10.128.34.93|     10.0.0.3|41442|   21|  6|    2|2705474059| 1252941224465
      10.0.0.3| 10.128.34.93|   21|41442|  6|    2|  11920380| 1252941224465
  10.128.44.78|     10.0.0.4|48081|   80|  6|    2|3144764506| 1252941276278
      10.0.0.4| 10.128.44.78|   80|48081|  6|    2|  15792091| 1252941276279

This result exactly matches that from rwcut.

When rwipfix2silk does not include the --interface-values=vlan switch, the result is:

 $ yaf < vlan.pcap                                              \
   | rwipfix2silk                                               \
   | rwcut --plugin=flowkey.so --fields=1-5,in,flowkey,stime    \
        --ipv6=ignore --timestamp=epoch --num-rec=9
           sIP|          dIP|sPort|dPort|pro|   in|   flowkey|         sTime|
  10.128.87.50|     10.0.0.4|32942|   80|  6|    0|2150512422|1252941224.465|
      10.0.0.4| 10.128.87.50|   80|32942|  6|    0|  13678552|1252941224.465|
  10.128.87.50|     10.0.0.4|32942|   80|  6|    0|2150512422|1252941224.505|
  10.128.34.93|     10.0.0.3|41443|46612|  6|    0|2707682314|1252941224.505|
      10.0.0.3| 10.128.34.93|46612|41443|  6|    0|3063211005|1252941224.505|

To get the same result from getFlowKeyHash, use the --snmp switch:

 $ yaf --uniflow < vlan.pcap    \
   | getFlowKeyHash --snmp      \
   | head -6
           sIP|          dIP|sPort|dPort|pro| vlan|      hash|            ms
  10.128.87.50|     10.0.0.4|32942|   80|  6|    0|2150512422| 1252941224465
      10.0.0.4| 10.128.87.50|   80|32942|  6|    0|  13678552| 1252941224465
  10.128.87.50|     10.0.0.4|32942|   80|  6|    0|2150512422| 1252941224505
  10.128.34.93|     10.0.0.3|41443|46612|  6|    0|2707682314| 1252941224505
      10.0.0.3| 10.128.34.93|46612|41443|  6|    0|3063211005| 1252941224505

To find SiLK flow records that have a particular flow key hash, use rwfilter(1):

 $ yaf < vlan.pcap                                              \
   | rwipfix2silk --interface-values=vlan                       \
   | rwfilter --plugin=flowkey.so --flowkey=2148415270,15775704 \
        --pass=stdout -                                         \
   | rwcut --plugin=flowkey.so --fields=1-5,in,flowkey,stime    \
        --ipv6=ignore --timestamp=epoch --num-rec=9
           sIP|          dIP|sPort|dPort|pro|   in|   flowkey|         sTime|
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270|1252941224.465|
      10.0.0.4| 10.128.87.50|   80|32942|  6|    2|  15775704|1252941224.465|
  10.128.87.50|     10.0.0.4|32942|   80|  6|    2|2148415270|1252941224.505|

When using rwfilter, it is best to specify the flow hash key for both the forward and reverse records.

Use rwuniq(1) to check if records with the same flow key hash appear more than twice.

 $ yaf < vlan.pcap                                              \
   | rwipfix2silk --interface-values=vlan                       \
   | rwuniq --plugin=flowkey.so --fields=flowkey --flows=3-
    flowkey|   Records|

Since no flow records are printed, the maximum number of times a flow key hash appears is 2.

ENVIRONMENT

SILK_PATH

This environment variable gives the root of the install tree. When searching for plug-ins, a SiLK application may use this environment variable. See the "FILES" section for details.

SILK_PLUGIN_DEBUG

When set to 1, the SiLK applications print status messages to the standard error as they attempt to find and open the flowkey.so plug-in. A typical invocation using this variable is:

 env SILK_PLUGIN_DEBUG=1 rwcut --plugin=flowkey.so --version

FILES

${SILK_PATH}/lib64/silk/flowkey.so
${SILK_PATH}/lib64/flowkey.so
${SILK_PATH}/lib/silk/flowkey.so
${SILK_PATH}/lib/flowkey.so
/usr/local/lib64/silk/flowkey.so
/usr/local/lib64/flowkey.so
/usr/local/lib/silk/flowkey.so
/usr/local/lib/flowkey.so: Possible locations for the plug-in.

NOTES

The flowkey plug-in was added in SiLK 3.15.0.