General Overview of rwflowpack

The settings that rwflowpack uses to categorize each flow record are determined by two textual configuration files and a compiled plug-in that is referred to as the packing logic.

The first of the configuration files is silk.conf(5) which specifies the classes, types, and sensors that rwflowpack uses when writing files and that rwfilter uses when selecting flow files.

The second configuration file is the sensor.conf(5) file. This file contains multiple sensor blocks, where each block contains information which the packing logic uses to categorize flow records collected by the probes specified for that sensor.

The combination of a silk.conf file and a particular packing logic plug-in define a site. By having the configuration and packing logic outside of the core tools, users can more easily configure SiLK for their particular installation and a single installation of SiLK can support multiple sites.

This manual page describes the packing logic for the twoway site. For a description of the packing logic at another site, see that site's manual page.

•

packlogic-generic(3)

Networks, Classes, and Types for the "twoway" Site

The packlogic-twoway.so plug-in specifies three network names to describe the logical address spaces that border the sensor:

internal

the space that is being monitored

external

the space outside the monitored network

null

the destination network for a flow that does not leave the router, because either the flow was blocked by the router's access control list or its destination was the router itself---e.g., a BGP message

There is an implicit fourth network, unknown, which is anything that does not match the three networks above.

Given these networks, the following table describes how flows can move between the networks. Traffic between the networks is successfully routed unless the description explicitly says "blocked".

 SOURCE    DESTINATION   DESCRIPTION
 external  internal      incoming traffic
 internal  external      outgoing traffic
 external  null          blocked incoming traffic
 internal  null          blocked outgoing traffic
 external  external      strictly external traffic
 internal  internal      strictly internal traffic
 null      any           unclear: null should never be a source
 external  unknown       unclear
 internal  unknown       unclear
 unknown   any           unclear

The silk.conf file and packlogic-twoway.so plug-in define a single class, all.

The type assigned to a flow record within the all class depends on the how the record moves between the networks, and the types follow from the table above:

in, inicmp, inweb

Incoming traffic. The traffic is split into multiple types, and these types allow the analysts to query a subset of the flow records depending on their needs. Each incoming flow record is split into the one of incoming types using the following rules:

out, outicmp, outweb

Outgoing traffic. The traffic is split among the types using rules similar to those for incoming traffic.

innull

Blocked incoming traffic

outnull

Blocked outgoing traffic

ext2ext

Strictly external traffic

int2int

Strictly internal traffic

other

Either traffic from the null network or traffic to or from the unknown network

Assigning a flow to source and destination networks

The first method sets the source and destination of all records to particular networks. This can be used, for example, when the physical network device at the sensor only sees one direction of the traffic. To do this, use the source-network and destination-network statements in the sensor block. The following sensor, S1, considers all traffic as blocked incoming:

 sensor S1
   ipfix-probes S1
   source-network external
   destination-network null
 end sensor

The second method to determine how a flow record moves between the networks is to define the networks and use characteristics of the flow record to determine its source and destination networks.

The sensor.conf file provides two ways to define a network: use the NET-ipblocks statement to specify the NET network as a list of IP address blocks, or use the NET-interfaces statement to specify the NET network using a list of SNMP interfaces.

For the source network of a flow record to be considered external, either the source IP (SiLK field "sIP") must appear in the list of external-ipblocks or the incoming SNMP interface (SiLK field "in") must appear in the list of external-interfaces. Note: If the probe block that specifies where the flow was collected contains an interface-values vlan statement, the SiLK "in" field contains the VLAN ID.

For the destination network of a flow record to be considered null, either the destination IP ("dIP") must appear in the list of null-ipblocks or the outgoing SNMP interface ("out") must appear in the list of null-interfaces.

Consider the following two sensors:

 sensor S2
   ipfix-probes S2
   external-ipblocks 172.16.0.0/16
   internal-ipblocks 172.20.0.0/16
 end sensor

 sensor S3
   ipfix-probes S3
   external-interfaces 17,18,19
   internal-interfaces 21,22,23
 end sensor

A flow record collected at probe S2 whose "sIP" is 172.16.1.1 and whose "dIP" is 172.20.2.2 is considered incoming.

A flow record collected at probe S3 whose "in" is 23 and whose "out" is 18 is considered outgoing. A flow on S3 whose "in" is 23 and whose "out" is 27 is written to other since the "out" field is not matched.

There are two constructs in the sensor.conf file that help when specifying these lists:

1.

The NET-interfaces or NET-ipblocks statement in a sensor block may use remainder to denote interfaces or IP blocks that do not appear elsewhere in the block.

2.

A group block can be used to give a name to a set of IP blocks or SNMP interfaces which a sensor block can reference.

For details, see the sensor.conf(5) manual page.

Valid sensors

• ipfix
• netflow-v5
• netflow-v9
• sflow
• silk

In addition, each sensor block must meet the following rules:

• If the sensor has the source-network and destination-network explicitly set, the sensor is valid and none of the following checks are performed. Otherwise,
• At least one of NET-interfaces or NET-ipblocks must be specified, where NET is either internal or external. And,
• A sensor cannot mix NET-ipblocks and NET-interfaces, with the exception that null-interfaces are always allowed. And,
• Only one network on the sensor may use remainder. And,
• If a sensor contains only one NET-ipblocks statement, that statement may not use remainder. (The NET-interfaces statement does not have this restriction.) And,
• When the remainder keyword is not used and only one of the internal or external networks is defined, the external or internal network, respectively, is defined as having the remainder.

Packing logic code

A single sensor block will assign the flow record to a single class and type, and processing of the flow for that sensor block stops as soon as a type is assigned. When multiple sensor blocks reference the same probe, the flow records collected by that probe are processed by each of those sensor blocks.

A flow record is always assigned to the class all unless the flow is ignored.

A textual description of the code used to assign the type is shown here. As of SiLK 3.8.0, the type may be determined by the presence of certain IPFIX or NetFlowV9 information elements.

• Ignore any flow record that matches a discard-when statement or does not match a discard-unless statement.
• If source-network is external, if "sIP" matches external-ipblocks, or if "in" matches external-interfaces, then

•

Else, if source-network is internal, if "sIP" matches internal-ipblocks, or if "in" matches internal-interfaces, then

• Else, pack as other.
• Potentially modify the type: If the probe has a quirks setting that includes "firewall-event" and if the incoming record contains the "firewallEvent" or "NF_F_FW_EVENT" information element whose value is 3 (flow denied), change the type where the flow is packed as follows: