Configuration File

The YAF configuration file can be used instead of or in addition to command line arguments. Lua must be installed for use of the yaf configuration file.

--config CONFIGURATION_FILE

If present, use the variables set in the CONFIGURATION_FILE. The CONFIGURATION_FILE is a Lua configuration file, a plain text file that can also be a Lua program. A sample configuration file can be found in etc/yaf.init. yaf will use the variables set in the configuration file along with any command line arguments.

Input Options

These options control where yaf will take its input from. yaf can read packets from a pcap dumpfile (as generated by tcpdump -w) or live from an interface via libpcap, libdag, or libnapatech, or Netronome API. By default, if no input options are given, yaf reads a pcap dumpfile on standard input.

--in INPUT_SPECIFIER

INPUT_SPECIFIER is an input specifier. If --live is given, this is the name of an interface (e.g. "eth0", "en0", "dag0", "nt3g", "nt3g0:1", "0:0") to capture packets from. Otherwise, it is a filename; the string - may be used to read from standard input (the default). See --live for more information on formats for Napatech, Dag, and Netronome Interface formats.

--caplist

If present, treat the filename in INPUT_SPECIFIER as an ordered newline-delimited list of pathnames to pcap(3) dumpfiles. Blank lines and lines beginning with the character '#' within this are ignored. All pathnames are evaluated with respect to the working directory yaf is run in. These dumpfiles are processed in order using the same flow table, so they must be listed in ascending time order. This option is intended to ease the use of yaf with rotated or otherwise split tcpdump(1) output.

--noerror

Used with the --caplist option. When present, this prevents yaf from exiting when processing a list of dumpfiles in the middle due to an error in a file. yaf will continue to process all files given in the INPUT_SPECIFIER despite errors within those files.

--live LIVE_TYPE

If present, capture packets from an interface named in the INPUT_SPECIFIER. LIVE_TYPE is one of pcap for packet capture via libpcap, pfring for packet capture via libpfring, or dag for packet capture via an Endace DAG interface using libdag, or napatech for packet capture via a Napatech Adapter, or netronome for packet capture via a Netronome NFE card, or zc for packet capture via PF_RING ZC. <pfring> is only available if yaf was built with PF_RING support. See the yafzcbalance(1) man page for using yaf with PF_RING ZC. dag is only available if yaf was built with Endace DAG support. napatech is only available if yaf was built with Napatech API support. If LIVE_TYPE is napatech, the INPUT_SPECIFIER given to --in should be in the form nt3g[<streamID>:<ports>]. StreamID and Ports are optional. StreamID if given, is the ID that the traffic stream will be assigned to on the incoming ports. Ports may be a comma-separated list of ports to listen on. If [ports] is not specified, the default is to listen on All ports. StreamID defaults to 0. netronome is only available if yaf was built with Netronome API support. If LIVE_TYPE is netronome, the INPUT_SPECIFIER given to --in should be in the form <device>:<ring> where device is the NFE card ID, typically 0. Ring is the capture ring ID which is configured via a modprobe configuration file and resides in /etc/modprobe.d/pcd.conf.

--export-interface

If present, the interface on which a packet was received will be noted internally within yaf. When flow records are exported from yaf, an "ingressinterface" and an "egressinterface" set of fields will be added to the output. The "ingressinterface" field will be the physical interface which captured the packet while the "egressinterface" will be the physical interface | 0x100. This can be used to separate traffic based on DAG physical ports. For use with the DAG card, traffic received on separate ports will be separated into different flows if yaf is configured with the --enable-daginterface option. Otherwise the physical port will simply be exported in the "ingressInterface" or "egressInterface" fields in the IPFIX record (flows can exist over multiple interfaces). This option requires building DAG, Netronome, or Napatech support in yaf with the --with-dag, --with-napatech, or --with-netronome switch. In previous versions of yaf this option was enabled using the --dag-interface or --napatech-interface switch. It is now enabled by default when yaf is built with DAG, Netronome, or Napatech support. It can be disabled by configuring yaf with <--enable-interface=no>. To separate traffic received on separate ports into separate flows, you must use --enable-daginterface when configuring yaf.

--filter BPF_FILTER

If present, enable Berkeley Packet Filtering (BPF) in yaf with BPF_FILTER as the incoming traffic filter. Syntax of BPF_FILTER follows the expression format described in the tcpdump(1) man page. This option is not currently supported if --live is set to dag or napatech or netronome as BPF filtering is implemented with libpcap. However, you may be able to use a BPF filter by running yaf with the DAG, Napatech, or Netronome implementations of libpcap.

--decompress DECOMPRESS_DIR

If present and the input file(s) are compressed (gzip'd), decompress the file to a temporary file within DECOMPRESS_DIR. If --caplist is also present, all files will be decompressed to DECOMPRESS_DIR. If this option is not present, yaf will decompress files to the variable specified by the TMPDIR environment variable or /tmp if TMPDIR is not set. The zlib library must be installed to use this feature.

Output Options

These options control where yaf will send its output. yaf can write flows to an IPFIX file or export flows to an IPFIX collector over SCTP, TCP, UDP, or Spread. By default, if no output options are given, yaf writes an IPFIX file to standard output.

--out OUTPUT_SPECIFIER

OUTPUT_SPECIFIER is an output specifier. If --ipfix is present, the OUTPUT_SPECIFIER specifies the hostname or IP address of the collector to which the flows will be exported. Otherwise, if --rotate is present, OUTPUT_SPECIFIER is the prefix name of each output file to write to. If --ipfix is present and set to spread, then OUTPUT_SPECIFIER should be set to the name of the Spread daemon to connect to (See below examples of spread daemon names). Otherwise, OUTPUT_SPECIFIER is a filename in which the flows will be written; the string - may be used to write to standard output (the default).

Examples

--ipfix TRANSPORT_PROTOCOL

If present, causes yaf to operate as an IPFIX exporter, sending IPFIX Messages via the specified transport protocol to the collector (e.g., SiLK's rwflowpack or flowcap facilities) named in the OUTPUT_SPECIFIER. Valid TRANSPORT_PROTOCOL values are tcp, udp, sctp, and spread; sctp is only available if yaf was built with SCTP support; spread is only available if yaf was built with Spread support. UDP is not recommended, as it is not a reliable transport protocol, and cannot guarantee delivery of messages. As per the recommendations in RFC 5101, yaf will retransmit templates three times within the template timeout period (configurable using --udp-temp-timeout or by default, 10 minutes). Use the --ipfix-port, --tls, --tls-ca, --tls-cert, --tls-key, and --group options to further configure the connection to the IPFIX collector.

--rotate ROTATE_DELAY

If present, causes yaf to write output to multiple files, opening a new output file every ROTATE_DELAY seconds in the input data. Rotated files are named using the prefix given in the OUTPUT_SPECIFIER, followed by a suffix containing a timestamp in "YYYYMMDDhhmmss" format, a decimal serial number, and the file extension .yaf.

--lock

Use lockfiles for concurrent file access protection on output files. This is recommended for interoperating with the Airframe filedaemon facility.

--stats INTERVAL

If present, causes yaf to export process statistics every INTERVAL seconds. The default value for INTERVAL is 300 seconds or every 5 minutes. yaf uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. If INTERVAL is set to zero, stats will not be exported.

--no-stats

If present, yaf will not export process statistics. yaf uses IPFIX Options Templates and Records to export flow, fragment, and decoding statistics. --no-stats takes precedence over --stats.

--no-tombstone

If present, yaf will not export tombstone records. yaf uses IPFIX Options Templates and Records to export tombstone records. Tombstone records will only be exported if stats exporting is also active.

--tombstone-configured-id IDENTIFIER

If present, overrides the default "exporterConfiguredId" value in tombstone records. This value should be less than 0xFFFF (65535). The default value is 0.

--no-output

If present, yaf will not export IPFIX data. It will ignore any argument provided to --out.

Decoder Options

These options are used to modify the yaf packet decoder's behavior. None of these options are required; the default behavior for each option when not present is noted.

--no-frag

If present, ignore all fragmented packets. By default, yaf will reassemble fragments with a 30 second fragment timeout.

--max-frags FRAG_TABLE_MAX

If present, limit the number of outstanding, not-yet reassembled fragments in the fragment table to FRAG_TABLE_MAX by prematurely expiring fragments from the table. This option is provided to limit yaf resource usage when operating on data from very large networks or networks with abnormal fragmentation. The fragment table may exceed this limit slightly due to limits on how often yaf prunes the fragment table (every 5 seconds). By default, there is no fragment table limit, and the fragment table can grow to resource exhaustion.

--ip4-only

If present, ignore all IPv6 packets and export IPv4 flows only. The default is to process both IPv4 and IPv6 packets.

--ip6-only

If present, ignore all IPv4 packets and export IPv6 flows only. The default is to process both IPv4 and IPv6 packets.

--gre-decode

If present, attempt to decode GRE version 0 encapsulated packets. Flows will be created from packets within the GRE tunnels. Undecodeable GRE packets will be dropped. Without this option, GRE traffic is exported as IP protocol 47 flows. This option is presently experimental.

Flow Table Options

These options are used to modify the flow table behavior within yaf. None of these options are required; the default behavior for each option when not present is noted.

--idle-timeout IDLE_TIMEOUT

Set flow idle timeout in seconds. Flows are considered idle and flushed from the flow table if no packets are received for IDLE_TIMEOUT seconds. The default flow idle timeout is 300 seconds (5 minutes). Setting IDLE_TIMEOUT to 0 creates a flow for each packet.

--active-timeout ACTIVE_TIMEOUT

Set flow active timeout in seconds. Any flow lasting longer than ACTIVE_TIMEOUT seconds will be flushed from the flow table. The default flow active timeout is 1800 seconds (30 minutes).

--udp-temp-timeout TEMPLATE_TIMEOUT

Set UDP template timeout in seconds if --ipfix is set to udp. As per RFC 5101 recommendations, yaf will attempt to export templates three times within TEMPLATE_TIMEOUT. The default template timeout period is 600 seconds (10 minutes).

--max-payload PAYLOAD_OCTETS

If present, capture at most PAYLOAD_OCTETS octets from the start of each direction of each flow. Non-TCP flows will only capture payload from the first packet unless --udp-payload is set. If not present, yaf will not attempt to capture payload. Payload capture must be enabled for payload export (--export-payload), application labeling (--applabel), and entropy evaluation (--entropy). Note that payload capture is still an experimental feature.

--max-flows FLOW_TABLE_MAX

If present, limit the number of open flows in the flow table to FLOW_TABLE_MAX by prematurely expiring the flows with the least recently received packets; this is analogous to an adaptive idle timeout. This option is provided to limit yaf resource usage when operating on data from large networks. By default, there is no flow table limit, and the flow table can grow to resource exhaustion.

--udp-payload

If present, capture at most PAYLOAD_OCTETS octets fom the start of each direction of each UDP flow, where PAYLOAD_OCTETS is set using the --max-payload flag.

--silk

If present, export flows in "SiLK mode". As of yaf 2.0, this will export TCP information (flags, ISN) in the main flow record instead of within the SubTemplateMultiList. This flag must be used when exporting to SiLK for it to collect TCP flow information. This also introduces the following incompatibilities with standard IPFIX export:

--force-read-all

If present, yaf will process out-of-sequence packets. However, it will still reject out-of-sequence fragments.

--no-vlan-in-key

If present, yaf will NOT use the VLAN ID in the flow key hash calculation for flows. This means that packets within the active/idle timeouts that have the same 5-tuple (sIP, dIP, sport, dport, protocol) but different VLAN IDs will be aggregated into 1 flow and the VLAN ID of the first packet in each direction will be exported in the vlanId and reverseVlanId fields.

Export Options

These options are used to modify the data exported by yaf.

--export-payload

If present, export payload from each direction of each flow. The amount of payload exported for each direction is the smaller of the arguments to --max-payload and --max-export. Non-TCP flows will only export payload from the first packet. By default, yaf will not export flow payload. See also --payload-applabel-select.

--payload-applabel-select APPLABEL_LIST

Enable payload export (as --export-payload) but only for the application labels specified in APPLABEL_LIST, a comma separated list of applabel values.

--max-export MAX_PAY_OCTETS

If present, export at most MAX_PAY_OCTETS from the start of each direction of each flow. MAX_PAY_OCTETS must not be more than the argument to --max-payload. By default, MAX_PAY_OCTETS is the value given to --max-payload if --export-payload is also present.

--uniflow

If present, export biflows using the Record Adjacency method in section 3 of RFC 5103. This is useful when exporting to IPFIX Collecting Processes that are not biflow-aware.

--mac

If present, export MAC-layer information; presently, exports source and destination MAC addresses.

--force-ip6-export

If present, force IPv4 flows to be exported with IPv6-mapped IPv4 addresses in ::FFFF/96. This will cause all flows to appear to be IPv6 flows.

--observation-domain DOMAIN_ID

Set the observationDomainID on each exported IPFIX message to the given integer value. If not present, the observationDomainId defaults to 0. This value is also used as the exportingProcessId in the yaf statistics Option Record as a Scope Field.

--udp-uniflow PORT

If present, export each UDP packet on the given port (or 1 for all ports) as a single flow, with flowEndReason set to YAF_END_UDPFORCE (0x1F). This will not close the flow. The flow will stay open until it closes naturally by the idle and active timeouts. Most useful with --export-payload in order to export every UDP payload on a specific port.

--flow-stats

If present, export extra flow attributes and statistics in the subTemplateMultiList field. This will maintain information such as small packet count, large packet count, nonempty packet count, average interarrival times, total data octets, and max packet size. See the flow statistics template below for more information about each of the fields yaf exports.

--delta

If present, export octet and packet total counts in the delta count information elements. octetTotalCount will be exported in octetDeltaCount (IE 1), reverseOctetTotalCount will be exported in reverseOctetDeltaCount. packetTotalCount will be exported in packetDeltaCount (IE 2), and reversePacketTotalCount will be exported in reversePacketDeltaCount.

--ingress INGRESS_INT

If present, set the ingressInterface field in the flow template to INGRESS_INT. This field will also be populated if yaf was configured with --enable-daginterface or --enable-napatechinterface or --with-bivio. If yaf is running on a dag, napatech, or bivio, and the physical interface is available, this value will override INGRESS_INT.

--egress EGRESS_INT

If present, set the egressInterface field in the flow template to EGRESS_INT. This field will also be populated if yaf was configured with --enable-daginterface or --enable-napatechinterface or --with-bivio. If yaf is running on a dag, napatech, or bivio, and the physical interface is available, this value will override EGRESS_INT.

--metadata-export

If present, export template metadata (name and description) and information element metadata before data records. NOTE: this option was renamed from --template-info in YAF 2.11.0

Application Labeler Options

If yaf is built with application labeler support enabled (using the --enable-applabel option to ./configure when yaf is built), then yaf can examine packet payloads and determine the application protocol in use within a flow, and export a 16-bit application label with each flow.

The exported application label uses the common port number for the protocol. For example, HTTP traffic, independent of what port the traffic is detected on, will be labeled with a value of 80, the default HTTP port. Labels and rules are taken from a configuration file read at yaf startup time.

Application labeling requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results.

Application labeling is presently experimental. SiLK does support IPFIX import and translation of the application label via rwflowpack, flowcap, and rwipfix2silk.

--applabel

If present, export application label data. Requires --max-payload to enable payload capture.

--applabel-rules RULES_FILE

Read application labeler rules from RULES_FILE. If not present, rules are read by default from /usr/local/etc/yafApplabelRules.conf.

nDPI Options

nDPI is a version of OpenDPI as maintained by ntop. You can read more about nDPI and the applications supported at: http://www.ntop.org/products/deep-packet-inspection/ndpi/

If yaf is built with nDPI support enabled (using the --enable-ndpi option to ./configure when yaf is built), then yaf can examine packet payloads and determine the application protocol in use within a flow, and export the applicaiton protocol and sub-protocol with each flow.

nDPI requires payload capture to be enabled with the --max-payload option. A minimum payload capture length of 384 octets is recommended for best results.

--ndpi

If present, export nDPI data. Requires --max-payload to enable payload capture.

--ndpi-protocol-file FILE

Specify protocol file for sub-protocol and port-based protocol detection

Entropy Measurement

If yaf is built with entropy measurement enabled (using the --enable-entropy option to ./configure when yaf is built,) then yaf can examine the packet payloads and determine a Shannon Entropy value for the payload. The entropy calculation does not include the network (IP) or transport (UDP/TCP) headers. The entropy is calculated in terms of bits per byte, (log base 2.) The calculation generates a real number value between 0.0 and 8.0. That number is then converted into an 8-bit integer value between 0 and 255. Roughly, numbers above 230 are generally compressed (or encrypted) and numbers centered around approximately 140 are English text. Lower numbers carry even less information content. Another useful piece of information is that SSL/TLS tends to zero pad its packets, which causes the entropy of those flows to drop quite low.

--entropy

If present, export the entropy values for both the forward and reverse payloads. Requires the --max-payload option to operate.

IPFIX Connection Options

These options are used to configure the connection to an IPFIX collector.

--ipfix-port PORT

If --ipfix is present, export flows to TCP, UDP, or SCTP port PORT. If not present, the default IPFIX port 4739 is used. If --tls is also present, the default secure IPFIX port 4740 is used.

--tls

If --ipfix is present, use TLS to secure the connection to the IPFIX collector. Requires the TRANSPORT_PROTOCOL to be tcp, as DTLS over UDP or SCTP is not yet supported. Requires the --tls-ca, --tls-cert, and --tls-key options to specify the X.509 certificate and TLS key information.

--tls-ca CA_PEM_FILE

Use the Certificate Authority or Authorities in CA_PEM_FILE to verify the remote IPFIX Collecting Process' X.509 certificate. The connection to the Collecting Process will fail if its certificate was not signed by this CA (or by a certificate signed by this CA, recursively); this prevents export to unauthorized Collecting Processes. Required if --tls is present.

--tls-cert CERT_PEM_FILE

Use the X.509 certificate in CERT_PEM_FILE to identify this IPFIX Exporting Process. This certificate should contain the public part of the private key in KEY_PEM_FILE. Required if --tls is present.

--tls-key KEY_PEM_FILE

Use the private key in KEY_PEM_FILE for this IPFIX Exporting Process. This key should contain the private part of the public key in CERT_PEM_FILE. Required if --tls is present. If the key is encrypted, the password must be present in the YAF_TLS_PASS environment variable.

--group SPREAD_GROUP_NAME

If --ipfix is present and set to spread, use --group to specify the spread group name(s) to publish output. It is possible to list more than one group name in a comma-seperated list. To use Spread as a manifold for different types of flows, use the format GROUP, GROUP_NAME:VALUE, GROUP_NAME:VALUE as the argument to --group and use the --groupby switch. This list should be contained in quotes if it contains spaces (yaf will ignore spaces in quotes). It is suggested to use one group as the catchall for all flows (no value listed) so flows are not lost. The --groupby switch must be used if --group uses GROUP:VALUE format. See the Spread Documentation, www.spread.org, for more details on Spread.

--groupby GROUPBY_TYPE

If --group is used with group values, use --groupby to specify what type of value should be used. Options are port, vlan, applabel, protocol, version. --groupby accepts only one argument. The port option is destination transport port of the flow. version is the IP version of the flow.

Privilege Options

These options are used to cause yaf to drop privileges when running as root for live capture purposes.

--become-user UNPRIVILEGED_USER

After opening the live capture device in --live mode, drop privilege to the named user. Using --become-user requires yaf to be run as root or setuid root. This option will cause all files written by yaf to be owned by the user UNPRIVILEGED_USER and the user's primary group; use --become-group as well to change the group yaf runs as for output purposes.

If running as root for live capture purposes and --become-user is not present, yaf will warn that privilege is not being dropped. We highly recommend the use of this option, especially in production environments, for security purposes.

--become-group UNPRIVILEGED_GROUP

--become-group can be used to change the group from the default of the user given in --become-user. This option has no effect if given without the --become-user option as well.

PCAP Options

These options are used to turn on and configure yaf's PCAP export capability.

--pcap PCAP_FILE_PREFIX

This option turns on rolling PCAP export in yaf. It will capture and write packets for all network traffic yaf has received and processed to PCAP files with the given PCAP_FILE_PREFIX. yaf will not create file directories. If yaf can't write to the file, yaf will turn off PCAP export. Pcap files will have names in the form of PCAP_FILE_PREFIX[datetime]_serialno.pcap". yaf will write to a file until the file size has reached --max-pcap or every --pcap-timer seconds (whichever happens first). By default, yaf rotates files every 5 MB. Files will be "locked" (".lock" will be appended to the filename) until yaf has closed the file. Be aware that your Operating System will have a limit on the maximum number of files in a directory and a maximum file size. If this limit is reached, yaf will write warning messages and terminate PCAP export. This may effect flow generation if yaf is also writing IPFIX files. Optionally, you can also export meta information about the flows in each rolling PCAP file with the --pcap-meta-file switch. If --pcap is used in conjunction with --hash and --stime, the PCAP_FILE_PREFIX should be the name of the PCAP file to write to (it will not be used as a file prefix).

--pcap-per-flow

If present, yaf will write a pcap file for each flow in the output directory given to --pcap. PCAP_FILE_PREFIX given to --pcap must be a file directory. This option is experimental and should only be used when reading pcap files of reasonable size. yaf only writes up to --max-payload bytes of each packet to the pcap file. Therefore, --max-payload must be set to an appropriate size to prevent packets from being truncated in the pcap file. yaf will use the last three digits of the flowStartMilliseconds as the directory and the flow key hash, flowStartMilliseconds, and serial number as the filename. See the included getFlowKeyHash program to easily calculate the name of the file for a given flow. When the pcap file has reached --max-pcap size, yaf will close the file, increment the serial number, and open a new pcap file with the same naming convention. Note that your operating system has a limit to the number of open file handles yaf can maintain at any given time. Therefore, the performance of yaf degrades when the number of open flows is greater than the maximum number of file handles.

--max-pcap MAX_FILE_MB

If present, set the maximum file size of pcap files to MAX_FILE_MB MB. The default is 5 MB.

--pcap-timer PCAP_ROTATE_DELAY

If present, yaf will rotate rolling pcap files every PCAP_ROTATE_DELAY seconds or when the file reaches --max-pcap size, whichever happens first. By default, yaf only rotates files based on file size.

--pcap-meta-file META_FILENAME

If present and --pcap is also present, yaf will export metadata on the flows contained in each rolling pcap file yaf is writing to the filename specified by META_FILENAME. yaf will write a line in the form:

--index-pcap

If present and --pcap and --pcap-meta-file are also present, export offset and length information about the packets yaf is writing to the rolling pcap files. This option can also be used when <B--pcap> is not present, in which case it will write information about the file it is reading. Adding this option will force yaf to write one line per packet to the pcap-meta-file in the form:

--hash FLOW_KEY_HASH

If present, only write PCAP data for the flow(s) with FLOW_KEY_HASH. This option is only valid with the --pcap option.

--stime FLOW_START_TIMEMS

If present, only write PCAP data for the flow(s) with FLOW_START_TIMEMS and FLOW_KEY_HASH given to --hash. This option is only valid when used with the --hash and --pcap options.

Logging Options

These options are used to specify how log messages are routed. yaf can log to standard error, regular files, or the UNIX syslog facility.

--log LOG_SPECIFIER

Specifies destination for log messages. LOG_SPECIFIER can be a syslog(3) facility name, the special value stderr for standard error, or the absolute path to a file for file logging. The default log specifier is stderr if available, user otherwise.

--loglevel LOG_LEVEL

Specify minimum level for logged messages. In increasing levels of verbosity, the supported log levels are quiet, error, critical, warning, message, info, and debug. The default logging level is warning.

--verbose

Equivalent to --loglevel debug.

--version

If present, print version and copyright information to standard error and exit.

Plugin Options

These options are used to load, configure, and run a yaf plugin.

--plugin-name LIBPLUGIN_NAME[,LIBPLUGIN_NAME...]

Specify the plugin to load. The loaded plugin must follow the yaf plugin framework. LIBPLUGIN_NAME must be the full path to the plugin library name. Two plugins are included with c<yaf>, a Deep Packet Inspection plugin, and a DHCP Fingerprinting plugin. This flag will only be recognized if yaf is configured with --enable-plugins. There are also configure options to export only DNS Authoritative and NXDomain responses. Read each plugin's documentation for more information.

--plugin-opts "OPTIONS[,OPTIONS...]"

Specify the arguments to the plugin given to --plugin-name. This flag will only be recognized if yaf is configured with --enable-plugins and --plugin-name is set to a valid plugin. For example, the DPI Plugin takes the well-known port of a protocol(s) to enable DPI (default for DPI is all protocols).

--plugin-conf CONF_FILE_PATH[,CONF_FILE_PATH...]

Specify the path to a configuration file for the plugin given to --plugin-name. This flag will only be recognized if yaf is configured with --enable-plugins and --plugin-name is set to a valid plugin. If this switch is not used, but the plugin requires a configuration file, the default location /usr/local/etc will be used.

Passive OS Fingerprinting (p0f)

These options are used to enable p0f in yaf. p0f is presently experimental. There is no support in yafscii or SiLK for printing p0f related data. Currently, yaf uses the p0f Version 2 SYN fingerprints (see p0f.fp).

--p0fprint

If present, export p0f data. This data consists of three related information elements; osName, osVersion, osFingerPrint. This flag requires yaf to be configured with --enable-p0fprinter.

--p0f-fingerprints

Location of the p0f fingerprint file(s), p0f.fp. Default is /usr/local/etc/p0f.fp. This version of yaf includes the updated CERT p0f fingerprints. See <https://tools.netsa.cert.org/p0f/index.html> for updates.

--fpexport

If present, enable export of handshake headers for external OS fingerprinters. The related information elements are firstPacketBanner and secondPacketBanner. This flag requires yaf to be configured with --enable-fpexporter.

Basic Flow Record

yaf's output consists of an IPFIX message stream. yaf uses a variety of templates for IPFIX data records; the information elements that may appear in these templates are enumerated below. For further information about the IPFIX information model and IPFIX message stream, see RFC 5102, RFC 5101, and RFC 5103. As of yaf 2.0, yaf nests some templates in an IPFIX subTemplateMultiList. In order to retain compatibility with the SiLK Tools, use --silk to prevent yaf from nesting TCP Information Elements. Below are descriptions of each of the templates yaf will export. See the Internet-Draft Export of Structured Data in IPFIX for more information on IPFIX lists.

yaf assigns information element numbers to reverse flow elements in biflow capture based on the standard IPFIX PEN 29305. This applies only for information elements defined in the standard IPFIX Information Model (RFC 5102) that do not have a reverse information element already defined. For information elements defined under the CERT PEN, a standard method is used to calculate their reverse element identifier. The method is that bit fourteen is set to one in the IE field, (e.g. 16384 + the forward IE number.)

flowStartMilliseconds IE 152, 8 octets, unsigned

Flow start time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.

flowEndMilliseconds IE 153, 8 octets, unsigned

Flow end time in milliseconds since 1970-01-01 00:00:00 UTC. Always present.

octetTotalCount IE 85, 8 octets, unsigned

Number of octets in packets in forward direction of flow. Always present (unless "--delta" is used.) May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseOctetTotalCount Reverse (PEN 29305) IE 85, 8 octets, unsigned

Number of octets in packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.

packetTotalCount IE 86, 8 octets, unsigned

Number of packets in forward direction of flow. Always present (unless "--delta" is used.) May be encoded in 4 octets using IPFIX reduced-length encoding.

reversePacketTotalCount Reverse (PEN 29305) IE 86, 8 octets, unsigned

Number of packets in reverse direction of flow. Present if flow has a reverse direction. May be encoded in 4 octets using IPFIX reduced-length encoding.

octetDeltaCount IE 1, 8 octets, unsigned

Number of octets in packets in forward direction of flow. Only present if "--delta" is used. May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseOctetDeltaCount Reverse (PEN 29305) IE 1, 8 octets, unsigned

Number of octets in reverse direction of flow. Only present if "--delta" is used and non-zero. May be encoded in 4 octets using IPFIX reduced-length encoding.

packetDeltaCount IE 2, 8 octets, unsigned

Number of packets in forward direction of flow. Only present if "--delta" is used. May be encoded in 4 octets using IPFIX reduced-length encoding.

reversePacketDeltaCount Reverse (PEN 29305) IE 2, 8 octets, unsigned

Number of packets in reverse direction of flow. Only present if "--delta" is used and non-zero. May be encoded in 4 octets using IPFIX reduced-length encoding.

reverseFlowDeltaMilliseconds CERT (PEN 6871) IE 21, 4 octets, unsigned

Difference in time in milliseconds between first packet in forward direction and first packet in reverse direction. Correlates with (but does not necessarily represent) round-trip time. Present if flow has a reverse direction.

sourceIPv4Address IE 8, 4 octets, unisigned

IPv4 address of flow source or biflow initiator. Present for IPv4 flows without IPv6-mapped addresses only.

destinationIPv4Address IE 12, 4 octets, unsigned

IPv4 address of flow source or biflow responder. Present for IPv4 flows without IPv6-mapped addresses only.

sourceIPv6Address IE 27, 16 octets, unsigned

IPv6 address of flow source or biflow initiator. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

destinationIPv6Address IE 28, 16 octets, unsigned

IPv6 address of flow source or biflow responder. Present for IPv6 flows or IPv6-mapped IPv4 flows only.

sourceTransportPort IE 7, 2 octets, unsigned

TCP or UDP port on the flow source or biflow initiator endpoint. Always present.

destinationTransportPort IE 11, 2 octets, usigned

TCP or UDP port on the flow destination or biflow responder endpoint. Always present. For ICMP flows, contains ICMP type * 256 + ICMP code. This is non-standard, and an open issue in yaf.

flowAttributes CERT (PEN 6871) IE 40, 2 octets, unsigned

Miscellaneous flow attributes for the forward direction of the flow. Always present (yaf 2.1 or later). Current flag values:

reverseFlowAttributes CERT (PEN 6871) IE 16424, 2 octets, unsigned

Miscellaneous flow attributes for the reverse direction of the flow. Always present (yaf 2.1 or later). Current flag values:

protocolIdentifier IE 4, 1 octet, unsigned

IP protocol of the flow. Always present.

flowEndReason IE 136, 1 octet, unsigned

Flow end reason code, as defined by the IPFIX Information Model. Always present. In --silk mode, the high-order bit is set if the flow was created by continuation.

silkAppLabel CERT (PEN 6871) IE 33, 2 octets, unsigned

Application label, defined as the primary well-known port associated with a given application. Present if the application labeler is enabled, and was able to determine the application protocol used within the flow.

vlanId IE 58, 2 octets, unsigned

802.1q VLAN tag of the first packet in the forward direction of the flow.

reverseVlanId Reverse (PEN 29305) IE 58, 2 octets, unsigned

802.1q VLAN tag of the first packet in the reverse direction of the flow. Present if the flow has a reverse direction.

ingressInterface IE 10, 4 octets, unsigned

The index of the IP interface where packets of this flow are being received. Use --ingress, --napatech-interface, --dag-interface or configure yaf with bivio for this field to be present in the flow template. Use --ingress to manually set this field.

egressInterface IE 14, 4 octets, unsigned

The index of the IP interface where packets of this flow are being received. Use --egress, --napatech-interface, --dag-interface or configure yaf with bivio for this field to be present in the flow template. If using napatech, dag, or bivio, "egressinterface" will be the physical interface | 0x100. Use --egress to manually set this field.

ipClassOfService IE 5, 1 octet, unsigned

For IPv4 packets, this is the value of the TOS field in the IPv4 header. For IPv6 packets, this is the Traffic Class field in the IPv6 header.

reverseIpClassOfService Reverse (PEN 29305) IE 5, 1 octet, unsigned

The TOS field in the IPv4 header for packets in the reverse direction, and Traffic Class field in the IPv6 header for packets in the reverse direction.

mplsTopLabelStackSection IE 70, 3 octets

The MPLS Label from the top of the MPLS label stack entry. yaf does not include the Experimental bits and Bottom of the Stack bit in the export field. yaf must have been enabled with MPLS support for export of this field. Note that this field is defined as an octet array in the default libfixbuf Information Model. yaf uses the length override feature in libfixbuf to redefine it from variable length to 3 bytes.

mplsLabelStackSection2, IE 71, 3 octets

The MPLS Label from the MPLS label stack entry immediately before the top entry. yaf does not include the Experimental bits and Bottom of the Stack bit in the export field. yaf must have been enabled with MPLS support for export of this field. Note that this field is defined as an octet array in the default libfixbuf Information Model. yaf uses the length override feature in libfixbuf to redefine it from variable length to 3 bytes.

mplsLabelStackSection3, IE 72, 3 octets

The MPLS Label from the third entry in the MPLS label stack. yaf does not include the Experimental bits and Bottom of the Stack bit in the export field. yaf must have been enabled with MPLS support for export of this field. Note that this field is defined as an octet array in the default libfixbuf Information Model. yaf uses the length override feature in libfixbuf to redefine it from variable length to 3 bytes.

subTemplateMultiList IE 293, variable length

Represents a list of zero or more instances of a structured data type, where the data type of each list element can be different and corresponds with different template definitions. The Information Element Number will change upon updates to the IPFIX lists specification and libfixbuf releases.

TCP Flow Template

The following six Information Elements will be exported as a template within the subTemplateMultiList unless --silk is used.

tcpSequenceNumber IE 184, 4 octets, unsigned

Initial sequence number of the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseTcpSequenceNumber Reverse (PEN 29305) IE 184, 4 octets, unsigned

Initial sequence number of the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

initialTCPFlags CERT (PEN 6871) IE 14, 1 octet, unsigned

TCP flags of initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

unionTCPFlags CERT (PEN 6871) IE 15, 1 octet, unsigned

Union of TCP flags of all packets other than the initial packet in the forward direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP). This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseInitialTCPFlags CERT (PEN 6871) IE 16398, 1 octet, unsigned

TCP flags of initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

reverseUnionTCPFlags CERT (PEN 6871) IE 16399, 1 octet, unsigned

Union of TCP flags of all packets other than the initial packet in the reverse direction of the flow. Present if the flow's protocolIdentifier is 6 (TCP) and the flow has a reverse direction. This element is contained in the yaf TCP template within the subTemplateMultiList unless --silk is used.

MPTCP Template

The following five Information Elements will be exported as a template within the subTemplateMultiList if any MPTCP options are seen.

mptcpInitialDataSequenceNumber, CERT (PEN 6871) IE 289, 8 octets, unsigned

The initial data sequence number found in the MPTCP Data Sequence Signal (DSS) Option.

mptcpReceiverToken, CERT (PEN 6871) IE 290, 4 octets, unsigned

The token used to identify an MPTCP connection over multiple subflows. This value is found in the MP_JOIN TCP Option for the initial SYN of a subflow.

mptcpMaximumSegmentSize, CERT (PEN 6871) IE 291, 2 octets, unsigned

The maximum segment size reported in the Maximum Segment Size TCP Option. This should be consistent over all subflows.

mptcpAddressID, CERT (PEN 6871), IE 292, 1 octet, unsigned

The address ID of the subflow found in the SYN/ACK of an MP_JOIN operation.

mptcpFlags, CERT (PEN 6871), IE 293, 1 octet, unsigned

Various MPTCP Values:

MAC Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.

sourceMacAddress, IE 56, 6 octets, unsigned

Source MAC Address of the first packet in the forward direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.

destinationMacAddress, IE 80, 6 octets, unsigned

Destination MAC Address of the first packet in the reverse direction of the flow. This element is contained in the yaf MAC template within the subTemplateMultiList.

Payload Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.

payload CERT (PEN 6871) IE 18, variable-length

Initial n bytes of forward direction of flow payload. Present if payload collection is enabled and payload is present in the forward direction of the flow. This element is contained in the yaf Payload template within the subTemplateMultiList.

reversePayload CERT (PEN 6871) IE 16402, variable-length

Initial n bytes of reverse direction of flow payload. Present if payload collection is enabled and payload is present in the reverse direction of the flow. This element is contained in the yaf Payload template within the subTemplateMultiList.

Entropy Flow Template

The following two Information Elements will be exported as a template within the subTemplateMultiList.

payloadEntropy CERT (PEN 6871) IE 35, 1 octet, unsigned

Shannon Entropy calculation of the forward payload data. This element is contained in the yaf Entropy template within the subTemplateMultiList.

reversePayloadEntropy CERT (PEN 6871) IE 16419, 1 octet, unsigned

Shannon Entropy calculation of the reverse payload data. This element is contained in the yaf Entropy template within the subTemplateMultiList.

p0f Flow Template

The following six Information Elements will be exported as a template within the subTemplateMultiList if present and only if p0f is enabled.

osName CERT (PEN 6871) IE 36, variable-length

p0f OS Name for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsName CERT (PEN 6871) IE 16420, variable-length

p0f OS Name for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

osVersion CERT (PEN 6871) IE 37, variable-length

p0f OS Version for the forward flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsVersion CERT (PEN 6871) IE 16421, variable-length

p0f OS Version for the reverse flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

osFingerPrint CERT (PEN 6871) IE 107, variable-length

p0f OS Fingerprint for the forward flow based on the SYN packet and p0f SYN fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

reverseOsFingerPrint CERT (PEN 6871) IE 16491, variable-length

p0f OS Fingerprint for the reverse flow based on the SYN packet and p0f SYN Fingerprints. Present only if p0f is enabled. This element is contained in the yaf p0f template within the subTemplateMultiList.

Fingerprint Exporting Template

The following four Information Elements will be exported as a template within the subTemplateMultiList if present and only if fpexport is enabled.

firstPacketBanner CERT (PEN 6871) IE 38, variable-length

IP and transport headers for first packet in forward direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

reverseFirstPacketBanner CERT (PEN 6871) IE 16422, variable-length

IP and transport headers for first packet in reverse direction to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

secondPacketBanner CERT (PEN 6871) IE 39, variable-length

IP and transport headers for second packet in forward direction (third packet in sequence) to be used for external OS Fingerprinters. Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

reverseSecondPacketBanner CERT (PEN 6871) IE 16423, variable-length

IP and transport headers for second packet in reverse direction (currently not used). Present only if fpexport is enabled. This element is contained in the yaf FPExport template within the subTemplateMultiList.

Hooks Templates

yaf can export other templates within the subTemplateMultiList if plugins are enabled in yaf. See yafdpi(1) for descriptions of the yaf Deep Packet Inspection Information Elements. See yafdhcp(1) for descriptions of the DHCP Fingerprint Information Elements.

Flow Statistics Template

yaf can maintain and export more information about each flow than what is exported in the Basic Flow Template. If yaf is run with --flow-stats yaf will export the following attributes with every flow as long as one of the following characteristics is nonzero. The following flow attributes have been known to help in traffic classification.

dataByteCount CERT (PEN 6871) IE 502, 8 octets, unsigned

Total bytes transferred as payload.

averageInterarrivalTime CERT (PEN 6871) IE 503, 8 octets, unsigned

Average number of milliseconds between packets.

standardDeviationInterarrivalTime CERT (PEN 6871) IE 504, 8 octets, unsigned

Standard deviation of the interarrival time for up to the first ten packets.

tcpUrgTotalCount IE 223, 4 octets, unsigned

The number of TCP packets that have the URGENT Flag set.

smallPacketCount CERT (PEN 6871) IE 500, 4 octets, unsigned

The number of packets that contain less than 60 bytes of payload.

nonEmptyPacketCount CERT (PEN 6871) IE 501, 4 octets, unsigned

The number of packets that contain at least 1 byte of payload.

largePacketCount CERT (PEN 6871) IE 510, 4 octets, unsigned

The number of packets that contain at least 220 bytes of payload.

firstNonEmptyPacketSize CERT (PEN 6871) IE 505, 2 octets, unsigned

Payload length of the first non-empty packet.

maxPacketSize CERT (PEN 6871) IE 506, 2 octets, unsigned

The largest payload length transferred in the flow.

standardDeviationPayloadLength CERT (PEN 6871) IE 508, 2 octets, unsigned

The standard deviation of the payload length for up to the first 10 non empty packets.

firstEightNonEmptyPacketDirections CERT (PEN 6871) IE 507, 1 octet, unsigned

Represents directionality for the first 8 non-empty packets. 0 for forward direction, 1 for reverse direction.

reverseDataByteCount CERT (PEN 6871) IE 16886, 8 octets, unsigned

Total bytes transferred as payload in the reverse direction.

reverseAverageInterarrivalTime CERT (PEN 6871) IE 16887, 8 octets, unsigned

Average number of milliseconds between packets in reverse direction.

reverseStandardDeviationInterarrivalTime CERT (PEN 6871) IE 16888, 8 octets, unsigned

Standard deviation of the interarrival time for up to the first ten packets in the reverse direction.

reverseTcpUrgTotalCount Reverse (PEN 29305), IE 223, 4 octets, unsigned

The number of TCP packets that have the URGENT Flag set in the reverse direction.

reverseSmallPacketCount CERT (PEN 6871) IE 16884, 4 octets, unsigned

The number of packets that contain less than 60 bytes of payload in reverse direciton.

reverseNonEmptyPacketCount CERT (PEN 6871) IE 16885, 4 octets, unsigned

The number of packets that contain at least 1 byte of payload in reverse direction.

reverseLargePacketCount CERT (PEN 6871) IE 16894, 4 octets, unsigned

The number of packets that contain at least 220 bytes of payload in the reverse direction.

reverseFirstNonEmptyPacketSize CERT (PEN 6871) IE 16889, 2 octets, unsigned

Payload length of the first non-empty packet in the reverse direction.

reverseMaxPacketSize CERT (PEN 6871) IE 16890, 2 octets, unsigned

The largest payload length transferred in the flow in the reverse direction.

reverseStandardDeviationPayloadLength CERT (PEN 6871) IE 16892, 2 octets, unsigned

The standard deviation of the payload length for up to the first 10 non empty packets in the reverse direction.

Statistics Options Template

yaf will export information about its process periodically using IPFIX Options Template Record. This record gives information about the status of the flow and fragment table, as well as decoding information. This can be turned off using the --no-stats option. The following Information Elements will be exported:

systemInitTimeMilliseconds IE 161, 8 octets, unsigned

The time in milliseconds of the last (re-)initialization of yaf.

exportedFlowRecordTotalCount IE 42, 8 octets, unsigned

Total amount of exported flows from yaf start time.

packetTotalCount IE 86, 8 octets, unsigned

Total amount of packets processed by yaf from yaf start time.

droppedPacketTotalCount IE 135, 8 octets, unsigned

Total amount of dropped packets according to statistics given by libpcap, libdag, or the Napatech or Netronome APIs.

ignoredPacketTotalCount IE 164, 8 octets, unsigned

Total amount of packets ignored by the yaf packet decoder, such as unsupported packet types and incomplete headers, from yaf start time.

notSentPacketTotalCount IE 167, 8 octets, unsigned

Total amount of packets rejected by yaf because they were received out of sequence.

expiredFragmentCount CERT (PEN 6871) IE 100, 4 octets, unsigned

Total amount of fragments that have been expired since yaf start time.

assembledFragmentCount CERT (PEN 6871) IE 101, 4 octets, unsigned

Total number of packets that been assembled from a series of fragments since yaf start time.

flowTableFlushEventCount CERT (PEN 6871) IE 104, 4 octets, unsigned

Total number of times the yaf flow table has been flushed since yaf start time.

flowTablePeakCount CERT (PEN 6871) IE 105, 4 octets, unsigned

The maximum number of flows in the yaf flow table at any one time since yaf start time.

exporterIPv4Address IE 130, 4 octets, unsigned

The IPv4 Address of the yaf flow sensor.

exportingProcessId IE 144, 4 octets, unsigned

Set the ID of the yaf flow sensor by giving a value to --observation-domain. The default is 0.

meanFlowRate CERT (PEN 6871) IE 102, 4 octets, unsigned

The mean flow rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.

meanPacketRate CERT (PEN 6871) IE 103, 4 octets, unsigned

The mean packet rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.

Tombstone Options Template

yaf will export tombstone records periodically using IPFIX Options Template Records. These records are intended to allow the analysis of the time it takes for records to be processed by each tool (eg. YAF, Super Mediator, SiLK) in your environment. Each tombstone record generated by yaf consists of six information elements: the observation domain ID set by the --observation- domain argument, the exporting process ID which is the PID of the YAF process, a user- settable tombstone ID, a sequentially increasing "tombstoneID" for each record, the timestamp of the record's creation, and a subTemplateList of the time each program interacted with the tombstone record. With ideal randomness and/or proper user arguments, the 4 IDs taken together should uniquely specify a record. Tombstone records are only active when stats are active and can be individually turned off using the --no-tombstone option.

The following Information Elements will be exported:

observationDomainId IE 149, 4 octets, unsigned

The (user-set) observation domain of the YAF sensor.

exportingProcessId IE 144, 2 octets, unsigned

The PID of the YAF sensor.

exporterConfiguredId CERT (PEN 6871) IE 551, 2 octets, unsigned

An identification number for the record that is user specifiable at runtime and shared across all records in a run of the given program.

tombstoneId CERT (PEN 6871) IE 550, 4 octets, unsigned

A sequentially increasing identification number unique to each tombstone record in a run of a given program.

observationTimeSeconds IE 322, 4 octets, dateTimeSeconds

The UNIX timestamp of when the record was created.

tombstoneAccessList CERT (PEN 6871) IE 554, variable length, subTemplateList

A subTemplateList consisting of Tombstone Access Templates (see below) that specify when each program that supports tombstone times-tamping interacted with the tombstone record.

Tombstone Access Template

The following two Information Elements will be exported as a template within the subTemplateList of a Tombstone Options Template:

certToolId CERT (PEN 6871) IE 553, 4 octets, unsigned

The identification number of the program that interacted with the record. yaf has and ID of 1.

observationTimeSeconds IE 322, 4 octets, dateTimeSeconds

The UNIX timestamp of when the program interacted with the record.