Operation Switches

--union

Perform the set union operation: The resulting IPset contains each IP address that is a member of any of the input IPsets.

--intersect

Perform the set intersection operation: The resulting IPset contains each IP address that is a member of all of the input IPsets.

--difference

Perform the set difference operation: The resulting IPset contains each IP address that is a member of the first IPset and not a member of any subsequent IPsets.

--symmetric-difference

Perform the symmetric difference operation: For two input sets, the resulting IPset contains each IP address that is a member of one of the input IPsets but not both. For each additional IPset, rwsettool computes the symmetric difference of the current result with the additional IPset. For three input sets, the output IPset contains each IP address that is a member of either one of the IPsets or of all three IPsets. Since SiLK 3.13.0.

--sample

Select a random sample of IPs from the input IPsets. The size of the subset must be specified by either the --size or --ratio switches described next. In the case of multiple input IPsets, the resulting IPset is the union of all IP addresses sampled from each of the input IPsets. That is, each IPset is individually sampled, and the results are merged.

--mask=NET_BLOCK_SIZE

Perform a (sparse) masking operation: The resulting IPset contains one IP address for each /NET_BLOCK_SIZE CIDR block in the input IPset(s) that contains one or more IP addresses in that CIDR block. That is, rwsettool visits each /NET_BLOCK_SIZE CIDR block in the IPset. If the block is empty, no change is made; otherwise the block is cleared (all IPs removed) and the lowest IP address in that block is made a member of the set. NET_BLOCK_SIZE should be value between 1 and 32 for IPv4 sets and between 1 and 128 for IPv6 sets. Contrast with --fill-blocks.

--fill-blocks=NET_BLOCK_SIZE

Perform a (non-sparse) masking operation: The resulting IPset contains a completely full /NET_BLOCK_SIZE block for each /NET_BLOCK_SIZE CIDR block in the input IPset(s) that contain one or more IP addresses in that CIDR block. That is, rwsettool visits each /NET_BLOCK_SIZE CIDR block in the IPset; if the block is empty, no change is made, otherwise all IP addresses in the block are made members of the set. NET_BLOCK_SIZE should be value between 1 and 32 for IPv4 sets and between 1 and 128 for IPv6 sets. Contrast with --mask.

Output Switches

--output-path=PATH

Write the resulting IPset to PATH, where PATH is a filename, a named pipe, the keyword "stderr" to write the output to the standard error, or the keyword "stdout" or "-" to write the output to the standard output. If PATH names an existing file, rwsettool exits with an error unless the SILK_CLOBBER environment variable is set, in which case PATH is overwritten. If this switch is not given, the output is written to the standard output. Attempting to write the binary output to a terminal causes rwsettool to exit with an error.

--record-version=VERSION

Specify the format of the IPset records that are written to the output. VERSION may be 2, 3, 4, 5 or the special value 0. When the switch is not provided, the SILK_IPSET_RECORD_VERSION environment variable is checked for a version. The default version is 0.

--invocation-strip

Do not record any command line history; that is, do not copy the invocation history from the input files to the output file, and do not record the current command line invocation in the output.

--note-strip

Do not copy the notes (annotations) from the input files to the output file. Normally notes from the input files are copied to the output.

--note-add=TEXT

Add the specified TEXT to the header of the output file as an annotation. This switch may be repeated to add multiple annotations to a file. To view the annotations, use the rwfileinfo(1) tool.

--note-file-add=FILENAME

Open FILENAME and add the contents of that file to the header of the output file as an annotation. This switch may be repeated to add multiple annotations. Currently the application makes no effort to ensure that FILENAME contains text; be careful that you do not attempt to add a SiLK data file as an annotation.

--compression-method=COMP_METHOD

Specify the compression library to use when writing output files. If this switch is not given, the value in the SILK_COMPRESSION_METHOD environment variable is used if the value names an available compression method. When no compression method is specified, output to the standard output or to named pipes is not compressed, and output to files is compressed using the default chosen when SiLK was compiled. The valid values for COMP_METHOD are determined by which external libraries were found when SiLK was compiled. To see the available compression methods and the default method, use the --help or --version switch. SiLK can support the following COMP_METHOD values when the required libraries are available.

Additional Switches

--help

Print the available options and exit.

--version

Print the version number and information about how SiLK was configured, then exit the application.

Set Union Examples

 +---------------------------------+----------------------------+
 | OPTIONS                         | RESULT                     |
 +---------------------------------+----------------------------+
 | --union A.set B.set             | { 1, 2, 3, 4, 5, 6, 7 }    |
 | --union A.set C.set             | { 1, 2, 3, 4, 6, 8 }       |
 | --union A.set B.set C.set       | { 1, 2, 3, 4, 5, 6, 7, 8 } |
 | --union C.set D.set             | { 1, 3, 6, 8 }             |
 | --union A.set                   | { 1, 2, 4, 6 }             |
 | --union A.set A.set             | { 1, 2, 4, 6 }             |
 +---------------------------------+----------------------------+

Set Intersection Examples

 +---------------------------------+----------------------------+
 | OPTIONS                         | RESULT                     |
 +---------------------------------+----------------------------+
 | --intersect A.set B.set         | { 1 }                      |
 | --intersect A.set C.set         | { 1, 6 }                   |
 | --intersect B.set C.set         | { 1, 3 }                   |
 | --intersect A.set B.set C.set   | { 1 }                      |
 | --intersect A.set D.set         | { }                        |
 | --intersect A.set               | { 1, 2, 4, 6 }             |
 | --intersect A.set A.set         | { 1, 2, 4, 6 }             |
 +---------------------------------+----------------------------+

Set Difference Examples

 +---------------------------------+----------------------------+
 | OPTIONS                         | RESULT                     |
 +---------------------------------+----------------------------+
 | --difference A.set B.set        | { 2, 4, 6 }                |
 | --difference B.set A.set        | { 3, 5, 7 }                |
 | --difference A.set B.set C.set  | { 2, 4 }                   |
 | --difference C.set B.set A.set  | { 8 }                      |
 | --difference C.set D.set        | { 1, 3, 6, 8 }             |
 | --difference D.set C.set        | { }                        |
 | --difference A.set              | { 1, 2, 4, 6 }             |
 | --difference A.set A.set        | { }                        |
 +---------------------------------+----------------------------+

Set Symmetric Difference Examples

 +---------------------------------+----------------------------+
 | OPTIONS                         | RESULT                     |
 +---------------------------------+----------------------------+
 | --symmetric A.set B.set         | { 2, 3, 4, 5, 6, 7 }       |
 | --symmetric A.set C.set         | { 2, 3, 4, 8 }             |
 | --symmetric A.set D.set         | { 1, 2, 4, 6 }             |
 | --symmetric C.set B.set         | { 5, 6, 7, 8 }             |
 | --symmetric A.set B.set C.set   | { 1, 2, 4, 5, 7, 8 }       |
 | --symmetric A.set               | { 1, 2, 4, 6 }             |
 | --symmetric A.set A.set         | { }                        |
 +---------------------------------+----------------------------+

Finding IP Addresses Unique to an Input Set

1.

First, use rwbagbuild(1) to create an empty bag file /tmp/b.bag.

 $ echo "" | rwbagbuild --bag-input=stdin --output-path=/tmp/b.bag
    

2.

For each input IPset, i.set, use rwbagbuild to create a bag from the IPset, and use rwbagtool(1) to add that bag to b.bag.

 $ rwbagbuild --set-input=i.set   \
   | rwbagtool --add - /tmp/b.bag --output-path=/tmp/b2.bag
 $ mv /tmp/b2.bag /tmp/b.bag
    

To do that in a loop, run

 $ for i in *.set ; do \
         rwbagbuild --set-input=$i  \
         | rwbagtool --add - /tmp/b.bag --output-path=/tmp/b2.bag ; \
         mv /tmp/b2.bag /tmp/b.bag ; \
   done
    

3.

Use rwbagtool to create a coverset named unique.set that contains the IP addresses in b.bag whose counter is 1.

 $ rwbagtool --maxcounter=1 --coverset --output-path=unique.set \
         /tmp/b.bag
    

A different approach may be used which does not require temporary files. Use rwsetcat(1) to convert the IPset files to text and feed that data to rwbagbuild. (When rwsetcat is invoked on multiple IPset files, it prints the contents of each individual IPset file, and as rwbagbuild processes the text, it increments an IP address's counter each time the IP appears in the input.) Use rwbagtool to create the IPset as shown in Step 3 above.

 $ rwsetcat --cidr-blocks=1 *.set   \
   | rwbagbuild --bag-input=-       \
   | rwbagtool  --maxcounter=1 --coverset --output=unique.set

Set Sampling Examples

The --size switch selects exactly SIZE IP addresses from each input set, but the number of IP addresses in the result may be less than the product of SIZE and the number of inputs when the input sets have IPs in common or when an IPset has fewer than SIZE members.

When using the --size switch, the probability of selecting an individual IP address varies with the number of IPs to be selected and the number of IPs remaining in the set. If N is the number of IPs in a set, the probability of selecting the first IP is SIZE/N. If that IP is selected, the probability of selecting the second is (SIZE-1)/(N-1), but if the first IP is not selected, the probability of selecting the second is SIZE/(N-1).

 +----------------------------------+----------------------------+
 | COMMAND                          | RESULT                     |
 +----------------------------------+----------------------------+
 | --sample --size 2 A.set          | { 1, 4 }                   |
 | --sample --size 2 A.set          | { 1, 6 }                   |
 | --sample --size 3 A.set          | { 2, 4, 6 }                |
 | --sample --size 2 A.set B.set    | { 1, 2, 5, 7 }             |
 | --sample --size 2 A.set B.set    | { 3, 4, 5, 6 }             |
 | --sample --size 2 A.set B.set    | { 1, 4, 5 }                |
 +----------------------------------+----------------------------+

The argument to the --ratio switch is the probability of choosing an individual IP address. For each IP address in the input, the IP is added to the output when a pseudo-random number between 0 and 1 is less then the argument to --ratio. The number of IP addresses in the result varies with each invocation.

 +----------------------------------+----------------------------+
 | COMMAND                          | RESULT                     |
 +----------------------------------+----------------------------+
 | --sample --ratio 0.5 A.set       | { 2, 6 }                   |
 | --sample --ratio 0.5 A.set       | { 4 }                      |
 | --sample --ratio 0.5 A.set B.set | { 1, 3 }                   |
 | --sample --ratio 0.5 A.set B.set | { 2, 3, 5, 6, 7 }          |
 +----------------------------------+----------------------------+

Set Masking and Block-Filling Examples

The --mask and --fill-blocks switches require a decimal argument that is a CIDR-block network mask size. For example, the argument 24 represents 256 IPv4 addresses. rwsettool visits each block of that size in the input IPset. If no IP addresses appear in that block, the result also has no IPs in the block. If one or more IP addresses appear in that block, the output IPset has either the lowest address in that block as a member (for --mask) or all IP addresses in that block as members (for --fill-blocks.

For example, consider the IPset s.set containing the three IP addresses.

 $ rwsetcat --cidr-blocks=1 s.set
 10.1.1.1
 10.1.1.2
 10.1.3.1

Specifying --mask=24 produces an IPset containing two IP addresses.

 $ rwsettool --mask=24 s.set | rwsetcat --cidr-blocks=1
 10.1.1.0
 10.1.3.0

Specifying --fill-blocks=24 produces an IPset containing 512 IP addresses.

 $ rwsettool --fill-block=24 s.set | rwsetcat --cidr-blocks=1
 10.1.1.0/24
 10.1.3.0/24

Consider t.set that contains four IP addresses.

 $ rwsetcat --cidr-blocks=1 t.set
 10.1.1.1
 10.1.1.2
 10.1.2.5
 10.1.3.1

Running --mask=24 and --fill-blocks=24 on that file produces the following.

 $ rwsettool --mask=24 t.set | rwsetcat --cidr-blocks=1
 10.1.1.0
 10.1.2.0
 10.1.3.0

 $ rwsettool --fill-block=24 t.set | rwsetcat --cidr-blocks=1
 10.1.1.0/24
 10.1.2.0/23

rwsetcat merges 10.1.2.0/24 and 10.1.3.0.24 into a single /23.

When multiple IPsets are specified on the command line, the union of the IPsets is computed prior to performing the mask or fill-blocks operation. The result is not dependent on the order of the IPsets.

Mixed IPv4 and IPv6 Examples

 $ echo '::ffff:0:0/96' | rwsetbuild - all-v4.set

 $ rwsettool --intersect mixed.set all-v4.set > subset-v4.set

To create an IPset file that contains only the IPv6 addresses, subtract all-v4.set from mixed.set:

 $ rwsettool --difference mixed.set all-v4.set > subset-v6.set

The previous two commands may also be performed without having to write create the all-v4.set IPset file.

 $ echo '::ffff:0:0/96'   \
   | rwsettool --intersect mixed.set - > subset-v4.set

 $ echo '::ffff:0:0/96'   \
   | rwsettool --difference mixed.set - > subset-v6.set

Comparing Two IPsets Example

 $ cp A.set A2.set
 $ rwsettool --symmetric-difference A.set A2.set  \
   | rwsetcat --count
 0

Changing a File's Format

It is not possible to change those aspects of the file directly. A new file must be created first, and then you may then replace the old file with the new file.

To create a new file that uses a different record-version or compression-method of the IPset file A.set, use rwsettool with the --union switch and specify the desired arguments:

 $ rwsettool --union --record-version=5 --output-path=A2.set A.set

 $ rwsettool --union --compression=none --output-path=A3.set A.set

 $ rwsettool --union --record-version=2 --compression=best \
        --output-path=A4.set A.set