|
NAMEdnstable-encoding - key-value encoding format used by dnstableDESCRIPTIONdnstable has been designed to import the data generated by the nmsg-dns-cache program in the NMSG SIE/dnsdedupe format. See the ISC Passive DNS Architecture whitepaper for details about how this data is actually generated. The data model used with the aid of this format preserves a number of important properties like Resource Record type transparency and Resource Record set atomicity. Additionally, dnstable preserves extra metadata available in the dnsdedupe payloads.dnstable makes use of the MTBL Sorted String Table (SSTable) format, which provides an immutable mapping of keys to optional values. This format allows fast lookups of key prefixes, so the precise encoding and layout of dnstable keys are chosen to take advantage of this property. MTBL keys and values are arbitrary byte arrays, and because MTBL keys are stored in sorted order it is possible to look up entire key-value entries by searching for the initial part of a key. The first byte of each dnstable key is a “type” byte, and there are currently six different “types” of dnstable key-value entries which are described below. These different types are used to encode the full passive DNS payload as well as “indices” into the payload and metadata. Each dnstable key is a compound key that packs individual fields together, and for the key types that include a value, the value is a compound value that also packs individual fields together. In the string literals given in the text below, C-style "\xNN" hex encoding may be used to represent particular byte values. For example, "\x03" is an octet with decimal value 3. Note that this is different than the \DDD encoding of arbitrary decimal values in DNS names. Key-value entry typesAll keys and values described below are byte arrays containing packed fields. Each field is concatenated together without delimiters. The following data types are used:•Byte.
•Fixed-width 16 bit integer in little-endian
order.
•Variable-width integer.
•Wire-format DNS record data.
•Wire-format DNS domain names. The domain name
"www.example.com" would be encoded as
"\x03www\x07example\x03com\x00".
•Label-reversed wire-format DNS domain names. The
domain name "www.example.com" would be encoded as
"\x03com\x07example\x03www\x00".
•RRtype union. This can contain one of multiple
types, depending upon the length of the value.
•If the length is 0, then it is interpreted the
same as if all defined RRtype bits were set.
•If the length is 1, then the value is an RRtype
valued from 0 to 255.
•If the length is 2, then the value is an RRtype
valued from 256 to 65535, in little-endian order.
•If the length is 3 or more, then the value is an
RRtype bitmap, as defined in RFC4034 section 4.1.2.
•Examples of RFC4034 bitmaps:
•RRtypes A, NS, and SOA are encoded as bitmap
00 01 62.
•RRtypes A, NS, NAPTR, SPF, SOA, MX, TXT, AAAA,
and LOC are encoded as bitmap 00 0d 62 01 80 0c 10 00 00 00 00 00 00 00
10.
ENTRY_TYPE_RRSET
This entry type encodes the complete passive DNS payload. The RRset owner name is encoded at the beginning of the key so that “forward” lookups can be performed on the owner name. The owner name is encoded with labels reversed from the usual DNS order to increase locality and allow for wildcard matches. (For example, "*.example.com".) The key is a packed tuple consisting of the following fields, in order: 1.Type byte. The constant "\x00".
2.RRset owner name. Label-reversed wire-format
DNS domain name.
3.RRset type. Variable-width integer.
4.Bailiwick domain name. Label-reversed
wire-format DNS domain name.
5.Rdata array. An array of one or more
wire-format DNS record data values. Each record data value is preceded by its
length, encoded as a variable-width integer.
The value consists of the following fields, in order: 1.First timestamp. Variable-width integer. Number
of seconds since the epoch.
2.Last timestamp. Variable-width integer. Number
of seconds since the epoch.
3.Count. Variable-width integer. Number of
observations between the first and last timestamps.
ENTRY_TYPE_RRSET_NAME_FWD
This entry type encodes only the RRset owner name in the usual “forward” direction. This entry type provides an “index” into the ENTRY_TYPE_RRSET “table” so that RRsets whose owner names start with a particular subsequence of labels can be found. (For example, looking up the label subsequence "www.example.*" using this index would reveal the existence of the domain names "www.example.com", "www.example.net", etc.) The key consists of the following fields, in order: 1.Type byte. The constant "\x01".
2.RRset owner name. Wire-format DNS domain
name.
In versions before 0.12.0, there was no value associated with this entry type. As of that version, the value is an RRtype union described above. ENTRY_TYPE_RDATA
This entry type encodes individual Resource Record data values; that is, an RRset containing multiple Resource Records will be encoded as multiple entries in this “table”. The key consists of the following fields, in order: 1.Type byte. The constant "\x02".
2.Rdata. Wire-format DNS record data.
3.RR type. Variable-width integer.
4.RR owner name. Label-reversed wire-format DNS
domain name.
5.Rdata length. Fixed-width 16 bit integer in
little-endian order. Number of bytes in the Rdata field.
The Rdata length field is encoded at the end of the key so that searches can be performed over the Rdata field without needing to know the length of the record data value in advance (or worse, needing to iterate over all possible length values). An additional ENTRY_TYPE_RDATA entry is produced for DNS records which encode a domain name but do not begin with one (i.e. DNS records which contain a DNS domain name offset into the latter part of the record data): •An MX record contains a domain name offset 2
octets into the record data value.
•An SVCB record contains a domain name offset 2
octets into the record data value.
•An HTTPS record contains a domain name offset 2
octets into the record data value.
•A SRV record contains a domain name offset 6
octets into the record data value.
If those record types had been laid out in the applicable DNS RFCs with the domain name at the beginning of the record data the following hack would not be necessary in order to make these record types easily searchable by domain, or for records of different rrtypes that reference the same domains to cluster together. This sub-type “slices” the record data so that the domain name in the latter part of the record appears at the beginning of the key and the initial bytes of the record appear just before the Rdata length field at the end of the key. That is, the key consists of the following fields, in order: 1.Type byte. The constant "\x02".
2.Latter data slice. Wire-format DNS record data.
This is the latter part of the record data.
3.RR type. Variable-width integer.
4.RR owner name. Label-reversed wire-format DNS
domain name.
5.Initial data slice. Wire-format DNS record
data. This is the initial part of the record data.
6.Latter data slice length. Fixed-width 16 bit
integer in little-endian order. Number of bytes in the latter data
slice.
The number of bytes in the initial data slice is implicit (the difference between the dnstable entry value length and the sum of the sizes of all previously described mandatory fields, once decoded.) An entry can be determined to be either a sliced or un-sliced record by whether or not this implicit initial data slice length is non-zero or zero, respectively. The value consists of the following fields, in order: 1.First timestamp. Variable-width integer. Number
of seconds since the epoch.
2.Last timestamp. Variable-width integer. Number
of seconds since the epoch.
3.Count. Variable-width integer. Number of
observations between the first and last timestamps.
This is the same encoding as the values for ENTRY_TYPE_RRSET entries. ENTRY_TYPE_RDATA_NAME_REV
This entry type encodes label-reversed domain names from the record data value for certain DNS record types that contain a domain name in the record data, specifically the SOA, NS, CNAME, DNAME, PTR, MX, SRV, SVCB, and HTTPS DNS record types. The "slicing" logic described above in ENTRY_TYPE_RDATA encompasses the extraction of these domain names. The key consists of the following fields, in order: 1.Type byte. The constant "\x03".
2.Rdata name. Label-reversed wire-format DNS
domain name.
In versions before 0.12.0, there was no value associated with this entry type. As of that version, the value is an RRtype union described above. ENTRY_TYPE_TIME_RANGE
This entry type encodes the overall time range of all ENTRY_TYPE_RRSET and ENTRY_TYPE_RDATA entries in a dnstable data set, i.e. the earliest first timestamp and latest last timestamp of all such entries. The key consists of: 1.Type byte. The constant "\xFE".
The value consists of the following fields, in order: 1.First timestamp. Variable-width integer. Number
of seconds since the epoch.
2.Last timestamp. Variable-width integer. Number
of seconds since the epoch.
ENTRY_TYPE_VERSION
This entry encodes a version number for the format of another entry type. Each of the above entry types, with the exception of ENTRY_TYPE_TIME_RANGE may have an associated version number entry. The key consists of the following fields, in order: 1.Type byte. The constant "0xFF".
2.Versioned Type. The Type byte to which this
entry applies.
The value consists of: 1.Version number. Variable-width integer.
EXAMPLESThis section gives two example RRsets and shows how they would be encoded using the above entry types. The first example is an NS delegation RRset for the domain name "example.com" and consists of two Resource Records; it will result in the generation of six key-value entries. The second example is an A-record for the domain name "www.isc.org" that consists of a single Resource Record; it will result in the generation of three key-value entries.In each example entry, a table of the broken down fields in each entry is given, and then the final, concatenated key and value byte strings. Example 1: example.com./NSGiven the following passive DNS payload:
The following entries will be generated: example.com entry #1 (ENTRY_TYPE_RRSET)
Key:
Value:
Final key: "\x00\x03com\x07example\x00\x02\x03com\x00\x11\x03ns1\x07example\x03com\x00\x11\x03ns2\x07example\x03com\x00" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x17" example.com entry #2 (ENTRY_TYPE_RRSET_NAME_FWD)
Key:
Value:
Final key: "\x01\x07example\x03com\x00" Final value: "\x02" example.com entry #3 (ENTRY_TYPE_RDATA)
Key:
Value:
Final key: "\x02\x03ns1\x07example\x03com\x00\x02\x03com\x07example\x00\x00\x11" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x17" example.com entry #4 (ENTRY_TYPE_RDATA)
Key:
Value:
Final key: "\x02\x03ns2\x07example\x03com\x00\x02\x03com\x07example\x00\x00\x11" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x17" example.com entry #5 (ENTRY_TYPE_RDATA_NAME_REV)
Key:
Value:
Final key: "\x03\x03com\x07example\x03ns1\x00" Final value: "\x02" example.com entry #6 (ENTRY_TYPE_RDATA_NAME_REV)
Key:
Value:
Final key: "\x03\x03com\x07example\x03ns2\x00" Final value: "\x02" Example 2: www.isc.org./AGiven the following passive DNS payload:
The following entries will be generated: www.isc.org entry #1 (ENTRY_TYPE_RRSET)
Key:
Value:
Final key: "\x00\x03org\x03isc\x03www\x01\x03org\x03isc\x00\x04\x95\x14\x40\x2a" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x01" www.isc.org entry #2 (ENTRY_TYPE_RRSET_NAME_FWD)
Key:
Value:
Final key: "\x01\x03www\x03isc\x03org\x00" Final value: "\x01" www.isc.org entry #3 (ENTRY_TYPE_RDATA)
Key:
Value:
Final key: "\x02\x95\x14\x40\x2a\x01\x03osc\x03isc\x03www\x00\x00\x04" Final value: "\x90\xb9\xe6\xfb\x04\xa0\x87\xe7\xfb\x04\x01" SEE ALSOISC Passive DNS Architecture, https://www.farsightsecurity.com/assets/media/download/passive-dns-architecture.pdfGoogle Protocol Buffers Encoding, https://developers.google.com/protocol-buffers/docs/encoding, for the base-128 variable-width integer encoding used in some of the fields in dnstable keys. Also see mtbl_varint(3) for a varint encoding implementation.
Visit the GSP FreeBSD Man Page Interface. |