Supporting yaf statistics (Option Templates)

[flexd]

I see here https://github.com/calmh/ipfix/blob/master/parser.go#L296 that option templates are not handled at all.

I am not very familiar with the IPFIX format. How hard would it be to implement support for Option Templates?

I am trying to grab statistics from yaf flow files. Relevant docs embedded below in this post, see https://tools.netsa.cert.org/yaf/yaf.html and search for "Statistics Option Template" for the source.

I tried contacting you on the gophers slack to talk, but I figured I'd post here as well :)

Statistics Option Template
yaf will export information about its process periodically using IPFIX Options Template Record. This record gives information about the status of the flow and fragment table, as well as decoding information. This can be turned off using the --no-stats option. The following Information Elements will be exported:

systemInitTimeMilliseconds IE 161, 8 octets, unsigned
The time in milliseconds of the last (re-)initialization of yaf.

exportedFlowRecordTotalCount IE 42, 8 octets, unsigned
Total amount of exported flows from yaf start time.

packetTotalCount IE 86, 8 octets, unsigned
Total amount of packets processed by yaf from yaf start time.

droppedPacketTotalCount IE 135, 8 octets, unsigned
Total amount of dropped packets according to statistics given by libpcap, libdag, or the Napatech or Netronome APIs.

ignoredPacketTotalCount IE 164, 8 octets, unsigned
Total amount of packets ignored by the yaf packet decoder, such as unsupported packet types and incomplete headers, from yaf start time.

notSentPacketTotalCount IE 167, 8 octets, unsigned
Total amount of packets rejected by yaf because they were received out of sequence.

expiredFragmentCount CERT (PEN 6871) IE 100, 4 octets, unsigned
Total amount of fragments that have been expired since yaf start time.

assembledFragmentCount CERT (PEN 6871) IE 101, 4 octets, unsigned
Total number of packets that been assembled from a series of fragments since yaf start time.

flowTableFlushEventCount CERT (PEN 6871) IE 104, 4 octets, unsigned
Total number of times the yaf flow table has been flushed since yaf start time.

flowTablePeakCount CERT (PEN 6871) IE 105, 4 octets, unsigned
The maximum number of flows in the yaf flow table at any one time since yaf start time.

exporterIPv4Address IE 130, 4 octets, unsigned
The IPv4 Address of the yaf flow sensor.

exportingProcessId IE 144, 4 octets, unsigned
Set the ID of the yaf flow sensor by giving a value to --observation-domain. The default is 0.

meanFlowRate CERT (PEN 6871) IE 102, 4 octets, unsigned
The mean flow rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.

meanPacketRate CERT (PEN 6871) IE 103, 4 octets, unsigned
The mean packet rate of the yaf flow sensor since yaf start time, rounded to the nearest integer.

[calmh]

Yeah, not active on slack. I'm not sure about options templates either, never had a use for them. You're welcome to investigate.

[flexd]

I'm looking into it, but I have never done any kind of parsing like this. It does not look that complicated, and the format is well documented in the RFC.

Looks like there are two types of messages, that either contain a Template Set and a Data Set, or a Option Template Set and a Data set? Or could a IPFIX Message contain any number of these combinations?

   IPFIX Message:

    +--------+------------------------------------------. . .
    |        | +--------------+ +------------------+
    |Message | | Template     | | Data             |
    | Header | | Set          | | Set              |   . . .
    |        | | (1 Template) | | (3 Data Records) |
    |        | +--------------+ +------------------+
    +--------+------------------------------------------. . .

         . . .-------------------------------------------+
               +------------------+ +------------------+ |
               | Options          | | Data             | |
        . . .  | Template Set     | | Set              | |
               | (1 Template)     | | (2 Data Records) | |
               +------------------+ +------------------+ |
         . . .-------------------------------------------+

The Options Template Set format is also described

 The example in Figure O shows an Options Template Set with mixed
   IANA-assigned and enterprise-specific Information Elements.  It
   consists of a Set Header, an Options Template Header, and several
   Field Specifiers.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |          Set ID = 3           |          Length               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |         Template ID = 258     |         Field Count = N + M   |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Scope Field Count = N     |0|  Scope 1 Infor. Element id. |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Scope 1 Field Length      |0|  Scope 2 Infor. Element id. |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Scope 2 Field Length      |             ...               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |            ...                |1|  Scope N Infor. Element id. |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Scope N Field Length      |   Scope N Enterprise Number  ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    ...  Scope N Enterprise Number   |1| Option 1 Infor. Element id. |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |    Option 1 Field Length      |  Option 1 Enterprise Number  ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    ... Option 1 Enterprise Number   |              ...              |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |             ...               |0| Option M Infor. Element id. |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Option M Field Length     |      Padding (optional)       |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

And with some examples from the RFC

A.4.  Options Template Set Examples

A.4.1.  Options Template Set Using IANA Information Elements

   Per line card (the router being composed of two line cards), we want
   to report the following Information Elements:

   - Total number of IPFIX Messages: exportedMessageTotalCount
     [IANA-IPFIX], with a length of 2 octets

   - Total number of exported Flows: exportedFlowRecordTotalCount
     [IANA-IPFIX], with a length of 2 octets

   The line card, which is represented by the lineCardId Information
   Element [IANA-IPFIX], is used as the Scope Field.

   Therefore, the Options Template Set will be:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |         Set ID = 3            |          Length = 24          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Template ID 258         |        Field Count = 3        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Scope Field Count = 1     |0|     lineCardId = 141        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |   Scope 1 Field Length = 4    |0|exportedMessageTotalCount=41 |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Field Length = 2        |0|exportedFlowRecordTotalCo.=42|
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Field Length = 2        |           Padding             |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

A.4.2.  Options Template Set Using Enterprise-Specific Information
        Elements

   Per line card (the router being composed of two line cards), we want
   to report the following Information Elements:

   - Total number of IPFIX Messages: exportedMessageTotalCount
     [IANA-IPFIX], with a length of 2 octets

   - An enterprise-specific number of exported Flows, with a type of 42
     and a length of 4 octets

   The line card, which is represented by the lineCardId Information
   Element [IANA-IPFIX], is used as the Scope Field.
The format of the Options Template Set is as follows:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |         Set ID = 3            |          Length = 28          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Template ID 259         |        Field Count = 3        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Scope Field Count = 1     |0|     lineCardId = 141        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |   Scope 1 Field Length = 4    |0|exportedMessageTotalCount=41 |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Field Length = 2        |1|Information Element id. = 42 |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Field Length = 4        |       Enterprise number      ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    ...       Enterprise number      |           Padding             |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

A.4.3.  Options Template Set Using an Enterprise-Specific Scope

   In this example, we want to export the same information as in the
   example in Appendix A.4.1:

   - Total number of IPFIX Messages: exportedMessageTotalCount
     [IANA-IPFIX], with a length of 2 octets

   - Total number of exported Flows: exportedFlowRecordTotalCount
     [IANA-IPFIX], with a length of 2 octets

   But this time, the information pertains to a proprietary scope,
   identified by enterprise-specific Information Element number 123.

The format of the Options Template Set is now as follows:

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |         Set ID = 3            |          Length = 28          |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Template ID 260         |        Field Count = 3        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Scope Field Count = 1     |1|Scope 1 Infor. El. id. = 123 |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |    Scope 1 Field Length = 4   |       Enterprise Number      ...
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    ...       Enterprise Number      |0|exportedMessageTotalCount=41 |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Field Length = 2        |0|exportedFlowRecordTotalCo.=42|
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |       Field Length = 2        |           Padding             |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

[flexd]

A Data Set is composed of Data Record(s).  No Template Record is
   included.  A Template Record or an Options Template Record defines
   the Data Record.

   A Template Set contains only Template Record(s).

   An Options Template Set contains only Options Template Record(s).

Following are some examples of IPFIX Messages:

   1. An IPFIX Message consisting of interleaved Template, Data, and
      Options Template Sets, as shown in Figure C.  Here, Template and
      Options Template Sets are transmitted "on demand", before the
      first Data Set whose structure they define.

     +--------+--------------------------------------------------------+
     |        | +----------+ +---------+     +-----------+ +---------+ |
     |Message | | Template | | Data    |     | Options   | | Data    | |
     | Header | | Set      | | Set     | ... | Template  | | Set     | |
     |        | |          | |         |     | Set       | |         | |
     |        | +----------+ +---------+     +-----------+ +---------+ |
     +--------+--------------------------------------------------------+

                     Figure C: IPFIX Message: Example 1

   2. An IPFIX Message consisting entirely of Data Sets, sent after the
      appropriate Template Records have been defined and transmitted to
      the Collecting Process, as shown in Figure D.

       +--------+----------------------------------------------+
       |        | +---------+     +---------+      +---------+ |
       |Message | | Data    |     | Data    |      | Data    | |
       | Header | | Set     | ... | Set     | ...  | Set     | |
       |        | +---------+     +---------+      +---------+ |
       +--------+----------------------------------------------+

                    Figure D: IPFIX Message: Example 2

   3. An IPFIX Message consisting entirely of Template and Options
      Template Sets, as shown in Figure E.  Such a message can be used
      to define or redefine Templates and Options Templates in bulk.

      +--------+-------------------------------------------------+
      |        | +----------+     +----------+      +----------+ |
      |Message | | Template |     | Template |      | Options  | |
      | Header | | Set      | ... | Set      | ...  | Template | |
      |        | |          |     |          |      | Set      | |
      |        | +----------+     +----------+      +----------+ |
      +--------+-------------------------------------------------+

I'm kind of using this issue as a notepad for useful information about it. At some point I'll understand this enough to implement it, and/or someone else is capable and willing.