RFC: pattern template support

[ihrwein]

In my very first proposal I mentioned the pattern reuse problem: there are similar patterns and we want to gather the common parts into one pattern which can be used by the others:

{                                                                                                                                                
    "patterns": [                                                                                                                                
        {                                                                                                                                        
            "pattern": "loading configuration from '%{GREEDY:file}'",                                                                            
            "name": "bind9",                                                                                                                     
            "uuid": "db36c4bf-0310-41d4-b59a-1dfe1929c4da"                                                                                       
        },                                                                                                                                       
        {                                                                                                                                        
            "pattern": "general: loading configuration from '%{GREEDY:file}'",                                                                   
            "name": "bind9",                                                                                                                     
            "uuid": "db36c4bf-0310-41d4-b59a-1dfe1929c4da"                                                                                       
        }                                                                                                                                        
    ]                                                                                                                                            
} 

The loading configuration from '%{GREEDY:file}' part is common in both pattern definitions.

The proposed syntax is the following:

{                                                                                                                                                
    "patterns": [                                                                                                                                
        {                                                                                                                                        
            "pattern": "%{BIND9_LOADING_CONFIG}",                                                                                                
            "name": "bind9",                                                                                                                     
            "uuid": "db36c4bf-0310-41d4-b59a-1dfe1929c4da"                                                                                       
        },                                                                                                                                       
        {                                                                                                                                        
            "pattern": "general: %{BIND9_LOADING_CONFIG}",                                                                                       
            "name": "bind9",                                                                                                                     
            "uuid": "db36c4bf-0310-41d4-b59a-1dfe1929c4da"                                                                                       
        }                                                                                                                                        
    ],                                                                                                                                           
    "templates": [                                                                                                                               
        {                                                                                                                                        
            "pattern": "loading configuration from '%{GREEDY:appacct.conf_file}'",                                                               
            "name": "BIND9_LOADING_CONFIG",                                                                                                      
            "test_messages": [                                                                                                                   
                {                                                                                                                                
                    "message": "loading configuration from '/etc/bind/unctions.overreact.tld'",                                                  
                    "values": {                                                                                                                  
                        "appacct.conf_file": "/etc/bind/unctions.overreact.tld"                                                                  
                    }                                                                                                                            
                }                                                                                                                                
            ]                                                                                                                                    
        }                                                                                                                                        
    ]                                                                                                                                            
}

At first I'd support only the SET, INT, GREEDY parsers in templates. If that's not enough we may enable cross references between templates, but we have to care about cycles in the graph.

I'd support the following attributes in pattern templates:

• name: it's mandatory
• pattern: it's mandatory
• tags: optional tags (copied into the referencing pattern)
• values: optional values (copied into the referencing pattern)
• test_messages: optional test messages. They aren't copied into the pattern, because the message wouldn't match every time.

Pattern templates doesn't have uuid:

• user would expect that a pattern can inherit it: but what if they forgot to include the uuid in the template?
• pattern templates are referenced by their names

I'm not sure about the "pattern template" expression. Maybe the user defined parser or user defined template would be a better option.

[ihrwein]

@faxm0dem @lbudai: what do you think about this idea?

[faxm0dem]

Definitely a great feature. I need some time to think about the uuid, examples and syntax though

[faxm0dem]

I gave this some thought, and have some concerns about the location of test_messages.
While it makes sense to place this into the template section, for deduplication reasons, it still makes sense to allow for test_messages in the patterns section: will this still be possible?

[ihrwein]

They will be allowed in both places.

[ihrwein]

Maybe we have to prefix the parsed variables with the name of the pattern template to avoid name collisions if a pattern template is used multiple times in the same pattern.

[faxm0dem]

• 👍 for patterns not having an uuid
• 👎 for prefixing: the whole point of having templates is to deduplicate stuff. If foo.bar is extracted from two different patterns, then it's the same thing, so let's call it the same name. Name collisions should throw an exception so things are clear

[ihrwein]

I wanted to solve this:

pattern= "from %{IP:a} to %{IP:b}"
IP template = "%{GREEDY:word}"

After parsing with the pattern we would get a.word and b.word. This is not the same as you would use it but probably we can support both features. I hope we will find a nice, general solution :)

[faxm0dem]

how about:

"templates": [
  {
    "pattern": "loading configuration from '%{GREEDY:conf_file}'", 
    "name": "BIND9_LOADING_CONFIG",
    "prefix": "bind."
  }
]

with possibility to leave prefix out?

[ihrwein]

This could be done if we use the following rules:

• if BIND9_LOADING_CONFIG is used only once in the pattern, no prefix used
• if BIND9_LOADING_CONFIG is used multiple times in a pattern, the name of the actual binding is used as a prefix (this would be more expressive with the IP template)

This will have consequences if we want to support more than 2 layers of embedding.

[faxm0dem]

I'd need a full example at this point 😕

[ihrwein]

Something like this (note how port is inserted into the values):

{
    "patterns": [
        {
            "pattern": "%{BIND9_LOADING_CONFIG}",
            "name": "bind9",
            "uuid": "db36c4bf-0310-41d4-b59a-1dfe1929c4da"
        },
        {
            "pattern": "general: %{BIND9_LOADING_CONFIG}",
            "name": "bind9",
            "uuid": "db36c4bf-0310-41d4-b59a-1dfe1929c4da"
        },
        {
            "pattern": "%{PORT:port1}%{PORT:port2}",
            "name": "ports_more_PORTs",
            "uuid": "eb36c4bf-0310-41d4-b59a-1dfe1929c4da",
            "test_messages": [
              {
                "message": "port: 5678port: 0000",
                "values": {
                  "port1.port": "5678",
                  "port2.port": "0000",
                }
              }
            ]
        },
        {
            "pattern": "%{PORT}",
            "name": "port_one_PORT",
            "uuid": "eb36c4bf-0310-41d4-b59a-1dfe1929c4da",
            "test_messages": [
              {
                "message": "port: 5678",
                "values": {
                  "port": "5678"
                }
              }
            ]
        }

    ],
    "templates": [
        {
            "pattern": "loading configuration from '%{GREEDY:appacct.conf_file}'",
            "name": "BIND9_LOADING_CONFIG",
            "test_messages": [
                {
                    "message": "loading configuration from '/etc/bind/unctions.overreact.tld'",
                    "values": {
                        "appacct.conf_file": "/etc/bind/unctions.overreact.tld"
                    }
                }
            ]
        },
        {
            "pattern": "port: %{INT:port}",
            "name": "PORT",
            "test_messages": [
                {
                    "message": "port 1234",
                    "values": {
                        "port": "1234"
                    }
                }
            ]
        }
    ]
}

[faxm0dem]

what would happen with this:?

{
            "pattern": "%{PORT:port}",
            "name": "port_one_PORT",
            "uuid": "eb36c4bf-0310-41d4-b59a-1dfe1929c4da",
}

[ihrwein]

Nice catch. If I were a user I'd expect that the port variable contains the whole match (not just an integer). This example really raises some questions about the corner cases. Maybe the best option is to move backward with one step and start to think about the use cases we want to solve.