Unable to send ASM logs to a HTTP endpoint via telemetry streaming

[harshnasitcrest]

Environment

• Telemetry Streaming Version: 1.36.0-1
• BIG-IP Version: 17.1.0.1

Summary

I'm able to generate ASM logs deployed in our on-premise setup but unable to send it via telemetry streaming to a HTTP endpoint (consumer type is Generic_HTTP).

Steps To Reproduce

Steps to reproduce the behavior:

1. Submit the following declaration:

{
    "class": "Telemetry",
     "My_ASM_Listener": {
        "class": "Telemetry_Listener",
        "port": 6514,  
        "trace": true
        "match": "ASM",
        "actions": [
            {
                "setTag": {
                    "application": "`ASM`"
                        },
                "enable": true
                }
            ]


    },
    "My_Consumer": {
        "class": "Telemetry_Consumer",
        "type": "Generic_HTTP",
        "host": "10.50.9.132",
        "protocol": "http",
        "port": 5151,
        "path": "/post",
        "headers": [
            {"name": "Authorization", "value": "12345689"},
            {"name": “ID1", "value": "ABC"},
            {"name": "ID2", "value": "XYZ"}


        ],
        "actions": [
            {
                "JMESPath": {},
                "expression": "{ logs: [@] }"
            }
        ]
    }
}

2. On submitting above declaration, we're getting 200 response code.
3. Verified that ASM logs are generated in the backend at /var/log/asmdata1/request_log and shown on the F5 UI at: Security --> Events logs --> Application --> Requests.
4. Log level was set to debug but unable to see API failure logs. Following are the contents of logs at /var/log/restnoded/restnoded.log:

Screenshot 1:

Screenshot 2:

6. Verified that enough resources are provided for this setup to work.

Expected Behavior

• Logs should be sent via Telemetry Streaming to HTTP endpoint defined.
• Logs should indicate the failure and document what should be the next troubleshooting steps. Followed this troubleshooting guide and tried multiple steps but didn't help.

[pgouband]

Hi @harshnasitcrest,

Have you tried to apply the following and fine tune memory management?

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/troubleshooting.html#why-is-my-big-ip-experiencing-occasional-high-cpu-usage-and-slower-performance

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/memory-monitor.html?highlight=beta

[pgouband]

Hi @harshnasitcrest,

Can you try to disable system poller and test it?
If the issue occurs even with system poller disabled, you should increase the amount of memory for host system.

[harshnasitcrest]

Referring to https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/telemetry-system.html#system-poller , I disabled system poller using following declaration:

"My_System_Minimal": {
     "class": "Telemetry_System",
     "systemPoller": {
         "enable": false
     }
 }

I increased memory and CPU both after disabling system poller.

Somehow, I don't see memory and tomcat restart errors now, but I still don't see any logs regarding failures in making API call to destination.

Does the last log saying Applying restrictions to incoming data causing the trouble? Any other steps I can try? Is it usually this cumbersome to configure and send ASM events to HTTP server?

[petrov-serg]

what is your full declaration? (you can mask/remove secrets from it)

[harshnasitcrest]

what is your full declaration? (you can mask/remove secrets from it)

@petrov-serg Here's the full declaration that I posted using POST request to https://<ip>/mgmt/shared/telemetry/declareAPI:

{
    "class": "Telemetry",
    "My_ASM_Listener": {
        "class": "Telemetry_Listener",
        "port": 6514,
        "trace": true,
        "match": "ASM",
        "actions": [
            {
                "setTag": {
                    "application": "`ASM`"
                },
                "enable": true
            }
        ]
    },
    "My_System_Minimal": {
        "class": "Telemetry_System",
        "systemPoller": {
            "enable": false
        }
    },
    "My_Consumer": {
        "class": "Telemetry_Consumer",
        "type": "Generic_HTTP",
        "host": "10.50.9.132",
        "protocol": "http",
        "port": 5151,
        "path": "/post",
        "headers": [
            {
                "name": "Authorization",
                "value": "12345689"
            },
            {
                "name": "ID1",
                "value": "ABC"
            },
            {
                "name": "ID2",
                "value": "XYZ"
            }
        ],
        "actions": [
            {
                "JMESPath": {},
                "expression": "{ logs: [@] }"
            }
        ]
    }
}