Repository navigation
log4net input should include global and logical thread context data #287
Description
I have a simple eventflow setup writing to elasticsearch with log4net input
class Program
{
static void Main(string[] args)
{
using (var pipeline = DiagnosticPipelineFactory.CreatePipeline(".\\eventFlowConfig.json"))
{
var logger = LogManager.GetLogger("EventFlowRepo", "MY_LOGGER_NAME");
using (LogicalThreadContext.Stacks["NDC"].Push("12345"))
{
logger.Debug("Hey! Listen!", new Exception("uhoh"));
}
}
Console.ReadLine();
}
}
eventflowconfig.json
{
"inputs": [
{
"type": "Log4net",
"logLevel": "Debug"
},
{
"type": "Log4net",
"logLevel": "Info"
},
{
"type": "Log4net",
"logLevel": "Warn"
},
{
"type": "Log4net",
"logLevel": "Error"
},
{
"type": "Log4net",
"logLevel": "Fatal"
}
],
"outputs": [
{
"type": "ElasticSearch",
"indexNamePrefix": "defaultindex",
"serviceUri": "http://servername:9200",
"basicAuthenticationUserName": "user",
"basicAuthenticationUserPassword": "password",
"eventDocumentTypeName": "diagData",
"numberOfShards": 1,
"numberOfReplicas": 0,
"refreshInterval": "5s"
}
],
"settings": {
"pipelineBufferSize": "1000",
"maxEventBatchSize": "100",
"maxBatchDelayMsec": "500",
"maxConcurrency": "8",
"pipelineCompletionTimeoutMsec": "30000"
},
"extensions": []
}
The issue is, the custom data [12345 in this case] I write to the NDC context, is not written to elasticsearch. How do I configure elastic output to recognize any custom data pushed to NDC?
The debug message along with the "exception" message does get written to ELK
Activity
karolz-ms commentedon Nov 15, 2018
@teeboy75 thanks for submitting this issue
First, it was not clear in the documentation, but the
logLevelproperty of the Log4net input indicates minimum level for captured events. So if you want to capture all events, with all levels, you need to declare the Log4net input just once, withlogLevelequal toDebug, as it is the lowest possible level. I have just updated the docs to clarify this.Also, I would suggest to use StdOutput to determine whether the problem is that the Log4net is not capturing the NDC context data, or is it a problem with Elasticsearch output. Can you please do another experiment and:
and let us know if that causes NDC data to show up? Thx!
pattisapu01 commentedon Nov 15, 2018
Added stdout and removed additional log levels from eventflowconfig. Does not make any difference. NDC output does not show up on console window and ELK. The only additional Json enriched fields that seems to be sent to elastic are "timestamp", "providernam","level" and "keywords". Ofcourse, "payload" element contains just the text "Hey! Listen!". Here is the json.
{
"_index": "index-2018.11.15",
"_type": "diagData",
"_id": "P0PeGGcBITRnpnnqT_iE",
"_version": 1,
"_score": null,
"_source": {
"timestamp": "2018-11-15T14:34:38.603181-05:00",
"providerName": "Log4netInput.MY_LOGGER_NAME",
"level": 5,
"keywords": 0,
"payload": {
"Message": "Hey! Listen!",
"Exception": {
"ClassName": "System.Exception",
"Message": "uhoh",
"Data": null,
"InnerException": null,
"HelpURL": null,
"StackTraceString": null,
"RemoteStackTraceString": null,
"RemoteStackIndex": 0,
"ExceptionMethod": null,
"HResult": -2146233088,
"Source": null,
"WatsonBuckets": null
}
}
},
"fields": {
"timestamp": [
"2018-11-15T19:34:38.603Z"
]
},
"sort": [
1542310478603
]
}
pattisapu01 commentedon Nov 15, 2018
Just to be clear, if I remove eventflow input and output and directly use log4nets log appenders, the NDC comment works..
karolz-ms commentedon Nov 16, 2018
OK, thanks. I looked a bit more closely into this and it looks like appending data from Log4net global context & logical thread context was never implemented.
For comparison here is how Stackify Retrace appender does this
@teeboy75 would you be willing to submit a PR to have these properties added?
CC Jeremy Smith (@jeremysmith1)
pattisapu01 commentedon Nov 16, 2018
Sure! I can do it late next week..
karolz-ms commentedon Nov 16, 2018
Thank you! Much appreciate your help @teeboy75
Adding support for global and thread context stacks for log4net
pattisapu01 commentedon Dec 2, 2018
Karol Zadora-Przylecki (@karolz-ms)
I have the code ready. Waiting from corporate to give me permission for PR submission. will update you.
karolz-ms commentedon Dec 3, 2018
@teeboy75 Roger!
karolz-ms commentedon Mar 26, 2019
pattisapu01 Do you mind if we (myself or some other Microsoft developer) "steal" the code from you (pattisapu01@c9c4d65) and finish this PR? I'd be happy if your time and work on this issue did not go to waste...
pattisapu01 commentedon Mar 26, 2019
[-]Eventflow output to ELK with log4net input and NDC stack[/-][+]log4net input should include global and logical thread context data[/+]