a little better documentation

[lcorsini]

Hello,
I'm trying to figure out how to use this plugin, I need to parse a oracle log, like this :

<msg time="'2017-02-14T02:00:00.209+01:00'" org_id="'oracle'" comp_id="'rdbms'type='UNKNOWN'" level="'16'" host_id="'lxoraopen0s.mil.esselunga.net'host_addr='10.12.1.21'" pid="'2461'"> <txt>Closing scheduler window</txt> </msg> <msg time="'2017-02-14T02:00:00.209+01:00'" org_id="'oracle'" comp_id="'rdbms'type='UNKNOWN'" level="'16'" host_id="'lxoraopen0s.mil.esselunga.net'host_addr='10.12.1.21'" pid="'2461'"> <txt>Closing Resource Manager plan via scheduler window</txt> </msg> <msg time="'2017-02-14T02:00:00.209+01:00'" org_id="'oracle'" comp_id="'rdbms'type='UNKNOWN'" level="'16'" host_id="'lxoraopen0s.mil.esselunga.net'host_addr='10.12.1.21'" pid="'2461'"> <txt>Clearing Resource Manager CDB plan via parameter</txt> </msg>
and forward it to elastic search.
I don't really understand how to parse the single attributes like timestamp or the pid, and then the message included in /msg/txt
the single example is not very explaining (maybe because the original log is not explained)
I'm not very expert of XML, could you provide some documentation on how to start using this plugin?

[toyokazu]

Thank you for your comment! I've added an example XML file as you indicated. As already written in the last part of README.md, please save your XML file as test.xml and read it from irb or pry. You can find how to specify your target entities by REXML (DOM style XML parser for ruby). To tell the truth, while my original motivation to maintain this plugin was to parse Libelium sensor data, since they changed their data format to JSON, I don't need to maintain this plugin anymore. If you still have requirements to maintain this plugin, please let me know. I'll try it as much as possible :)

[lcorsini]

Thanks, Oracle still uses XML, or provides a plain text version, which is multiline (and it's worse to parse imho) so, I tought XML could be a better option. I'll try to make it work with your examples. Thanks for your help, I hope that the plugin can work as is, then I can try to study the source code (I'm more a python guy) and maybe help you maintain it, in the meantime thanks for your support

…

On Wed, Feb 15, 2017 at 12:38 AM, Toyokazu Akiyama ***@***.*** > wrote: Thank you for your comment! I've added an example XML file as you indicated. As already written in the last part of README.md, please save your XML file as test.xml and read it from irb or pry. You can find how to specify your target entities by REXML (DOM style XML parser for ruby). To tell the truth, while my original motivation to maintain this plugin was to parse Libelium sensor data, since they changed their data format to JSON, I don't need to maintain this plugin anymore. If you still have requirements to maintain this plugin, please let me know. I'll try it as much as possible :) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABbq6uCKSjadKH7MTWOeGyX3_WIftomYks5rcjsPgaJpZM4MAlMs> .

[lcorsini]

Hi, just a quick update
I've tried to parse that log, but without success,

got this error in log:

[2017-02-15 21:00:33 +0100 [warn]: Parse error error="malformed XML: missing tag start\nLine: 1\nPosition: 73\nLast 80 unconsumed characters:\n<msg time='2017-02-15T21:00:33.613+01:00' org_id='oracle' comp_id='rdbms'"
2017-02-15 21:00:33 +0100 [warn]: error error="undefined method `text' for class `NilClass'"
2017-02-15 21:00:33 +0100 [warn]: error error="undefined method `text' for class `NilClass'"
2017-02-15 21:00:33 +0100 [warn]: Parse error error="No close tag for /txt\nLine: 1\nPosition: 57\nLast 80 unconsumed characters:\n"
2017-02-15 21:00:33 +0100 [warn]: Parse error error="Missing end tag for '' (got \"txt\")\nLine: 1\nPosition: 7\nLast 80 unconsumed characters:\n"
2017-02-15 21:00:33 +0100 [warn]: Parse error error="Missing end tag for '' (got \"msg\")\nLine: 1\nPosition: 6\nLast 80 unconsumed characters:\n"
2017-02-15 21:00:33 +0100 [warn]: Parse error error="malformed XML: missing tag start\nLine: 1\nPosition: 73\nLast 80 unconsumed characters:\n<msg time='2017-02-15T21:00:33.616+01:00' org_id='oracle' comp_id='rdbms'"]

this is the log entity mentioned above

<msg time='2017-02-15T21:00:33.616+01:00' org_id='oracle' comp_id='rdbms'
type='UNKNOWN' level='16' host_id='lxoraopen0s.mil.esselunga.net'
host_addr='10.12.1.21' pid='2264'>
<txt>  Current log# 1 seq# 193 mem# 0: /oradata/WAB0S/redologs/redo01.log
</txt>
</msg>

This is my config

<source>
  type tail
  path /u01/app/oracle/diag/rdbms/wab0s/WAB0S/alert/log.xml
  format xml
  time_xpath '["msg[@time]", "text"]'
  time_key '@timestamp'
  #attr_xpaths '[["cap:alert/cap:info/cap:parameter/cap:valueName", "text"]]'
  value_xpaths '[["msg/txt", "text"]]'
  tag debug
</source>

I think that there is a series of problems here:
Don't know if your plugin is able to extract attributes from xml tag (see time= pid= etc etc, inside the msg tag)
probably that kind of file isn't a valid xml file, so I think that I need to build something specific, maybe based on your code

[toyokazu]

Hi. Sorry for being a late reply.

First of all, the plugin basically assumes MQTT I/O plugin. So, if you try to use with tail input plugin, you need to input your whole xml part as one chunk. The first warning is shown because of that. I didn't tested yet, but it may be solved by multiline option of tail plugin.

http://docs.fluentd.org/v0.12/articles/in_tail

However, if you can extract multiline easily by regexp, it may be easier to use regexp to extract the required information from your log directly...

The other issue is that in order to specify an attribute of xml, you need to use /@xxx for xpath. And the method name to obtain the target value is 'value'. So thus, please try the following.

time_xpath '["msg/@time", "value"]'

That's all from the current information.

Best Regards

[lcorsini]

Don't worry, thank you very much for your help. I'll try what you suggest and report back Thanks again

…

On Wed, Feb 22, 2017 at 5:25 PM, Toyokazu Akiyama ***@***.***> wrote: Hi. Sorry for being a late reply. First of all, the plugin basically assumes MQTT I/O plugin. So, if you try to use with tail input plugin, you need to input your whole xml part as one chunk. The first warning is shown because of that. I didn't tested yet, but it may be solved by multiline option of tail plugin. http://docs.fluentd.org/v0.12/articles/in_tail However, if you can extract multiline easily by regexp, it may be easier to use regexp to extract the required information from your log directly... The other issue is that in order to specify an attribute of xml, you need to use ***@***.*** <https://github.com/xxx> for xpath. And the method name to obtain the target value is 'value'. So thus, please try the following. time_xpath ***@***.*** <https://github.com/time>", "value"]' That's all from the current information. Best Regards — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABbq6hHOo3IZDvRapIPDa4b-kdJeIKk0ks5rfGGIgaJpZM4MAlMs> .

[himmatb]

Can you please help us by Updating this plugin .. we really need this plugin becoz almost all the application logs that we have are in xml format its very difficult to write regex for all logs to extract specific fields if we have this plugin we will really get help.
I think not only me most of guys have hope on you to get this plugin it will really help everyone!!

Help will be really appreciated!!
Thanks in advance.!!

[acechef]

@lcorsini Has your problem been solved?