Automated classification of unstructured log records
In a browser now: logcaliper.app/profiler does this job on a log sample with Drain, the log-parsing algorithm, in your browser with nothing uploaded, and adds the bytes per event, the events per second and what a year of the log takes to keep. This script stays as it is, for edit-distance and TLSH classification from the command line.
This Python script can automatically discover message types from sample log files. It supports the Levenshtein distance / distance-ratio and Trend Micro Locality Sensitive Hash (TLSH) methods for measuring the similarity between two sequences.
https://pypi.org/project/python-Levenshtein/
https://pypi.org/project/py-tlsh/
python3 logstats.py -f input.txt -o output.txt -r 0.5 -m distance-ratio -v
For the distance-ratio method, [-r] is the Levenshtein ratio is a value between 0 and 1 used to adjust the granularity of the classification. The higher the ratio is, the more granular will be the classification.
python3 logstats.py -f input.txt -o output.txt -r 30 -m distance -v
For the distance method, [-r] is the minimum number of difference to consider the strings similar.
python3 logstats.py -f input.txt -o output.txt -r 30 -m tlsh -v
For the tlsh method, [-r] is the minimum TLSH distance score (a.k.a. radius) to consider the strings similar. 0 for perfect match, 1 .. 50 for near perfect (1) to good match (50), 51 .. 100 for possible matches and 2000 for very distant strings.
