Improve core logic to support non space tokenized languages like japenese

[mayankpunetha007]

The tokenization logic should be more generic.

Could use something like: https://www.atilika.org/ , to tokenize Japanese.