Fix Term_Frequency

[mk2510]

The term_frequency function is currently implemented incorrectly. Example:

>>> s = pd.Series(["Text Text of doc one", "Text of of doc two", "Aha hi bnd one"]).pipe(hero.tokenize)
>>> s
0    [Text, Text, of, doc, one]
1      [Text, of, of, doc, two]
2           [Aha, hi, bnd, one]
dtype: object

Old implementation of term frequency returns:

  term_frequency                                                                      
             Aha      Text       bnd       doc        hi        of       one       two
0       0.000000  0.142857  0.000000  0.071429  0.000000  0.071429  0.071429  0.000000
1       0.000000  0.071429  0.000000  0.071429  0.000000  0.142857  0.000000  0.071429
2       0.071429  0.000000  0.071429  0.000000  0.071429  0.000000  0.071429  0.000000

New version returns:

  term_frequency                                      
            Aha   Text  bnd  doc    hi   of   one  two
0           0.00  0.4  0.00  0.2  0.00  0.2  0.20  0.0
1           0.00  0.2  0.00  0.2  0.00  0.4  0.00  0.2
2           0.25  0.0  0.25  0.0  0.25  0.0  0.25  0.0

We can see that the old version simply did count for each document and then divided each count value by the total number of terms across all documents (the .sum() in the old code). That does not make sense for term_frequency. The new version will also do count, but then divide the count values of each document by the number of terms in that document to produce the correct term frequencies. Of course, that's the same as L1-Normalizing every row, so that's how we implemented it 🚥 .

Note: only so many lines changed as this builds upon the DocumentTermDF (see #156)

[mk2510]

now, this branch is based on the master branch again and is ready for review/merge 🥇 🌵

[jbesomi]

Looks good and it has been merged. What would be nice ( probably to be done in a separate PR) is to have more friendly and funny doctest text content (instead of "Aha", "Text", ...). One idea, for instance, is to use famous sentences said by movie Superheroes. Here are a few examples:

• I have the power!
• Flame on!
• HULK SMASH!
• Holy ____ Batman!
• I am the vengeance, I am the night, I am BATMAN!
• I am GROOT.
• I’m going ghost!
• I am the law!
• SPOOOON!!!

Opinions?

[henrifroese]

Added an issue for that, see #189

[Iota87]

Great idea, I have already started to use Hero related text examples in the Docs and Tutorials I am working on.