Artifacts from cleaning?

[gabrielStanovsky]

Some sentences seem very weird:
For example:

Boy Scouts ' ` perversion ' files set to be released travel

@rachelvov, What's the original tweet?
@kleinay, do you know the tweet id?

[rachelvov]

this is the original tweet: 258960355655548930 20007 Boy Scouts' 'perversion' files set to be released #travel http://t.co/RrOpIc2Y Seems that the cleaning process removed the #

…

On Tue, Aug 15, 2017 at 1:19 PM, Gabriel Stanovsky ***@***.*** > wrote: Assigned #18 <#18> to @rachelvov <https://github.com/rachelvov>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AP92_xbtINdu2TIbwezAufoAaOUJVfZjks5sYXChgaJpZM4O3a_T> .

[gabrielStanovsky]

@OriShapira, maybe we can remove hashtags altogether.

[rachelvov]

I think the reason I removed only the "#" and not the word after it is that it sometimes appears in the middle of the tweet with a content word, like here:
#Baghdad: 25 killed as car bomb targets #Iraq army recruits
It's worth checking how often this happens.

[gabrielStanovsky]

Maybe it's applicable to remove hashtags at the end of tweets?

[kleinay]

Who is the owner of the pre-processing step (cleaning the tweets)? @OriShapira ? we should perhaps integrate the pre-process code into the project?