NLP Server

Introduction of NLP Server

The NLP server is a Flask based application running on Google Cloud that reduces redundancy between different questions as well as reduce redundancy between answers. For example the system can detect that. "What is the color of the dog ?" is the same as "The dog is what color ?" Similarly for answers the system can detect "The dog is brown" is the same as "The dog's color is brown."

Frameworks/Modules Used

The application is built using the following tools -

• Flask - Framework to build the application
• Python - Language of choice. V3.6+
• SpaCy - NLP of choice to compare the answers and questions to each other.

Structure of Application / Dataflow

Every-time the user hits the submit button on the page, the NLP server is given the questions that he has entered and given the old questions from the database. With these results the redundancy in the new questions are removed and unique questions are given back and long with those a dictionary showing the merges and another dictionary showing which question is better is given back.

How similarity score is calculated.

For every question pair that can be made using the new questions and old questions, a similarity score for that pair is calculated. This is a multistep process.

• The most common words are removed from each of the questions by referring to a taboo list. For example the words such as "What", "Which", "is", "the" etc.
• If the count of the remaining words is less than 3, then all the common words between the sentences are removed. Then the similarity score is calculated.
• If the length after the first step is more than 3 then we proceed directly to calculate the similarity score without removing any more common words.
• The similarity score is the average of all the individual similarity scores of the words in a sentence. The similarity score for each word pair is calculated by getting its cosine similarity for each of the word vectors representing that word.

Sentence Quality Calculator

For each question pair that is found to be similar we also calculate a quality score to choose the better sentence out of the two. This is done by assigning a score to each sentence based on certain grammatical properties given below. Once the score is calculated we keep the sentence with a higher score.

• For each noun-subject in the sentence it is given a point(1).
• For each dependent subject they are given a point and half(1.5).
• For each noun-chunk they are given an additional point(1).