Difficulties in reproducing the results described in ISWC'20 paper

[flaviomerenda]

The repository seems to contain several errors partially fixed by the merge request #1. However, despite the new modifications that guarantee the reproducibility of the scripts, the results obtained during the evaluation are lower than the ones reported in the paper, especially for coinfo250 and FakeNewsNet. @rdenaux do you have any ideas and do you know where the problem could be within the pipeline?

[rdenaux]

In the past, reproducibility issues typically arose due to:

1. unstable documents to be reviewed. For example, deleted or no longer accessible news articles (FakeNewsNet) or tweets (coinfo250), or linked articles (e.g. articles linked in a tweet). I think we aimed to have hashes of the raw texts as a way to double check this without having to store the text (due to copyright issues), but not sure we published these hashcodes (which could be useful to compare the datasets). E.g. if a tweet linked to a webpage which is no longer accessible, the credibility rating may change, and thus also the overall accuracy.
2. backwards compatibility of the various deep learning models. When loading finetuned models (e.g. for semantic similarity or stance-detection), using newer library versions (e.g. numpy, cuda, pytorch) can result in a decay of the model performance which would translate in an overall acred performance decay. There's no fully automated test/validation for this, although there's typically some description for each model specifying the test results that the model should achieve. For example the worthiness model should achieve 95% accuracy, the claimencoder should achieve 83% accuracy on STS-B. You'd need to test each model individually/separately to rule out backwards compatibility issues.

[flaviomerenda]

Hi @rdenaux,

Thank you for your quick response and help. Regarding the points you discussed:

1. We successfully collected the datasets necessary for the evaluation and the number of samples correspond to the number described for each evaluation dataset (clef18, coinfo250 and FakeNewsNet). Unfortunately, it is quite difficult to follow the validity of linked sources present in texts (such as URLs pointing to external tweets or articles).
2. We successfully tested the performances of the models. The claimencoder achieves 83% accuracy on STS-B and the worthinesschecker achieves 95% accuracy on its test set. In addition, we also tried to test the claimneuralindex with some sentence similarity test call.

The results currently achieved for each dataset are:

• Clef18 accuracy: 0.3669
• Coinform250 accuracy: 0.1774
• FakeNewNet accuracy: 0.4726

As far as you know, is it possible that there is some other point in the pipeline that can cause such errors?