Hello! Going over your paper: Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline, I'm wondering how can I reproduce the results in Table 3 and 4 in particular, using what is released in this repository?
In particular, what I'm interested in are the training and test datasets. I see the manual datasets directory, however, I'm not able to find my way around it.
To be specific, as I understand the problem, an input to the model (or rather, a pipeline) should be a summary, and a document (or a set of documents if the problem is MDS). Assuming the data (both summary and document) is already split into IUs (by an OIE system), the task is to align each document IU, to the highest scoring summary IU. Therefore, multiple document IUs might be aligned to the same summary IU, while each summary IU has to be aligned to at least one document IU.
From the datasets released in the manual datasets directory, I can obtain all unique summary IUs (there denoted as SCUs) for each document. Then, I can align each document IU to highest scoring summary IU. In theory, this yields a list of alignments (document IU -> summary IU), where for each link I can get the corresponding text span in the summary/document. Finally, given the spans of both, I can measure the metrics (coverage, recall, precision, F1) reported in the paper.
Assuming my reasoning is correct, I should be able to run inference and measure performance on your data as is (nothing extra needed), but I'm not entirely sure. I would sincerely appreciate your comments on my train-of-thought dump above 😃
Going further, to me it seems that the train dataset is only partially released, and the current released version cannot be used as is. Can you please point to the remaining steps that need to be partaken in order for me reproduce your trained model - SuperPAL? 😃
Thank you for the great work! 😃
Hello! Going over your paper: Summary-Source Proposition-level Alignment: Task, Datasets and Supervised Baseline, I'm wondering how can I reproduce the results in Table 3 and 4 in particular, using what is released in this repository?
In particular, what I'm interested in are the training and test datasets. I see the manual datasets directory, however, I'm not able to find my way around it.
To be specific, as I understand the problem, an input to the model (or rather, a pipeline) should be a summary, and a document (or a set of documents if the problem is MDS). Assuming the data (both summary and document) is already split into IUs (by an OIE system), the task is to align each document IU, to the highest scoring summary IU. Therefore, multiple document IUs might be aligned to the same summary IU, while each summary IU has to be aligned to at least one document IU.
From the datasets released in the manual datasets directory, I can obtain all unique summary IUs (there denoted as SCUs) for each document. Then, I can align each document IU to highest scoring summary IU. In theory, this yields a list of alignments (document IU -> summary IU), where for each link I can get the corresponding text span in the summary/document. Finally, given the spans of both, I can measure the metrics (coverage, recall, precision, F1) reported in the paper.
Assuming my reasoning is correct, I should be able to run inference and measure performance on your data as is (nothing extra needed), but I'm not entirely sure. I would sincerely appreciate your comments on my train-of-thought dump above 😃
Going further, to me it seems that the train dataset is only partially released, and the current released version cannot be used as is. Can you please point to the remaining steps that need to be partaken in order for me reproduce your trained model - SuperPAL? 😃
Thank you for the great work! 😃