MQA-RC-Reading Comprehension

Abstract

We present a novel reading comprehension eye tracking dataset - MQA-RC - which allows researchers to observe changes in reading behavior in three comprehension tasks. Our extension uses 32 movie plots, with corresponding question-answer (QA) pairs, from the benchmark MovieQA dataset (Tapaswi et al., 2015). To the best of our knowledge, this is the first eye tracking dataset over a QA corpus and thus provides a gold standard to compare and synchronize model versus human visual attention in machine comprehension tasks.

The full dataset can be requested by filling out an EULA license agreement and send the agreement to cai-office@vis.uni-stuttgart.de.

Contact: Mrs. Daniela Milanese

The data is only to be used for non-commercial scientific purposes. If you use this dataset in a scientific publication, please cite the following paper:

1. 

   Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension

   Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, Ngoc Thang Vu

   Proc. ACL SIGNLL Conference on Computational Natural Language Learning (CoNLL), pp. 12-25, 2020.

   Abstract Links BibTeX Project

   While neural networks with attention mechanisms have achieved superior performance on many natural language processing tasks, it remains unclear to which extent learned attention resembles human visual attention. In this paper, we propose a new method that leverages eye-tracking data to investigate the relationship between human visual attention and neural attention in machine reading comprehension. To this end, we introduce a novel 23 participant eye tracking dataset - MQA-RC, in which participants read movie plots and answered pre-defined questions. We compare state of the art networks based on long short-term memory (LSTM), convolutional neural models (CNN) and XLNet Transformer architectures. We find that higher similarity to human attention and performance significantly correlates to the LSTM and CNN models. However, we show this relationship does not hold true for the XLNet models – despite the fact that the XLNet performs best on this challenging task. Our results suggest that different architectures seem to learn rather different neural attention strategies and similarity of neural to human attention does not guarantee best performance.

   doi: 10.18653/v1/P17

   Paper: sood20_conll.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/visualizing-human-and-neural-attention

   Dataset: https://perceptualui.org/research/datasets/MQA-RC/

   @inproceedings{sood20_conll, title = {Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension}, author = {Sood, Ekta and Tannert, Simon and Frassinelli, Diego and Bulling, Andreas and Vu, Ngoc Thang}, booktitle = {Proc. ACL SIGNLL Conference on Computational Natural Language Learning (CoNLL)}, year = {2020}, pages = {12-25}, doi = {10.18653/v1/P17}, publisher = {Association for Computational Linguistics} }