Adnen Abdessaied

Institute for Visualisation and Interactive Systems, University of Stuttgart

+49 711 685 60052 Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building, Room 01.035 Google Scholar

Biography

Adnen Abdessaied is a PhD student in the Collaborative Artificial Intelligence group since March 2021. He holds a Bachelor's and Master's degree in Simulation Technology (SimTech) from the University of Stuttgart. The latter was obtained with an overall verdict of very good with distinction and won the Outstanding Work Award from the Industrial Consortium SimTech (IC SimTech). In his Bachelor's studies, he focused on numerical mathematics and classical machine learning techniques. In his master's studies, he got involved in deep multi-modal memory-enhanced neural networks that are inspired by the human brain. His research interests lie in the design and interpretability of multi-modal hybrid Neuro-Symbolic AI models.

Teaching

2022

Mensch-Computer Interaktion (Teaching Assistant)Bachelor

2021

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

---

Supervision

2022

Attention-Based Reasoning over Multimodal Embeddings in Video-Grounded Dialog (M.Sc.) Manuel Hochmeister

 

Multimodal Feature Fusion via Graph Neural Networks for Visual Dialog (M.Sc.) Nishan Chatterjee

---

Open Thesis Projects

 

Diagnostic Dataset for Video Dialog based on CLEVRERMaster

 

Multimodal Models Meet Unbiased Scene GraphsMaster

---

Publications

(Abdessaied et al., 2024; Abdessaied et al., 2024; Abdessaied et al., 2024; Bortoletto et al., 2024; Abdessaied* et al., 2022; Abdessaied et al., 2022)

1. 

   Multi-Modal Video Dialog State Tracking in the Wild

   Adnen Abdessaied, Lei Shi, Andreas Bulling

   Proc. 18th European Conference on Computer Vision (ECCV), pp. 1–25, 2024.

   Abstract Links BibTeX Project

   We present MST-MIXER – a novel video dialog model operating over a generic multi-modal state tracking scheme. Current models that claim to perform multi-modal state tracking fall short of two major aspects: (1) They either track only one modality (mostly the visual input) or (2) they target synthetic datasets that do not reflect the complexity of real-world in the wild scenarios. Our model addresses these two limitations in an attempt to close this crucial research gap. Specifically, MST-MIXER first tracks the most important constituents of each input modality. Then, it predicts the missing underlying structure of the selected constituents of each modality by learning local latent graphs using a novel multi-modal graph structure learning method. Subsequently, the learned local graphs and features are parsed together to form a global graph operating on the mix of all modalities which further refines its structure and node embeddings. Finally, the fine-grained graph node features are used to enhance the hidden states of the backbone Vision-Language Model (VLM). MST-MIXER achieves new state-of-the-art results on five challenging benchmarks.

   Paper: abdessaied24_eccv.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/MST-MIXER

   @inproceedings{abdessaied24_eccv, author = {Abdessaied, Adnen and Shi, Lei and Bulling, Andreas}, title = {Multi-Modal Video Dialog State Tracking in the Wild}, booktitle = {Proc. 18th European Conference on Computer Vision (ECCV)}, year = {2024}, pages = {1--25} }
2. 

   OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog

   Adnen Abdessaied, Manuel Hochmeister, Andreas Bulling

   Proc. 31st Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING), pp. 1–11, 2024.

   Abstract Links BibTeX Project

   We present the Object Language Video Transformer (OLViT) – a novel model for video dialog operating over a multi-modal attention-based dialog state tracker. Existing video dialog models struggle with questions requiring both spatial and temporal localization within videos, long-term temporal reasoning, and accurate object tracking across multiple dialog turns. OLViT addresses these challenges by maintaining a global dialog state based on the output of an Object State Tracker (OST) and a Language State Tracker (LST): while the OST attends to the most important objects within the video, the LST keeps track of the most important linguistic co-references to previous dialog turns. In stark contrast to previous works, our approach is generic by nature and is therefore capable of learning continuous multi-modal dialog state representations of the most relevant objects and rounds. As a result, they can be seamlessly integrated into Large Language Models (LLMs) and offer high flexibility in dealing with different datasets and tasks. Evaluations on the challenging DVD (response classification) and SIMMC 2.1 (response generation) datasets show that OLViT achieves new state-of-the-art performance across both datasets.

   Paper: abdessaied24_coling.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/OLViT

   @inproceedings{abdessaied24_coling, author = {Abdessaied, Adnen and von Hochmeister, Manuel and Bulling, Andreas}, title = {OLViT: Multi-Modal State Tracking via Attention-Based Embeddings for Video-Grounded Dialog}, booktitle = {Proc. 31st Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)}, year = {2024}, pages = {1--11} }
3. 

   VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs

   Adnen Abdessaied, Lei Shi, Andreas Bulling

   Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 5805–5814, 2024.

   Abstract Links BibTeX Project

   We propose VD-GR – a novel visual dialog model that combines pre-trained language models (LMs) with graph neural networks (GNNs). Prior works mainly focused on one class of models at the expense of the other, thus missing out on the opportunity of combining their respective benefits. At the core of VD-GR is a novel integration mechanism that alternates between spatial-temporal multi-modal GNNs and BERT layers, and that covers three distinct contributions: First, we use multi-modal GNNs to process the features of each modality (image, question, and dialog history) and exploit their local structures before performing BERT global attention. Second, we propose hub-nodes that link to all other nodes within one modality graph, allowing the model to propagate information from one GNN (modality) to the other in a cascaded manner. Third, we augment the BERT hidden states with fine-grained multi-modal GNN features before passing them to the next VD-GR layer. Evaluations on VisDial v1.0, VisDial v0.9, VisDialConv, and VisPro show that VD-GR achieves new state-of-the-art results across all four datasets

   Paper: abdessaied24_wacv.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/VDGR

   @inproceedings{abdessaied24_wacv, author = {Abdessaied, Adnen and Shi, Lei and Bulling, Andreas}, title = {VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs}, booktitle = {Proc. IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)}, year = {2024}, pages = {5805--5814} }
4. 

   Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition

   Matteo Bortoletto, Constantin Ruhdorfer, Adnen Abdessaied, Lei Shi, Andreas Bulling

   Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1–16, 2024.

   Abstract Links BibTeX Project

   Recent work on dialogue-based collaborative plan acquisition (CPA) has suggested that Theory of Mind (ToM) modelling can improve missing knowledge prediction in settings with asymmetric skill-sets and knowledge. Although ToM was claimed to be important for effective collaboration, its real impact on this novel task remains under-explored. By representing plans as graphs and by exploiting task-specific constraints we show that, as performance on CPA nearly doubles when predicting one’s own missing knowledge, the improvements due to ToM modelling diminish. This phenomenon persists even when evaluating existing baseline methods. To better understand the relevance of ToM for CPA, we report a principled performance comparison of models with and without ToM features. Results across different models and ablations consistently suggest that learned ToM features are indeed more likely to reflect latent patterns in the data with no perceivable link to ToM. This finding calls for a deeper understanding of the role of ToM in CPA and beyond, as well as new methods for modelling and evaluating mental states in computational collaborative agents.

   doi:

   Paper: bortoletto24_acl.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/limits-of-tom

   @inproceedings{bortoletto24_acl, author = {Bortoletto, Matteo and Ruhdorfer, Constantin and Abdessaied, Adnen and Shi, Lei and Bulling, Andreas}, title = {Limits of Theory of Mind Modelling in Dialogue-Based Collaborative Plan Acquisition}, booktitle = {Proc. 62nd Annual Meeting of the Association for Computational Linguistics (ACL)}, year = {2024}, pages = {1--16}, doi = {} }

1. 

   Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA

   Adnen Abdessaied*, Ekta Sood*, Andreas Bulling

   Proc. of the 7th Workshop on Representation Learning for NLP (Repl4NLP), pp. 1–12, 2022.

   Abstract Links BibTeX Project

   [Equal contribution by the first two authors.] We propose the Video Language Co-Attention Network (VLCN) – a novel memory-enhanced model for Video Question Answering (VideoQA). Our model combines two original contributions: A multimodal fast-learning feature fusion (FLF) block and a mechanism that uses self-attended language features to separately guide neural attention on both static and dynamic visual features extracted from individual video frames and short video clips. When trained from scratch, VLCN achieves competitive results with the state of the art on both MSVD-QA and MSRVTT-QA with 38.06% and 36.01% test accuracies, respectively. Through an ablation study, we further show that FLF improves generalization across different VideoQA datasets and performance for question types that are notoriously challenging in current datasets, such as long questions that require deeper reasoning as well as questions with rare answers.

   Paper: abdessaied22_repl4NLP.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/vlcn

   @inproceedings{abdessaied22_repl4NLP, author = {Abdessaied*, Adnen and Sood*, Ekta and Bulling, Andreas}, title = {Video Language Co-Attention with Multimodal Fast-Learning Feature Fusion for VideoQA}, booktitle = {Proc. of the 7th Workshop on Representation Learning for NLP (Repl4NLP)}, year = {2022}, pages = {1--12} }
2. 

   Neuro-Symbolic Visual Dialog

   Adnen Abdessaied, Mihai Bâce, Andreas Bulling

   Proc. 29th International Conference on Computational Linguistics (COLING), pp. 1–11, 2022.

   Abstract Links BibTeX Project

   We propose Neuro-Symbolic Visual Dialog (NSVD) —the first method to combine deep learning and symbolic program execution for multi-round visually-grounded reasoning. NSVD significantly outperforms existing purely-connectionist methods on two key challenges inherent to visual dialog: long-distance co-reference resolution as well as vanishing question-answering performance. We demonstrate the latter by proposing a more realistic and stricter evaluation scheme in which we use predicted answers for the full dialog history when calculating accuracy. We describe two variants of our model and show that using this new scheme, our best model achieves an accuracy of 99.72% on CLEVR-Dialog —a relative improvement of more than 10% over the state of the art —while only requiring a fraction of training data. Moreover, we demonstrate that our neuro-symbolic models have a higher mean first failure round, are more robust against incomplete dialog histories, and generalise better not only to dialogs that are up to three times longer than those seen during training but also to unseen question types and scenes.

   Paper: abdessaied22_coling.pdf

   Code: Contact Author for further information

   @inproceedings{abdessaied22_coling, author = {Abdessaied, Adnen and Bâce, Mihai and Bulling, Andreas}, title = {Neuro-Symbolic Visual Dialog}, booktitle = {Proc. 29th International Conference on Computational Linguistics (COLING)}, year = {2022}, pages = {1--11} }