Florian Strohm

Institute for Visualisation and Interactive Systems, University of Stuttgart

+49 711 685 61723 Pfaffenwaldring 5a, 70569 Stuttgart, Germany University of Stuttgart, SimTech Building, Room 01.031 Google Scholar

Biography

Florian Strohm is a PhD student in the Perceptual User Interfaces group since January 2020. He holds a Bachelor's and Master’s degree in Computer Science from the University of Stuttgart. During his Master’s studies he focused on the fields of Computer Vision and Machine Learning, and he won the Rul Gunzenhäuser Prize for his master thesis. His research interests are analyzing cognitive processes such as how humans encode visual stimuli in memory and the synthetic reconstruction of such stimuli.

Teaching

2022

Machine Perception and Learning (Tutor) (summer)Master

 

Machine Perception and Learning (Tutor) (winter)Master

2021

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

 

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

2020

Machine Learning and Computer Vision for HCI (Fachpraktikum)Master

 

Mensch-Computer-Interaktion (Teaching Assistant)Bachelor

2018

Information Retrieval and Text Mining (Teaching Assistant)Bachelor/Master

2017

Rechnerorganisation (Tutor)Bachelor

---

Supervision

2023

Web Interface for Interactive Face Reconstruction (B.Sc.) Lewin Fritzenschaft

 

Disentangled Face Embeddings (B.Sc.) Michael Linder

2022

Video-based Representation Learning for Emotion Recognition (M.Sc.) Shangrui Nie**

 

Interpreting Neural Attention in NLP with Human Visual Attention (M.Sc. Research Project) Shangrui Nie**

 

Conditional GAN-based Mask Guided Portrait Editing (M.Sc. Research Project) Prabhakar Panday

 

Augmenting fMRI-based Image Reconstruction with Image Labelling (M.Sc.) Jamie Ullerich

 

Intention Recognition via Eye Gaze for Hierarchical Imitation Learning (M.Sc.) Anna Penzkofer

 

Fine-Grained Facial Composite Generation using Facial Landmarks (B.Sc.) Markus Kaltenecker

2021

Image Reconstruction from Human Brain Activity by Variational Autoencoder and Adversarial Learning (M.Sc.) Mariia Podguzova**

 

Analysing the Influence of Facial Feature Alteration on Person Identification (B.Sc.) Britta Schulz

*Co-supervised with Ekta Sood

---

Publications

(Zermiani et al., 2024; Strohm et al., 2024; Strohm et al., 2024; Strohm et al., 2023; Penzkofer et al., 2023; Strohm et al., 2023; Strohm et al., 2021; Sood et al., 2021; Strohm & Klinger, 2018)

1. 

   Individual differences in visuo-spatial working memory capacity and prior knowledge during interrupted reading

   Francesca Zermiani, Prajit Dhar, Florian Strohm, Sibylle Baumbach, Andreas Bulling, Maria Wirzberger

   Frontiers in Cognition, 3, pp. 1–9, 2024.

   Abstract Links BibTeX Project

   Interruptions are often pervasive and require attentional shifts from the primary task. Limited data are available on the factors influencing individuals’ efficiency in resuming from interruptions during digital reading. The reported investigation -conducted using the InteRead dataset -examined whether individual differences in visuo-spatial working memory capacity (vsWMC) and prior knowledge could influence resumption lag times during interrupted reading. Participants’ vsWMC capacity was assessed using the symmetry span (SSPAN) task, while a pre-test questionnaire targeted their background knowledge about the text. While reading an extract from a Sherlock Holmes story, they were interrupted six times and asked to answer an opinion question. Our analyses revealed that the interaction between vsWMC and prior knowledge significantly predicted the time needed to resume reading following an interruption. The results from our analyses are discussed in relation to theoretical frameworks of task resumption and current research in the field.

   doi: 10.3389/fcogn.2024.1434642

   Paper: zermiani24_fic.pdf

   @article{zermiani24_fic, title = {Individual differences in visuo-spatial working memory capacity and prior knowledge during interrupted reading}, author = {Zermiani, Francesca and Dhar, Prajit and Strohm, Florian and Baumbach, Sibylle and Bulling, Andreas and Wirzberger, Maria}, year = {2024}, doi = {10.3389/fcogn.2024.1434642}, pages = {1--9}, volume = {3}, journal = {Frontiers in Cognition} }
2. 

   Learning User Embeddings from Human Gaze for Personalised Saliency Prediction

   Florian Strohm, Mihai Bâce, Andreas Bulling

   Proc. ACM on Human-Computer Interaction (PACM HCI), 8 (ETRA), pp. 1–18, 2024.

   Abstract Links BibTeX Project

   Reusable embeddings of user behaviour have shown significant performance improvements for the personalised saliency prediction task. However, prior works require explicit user characteristics and preferences as input, which are often difficult to obtain. We present a novel method to extract user embeddings from pairs of natural images and corresponding saliency maps generated from a small amount of user-specific eye tracking data. At the core of our method is a Siamese convolutional neural encoder that learns the user embeddings by contrasting the image and personal saliency map pairs of different users. Evaluations on two saliency datasets show that the generated embeddings have high discriminative power, are effective at refining universal saliency maps to the individual users, and generalise well across users and images. Finally, based on our model’s ability to encode individual user characteristics, our work points towards other applications that can benefit from reusable embeddings of gaze behaviour.

   doi: 10.1145/3655603

   Paper: strohm24_etra.pdf

   @article{strohm24_etra, title = {Learning User Embeddings from Human Gaze for Personalised Saliency Prediction}, author = {Strohm, Florian and Bâce, Mihai and Bulling, Andreas}, year = {2024}, journal = {Proc. ACM on Human-Computer Interaction (PACM HCI)}, pages = {1--18}, volume = {8}, number = {ETRA}, doi = {10.1145/3655603} }

1. 

   SeFFeC: Semantic Facial Feature Control for Fine-grained Face Editing

   Florian Strohm, Mihai Bâce, Markus Kaltenecker, Andreas Bulling

   arXiv:2403.13972, pp. 1–18, 2024.

   Abstract Links BibTeX Project

   We propose Semantic Facial Feature Control (SeFFeC) - a novel method for fine-grained face shape editing. Our method enables the manipulation of human-understandable, semantic face features, such as nose length or mouth width, which are defined by different groups of facial landmarks. In contrast to existing methods, the use of facial landmarks enables precise measurement of the facial features, which then enables training SeFFeC without any manually annotated labels. SeFFeC consists of a transformer-based encoder network that takes a latent vector of a pre-trained generative model and a facial feature embedding as input, and learns to modify the latent vector to perform the desired face edit operation. To ensure that the desired feature measurement is changed towards the target value without altering uncorrelated features, we introduced a novel semantic face feature loss. Qualitative and quantitative results show that SeFFeC enables precise and fine-grained control of 23 facial features, some of which could not previously be controlled by other methods, without requiring manual annotations. Unlike existing methods, SeFFeC also provides deterministic control over the exact values of the facial features and more localised and disentangled face edits.

   Paper Access: https://arxiv.org/abs/2403.13972

   @techreport{strohm24_arxiv_2, title = {SeFFeC: Semantic Facial Feature Control for Fine-grained Face Editing}, author = {Strohm, Florian and Bâce, Mihai and Kaltenecker, Markus and Bulling, Andreas}, year = {2024}, pages = {1--18}, url = {https://arxiv.org/abs/2403.13972} }

1. 

   Usable and Fast Interactive Mental Face Reconstruction

   Florian Strohm, Mihai Bâce, Andreas Bulling

   Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 1–15, 2023.

   Abstract Links BibTeX Project

   We introduce an end-to-end interactive system for mental face reconstruction – the challenging task of visually reconstructing a face image a person only has in their mind. In contrast to existing methods that suffer from low usability and high mental load, our approach only requires the user to rank images over multiple iterations according to the perceived similarity with their mental image. Based on these rankings, our mental face reconstruction system extracts image features in each iteration, combines them into a joint feature vector, and then uses a generative model to visually reconstruct the mental image. To avoid the need for collecting large amounts of human training data, we further propose a computational user model that can simulate human ranking behaviour using data from an online crowd-sourcing study (N=215). Results from a 12-participant user study show that our method can reconstruct mental images that are visually similar to existing approaches but has significantly higher usability, lower perceived workload, and is 40% faster. In addition, results from a third 22-participant lineup study in which we validated our reconstructions on a face ranking task show a identification rate of 55.3%, which is in line with prior work. These results represent an important step towards new interactive intelligent systems that can robustly and effortlessly reconstruct a user’s mental image.

   doi: https://doi.org/10.1145/3586183.3606795

   Paper: strohm23_uist.pdf

   @inproceedings{strohm23_uist, author = {Strohm, Florian and B{\^a}ce, Mihai and Bulling, Andreas}, title = {Usable and Fast Interactive Mental Face Reconstruction}, booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)}, year = {2023}, pages = {1--15}, doi = {https://doi.org/10.1145/3586183.3606795} }
2. 

   Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning

   Anna Penzkofer, Simon Schaefer, Florian Strohm, Mihai Bâce, Stefan Leutenegger, Andreas Bulling

   Proc. Adaptive and Learning Agents Workshop (ALA), pp. 1–7, 2023.

   Abstract Links BibTeX Project

   While deep reinforcement learning (RL) agents outperform humans on an increasing number of tasks, training them requires data equivalent to decades of human gameplay. Recent hierarchical RL methods have increased sample efficiency by incorporating information inherent to the structure of the decision problem but at the cost of having to discover or use human-annotated sub-goals that guide the learning process. We show that intentions of human players, i.e. the precursor of goal-oriented decisions, can be robustly predicted from eye gaze even for the long-horizon sparse rewards task of Montezuma’s Revenge – one of the most challenging RL tasks in the Atari2600 game suite. We propose Int-HRL: Hierarchical RL with intention-based sub-goals that are inferred from human eye gaze. Our novel sub-goal extraction pipeline is fully automatic and replaces the need for manual sub-goal annotation by human experts. Our evaluations show that replacing hand-crafted sub-goals with automatically extracted intentions leads to a HRL agent that is significantly more sample efficient than previous methods.

   Paper: penzkofer23_ala.pdf

   @inproceedings{penzkofer23_ala, author = {Penzkofer, Anna and Schaefer, Simon and Strohm, Florian and Bâce, Mihai and Leutenegger, Stefan and Bulling, Andreas}, title = {Int-HRL: Towards Intention-based Hierarchical Reinforcement Learning}, booktitle = {Proc. Adaptive and Learning Agents Workshop (ALA)}, year = {2023}, pages = {1--7} }
3. 

   Facial Composite Generation with Iterative Human Feedback

   Florian Strohm, Ekta Sood, Dominike Thomas, Mihai Bâce, Andreas Bulling

   Proc. The 1st Gaze Meets ML workshop, PMLR, pp. 165–183, 2023.

   Abstract Links BibTeX Project

   We propose the first method in which human and AI collaborate to iteratively reconstruct the human’s mental image of another person’s face only from their eye gaze. Current tools for generating digital human faces involve a tedious and time-consuming manual design process. While gaze-based mental image reconstruction represents a promising alternative, previous methods still assumed prior knowledge about the target face, thereby severely limiting their practical usefulness. The key novelty of our method is a collaborative, it- erative query engine: Based on the user’s gaze behaviour in each iteration, our method predicts which images to show to the user in the next iteration. Results from two human studies (N=12 and N=22) show that our method can visually reconstruct digital faces that are more similar to the mental image, and is more usable compared to other methods. As such, our findings point at the significant potential of human-AI collaboration for recon- structing mental images, potentially also beyond faces, and of human gaze as a rich source of information and a powerful mediator in said collaboration.

   Paper: strohm23_gmml.pdf

   Paper Access: https://proceedings.mlr.press/v210/strohm23a.html

   @inproceedings{strohm23_gmml, title = {Facial Composite Generation with Iterative Human Feedback}, author = {Strohm, Florian and Sood, Ekta and Thomas, Dominike and B{\^a}ce, Mihai and Bulling, Andreas}, booktitle = {Proc. The 1st Gaze Meets ML workshop, PMLR}, pages = {165--183}, year = {2023}, editor = {Lourentzou, Ismini and Wu, Joy and Kashyap, Satyananda and Karargyris, Alexandros and Celi, Leo Anthony and Kawas, Ban and Talathi, Sachin}, volume = {210}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, pdf = {https://proceedings.mlr.press/v210/strohm23a/strohm23a.pdf}, url = {https://proceedings.mlr.press/v210/strohm23a.html} }

1. 

   Neural Photofit: Gaze-based Mental Image Reconstruction

   Florian Strohm, Ekta Sood, Sven Mayer, Philipp Müller, Mihai Bâce, Andreas Bulling

   Proc. IEEE International Conference on Computer Vision (ICCV), pp. 245-254, 2021.

   Abstract Links BibTeX Project

   We propose a novel method that leverages human fixations to visually decode the image a person has in mind into a photofit (facial composite). Our method combines three neural networks: An encoder, a scoring network, and a decoder. The encoder extracts image features and predicts a neural activation map for each face looked at by a human observer. A neural scoring network compares the human and neural attention and predicts a relevance score for each extracted image feature. Finally, image features are aggregated into a single feature vector as a linear combination of all features weighted by relevance which a decoder decodes into the final photofit. We train the neural scoring network on a novel dataset containing gaze data of 19 participants looking at collages of synthetic faces. We show that our method significantly outperforms a mean baseline predictor and report on a human study that shows that we can decode photofits that are visually plausible and close to the observer’s mental image. Code and dataset available upon request.

   doi: 10.1109/ICCV48922.2021.00031

   Paper: strohm21_iccv.pdf

   Code: Available upon request.

   Dataset: Available upon request.

   @inproceedings{strohm21_iccv, title = {Neural Photofit: Gaze-based Mental Image Reconstruction}, author = {Strohm, Florian and Sood, Ekta and Mayer, Sven and Müller, Philipp and Bâce, Mihai and Bulling, Andreas}, year = {2021}, booktitle = {Proc. IEEE International Conference on Computer Vision (ICCV)}, doi = {10.1109/ICCV48922.2021.00031}, pages = {245-254} }
2. 

   VQA-MHUG: A gaze dataset to study multimodal neural attention in VQA

   Ekta Sood, Fabian Kögel, Florian Strohm, Prajit Dhar, Andreas Bulling

   Proc. ACL SIGNLL Conference on Computational Natural Language Learning (CoNLL), pp. 27–43, 2021.

   Abstract Links BibTeX Project Oral Presentation

   We present VQA-MHUG - a novel 49-participant dataset of multimodal human gaze on both images and questions during visual question answering (VQA) collected using a high-speed eye tracker. We use our dataset to analyze the similarity between human and neural attentive strategies learned by five state-of-the-art VQA models: Modulated Co-Attention Network (MCAN) with either grid or region features, Pythia, Bilinear Attention Network (BAN), and the Multimodal Factorized Bilinear Pooling Network (MFB). While prior work has focused on studying the image modality, our analyses show - for the first time - that for all models, higher correlation with human attention on text is a significant predictor of VQA performance. This finding points at a potential for improving VQA performance and, at the same time, calls for further research on neural text attention mechanisms and their integration into architectures for vision and language tasks, including but potentially also beyond VQA.

   doi: 10.18653/v1/2021.conll-1.3

   Paper: sood21_conll.pdf

   Code: https://git.hcics.simtech.uni-stuttgart.de/public-projects/vqa-mhug-interpretability

   Dataset: https://perceptualui.org/research/datasets/VQA-MHUG/

   @inproceedings{sood21_conll, title = {VQA-MHUG: A gaze dataset to study multimodal neural attention in VQA}, author = {Sood, Ekta and Kögel, Fabian and Strohm, Florian and Dhar, Prajit and Bulling, Andreas}, booktitle = {Proc. ACL SIGNLL Conference on Computational Natural Language Learning (CoNLL)}, year = {2021}, pages = {27--43}, doi = {10.18653/v1/2021.conll-1.3}, publisher = {Association for Computational Linguistics} }

1. 

   An Empirical Analysis of the Role of Amplifiers, Downtoners, and Negations in Emotion Classification in Microblogs

   Florian Strohm, Roman Klinger

   IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 673-681, 2018.

   Abstract Links BibTeX Project

   The effect of amplifiers, downtoners, and negations has been studied in general and particularly in the context of sentiment analysis. However, there is only limited work which aims at transferring the results and methods to discrete classes of emotions, e.g., joy, anger, fear, sadness, surprise, and disgust. For instance, it is not straight-forward to interpret which emotion the phrase "not happy" expresses. With this paper, we aim at obtaining a better understanding of such modifiers in the context of emotion-bearing words and their impact on document-level emotion classification, namely, microposts on Twitter. We select an appropriate scope detection method for modifiers of emotion words, incorporate it in a document-level emotion classification model as additional bag of words and show that this approach improves the performance of emotion classification. In addition, we build a term weighting approach based on the different modifiers into a lexical model for the analysis of the semantics of modifiers and their impact on emotion meaning. We show that amplifiers separate emotions expressed with an emotion-bearing word more clearly from other secondary connotations. Downtoners have the opposite effect. In addition, we discuss the meaning of negations of emotion-bearing words. For instance we show empirically that "not happy" is closer to sadness than to anger and that fear words in the scope of downtoners often express surprise.

   doi: 10.1109/DSAA.2018.00087

   Paper: strohm18_dsaa.pdf

   @inproceedings{strohm18_dsaa, author = {Strohm, Florian and Klinger, Roman}, booktitle = {IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA)}, title = {An Empirical Analysis of the Role of Amplifiers, Downtoners, and Negations in Emotion Classification in Microblogs}, year = {2018}, pages = {673-681}, doi = {10.1109/DSAA.2018.00087} }