Publications
2025
HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality
Zhiming Hu, Guanhua Zhang, Zheming Yin, Daniel Häufle, Syn Schmitt, Andreas Bulling
IEEE Transactions on Visualization and Computer Graphics (TVCG), ,2025.
AbstractLinksBibTeXProject
Human hand and head movements are the most pervasive input modalities in extended reality (XR) and are significant for a wide range of applications. However, prior works on hand and head modelling in XR only explored a single modality or focused on specific applications. We present HaHeAE - a novel self-supervised method for learning generalisable joint representations of hand and head movements in XR. At the core of our method is an autoencoder (AE) that uses a graph convolutional network-based semantic encoder and a diffusion-based stochastic encoder to learn the joint semantic and stochastic representations of hand-head movements. It also features a diffusion-based decoder to reconstruct the original signals. Through extensive evaluations on three public XR datasets, we show that our method 1) significantly outperforms commonly used self-supervised methods by up to 74.1% in terms of reconstruction quality and is generalisable across users, activities, and XR environments, 2) enables new applications, including interpretable hand-head cluster identification and variable hand-head movement generation, and 3) can serve as an effective feature extractor for downstream tasks. Together, these results demonstrate the effectiveness of our method and underline the potential of self-supervised methods for jointly modelling hand-head behaviours in extended reality.
@article{hu25_tvcg,
title = {HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality},
author = {Hu, Zhiming and Zhang, Guanhua and Yin, Zheming and Häufle, Daniel and Schmitt, Syn and Bulling, Andreas},
year = {2025},
journal = {IEEE Transactions on Visualization and Computer Graphics (TVCG)},
doi = {10.1109/TVCG.2025.3576999},
}
SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation
Guanhua Zhang, Mohamed Ahmed, Zhiming Hu, Andreas Bulling
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--23, 2025.
AbstractLinksBibTeXProject
Recent work has highlighted the potential of modelling interactive behaviour analogously to natural language. We propose interactive behaviour summarisation as a novel computational task and demonstrate its usefulness for automatically uncovering latent user intentions while interacting with graphical user interfaces. To tackle this task, we introduce SummAct – a novel hierarchical method to summarise low-level input actions into high-level intentions. SummAct first identifies sub-goals from user actions using a large language model and in-context learning. High-level intentions are then obtained by fine-tuning the model using a novel UI element attention to preserve detailed context information embedded within UI elements during summarisation. Through a series of evaluations, we demonstrate that SummAct significantly outperforms baselines across desktop and mobile interfaces as well as interactive tasks by up to 21.9%. We further show three exciting interactive applications benefited from SummAct: interactive behaviour forecasting, automatic behaviour synonym identification, and language-based behaviour retrieval.
@inproceedings{zhang25_chi,
title = {SummAct: Uncovering User Intentions Through Interactive Behaviour Summarisation},
author = {Guanhua Zhang and Mohamed Ahmed and Zhiming Hu and Andreas Bulling},
year = {2025},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--23},
doi = {10.1145/3706598.3713190},
}
2024
DiffEyeSyn: Diffusion-based User-specific Eye Movement Synthesis
Chuhan Jiao, Guanhua Zhang, Zhiming Hu, Andreas Bulling
arXiv:2409.01240, pp. 1--22, 2024.
AbstractLinksBibTeXProject
High-frequency components in eye gaze data contain user-specific information promising for various applications, but existing gaze modelling methods focus on low frequencies of typically not more than 30 Hz. We present DiffEyeSyn – the first computational method to synthesise high-frequency gaze data, including eye movement characteristics specific to individual users. The key idea is to consider the high-frequency, user-specific information as a special type of noise in eye movement data. This perspective reshapes eye movement synthesis into the task of injecting this user-specific noise into any given eye movement sequence. We formulate this injection task as a conditional diffusion process in which the synthesis is conditioned on user-specific embeddings extracted from the gaze data using pre-trained models for user authentication. We propose user identity guidance – a novel loss function that allows our model to preserve user identity while generating human-like eye movements in the spatial domain. Experiment results on two public high-frequency eye movement biometric datasets show that our synthetic eye movements are indistinguishable from real human eye movements. Furthermore, we demonstrate that DiffEyeSyn can be used to synthesise eye gaze data at scale and for different downstream tasks, such as gaze data imputation and gaze data super-resolution. As such, our work lays the methodological foundations for personalised eye movement synthesis that has significant application potential, such as for character animation, eye movement biometrics, or gaze-based activity and context recognition.
@techreport{jiao24_arxiv_2,
title = {DiffEyeSyn: Diffusion-based User-specific Eye Movement Synthesis},
author = {Chuhan Jiao and Guanhua Zhang and Zhiming Hu and Andreas Bulling},
year = {2024},
pages = {1--22},
url = {https://arxiv.org/abs/2409.01240},
}
DisMouse: Disentangling Information from Mouse Movement Data
Guanhua Zhang, Zhiming Hu, Andreas Bulling
Proc. ACM Symposium on User Interface Software and Technology (UIST), pp. 1--13, 2024.
AbstractLinksBibTeXProject
Mouse movement data contain rich information about users, performed tasks, and user interfaces, but separating the respective components remains challenging and unexplored. As a first step to address this challenge, we propose DisMouse – the first method to disentangle user-specific and user-independent information and stochastic variations from mouse movement data. At the core of our method is an autoencoder trained in a semi-supervised fashion, consisting of a self-supervised denoising diffusion process and a supervised contrastive user identification module. Through evaluations on three datasets, we show that DisMouse 1) captures complementary information of mouse input, hence providing an interpretable framework for modelling mouse movements, 2) can be used to produce refined features, thus enabling various applications such as personalised and variable mouse data generation, and 3) generalises across different datasets. Taken together, our results underline the significant potential of disentangled representation learning for explainable, controllable, and generalised mouse behaviour modelling.
@inproceedings{zhang24_uist,
title = {DisMouse: Disentangling Information from Mouse Movement Data},
author = {Guanhua Zhang and Zhiming Hu and Andreas Bulling},
year = {2024},
booktitle = {Proc. ACM Symposium on User Interface Software and Technology (UIST)},
pages = {1--13},
doi = {https://doi.org/10.1145/3654777.3676411},
}
DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images
Chuhan Jiao, Yao Wang, Guanhua Zhang, Mihai Bâce, Zhiming Hu, Andreas Bulling
arXiv:2403.17477, pp. 1--13, 2024.
AbstractLinksBibTeXProject
We present DiffGaze, a novel method for generating realistic and diverse continuous human gaze sequences on 360° images based on a conditional score-based denoising diffusion model. Generating human gaze on 360° images is important for various human-computer interaction and computer graphics applications, e.g. for creating large-scale eye tracking datasets or for realistic animation of virtual humans. However, existing methods are limited to predicting discrete fixation sequences or aggregated saliency maps, thereby neglecting crucial parts of natural gaze behaviour. Our method uses features extracted from 360° images as condition and uses two transformers to model the temporal and spatial dependencies of continuous human gaze. We evaluate DiffGaze on two 360° image benchmarks for gaze sequence generation as well as scanpath prediction and saliency prediction. Our evaluations show that DiffGaze outperforms state-of-the-art methods on all tasks on both benchmarks. We also report a 21-participant user study showing that our method generates gaze sequences that are indistinguishable from real human sequences. Taken together, our evaluations not only demonstrate the effectiveness of DiffGaze but also point towards a new generation of methods that faithfully model the rich spatial and temporal nature of natural human gaze behaviour.
@techreport{jiao24_arxiv,
title = {DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images},
author = {Chuhan Jiao and Yao Wang and Guanhua Zhang and Mihai B{\^a}ce and Zhiming Hu and Andreas Bulling},
year = {2024},
pages = {1--13},
url = {https://arxiv.org/abs/2403.17477},
}
Mouse2Vec: Learning Reusable Semantic Representations of Mouse Behaviour
Guanhua Zhang, Zhiming Hu, Mihai Bâce, Andreas Bulling
Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 1--17, 2024.
AbstractLinksBibTeXProject
The mouse is a pervasive input device used for a wide range of interactive applications. However, computational modelling of mouse behaviour typically requires time-consuming design and extraction of handcrafted features, or approaches that are application-specific. We instead propose Mouse2Vec – a novel self-supervised method designed to learn semantic representations of mouse behaviour that are reusable across users and applications. Mouse2Vec uses a Transformer-based encoder-decoder architecture, which is specifically geared for mouse data: During pretraining, the encoder learns an embedding of input mouse trajectories while the decoder reconstructs the input and simultaneously detects mouse click events. We show that the representations learned by our method can identify interpretable mouse behaviour clusters and retrieve similar mouse trajectories. We also demonstrate on three sample downstream tasks that the representations can be practically used to augment mouse data for training supervised methods and serve as an effective feature extractor.
@inproceedings{zhang24_chi,
title = {Mouse2Vec: Learning Reusable Semantic Representations of Mouse Behaviour},
author = {Guanhua Zhang and Zhiming Hu and Mihai B{\^a}ce and Andreas Bulling},
year = {2024},
booktitle = {Proc. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI)},
pages = {1--17},
doi = {10.1145/3613904.3642141},
}
2023
Exploring Natural Language Processing Methods for Interactive Behaviour Modelling
Guanhua Zhang, Matteo Bortoletto, Zhiming Hu, Lei Shi, Mihai Bâce, Andreas Bulling
Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT), pp. 1--22, 2023.
AbstractLinksBibTeXProject Best Student Paper Nomination
Analysing and modelling interactive behaviour is an important topic in human-computer interaction (HCI) and a key requirement for the development of intelligent interactive systems. Interactive behaviour has a sequential (actions happen one after another) and hierarchical (a sequence of actions forms an activity driven by interaction goals) structure, which may be similar to the structure of natural language. Designed based on such a structure, natural language processing (NLP) methods have achieved groundbreaking success in various downstream tasks. However, few works linked interactive behaviour with natural language. In this paper, we explore the similarity between interactive behaviour and natural language by applying an NLP method, byte pair encoding (BPE), to encode mouse and keyboard behaviour. We then analyse the vocabulary, i.e., the set of action sequences, learnt by BPE, as well as use the vocabulary to encode the input behaviour for interactive task recognition. An existing dataset collected in constrained lab settings and our novel out-of-the-lab dataset were used for evaluation. Results show that this natural language-inspired approach not only learns action sequences that reflect specific interaction goals, but also achieves higher F1 scores on task recognition than other methods. Our work reveals the similarity between interactive behaviour and natural language, and presents the potential of applying the new pack of methods that leverage insights from NLP to model interactive behaviour in HCI.
@inproceedings{zhang23_interact,
title = {Exploring Natural Language Processing Methods for Interactive Behaviour Modelling},
author = {Zhang, Guanhua and Bortoletto, Matteo and Hu, Zhiming and Shi, Lei and B{\^a}ce, Mihai and Bulling, Andreas},
year = {2023},
booktitle = {Proc. IFIP TC13 Conference on Human-Computer Interaction (INTERACT)},
pages = {1--22},
publisher = {Springer},
}
2022
Predicting Next Actions and Latent Intents during Text Formatting
Guanhua Zhang, Susanne Hindennach, Jan Leusmann, Felix Bühler, Benedict Steuerlein, Sven Mayer, Mihai Bâce, Andreas Bulling
Proc. the CHI Workshop Computational Approaches for Understanding, Generating, and Adapting User Interfaces, pp. 1--6, 2022.
AbstractLinksBibTeXProject
In this work we investigate the challenging task of predicting user intents from mouse and keyboard input as well as gaze behaviour. In contrast to prior work we study intent prediction at two different resolutions on the behavioural timeline: predicting future input actions as well as latent intents to achieve a high-level interaction goal. Results from a user study (N=15) on a sample text formatting task show that the sequence of prior actions is more informative for intent prediction than gaze. Only using the action sequence, we can predict the next action and the high-level intent with an accuracy of 66% and 96%, respectively. In contrast, accuracy when using features extracted from gaze behaviour was significantly lower, at 41% and 46%. This finding is important for the development of future anticipatory user interfaces that aim to proactively adapt to user intents and interaction goals.
@inproceedings{zhang22_caugaui,
title = {Predicting Next Actions and Latent Intents during Text Formatting},
author = {Guanhua Zhang and Susanne Hindennach and Jan Leusmann and Felix Bühler and Benedict Steuerlein and Sven Mayer and Mihai Bâce and Andreas Bulling},
year = {2022},
booktitle = {Proc. the CHI Workshop Computational Approaches for Understanding, Generating, and Adapting User Interfaces},
pages = {1--6},
}
2021
MultiMediate: Multi-modal Group Behaviour Analysis for Artificial Mediation
Philipp Müller, Dominik Schiller, Dominike Thomas, Guanhua Zhang, Michael Dietz, Patrick Gebhard, Elisabeth André, Andreas Bulling
Proc. ACM Multimedia (MM), pp. 4878--4882, 2021.
AbstractLinksBibTeXProject
Artificial mediators are promising to support human group conversations but at present their abilities are limited by insufficient progress in group behaviour analysis. The MultiMediate challenge addresses, for the first time, two fundamental group behaviour analysis tasks in well-defined conditions: eye contact detection and next speaker prediction. For training and evaluation, MultiMediate makes use of the MPIIGroupInteraction dataset consisting of 22 three- to four-person discussions as well as of an unpublished test set of six additional discussions. This paper describes the MultiMediate challenge and presents the challenge dataset including novel fine-grained speaking annotations that were collected for the purpose of MultiMediate. Furthermore, we present baseline approaches and ablation studies for both challenge tasks.
@inproceedings{mueller21_mm,
title = {MultiMediate: Multi-modal Group Behaviour Analysis for Artificial Mediation},
author = {M{\"{u}}ller, Philipp and Schiller, Dominik and Thomas, Dominike and Zhang, Guanhua and Dietz, Michael and Gebhard, Patrick and André, Elisabeth and Bulling, Andreas},
year = {2021},
booktitle = {Proc. ACM Multimedia (MM)},
pages = {4878--4882},
doi = {10.1145/3474085.3479219},
}
SparseDGCNN: Recognizing Emotion from Multichannel EEG Signals
Guanhua {Zhang}, Minjing {Yu}, Yong-Jin {Liu}, Guozhen {Zhao}, Dan {Zhang}
IEEE Transactions on Affective Computing (TAFFC), ,pp. 1--12, 2021.
AbstractLinksBibTeXProject
Emotion recognition from EEG signals has attracted much attention in affective computing. Recently, a novel dynamic graph convolutional neural network (DGCNN) model was proposed, which simultaneously optimized the network parameters and a weighted graph G characterizing the strength of functional relation between each pair of two electrodes in the EEG recording equipment. In this paper, we propose a sparse DGCNN model which improves the DGCNN by imposing a sparseness constraint on G. Our work is based on an important observation: the tomography study reveals that different brain regions sampled by EEG electrodes may be related to different functions of the brain and then the functional relations among electrodes are possibly highly localized and sparse. However, introducing sparseness constraint into the graph G makes the loss function of sparse DGCNN non-differentiable at some singular points. To ensure that the training process of sparse DGCNN converges, we apply the forward-backward splitting method. To evaluate the performance of sparse DGCNN, we compare it with four representative recognition methods as well as different features and spectral bands. The results show that (1) sparse DGCNN has consistently better accuracy than representative methods and has a good scalability, and (2) DE, PSD and ASM features on γ bands convey most discriminative emotional information, and fusion of separate features and frequency bands can improve recognition performance.
@article{zhang21_taffc,
title = {SparseDGCNN: Recognizing Emotion from Multichannel EEG Signals},
author = {Guanhua {Zhang} and Minjing {Yu} and Yong-Jin {Liu} and Guozhen {Zhao} and Dan {Zhang}},
year = {2021},
journal = {IEEE Transactions on Affective Computing (TAFFC)},
pages = {1--12},
doi = {10.1109/TAFFC.2021.3051332},
}
2020
Multi-target Positive Emotion Recognition from EEG Signals
Guozhen Zhao, Yulin Zhang, Guanhua Zhang, Dan Zhang, Yong-Jin Liu
IEEE Transactions on Affective Computing (TAFFC), ,pp. 1-13, 2020.
AbstractLinksBibTeXProject
Compared with the widely studied negative emotions in which different classes are easy to distinguish, nowadays less attention is paid to the recognition of positive emotions that are not fully independent. In this paper, we propose to recognize multiple positive emotions by analyzing brain activities and explore the neural representation of different positive emotions. Thirty-seven participants volunteered to participate in our study, in which their brain activities were recorded when watching five selected film clips. First, 150 well-known power features extracted from Electroencephalography (EEG) signals and 105 multimedia content analysis features were collected as the pool of candidate features. Second, based on the collected features, we propose to use a linear model and a nonlinear model to predict the percentage of five positive emotions. Then, percentage values were converted to ranking numbers and Kendall rank correlation coefficients were calculated. Our results showed that (1) ensemble of regressor chains using LSTM as unit regressor obtained both the best regression results and the best Kendall rank correlation coefficient on EEG features merely, and (2) top features from alpha frequency bands of EEG signals could represent different positive emotions. These results demonstrate the effectiveness of selective EEG features on recognizing different positive emotions.
@article{zhao20_taffc,
title = {Multi-target Positive Emotion Recognition from EEG Signals},
author = {Guozhen Zhao and Yulin Zhang and Guanhua Zhang and Dan Zhang and Yong-Jin Liu},
year = {2020},
journal = {IEEE Transactions on Affective Computing (TAFFC)},
pages = {1-13},
doi = {10.1109/TAFFC.2020.3043135},
}
An Efficient LSTM Network for Emotion Recognition from Multichannel EEG Signals
Xiaobing Du, Cuixia Ma, Guanhua Zhang, Jinyao Li, Yu-Kun Lai, Guozhen Zhao, Xiaoming Deng, Yong-Jin Liu, Hongan Wang
IEEE Transactions on Affective Computing (TAFFC), ,pp. 1-12, 2020.
AbstractLinksBibTeXProject
Most previous EEG-based emotion recognition methods studied hand-crafted EEG features extracted from different electrodes. In this paper, we study the relation among different EEG electrodes and propose a deep learning method to automatically extract the spatial features that characterize the functional relation between EEG signals at different electrodes. Our proposed deep model is called ATtention-based LSTM with Domain Discriminator (ATDD-LSTM) that can characterize nonlinear relations among EEG signals of different electrodes. To achieve state-of-the-art emotion recognition performance, the architecture of ATDD-LSTM has two distinguishing characteristics: (1) By applying the attention mechanism to the feature vectors produced by LSTM, ATDD-LSTM automatically selects suitable EEG channels for emotion recognition, which makes the learned model concentrate on the emotion related channels in response to a given emotion; (2) To minimize the significant feature distribution shift between different sessions and/or subjects, ATDD-LSTM uses a domain discriminator to modify the data representation space and generate domain-invariant features. We evaluate the proposed ATDD-LSTM model on three public EEG emotional databases (DEAP, SEED and CMEED) for emotion recognition. The experimental results demonstrate that our ATDD-LSTM model achieves superior performance on subject-dependent (for the same subject), subject-independent (for different subjects) and cross-session (for the same subject) evaluation.
@article{du20_taffc,
title = {An Efficient LSTM Network for Emotion Recognition from Multichannel EEG Signals},
author = {Du, Xiaobing and Ma, Cuixia and Zhang, Guanhua and Li, Jinyao and Lai, Yu-Kun and Zhao, Guozhen and Deng, Xiaoming and Liu, Yong-Jin and Wang, Hongan},
year = {2020},
journal = {IEEE Transactions on Affective Computing (TAFFC)},
pages = {1-12},
doi = {10.1109/TAFFC.2020.3013711},
}
2019
A Review of EEG Features for Emotion Recognition (in Chinese)
Guanhua Zhang, Minjing Yu, Guo Chen, Yiheng Han, Dan Zhang, Guozhen Zhao, Yong-Jin Liu
SCIENTIA SINICA Informationis, 49 (9),pp. 1097-1118, 2019.
AbstractLinksBibTeXProject
Emotion recognition is an important research topic in the human-machine interaction field, and it can be applied to medicine, education, psychology, military, and other areas. Electroencephalogram (EEG) signals are mostly used among various indices of emotion recognition. High accuracy of emotion classifiers can be achieved by extracting the most relevant and discriminant features of emotion states. This study surveys EEG features that are extensively used in current emotion recognition studies by introducing EEG features from the following four viewpoints: time domain, frequency domain, time–frequency domain, and space domain. An SLDA algorithm is imported to three public EEG-emotion datasets (SEED, DREAMER, and CAS-THU) to evaluate feature capabilities that distinguish emotion valence. Existing problems and future investigations are also discussed in this paper.
@article{zhang19_ssi,
title = {A Review of EEG Features for Emotion Recognition (in Chinese)},
author = {Zhang, Guanhua and Yu, Minjing and Chen, Guo and Han, Yiheng and Zhang, Dan and Zhao, Guozhen and Liu, Yong-Jin},
year = {2019},
journal = {SCIENTIA SINICA Informationis},
volume = {49},
number = {9},
pages = {1097-1118},
doi = {10.1360/N112018-00337},
}
