Dually Noted: Layout-Aware Annotations with Smartphone Augmented Reality

Jing Qian, Qi Sun, Curtis Wigington, Han Han, Tong Sun, Jennifer Healey, James Tompkin, Jeff Huang
ACM Conference on Human Factors in Computing Systems (CHI) 2022
PDF Video

Abstract

Sharing annotations encourages feedback, discussion, and knowledge passing among readers and can be beneficial for personal and public use. Prior augmented reality (AR) systems have expanded these benefits to both digital and printed documents. However, despite smartphone AR now being widely available, there is a lack of research about how to use AR effectively for interactive document annotation. We propose Dually Noted, a smartphone-based AR annotation system that recognizes the layout of structural elements in a printed document for real-time authoring and viewing of annotations.
We conducted experience prototyping with eight users to elicit potential benefits and challenges within smartphone AR, and this informed the resulting Dually Noted system and annotation interactions with the document elements.
AR annotation is often unwieldy, but during a 12-user empirical study our novel structural understanding component allows Dually Noted to improve precise highlighting and annotation interaction accuracy by 13%, increase interaction speed by 42%, and significantly lower cognitive load over a baseline method without document layout understanding. Qualitatively, participants commented that Dually Noted was a swift and portable annotation experience. Overall, our research provides new methods and insights for how to improve AR annotations for physical documents.

Bibtex

@inproceedings{qian2022dually,
title={Dually Noted: Layout-Aware Annotations with Smartphone Augmented Reality},
author={Qian, Jing and Sun, Qi and Wigington, Curtis and Han, Han L and Sun, Tong and Healey, Jennifer and Tompkin, James and Huang, Jeff},
booktitle={CHI Conference on Human Factors in Computing Systems},
pages={1–15},
year={2022}
}