Virtual-to-Physical Surface Alignment and Refinement Techniques for Handwriting, Sketching, and Selection in XR

This repository contains the source code of our alignment and refinement techniques using the Meta Quest Pro. We used the Meta Quest Touch Pro Controller with Stylus Tip attachment. Other controllers can also be used when extending the reference implementation. Please feel free to contact us!

The publication can be found here: 2023-ieeevr-workshop-alignment-refinement-techniques-publication.pdf.

DOI: https://doi.org/10.1109/VRW58643.2023.00109

Abstract

The alignment of virtual to physical surfaces is essential to improve symbolic input and selection in XR. Previous techniques optimized for efficiency can lead to inaccuracies. We investigate regression-based refinement techniques and introduce a surface accuracy evaluation. The results revealed that refinement techniques can highly improve surface accuracy and show that accuracy depends on the gesture shape and surface dimension.

The figure shows the 3ViSuAl (3V) (Kern et al. 2021) alignment technique and our refinement techniques. 4ViSuAl (4V) extends 3V by adding a fourth reference point at the top left for a more uniform distribution of points over the surface. Since three single measurements can lead to inaccuracies, the 3V refinement technique (3VRT) and the 4V refinement technique (4VRT) record multiple measurements per reference point. Based on continuous motion, we consider the line refinement technique (LRT), the rectangular refinement technique (RRT), and the cross refinement technique (XRT). Finally, as a hybrid between discrete and continuous motion, we propose the circle refinement technique (CRT).

Reference Implementation

The figure shows 3V (Kern et al. 2021) and three selected refinement techniques (3VRT, RRT, and CRT). Gray spheres represent the three points for alignment with 3ViSuAl, blue spheres an arbitrary number of measured points for refinement, and orange spheres the realigned 3V reference points. Gray areas show the surfaces aligned with 3V, and the blue areas the surfaces refined.

The demo video can be found here:

Reference

We licensed our application under the MIT-License. When using our alignment/refinement techniques please reference:

@INPROCEEDINGS{10108564,
  author={Kern, Florian and Tschanter, Jonathan and Latoschik, Marc Erich},
  booktitle={2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)}, 
  title={Virtual-to-Physical Surface Alignment and Refinement Techniques for Handwriting, Sketching, and Selection in XR}, 
  year={2023},
  volume={},
  number={},
  pages={502-506},
  doi={10.1109/VRW58643.2023.00109}
  }

This publication is part of the Off-The-Shelf Stylus (OTSS) framework, a framework for 2D interaction (in 3D) as well as for handwriting and sketching with digital pen, ink, and paper on physically aligned virtual surfaces in Virtual, Augmented, and Mixed Reality (VR, AR, MR: XR for short).

@article{10.3389/frvir.2021.684498,
  author = {Kern, Florian and Kullmann, Peter and Ganal, Elisabeth and Korwisi, Kristof and Stingl, René and Niebling, Florian and Latoschik, Marc Erich},
  title = {Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces},
  journal = {Frontiers in Virtual Reality},
  volume = {2},
  year = {2021},
  url = {https://www.frontiersin.org/articles/10.3389/frvir.2021.684498},
  doi = {10.3389/frvir.2021.684498}
}

We have used the circle refinement technique (CRT) in a follow-up work on handwriting text input in VR and VST AR, facilitated by mid-air and physically aligned surfaces. In this publication, we chose the Meta Quest Pro with Meta Quest Touch Pro controllers, offering a pressure-sensitive stylus tip.

@article{10460576,
  title        = {Handwriting for Text Input and the Impact of XR Displays, Surface Alignments, and Sentence Complexities},
  author       = {Kern, Florian and Tschanter, Jonathan and Latoschik, Marc Erich},
  journal      = {IEEE Transactions on Visualization and Computer Graphics},
  year         = {2024},
  pages        = {1-11},
  url          = {https://ieeexplore.ieee.org/document/10460576},
  doi          = {10.1109/TVCG.2024.3372124}
}

Institution

University of Wuerzburg
Chair for Human-Computer Interaction
Prof. Dr. Marc Erich Latoschik
https://hci.uni-wuerzburg.de/

Corresponding Person

Florian Kern
florian.kern@uni-wuerzburg.de
http://hci.uni-wuerzburg.de/people/florian-kern/

Other References

We would also like to refer to our other publications als listed below:

Florian Kern, Florian Niebling, Marc Erich Latoschik, Text Input for Non-Stationary XR Workspaces: Investigating Tap and Word-Gesture Keyboards in Virtual and Augmented Reality, In IEEE Transactions on Visualization and Computer Graphics (TVCG). 2023.
Publication: https://doi.org/10.1109/TVCG.2023.3247098
Source Code: https://go.uniwue.de/tap-and-word-gesture-keyboards

Florian Kern, Jonathan Tschanter, Marc Erich Latoschik, Virtual-to-Physical Surface Alignment and Refinement Techniques for Handwriting, Sketching, and Selection in XR, In IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 2023. To be published.
Publication: https://doi.org/willbeadded
Source Code: https://go.uniwue.de/alignment-and-refinement-techniques

Florian Kern, Matthias Popp, Peter Kullmann, Elisabeth Ganal, Marc Erich Latoschik, 3D Printing an Accessory Dock for XR Controllers and its Exemplary Use as XR Stylus, In 27th ACM Symposium on Virtual Reality Software and Technology, pp. 1-3. Osaka, Japan: Association for Computing Machinery, 2021.
Publication: https://doi.org/10.1145/3489849.3489949
Source Code: https://go.uniwue.de/hci-otss-accessory-dock

Florian Kern, Peter Kullmann, Elisabeth Ganal, Kristof Korwisi, Rene Stingl, Florian Niebling, Marc Erich Latoschik, Off-The-Shelf Stylus: Using XR Devices for Handwriting and Sketching on Physically Aligned Virtual Surfaces, In Daniel Zielasko (Ed.), Frontiers in Virtual Reality, Vol. 2, p. 69. 2021.
Publication: https://doi.org/10.3389/frvir.2021.684498
Source Code: https://go.uniwue.de/hci-otss

Florian Kern, Thore Keser, Florian Niebling, Marc Erich Latoschik, Using Hand Tracking and Voice Commands to Physically Align Virtual Surfaces in AR for Handwriting and Sketching with HoloLens 2, In 27th ACM Symposium on Virtual Reality Software and Technology, pp. 1-3. Osaka, Japan: Association for Computing Machinery, 2021. Best poster award.
Publication: https://doi.org/10.1145/3489849.3489940
Source Code: https://go.uniwue.de/hci-otss-hololens

Architecture

The example scenes for are located in the Scenes folder.

Currently, we provide an example for Meta devices. For supporting additional devices, add the XR plugin device plugin (e.g., the PICO Unity Integration SDK ). Copy the MetaDevices scene and replace the OVRCameraRigCustom. In addition, the techniques require a reference to an interaction point for point recording. Add respective interaction points (Anchor in the image) to each controller point. Note, when the target XR controller does not provide a thin tip, the alignment/refinement result will be influenced by controller rotations. We provide concepts for an off-the-shelf stylus or the 3D-printed accessory dock.

If you want to use another alignment/refinement technique or change the primary/secondy hand, please update script references and hand assignments. Feel free to add a reference manager so simplify the process.