Virtual keyboards are one of the biggest pain points in augmented reality. They’re slow, they’re awkward, and holding your arm up to type in midair gets tiring fast. A UT Dallas team thinks they’ve found a better way. The researchers said improving text input could help make AR more practical for everyday and professional use.

Researchers in the Erik Jonsson School of Engineering and Computer Science have developed a new AR interface called PropType that lets users turn everyday objects, like a coffee mug, book, or water bottle, into typing surfaces. The patent-pending technology overlays an augmented keyboard onto a handheld object and adapts to different shapes, including curved ones.

“By integrating objects already present in the user’s surroundings, PropType reimagines text input in AR, creating a seamless connection between the physical and virtual worlds,” said Jin Ryong Kim, assistant professor of computer science. “PropType capitalizes on the tactile feedback provided by the objects themselves, offering better key confirmation and reducing reliance on visual cues.”

A more natural way to type in AR

Student researchers in Kim’s Multimodal Interaction Lab created a video showing PropType in action across several objects. The team said the goal is to offer an option that feels more intuitive than virtual keyboards or external physical ones, which can break the immersive experience.

“PropType leverages the familiarity of handheld objects to offer a more intuitive and accessible alternative to traditional keyboards, particularly in mobile or hands-free scenarios where conventional input methods are impractical,” Kim said.

The team studied how people naturally hold and interact with different objects. They observed 16 participants to understand grab postures and typing gestures, then designed custom keyboard layouts for each object type. The prototype also includes an editing tool so users can adjust layouts and visual effects.

PropType earned a Best Paper Honorable Mention Award in April at the ACM CHI Conference on Human Factors in Computing Systems in Yokohama, Japan, and was later featured at the ACM Symposium on User Interface Software and Technology in Busan, South Korea.

Touch in virtual worlds

PropType builds on Kim’s broader research in haptics, a field that examines how touch, temperature, and physical feedback can make digital systems feel more lifelike. His team has earned national and international recognition for work ranging from thermal-tactile integration to thermal masking, a phenomenon that tricks the brain into feeling heat or cold in a different location than where the sensation is produced.

“Touch carries a lot of information; it’s another form of communication that is underexplored in virtual and augmented reality,” Kim said.

The lab’s work on thermal masking, presented at the 2024 ACM CHI Conference, demonstrates how vibrations and heat can be combined to influence sensation. Kim said the research could eventually support more immersive interfaces for virtual reality, medical simulation, and other applications.

Kim’s co-authors on the PropType project include Hyunjae Gil, former UTD postdoctoral researcher and now assistant professor at the Daegu Gyeongbuk Institute of Science and Technology; Iniyan Joseph; and Ashish Pratap, a computer science doctoral student. The work was supported by a grant from the Institute of Information & Communications Technology Planning & Evaluation of South Korea.

Don’t miss what’s next. Subscribe to Dallas Innovates.

Track Dallas-Fort Worth’s business and innovation landscape with our curated news in your inbox Tuesday-Thursday.