Hand interaction in commercially available Virtual Reality systems

Physical and virtual correlation

As humans, one of our most important interaction devices with the physical world is our hands. Nowadays, when more users experience Virtual Reality (VR) for the first time one of the first reactions is to try and use their hands to interact with objects.

In this article I discuss the hand interaction device types of upcoming Virtual Reality Systems and how they match with what your brain expects.

The power of the brain

The desire to use your hands in VR is a reaction stemming from the fact that the brain is starting to processing the surroundings as if you are actually being physically located in the virtual world. Over the years we have equipped computers with different devices such as a the keyboard, game pads and space wands. These have been used successfully in both leisure and professional applications. Yes, they can be used in VR to a certain degree but are inadequate for more novel and natural interactions – due to their discrete nature (button press on/off, mouse movement left/right). Using these tools it is impossible for you as a user to map your actual physical movement to the same motion in the virtual world.

When using a novel VR system your brain begins to tell your consciousness that you are being “teleported” to an alternate space. Remember that your physical body accompanies your brain to this place. If you are able to use your body in this place as your normally do the level of presence can increase substantially. However this requires a good correlation between what you see and what your body experiences.

Toward more natural input devices

Traditional Hand Controllers

Traditional Hand Controllers

Beyond keyboards game pads  there are various modern technologies like Leap Motion which uses light to track the state of hands in front of a VR Head-Mounted Display (HMD). Currently the hand tracking is being performed from one physical location.

The limitation with systems like the Leap Motion is that when your hands move out of the tracked space they can no longer be updated in the virtual environment (VE). To be concrete this means that you cannot interact with virtual objects that are located behind your physical self. Although having improved substantially in the latest software update the possibility for fine tuned interaction with objects using Leap Motion is limited. To be able to work with virtual objects with millimetre precision we have to use other tracking systems. Fortunately there are three VR systems right around the corner that regular consumers finally will be able to buy and enjoy this year.

The three major upcoming VR systems

HTC Vive

The HTC Vive VR system uses two cubes fitted with laser emitter that you position in the opposite corner of a room. Its HMD and the two stick-shaped hand controllers utilises internal sensors and a cyclic light pattern from the laser emitters to continuously calculate their orientation and position in the room. In practise this makes it possible to interact with objects located in front of your face (as in the case with Leap Motion) as well as behind your body. The hand controllers are however not able to track your individual fingers.

Oculus Touch

In comparison to the hand controllers in HTC Vive are the Oculus Touch hand controllers from Oculus VR. Formed basically as a pistol grip their orientation and position is being tracked by internal sensors and external cameras. Its HMD is less bulky in comparison to the HTC Vive. The Oculus Touch controllers will not be shipped in the initial consumer release from Oculus but are to be sold separately several months later. The player instead has to use her keyboard, mouse and an XBox 360 controller.

Playstation VR

The Playstation VR system is expected to arrive in October 2016and comes with a HMD and two hand controllers – these are also shaped like sticks as in the HTC Vive system. This system connects to a Playstation 4 meaning that there is no need to upgrade your current console compared to having to upgrade your old PC or buy a new PC for about 1500 USD.

Possibility to create unique experiences based on input solution

During 2016 people will be able to experience very novel and technically advanced VR systems at a fraction of the cost of current state of the art VR systems. With a number of differently designed hand controllers developers will create experiences tailored for each specific system. In the case of the Oculus Touch you are able to see a virtual hand holding typical objects such as sticks and weapons. This could prove especially useful in scenarios when there is a match with what you actually see in the VE and what you normally see and feel in the physical world. For HTC Vive the player instead encounters a representative 3D model of the actual physical game controller, meaning there is little difference between what you see and what you actually hold physically.

Leap Motion

Leap motion mounted on an Oculus Rift

A market for small companies

Besides these three big players there are many more smaller companies working on developing input devices for hand interaction. Many of these typically utilise vibration actuators and sliding contactors to provide haptic feedback. Several of these focus on developing wireless support for the Mobile VR systems (mobile phones capable of rendering simplified environments) which as of now lack a “standard” input device besides various standard Bluetooth controllers. How they will succeed in the long term will be decided by the user when they literally get their hands on them.

In the market of VR the solutions that ultimately will succeed are likely to be the ones where the expectation of the user is met by an adequate match between what your brain thinks and what your body feels. Thus a natural selection processes will determine which technologies will prove viable.


2 thoughts on “Hand interaction in commercially available Virtual Reality systems

  1. I use a Beboncool bluetooth controller for my Gear VR. Actually just got it working today. What are the chances of getting a Leap Motion to work w/ Gear VR

    • That is currently not supported by default. There are some people who have done hacks with Gear VR but as of right now that is a bit of a hard time.

Leave a Reply

Your email address will not be published. Required fields are marked *