Research - VR Barriers - Barriers

Methodology

Our approach for gathering data on barriers in VR environments.

Introduction:

The ΒιΆΉΤΌΕΔ VR Barriers Project ran from early 2018 to mid 2020 and involved over 100 participants with a range of impairments. The goal of the project was to capture and document the barriers present in Virtual Reality environments.

The project has four high level goals:

  1. To capture a dataset of observed barriers faced by a wide range of users within VR environments.
  2. To consider if the current inclusive design principles encompass all observed barriers. See Findings
  3. To develop a methodology for running user testing within VR environments
  4. To develop suitable environments for testing to take place within.

This document outlines the methodology across the following ares:

  • Overview

    • Analysis method
    • Test Environments

  • Practices

    • Recruitment
    • Participants
    • Locations
    • Equipment
    • Travel
    • Safety
    • Legal & Safeguarding

Overview

The main deliverable of the project is a dataset of barriers. We collected this dataset by visiting participating organisations to work directly with participants. See recruitment for more information about how we approached working with organisations and recruiting participants.

We sought to work with users from all impairment groups, however our project was cut short by the covid-19 pandemic. We were unable to work with users with minimal limited vision or hearing.

In order to maintain control, repeatability and ensure participant safety we developed a VR environment based on a school library (see appendix A for images).

Each session was actively facilitated (see facilitation guide) with the facilitator supporting the user to complete three tasks relating to navigation, interaction and information gathering.

For each session we recorded the following information:

  1. Any barriers experienced
  2. The impairment groups the participant was present in.
  3. If the user was the youngest or oldest user we had worked with.

We did not record identifying information. The final barriers dataset is indicative or barriers, but is not of sufficient size for statistically significant conclusions broken down by age or gender.

See the analysis section of this document to learn more about our approach to analysing the data.

Equipment

To conduct the research we had to assemble a portable testing rig. We traveled with the rig to each of the participant organisations in order to conduct test sessions.

Initially we looked at running the sessions within a tent, but this proved to be impractical as many places did not have enough height. For most sessions we were able to conduct the session in one room, with participants waiting in another room.

The rig was contained within a custom made 1150 Pelicase. Having as much of the equipment as possible inside a single case was key. It helped with portability on public transport and helped to ensure nothing got lost.

The case is also waterproof and kept the equipment safe during a sudden downpour while traveling to a participant organisation. The case could also be locked for security.

We over-specced the computing hardware in order to ensure we had ample performance headroom for development. Choosing a notebook platform benefited portability but constrained performance. This trade off worked well.

We used the following equipment for the project:

HMD

For HMD we went with a 2016 HTC Vive. We felt this headset provided a good balance of visual quality, weight, software stability and development ease.

We used standard HTC vive controllers and lighthouses. We augmented the standard controllers with switches where needed. Please see the Switch Control document for details.

We used a set of 2.1 meter tall stands for the lighthouses placing them in opposing corners of the test space. For safety reasons we ensured that the lighthouses were places higher than our tallest participant. This minimizes the chance of stutters and dropper frames.

For audio we used the . The strap also helped to balance the mass of the headset front to back.

If we were running the research again we would seek out a headset which was wireless and better balanced front to back. Some participants with motor impairment found the headset to be front heavy.

Computing

We used a ASUS ROG Zephyrus (model ID: GX501VI-XS74) notebook for generating the Virtual Reality environments. It has the following specifications:

Component Specification
CPU Intel Core i7-7700HQ
GPU NVIDIA GeForce GTX 1080 8GB Max Q
RAM 16gb DDR4
SSD 512SSD

We tuned our environments to deliver stable performance on this hardware. Please see our performance tuning document for details on how we evaluated and tuned the performance of the computer and our software.

Mounts

In some locations we where unable to use the 2.1m tall lighthouse stands. In these situations we TinyRig mounts to attach the lighthouses to fixtures within the venues.

Logistics:

Travel

We chose primarily to carry out our research at locations which were controlled by, or familiar to, the participants. The reason for this was to create an environment where the participants would be as comfortable as possible, allowing us to more easily work with them and get good observations quickly without unnecessary pressure.

In order to do this, we had to consider how we would create an effective mobile VR environment and subsequently, get it to the participants. Ultimately this meant using personal transport (car) as the mobile environment consisted of multiple cases and bags, which would be impractical on public transport.

In addition, having a car allowed us greater flexibility around research locations and enabled multi-day research trips in one or multiple locations (i.e. it enabled us to stay in a wide range of hotels). Finally, the car represented a known β€˜escape route’ or safe space for ND members of the team, which ultimately made the research viable.

Location

Research locations were first defined by their proximity to the participants. As the participants were recruited, we would also be investigating (considering?) where we would carry out the research. Often, this was relatively simple as the participating organisation would have such a space available in the form of a hall or private large office space. This also would mean that practical considerations would have been made for participant access, such as wheelchair access or other needs, as the space would be familiar or tailored to the participants' needs already.

Other than proximity, a key factor in a usable location for our VR research was for the space to be physically large enough for all related elements. This includes the VR play space, which would ideally be at least 2.5m square with a buffer zone of roughly another 0.5m before any physical objects. It also includes chairs, tables for the equipment and a monitor for watching the virtual environment.

Privacy and control of the space is also key, as this allows us to ensure that participants and observers are comfortable and focused. This would ideally mean a space which is used only by the team while we are working, is behind a closed door and there are no unexpected interruptions (the organisation is aware of our presence).

There are practical technical considerations also, such as having enough power sockets, an internet connection in case of updates, a monitor / television and somewhat unexpectedly, no IR noise - as this can affect the VR tracking (we observed that LED disco / house lights can affect this).

In addition to these considerations, it may be that the participant is coming to us at the ΒιΆΉΤΌΕΔ for the session. In these instances, one must also consider things such as how will the participant travel to site, participant access (guest passes and chaperones) and comfort-making (food, drink, bathroom access).

Safety

Safety is an important concern at all times, particularly when dealing with groups of people with diverse needs and with new technologies / content.

In order to ensure the safety of all involved, we undertook an initial risk assessment while keeping aware of emerging risks as we went ahead with the research, adjusting our processes as required.

Participant safety was our first priority, and was considered from two main perspectives - physical safety and mental / sensory.

Physically, we had to ensure that the play space was set up and maintained to be clear of obstacles, as the participant would not be aware of this while inside of the Virtual experience. This meant clearing an area of roughly 9m2 with an additional 0.5m buffer before any physical objects. This space would be marked by physical objects, such as chairs or tables. Initially we used tape marks on the floor but these were often not effective at keeping the play space clear.

If a chair or similar had to be used (as was the case for some user groups), it would have to be consciously removed once it was no longer required. The facilitator had to keep aware of the participants motion in relation to the space and physical objects (including themselves) as well as the VR equipment cabling, ensuring that the participant would not trip or otherwise be injured.

Mental / Sensory safety refers to the participants not being harmed by means of the VR experience itself. Considerations include the impact on pre-existing medical concerns, motion sickness, disorientation and the impacts these may have (sickness, uncoordinated motion, accidents). To address the concerns around pre-existing medical conditions, we collaborated with the ΒιΆΉΤΌΕΔ’s legal team and produced a consent form which would exclude those who would be too high risk (CITATION?LINK). This was frustrating at times, as it did exclude a sizable portion of potential participants. For any participants which were unclear, we consulted with the Research Lead for confirmation.

Around Sensory safety, the participants were initially asked if they had tried VR before and how they felt these times, if they had not, they were asked about any experiences with motion sickness. Once inside the experience, the facilitator had to stay aware of the participant’s comfort, asking questions and reassuring / ending the experience as necessary.

It became evident in our initial sessions that noise in the physical environment could disorient the participants, in particular having multiple speak from different directions. To prevent this, we controlled who was present in the physical space and the facilitator would be the primary communicator.

Documents

A number of pieces of documentation had to be produced in order to complete the research, these included informative posters, legal documents and functional documents.

Information was conveyed to participants orally as well as visually by means of posters detailing the main points of the research. This would be displayed in the β€˜waiting area’ so that participants could become familiar with our aims. That said, this became less and less relevant as the study progressed, as either participants did not have to wait long or the information was not utilised.

In future, we would reconsider this method of informing participants. Informing participants varied depending on location and other factors. We initially believed the posters would be a key way to share information, however the participants tended not to read them and instead waited for the oral explanation.

From the legal perspective, the final version of our 5 page consent form had to cover multiple aspects of risk and liability, this is available at.
(Do we now discuss these, what are the areas?)

In order to record observations, gather feedback data and ensure that the research flowed smoothly, we produced additional functional documents.

Checklists enable us to ensure that we are hitting main points and that the process is consistent, as well as ensuring that all equipment was packed and accounted for.

Standardised feedback forms ensured that observations were consistently formatted (for ease of data entry) and that additional assistants could be asked to observe with minimal training. Feedback forms were produced, with a focus on a simple method to give feedback. The participant was asked a question (For example, β€œHow was holding the controller?”) and then was presented with three options: Easy, OK or Difficult. These options had visual cues in the form of traffic light coloured β€˜thumbs-up’ or β€˜thumbs-down’ check boxes, with an area for comments if they wished to give additional information.