Remote vs. In-Person Testing
Quick Tip!
In-person testing is often the preferred and more robust option for usability testing at the Digital Corps. However, the work-at-home protocol and social distancing have necessitated the need to adopt remote testing procedures.
Helpful Staff for this Topic
Usability testing is conducted to ensure the websites, applications, and experiences created by the Digital Corps are usable and functional. In its purest form, usability testing includes a proctor to facilitate the test, participants to interact with the system, and tasks for the participants to complete.
This article will describe the differences and similarities between in-person and remote procedures for each of the basic elements of usability testing, including:
- Location
- Device/System Interaction and Context
- Scheduling
- Script
- Participants
- Observations
Quick Tip!
For a refresher on user testing at the Digital Corps, review User Testing 101.
Location
In-Person
- an empty room or quiet space in the office
- the proctor, note-taker, and participants are all in the same physical space
Remote
- a web conferencing service, such as Zoom or WebEx
- the proctor, note-taker, and participant are all in separate physical spaces
Remote testing’s dependency on technology necessitates the need to check for and resolve connection issues with these third party services before you schedule time with your participants. These issues can include connection quality/speed, lack of a webcam, and if the service itself is available. If a UX-er cannot communicate with their participant, testing is effectively cancelled, making this a major weakness of remote testing.
Although you may check connection issues before the session begins, be prepared for the unexpected – power outage, dropped connection, external interruptions, or other issues that may affect the quality and ease of data collection. This will likely extend the time needed to complete the testing portion of the project.
Device/System Interaction and Context
The Digital Corps prides itself in developing systems for the latest technology. Augmented Reality (AR), Virtual Reality (VR), and interactive touch devices are all unique pieces of technology that require specific affordances in usability testing.
In-Person
- allows for flexibility in testing environments and provides the opportunity to simulate the context of a system during testing
- e.g. when the UX team evaluated the Hololens project, they created an environment with music playing, people talking, and friends distracting the participant because the intended environment was crowded and noisy.
Remote
- these affordances are not usually possible in a remote setting, or they may require additional considerations and planning
- e.g. a system designed to be used on a large wall display in a crowded, public space cannot be effectively tested from someone’s quiet bedroom on a laptop.
Additionally, with remote testing, participants may not have access to the devices that the system requires for testing (e.g. a monitor of specific dimensions or a certain mobile device). If you encounter this situation on a project, consider the following solutions:
- Conduct a pre-test questionnaire to determine if a participant is eligible before scheduling the testing session.
- Make the requirements of testing clear in all announcement and recruitment messaging.
Quick Tip!
A lack of context can be a major roadblock depending on the system you are evaluating. Connect with the UX team lead to determine if remote testing is feasible and/or may require additional in-person testing.
Scheduling
When it comes to scheduling, in-person and remote testing are fairly equal. You still need to balance several peoples’ schedules and send out calendar invitations. Remote testing does have an advantage because you do not need to check the availability of a testing space.
Note: depending on where your participants are located, you may need to account for differences in time zones.
Script
The differences in the in-person and remote scripts are minor and typically include changes to accomodate for the change in methods.
In-Person
- “I will hand each task to you on an index card”
- make the participant aware if you intend to record any aspect of testing
Remote
- “I will read each task aloud.”
- explain how to record or share the participant’s screen
Participants
The number of ideal participants is five, no matter how you’re testing. This number provides the most information proportionally, as described by the Nielsen Norman Group.
During any kind of testing, it is normal for a participant to become confused. It can be easier to address this during in-person testing because the proctor is physically present with the participant and can assist in working through the issue (e.g. if a page does not load properly or a device shuts off unexpectedly) In remote testing, you have to walk the participant through troubleshooting the issue on their own.
Observations
In-Person
- able to observe the participants’ body language and interpret their actions and words
- Note-taker and proctor should be two separate roles
Remote
- unable to observe body language
- screen recordings allows a UX-er to review the actions and behaviors of a participant
- Note-taker and proctor should be two separate roles
In remote testing, you may not be able to require participants to enable their video camera, but it is recommended that you encourage them to do so and explain the benefits.
Quality Assurance Testing
Usability testing is clearly affected by whether it is conducted in-person or remotely. However, Quality Assurance (QA) testing has proven to be relatively unchanged by a shift to remote work.
The standard procedure for QA testing can be followed during times of remote work. Review the “Creating a Testing Plan” article for a breakdown of this process.
Now what?
Based on these strengths and weaknesses, in-person testing is typically the preferred method of testing at the Digital Corps. Remote testing guidelines were not established at the Digital Corps until the COVID-19 pandemic required flexibility and quick thinking. With the restrictions in place for the Fall 2020 semester (and potentially beyond), remote testing will be a requirement for many of the Corps projects.
Remote testing should only occur when in-person testing is not possible. Further, this decision should be made with the whole project team, rather than by an individual UX-er.
