For the last 6 months, I’ve had the pleasure of championing the UX design for a multi-platform app that improves the travel experience for over 100 thousand Sydney commuters every day – the Opal Travel app (“Opal”).

Opal Travel App - UX Design Team

For those readers unfamiliar with Opal, it is equivalent to the myki card in Melbourne, Oyster card in the UK, or Metrocard in NYC.

The Opal user base has a very wide demographic with many users (including me), relying on it most when we’re running late for the train, the bus, the ferry, have three bags and a laptop, a shoelace is untied, feel in a panic and… it’s only 8 am.

To tap into the inner minds of these flustered, overworked commuters, we use (or used to use pre-COVID), guerrilla-style usability testing to collect feedback on new concepts for the app.

Approaching random commuters, at Central Station and the bus shelters of Sydney, was both surprisingly, exhilarating and a rapid way to collect contextual insights.

This style of approaching random members of the general public to huddle over a phone while they wait for the train was very effective…..Then came the coronavirus 😅😐😞.

I’m sure as other designers can relate, a user-centered logistical nightmare. With social distancing, our traditional face-to-face methods of user research and testing have been turned completely upside down.

Deserted train stations of Sydney, where we formerly conducted guerrilla user testing

The recent pandemic has presented us with a new form of problem statement

“How to connect with users when we’re physically disconnected?”.

Over the last couple of weeks, my partner in crime from the ‘Transport for NSW’ design team (@Alex.Junus) and I have recruited and conducted 15 user validation sessions over two rounds of testing.

We’ve learnt a lot about how to improve our new feature design, but also how to conduct successful remote user testing sessions using:

  1. The constricted resources available to us, and
  2. Minimal set-up fiction for our participants.

This article touches on our approach, what we learnt, and how we adapted in these uncertain times.

Our approach

Tools and Resources

The first thing we had to consider was the budget and choice of remote meeting tools. When it comes to remote user testing, there are a number of moderated and unmoderated tools on the market such as Loop11 and Validately.

Although brilliant, these tools are upwards of $200 a month for the ‘basic’ features. Given the ultimate provider for our project is down 80% in public transport usage (#stayhome), I’d have a difficult time convincing stakeholders to invest in these tools when there are cheaper options available.

So, we used Teams for moderated testing, using video and screen share. This worked really well, since the client already had a license, and it was easy for non-tech savvy participants to click one web link, and follow a few simple permission prompts.

Another compatible tool would be Zoom, which has a minimal barrier to entry (ie. it’s a web extension to add to your calendar), and will set you back about $20 a month.

I personally like Teams, as it also has as a feature for participants to screen share their mobile device – if they download the app.

Remote user testing - Work from home setup

Recruitment

The second thing we had to consider was time & recruitment. Due to business needs, we had to complete user testing quickly, without guerrilla-style testing, or the timely and pricey luxury of passing recruitment on to a third party. So with this in mind, we recruited from the people we could easily reach.

This included friends, family, and colleagues at Tigerspike and Transport for NSW who weren’t across the project, and were primary users of our Opal Travel app.

For those who don’t have close access to users, consider crowdsourcing through other means, such as a blast on social media and/or a banner on the website. Note, this method may require a little extra time to get approval from teams that manage these channels.

Since our recruitment didn’t involve the general public, we did not need to offer incentives. However, if your target is the general public, I’d recommend offering incentives appropriate for the times, such as Uber eats vouchers or Netflix gift cards.

Participant Consent

Also with the internal recruitment approach, we chose to by-pass the standard user testing practice of signing consent forms; including this as a written requirement in the communication email. “By signing up”, participants “agreed to the recording of video and audio, for sharing within the project team”.

For sessions that require a more formal approach with the general public, user testing consent forms could be shared and signed prior to the session.

Scheduling

Without a third-party recruiter or dedicated admin, the next hill we had to climb was scheduling. No one wants to spend their days endlessly calling to organise and shuffle the jigsaw puzzle which is scheduling.

For this we used Calendly. No joke, a lifesaver, as it allows you to set your availability and have participants schedule, re-schedule and/or cancel their time using a simple calendar link. The basic version is all you need…and it’s free.

Additional ‘planning’ tips

To wrap up this section of the approach  (i.e the planning), I’d also highly recommend the following that helped us sail smoothly through user testing.

  • Send participants an SMS ahead of time that requires a Y/N to confirm. Adding this ‘action’ will help decrease the chance that they will forget, and you can call the no replies.
  • Create a brief but instructional “how to set up guide” as a PDF.
    Include screenshots, prototype links, and share it with participants ahead of time.

Have a plan B in case tech/connectivity issues arise.
For us, it was a simple phone call and a prototype link shared with the participant (without screen share). Provided you encourage participants to speak out loud on what they are seeing and/or thinking – this plan B is not ideal, but does work.

Example - ‘how to set up PDF guide'

Key learnings

So after all the online user research and testing is conducted, what do we do next?

As masters of the trade we combine all the data, extract insights, and reflect on what we could do better next time.

In the realms of the remote user testing world, based on my experience I learnt:

1. The hypothesis, ‘Conducting user testing remotely wouldn’t be as insightful as face-to-face methods’…….was proven false. In fact, participants were highly engaged and honest, which gave us great insight into how to improve the experience. Perhaps the less confronting, informal setting, from the comfort of home, allowed them to feel more open to share their thoughts.

2. Never assume every participant has pre-read the ‘setup guide’ (a shoutout and big thank you to those that do. Woohoo!)

3. Scheduling an extra 15min per participant – is time well allocated (ie. be prepared for connectivity and tech issues).

4. Knowing the settings required per device is valuable, so you can swiftly help guide your participants through permissions for video, mic and screen share. Like – which everyday Joe would automatically think to find screen share on mac through ‘setting > privacy > screen recorder’? – I have my doubts.

5. Permissions, again!! Turning on permissions during a Teams or Zooms call will require participants to jump off and restart.
Kindly remind your participants to pre-read the setup guide, and pray to the design gods that they do. But hey, if they don’t, no biggie thanks to your pre-planning and lesson learnt #5.

6. Having a key facilitator and dedicated note taker in the session was super useful. They say 2-3 people are great company, while 4+ is a crowd. Be sure to keep the number of people to a minimum and introduce everyone on the line to help the participant feel comfortable.

#LessonLearnt for us, also ensure the facilitator is a meeting host with access to mute other participants. This is especially important with client observers on the line, who might take other calls and watch TV mid-session.

7. Be in tune with all your senses. One downside of remote user testing is the restricted ability to observe body language and facial expressions. Listen out carefully for changes in tone of voice, and long pauses as a cue to probe participants further.

8. And lastly, accept that we’re in this for the long haul.
The use of physically disconnected methods of user research and testing, will likely grow and become the new norm.

Embracing the future

As much as I feel like I’m a better, more empathetic user-centered designer, through being in physical contact with people; it’s going to be a long journey back to any routine or process that was considered normal as part of 2019.

Let’s be honest – Thinking back to my former user testing methods; In the near future will any person trust a random stranger approaching them on a train platform to say  “Hey, please provide feedback for our app – would you mind standing really close to me, while you tap a device that has been touched by many other people” – yeah probably not!!!

Without devaluing the benefits of in-person user research/testing, taking the half glass full approach; this pandemic will change expectations on how we connect with people to deliver value for our clients, and increase the ‘range’ in which we can connect.

Prior to this experience, I was a firm believer that face-to-face user research and testing methods were required to get the best result. However, the demand for face-to-face connection comes with the constraints of location, time availability, and lifestyles of your users.

If user research/testing participants no longer have to travel during ‘reasonable’ work hours, then they can take part remotely in a lunch break, while taking a walk, in a different city, in a different state…you get the picture.

As Covid-19 restrictions begin to ease, I’d like to continue to build upon the benefits of remote user testing, while finding a way to add real-life context to our tests.

This reminds me of a UX-Research Sydney meetup that I attended at Atlassian. It showcased the work of 2 UX Designers from Google, who transformed a typical user testing lab with limited budget, to emulate a driving setting for Google maps. The lab included an Xbox steering wheel, a car chair from a junkyard, and good old duct tape.

Using this story as inspiration, perhaps the next round of Opal user testing could involve participants (with minimal effort ) screen-sharing their device when using the app in-the moment for their commute. Not only is this contactless, but it adds environmental context, like the google maps example.

Potential for remote usability testing - in the environment

With this in mind, what we can, and should continuously strive to do is to use our vivid imaginations to get the best possible outcome for our user/customers, based on what we have available. Regardless of whether it’s limited by time, resources, user constraints, and/ or the inability to physically connect.

To finish my first ever blog post

As much I am profoundly saddened by the current state of the world, and give my deepest condolences to all families and health workers affected, this experience has taught me that we are extremely strong and resilient. This is proven – through something as small as modifying design user testing methods (i.e. this article), through to the much bigger generosity of the community, and efforts to find a vaccine/cure.

We are creatively capable of adapting to whatever lies ahead and can effectively connect, even when we are physically disconnected.

Renee Cuda

Senior UX Designer,Sydney

Renee Cuda

Senior UX Designer,Sydney