Remote user testing for medical device development: advantages as well as challenges abound

Remote user testing allowed us to substantially accelerate the development of the CoVent™ ventilator. Dan Lock reviews the impact of social distancing on user research and finds that many of our necessary adaptations will continue to have value. Yet, each new study challenges us to come up with new approaches to inform medical device development.

During the CoVent ventilator development, and in many other medical device development projects since, the new imperative for social distancing has forced us to think a bit harder about how to use technology to support remote user studies. Despite initial reservations, we are finding that many aspects work surprisingly well.

In this blog, we share five things that will likely have a permanent impact on user research and consider the next challenges for remote user testing.

1. COVID-19 has helped people become more familiar with video conferencing tools 

2. Remote testing can help you access more and different types of people 

3. Remote testing allows researchers to get closer to the action

4. Remote testing accelerates user research and may make it more productive

5. Remote user testing brings multiple efficiency gains

Before COVID-19, we would have been reluctant to conduct user research studies via video chat apps – especially within medical device development where a significant number of participants could be older and, possibly, less familiar with technology. We may have been concerned that a significant proportion of the call would have been consumed with network issues, microphone, and camera settings, logging in and other distractions.

During lockdown people have been forced to get to grips with the technology in order to stay in touch with friends and family. In this way, COVID-19 has upskilled our participants, of all ages, and made it easier for us to use video chat apps for research. 

It has also given us more options. Whereas before people might only have used Skype, now their weekly pub quiz has made them familiar with, e.g., Zoom as well, giving us fall back options in interviews where we weren’t able to get our first choice of platform to work for whatever reason. 

This trend has made it much easier to run early-stage exploratory interviews where we are trying to understand user requirements, context of use, and to get feedback on designs. It has also enabled us to get a more representative sample of people online, whereas previously they would have been self-selected for only the more technologically confident users.

During the CoVent ventilator development, we were able to test our product with anaesthetists, ICU nurses and intensivists working at hospitals ranging from Northern Ireland to the West Country to Greater London and East Anglia. Many of these people were self-isolating at the time and keen to use their time productively. In more normal circumstances conducting face-to-face research, we would not have been able to get this level of diversity that quickly at the sample sizes we required. 

These advantages are not restricted to the product development space of course. They are also having a broader impact in healthcare and beyond. For example, physicians have told us that some health services traditionally delivered via hospital appointments, that previously would have required patients to take half a day off work, are being switched to 15 minute video calls, while local health centres take blood samples for the associated tests. This can be far more convenient for patients with chronic conditions and more cost effective, and the added convenience also applies to remote user research.

For hands-on product usability type tests, it can be harder to conduct remote research, but we have had success getting prototypes and products delivered in advance of the study. By using the laptops and smartphones available on site, we have been able to set up multiple camera angles and switch between them, akin to a TV producer, as we observe and ask questions via video link. This kind of set-up has allowed us to get closer to the user and to get better views of them using the product than in traditional in-person user research, without being intrusive or getting in the way. 

This has worked best for products intended for a commercial environment where the client has access to a pool of users in-house. In addition, our remoteness makes it easier for them to forget we are there; we are less of a presence in the room – this means less risk of influencing their behaviour.  

On the CoVent ventilator development project, we were able to do this in another way by keeping the physical prototypes on site and asking users to dial in. Having multiple cameras dialled in to the call offered physicians a variety of viewing angles and a person who is able to interact with controls on their behalf. 

A related advantage is that remote video chatting should also allow you to get into people’s homes, where they are comfortable, and see the context of use better than when interviewing at a neutral location. 

All of this also requires new solutions for protecting IP, as there is nothing to stop users taking screenshots or, if the product is shipped to them, showing it to other people – one of the new challenges for remote user research.

One of the ways we were able to accelerate the development of the CoVent ventilator, was by having designers and engineers on the call working in the background while we were getting feedback from physicians on the user interface design. This enabled us to edit and iterate the user interface with them in real time. Ten minutes later, we were able to show them a new version of the user interface incorporating our interpretation of their feedback, or to propose an idea inspired by what we had learned in the interview.

This massively accelerated a development cycle that previously would have taken days, if not weeks. In early stage exploratory research, this approach could facilitate co-development with the user or client to identify new opportunities or enable them to better qualify their requirements – respondents often re-evaluate what they want when they see it brought to life.

Remote testing is mediated by PCs via an ever-expanding suite of communication tools. While in traditional user research sessions we may have had a moderator and a separate observer taking notes, with remote testing it is more acceptable for the moderator to simultaneously take her own notes. With appropriate permissions, it is also possible to directly record from the screen (and generate automatic interview transcripts with varying degrees of accuracy). 

Furthermore, the widespread availability of video chat apps and the ability to connect multiple callers means that there is a much lower barrier for members of the development team from other disciplines to dial in and this can be arranged quickly, easily and cheaply using tools that are familiar to them (unlike the live streaming offered by research facilities). Plus conducting user research via video chat app also has significant logistical advantages with respect to travel and venue hire and is not subject to viewing room size restrictions (or social distancing limits).

COVID-19 is accelerating many pre-existing trends, and for user research this has not all been bad. In particular, people are now so much more familiar and comfortable with video chatting. For many types of user research, video chats can be advantageous, especially with regard to logistics and efficiency. At the same time, each new study challenges us to come up with novel approaches to elicit robust answers via remote user research. 

The next challenge will be to tackle studies that require getting products into people’s hands in their homes while protecting the IP/confidentiality. Logistics improvements could help with this. Instead of paying for participant travel, one could envisage special courier services that deliver a product, hand it over, and collect and bring the product back again one hour later, after the interview. Ongoing or unpredictable restrictions on travel may further increase the attractiveness of holding such studies remotely. 

Improvements in technology could also mean that video chats no longer degrade communication. Currently, it can still be difficult to make out what participants are saying, harder to hear their tone of voice, and harder to see their gestures — but cameras and internet speeds are only getting better.

Of course, video chats are unlikely to entirely replace face-to-face user research activities, for example where physical interaction with a device is required for regulatory approval. During the CoVent ventilator development, we progressed and accelerated the development by means of remote user testing as described above. Only when the prototype devices were ready and as deployment of the ventilator into hospitals was imminent within weeks, did we conduct socially distanced in-person testing of the device. 

In this respect it is likely that regulatory requirements will evolve as experience accumulates. User interfaces, graphical or physical, may be particularly amenable to remote testing. As developments progress towards regulatory submissions, we will no doubt collect further experience and insight into the stance regulators take on remote user testing. 

Even if lockdowns and restrictions become the new normal for a long while, our experience so far gives me confidence that we will continue to find ways to maximise the value we can get from remote user research.

How we help our clients | Medical Devices | Healthcare Blogs | Healthcare Case Studies