The (un)importance of online interview modes for security and privacy studies

In 2018, a team of CyLab researchers conducted a pilot interview study to learn about consumers’ privacy concerns when purchasing sensitive items such as sex aids, illegal drugs, or medical items online. Fearing that participants would be wary of disclosing their experiences, the researchers invested substantial effort to conduct the study using an anonymous instant messaging platform. Unfortunately, the platform was difficult to use, and many participants still provided their real names.

This experience led the researchers to question whether an anonymous chat interview would indeed lead interview participants to be more forthcoming than they would in a non-anonymous chat interview, or whether audio, video, survey, or other interview formats might yield better results. They read interview methods guides looking for answers and came up empty handed.

It turned out: no one knew which online interview mode was best, because no one had ever studied it before.

In a rigorous new study just published in the journal PLOS ONE, a team of CyLab researchers show that any of the seven online interview modes they studied will yield valid data—that there is little evidence to suggest that interviews with crowd workers about sensitive topics are significantly impacted by the chosen online interview mode.

As a researcher, that's reassurring.

Kyle Crichton, Ph.D. student, Engineering and Public Policy

“As a researcher, that’s reassuring,” says CyLab’s Kyle Crichton, a Ph.D. student in Engineering and Public Policy and a co-author of the new study. “Our results show that if I choose a certain online method of conducting my interview study, I can be confident that the results will be valid.”

Even prior to the Covid-19 pandemic, factors like the cost of conducting interviews, the logistics surrounding interviews themselves such as scheduling and finding private places to conduct them, and participant anonymity pushed many interviews for academic studies online, providing even more motivation to study the validity of results from different types of online interviews.

In their study, the researchers conducted 154 interviews with sensitive questions across seven randomly assigned online interview conditions: video, audio, chat, non-anonymous chat, email, survey, and scheduled survey. After each interview, both the interviewer and interviewee completed a survey about their perceptions on rapport, anonymity, and honesty.

The questions for the interviews focused on sensitive topics, as the researchers hypothesized that any differences in self-disclosure, perceived anonymity, and honesty would probably be most pronounced when the questions were sensitive to the participant. Those questions are summarized below.

1. What are your favorite things to do in your free time?
2. What characteristics of yourself are you most proud of?
3. What are your feelings and attitudes about death?
4. What has been the biggest disappointment in your life?
5. What is your most common sexual fantasy?
6. What have you done in your life that you feel most guilty about?
7. What characteristics of your best friend really bother you?
8. Is there anything you want to add? 

While the researchers found some differences across the different interview modes related to self-reported rapport, disclosure, and perceived anonymity, they found little evidence to suggest that study participants’ responses were significantly affected by the modes themselves. 

The biggest differences between modes related to the ease of recruiting participants: the researchers found fewer participants interested in participating in audio and video interviews compared with the other modes. Furthermore, a larger fraction of recruited audio and video participants failed to show up for their interviews.

“Oftentimes, what security and privacy researchers are most interested in are behaviors that people are engaging in online that they’re not necessarily most willing to openly share,” says Crichton. “This study shows that researchers should be less worried about interview mode affecting their data and be more concerned with the mode that can best reach participants and how they can make their participants feel comfortable.”

Paper reference

Audio, video, chat, email, or survey: How much does online interview mode matter?

• Maggie Oates, Carnegie Mellon University
• Kyle Crichton, Carnegie Mellon University
• Lorrie Cranor, Carnegie Mellon University
• Storm Budwig, Carnegie Mellon University
• Erica Weston, Carnegie Mellon University
• Bridgette Bernagozzi, Carnegie Mellon University
• Julie Pagaduan, Carnegie Mellon University