Dialsmith

           Sarah Evans, PhD.                          Joy Hackenbracht, PhD. 
     (Images courtesy of Fors Marsh Group)

We’re always on the lookout for expert insights and opinions on moment-to-moment research.  A recent article entitled, “Hooked On A Feeling: Implicit Measurement of Emotion Improves Utility of Concept Testing” in the current edition (4^th Quarter 2014) of MRA’s ALERT! Magazine caught our attention. The article’s authors, researchers Sarah Evans, PhD and Joy Hackenbracht, PhD of Fors Marsh Group, offer a convincing argument for why measuring emotional response is critical to understanding our decision-making and therefore, “invaluable in the creative message process.” Drs. Evans and Hackenbracht go on to argue in favor of moment-to-moment testing in addition to self-reporting techniques (we certainly agree!) and describe an emotional response study they conducted using multiple modes of moment-to-moment data capture—a physiological method using electrodermal activity (EDA) as well as dial testing—and provide an interesting comparison of results.

With their experience using multi-modal, moment-to-moment approaches, we thought it interesting to go more in-depth with these two researchers to learn more about their experiences with these approaches and the type of results they’ve produced.

Q: Can you tell us briefly about the type of research you conduct and the type of clients you work with?

A: Generally, Fors Marsh Group is an applied research firm in the business of measuring, understanding, and influencing the way people think, make decisions, and behave. Our Communication Research team focuses on the role of messages in that process. Through research, we support clients in identifying message strategies likely to influence behavior and in testing how messages and communication interventions perform at various stages of development (i.e., from creative concepts to rough cuts to large scale campaign evaluations). Most of our clients are government agencies – or advertising/marketing firms working on behalf of government agencies – because there’s an expectation for rigorous research and evidenced-based messaging in that space.

Q: In the article, you run parallel EDA and dial tests. In your research, do you typically do both? How much experience do you have with dial testing and where do you think it adds value?

A: It depends on the research question, but we always couple EDA with other types of measures – for example, dial testing or self-report items.  EDA provides a measure of arousal, but that alone doesn’t provide enough information to draw accurate conclusions about a viewer’s response to an advertisement.  We added dial testing to our repertoire about two years ago as a means to provide context to the direction of a response (i.e., positive or negative). The advice we constantly return to is that each measurement tool provides a unique piece of the puzzle – and the understanding you gain in combination is far greater than you could obtain by simply measuring one or the other in isolation. The combination that makes sense differs for each client and each question asked, making study design so important (and fun).

Q: Do clients and other researchers “get” the value of moment-to-moment measurement or does it take convincing?

A: We’ve found that clients don’t get jazzed about the types of tools in your toolbox but rather about how the effectiveness of their message, commercial, website, or other product can be improved as a result of the information gathered through those tools (and rightly so!). A lot of people focus on the technology, but the technology is only as useful as the people interpreting/applying its output. For this reason, we don’t pitch moment-to-moment measurement per se – what we’re really pitching is the unique insight it provides above and beyond more traditional market research measures. It certainly helps that the output is really visually compelling. Often, we’re able to pique interest with the visuals, and then it’s a matter of identifying pressing questions that can only be answered through the combination of tools we are employing.

Q: Can you provide an example from your research where moment-to-moment measurement provided a specific result or insight that was unexpected and/or led to a significant change in a client’s strategy or decision-making?

A: Unfortunately, confidentiality agreements prevent us from sharing any specifics here. But, we can say that we consistently see moment-to-moment measurement, especially of the implicit variety, provide information above and beyond more traditional measures. For example, self-reported emotion rarely provides the level of detail needed to make diagnostic recommendations. On the other hand, EDA allows us to show, for example, that an ad really didn’t get a reaction until 20 seconds in. By then you may not even have an engaged viewer outside of the lab setting. This makes ad testing tremendously more useful and appealing to a wider audience – including some who may have been formerly in the “testing makes advertising bland” camp. Seeing that evolution in thinking has been really exciting.

Q: What types of challenges have you had to overcome when conducting a moment-to-moment study?

A: The precise and time sensitive nature of moment-to-moment data is part of what makes it so valuable, but it also makes measurement logistics complex. Some of the first challenges we had to overcome were related to integrating multiple sources of data – for example, EDA and continuous rating dial – into one data collection platform, and linking the data with the stimulus presentation program. Both of these steps are critical for ensuring the quality of the data and the ability to make diagnostic recommendations based on viewer reactions at precise points in time. We’re now working on increasing the real time element of this process such that we can sit with the client in a viewing room, watch what the participant is viewing in real time, and guide follow on questions or probes on the fly as we are learning how they are reacting and thinking about the message or execution.

Q: Are there other forms of moment-to-moment measurement that you’ve used or experimented with?

A: We also frequently leverage eye-tracking in our research. Often the process goes something like this: (1) we use moment-to-moment data, such as EDA, to pinpoint specific points in time that are eliciting viewer reactions, and (2) with eye tracking, we identify what specific elements of a particular moment or sequence of moments are most attracting viewers’ attention. We can do this by quantifying viewer gaze patterns and recording the length of time viewers spend looking at different parts of the frame (like a logo or call to action, for example). This is yet another case of a multi-measure approach improving the recommendations we can make.

Q: Where are we headed with moment-to-moment measurement? New technologies? New methods or approaches to collect moment-to-moment?

A: This is a matter quite literally of where we are headed – on the road. Increasingly, the mobility of moment-to-moment measurement tools is of paramount importance. Maximizing the utility of these technologies is not only about the ability to collect data in Seattle one week and Miami the next, but it’s also about the ability to collect in less traditional, more realistic environments. Related is a trend toward less intrusive, cumbersome measurement tools. We’ve come a long way from the wired labs of our graduate school days!

Thanks to Drs. Evans and Hackenbracht for their time and giving us more to think about in regards to moment-to-moment testing. You can keep tabs on their research by following the Fors Marsh Group on Twitter or by checking out the Fors Marsh Group Blog.