Can You Hear Me Now? General reflections on communications research
Living with two teenage girls in the house makes you acutely aware of the advantages of brevity. You see the familiar glaze in their eyes right around the fifth word of your sentence and by the eighth word you may as well be talking to the dog (who isn't really listening either). There is an old aphorism that goes: there's a great power in words, if you don't hitch too many of them together. It is worth remembering, both when talking to teens and when talking to customers, especially when those customers are harried health care professionals with eleven minutes per patient and a waiting room that is already backed up.
One would think that concise, focused messaging is an obvious element of marketing strategy. That does not appear to be the case, as many companies employ a more-is-better approach to the frequency and volume of their interactions with customers. The instinct is understandable. If you have twelve things worth saying about your product, saying only four of them feels like leaving something on the table. The difficulty is that the customer is not keeping score of how much you told them. They are deciding whether they can hold onto anything at all.
This idea of streamlined communication applies with equal force to how we design and conduct message optimization research. The specifics of any design will depend on the goals of the study, but I have come to believe that message optimization research should adhere to a small number of principles regardless of method.
Reflect brand positioning
Good brand communication supports and reinforces the positioning of the brand. Why then waste scarce research dollars testing dozens of unrelated messages? Other than a few categories that complete the brand story, the focus should be on messages that give customers a consistent and clear image of the brand. This assumes, of course, that the brand team has already done the positioning work based on key product benefits and differentiators. Where that work has not been done, message testing becomes a way of discovering the positioning by accident, which is an expensive way to arrive somewhere you could have reasoned your way to.
Reflect brand usage occasions
Physicians make occasion-specific prescribing decisions based on associations that link the situation in front of them with the products they believe will address it. Those associations are built by communications tailored to the occasions the brand actually competes in. This understanding has to precede the messaging research. Brand teams need to know the situations in which they enter the consideration set before the research can focus on the elements relevant to those situations. The alternative is the throw-everything-at-the-wall approach, which spends an expensive quantitative sample discovering that some of your messages are irrelevant to the occasions you compete in, something a well-run qualitative phase would have told you for a fraction of the cost.
Restrict wordsmithing to the qualitative and creative phases
Quantitative research is poorly suited to testing minor variations in language and will more often than not yield misleading results. Higher level differences swamp the effects of lower level distinctions. The difference between an efficacy message and a tolerability message is substantial and detectable. The difference between two slightly different phrasings of the same efficacy claim is neither. Regardless of how the preference exercise is constructed, our sample sizes are rarely powered to capture distinctions that subtle, and a difference that is not detectable is not a difference you should be making decisions about.
Apply the behavioral lens at development, not at measurement
This principle has become more relevant in the last several years, as there is now a veritable cottage industry of agencies offering message testing built explicitly on behavioral science. The selling point here is that conventional message research captures stated preference while behavioral methods capture something closer to how physicians actually decide.
The underlying observation is fair. How a message is framed matters, sometimes more than what it says. A claim framed as a loss carries different weight than the same claim framed as a gain. A message that invokes what peers are doing operates differently from one that presents the same information as a bare fact. None of this is controversial.
My quibble (to the extent I have one), is a question about sequence rather than a question about measurement. If framing effects are as consequential as the behavioral literature suggests, the place to act on them is during message development, when a copywriter and a brand team are deciding how to construct the claim in the first place. A message built with attention to framing, to reference points, to social proof, and to what the physician is being asked to give up, arrives at the testing stage having already absorbed most of the available benefit. What remains is the ordinary question of which messages resonate with which physicians, and conventional methods answer that question perfectly well.
I want to be careful about the limits of my own expertise here. I am not a behavioral scientist, and I am not positioned to evaluate the behavioral constructs these agencies claim to measure. My reservation is narrower and sits inside my own territory, which is research design.
It is this. The methodological innovation these offerings advertise is capacity. The pitch is that conventional methods can test twenty or thirty messages, while the new approach can handle a hundred and fifty, using various forms of adaptive questioning to make a task of that size survivable for a respondent. Set aside whether the adaptation works as described. Ask instead why it is needed.
A brand team arrives at quantitative testing with a hundred and fifty candidate messages for one reason, which is that nobody was willing to say no earlier. If the messages reflected the brand positioning, and if they reflected the occasions the brand actually competes in, there would not be a hundred and fifty of them. There would be thirty or forty. The exotic testing apparatus is solving a problem the process created upstream, and it is being sold as a capability rather than as a workaround for a failure of discipline.
There is an irony here worth contemplating. A field that exists to help brands say fewer things has developed a testing apparatus whose principal selling point is the ability to evaluate more things. The methods are being optimized in precisely the opposite direction from the advice we give our clients.
None of which means the behavioral work is without value. The development case is strong, and I would happily see more of it. But if a brand team is choosing where to spend, applying the lens during development, when the messages are being written, buys more than applying it during testing, when the messages are already fixed and all that remains is to sort them.
From many to a few
Developing an optimized messaging platform is a multi-phase process. It usually begins with qualitative research, proceeds through the input of creative agencies, and is then validated with quantitative research. Sometimes the agency output goes through an additional round of qualitative work to reduce the message count to something manageable, where manageable is defined as whatever the research agency says is feasible given the method and the available sample.
Quantitative message testing methods fall into two broad families, which I will call aggregative and dis-aggregative.
Aggregative methods
Aggregative methods treat message optimization as a block-building exercise. Assess the value of each message element individually, then assemble platforms from the elements with the greatest collective value. Message value is typically captured through scaled questions measuring motivational power, or through a MaxDiff exercise. The approach is usually paired with TURF analysis to identify combinations with the widest reach.
The advantages are substantial:
Straightforward to implement and analyze
Capable of testing a large number of messages, particularly with a blocked MaxDiff design
Provides a defensible answer to the question of how many messages the platform should contain
That third point deserves emphasis, because it is the strongest argument for the aggregative family and it is often undersold. TURF measures unduplicated reach. An additional message that appeals only to physicians already reached by other messages already in the platform adds nothing, because those physicians are already counted. The performance of a message platform is therefore bounded by whether or not additional messages pull in new target customers and the curve flattens because there is a real ceiling.
This matters because it distinguishes TURF from what happens in an additive preference model (which lies at the heart of dis-aggregative methods), where every message with a positive coefficient increases predicted preference without limit. TURF has a structural answer to the more-is-better problem. The dis-aggregative methods discussed below do not, and we will come back to that.
One practical note: MaxDiff-based TURF requires individual-level utilities, which means hierarchical Bayes estimation rather than aggregate logit. Reach is a count of individuals clearing a threshold, and you cannot count individuals from an aggregate model.
The disadvantages are those common to all direct elicitation methods:
Artificiality of task. Respondents evaluate individual messages rather than the cohesive platform they would encounter in practice.
No synergies. Interaction effects are not captured, so messages that score modestly alone but work well together cannot be identified.
No logical structure. The design does not distinguish between categories such as efficacy, safety, or dosing, which means the platform with the greatest reach may not represent the brand in a balanced way. This last point is the strongest argument for the alternative family of methods and is worth holding onto.
Dis-aggregative methods
The dis-aggregative approach applies principles of multi-attribute modeling to messages. Respondents evaluate experimentally designed message platforms rather than individual messages, and a discrete choice model recovers the value of the individual elements. The task is usually a pair-wise exercise in which respondents select the platform they find more motivating.
The approach works best when messages can be organized into logical buckets, with each platform drawing at most one element from each bucket. This is directly analogous to attributes and levels in a product conjoint. It is possible to construct designs that treat messages as fully independent, but with a large message set this requires constraining the design so that no platform contains too many elements, which in turn requires large, blocked designs and correspondingly large samples.
What the model gives you
The choice model produces a coefficient for each message element capturing its contribution to the probability that a platform is selected. This is analogous to part-worth utilities from a traditional conjoint model and can be utilized in a similar fashion to identify optimal message platforms of different sizes and the relative importance of each “attribute”, i.e., bucket.
Bucket importance is typically computed from the range of coefficients within each bucket. Where buckets carry unequal numbers of elements, and in message research they usually do, this measure needs a caveat. A bucket with six elements has more opportunity to produce a wide range than a bucket with three, which inflates its apparent importance for reasons that have nothing to do with the content.
I should also be honest about a limitation of the approach that is sometimes obscured in how it is sold. One of the arguments for choice-based message testing is that it captures synergies between elements that MaxDiff cannot. This is true in principle. However, it is not true of a main effects design at feasible sample sizes, which is what most message studies actually field. Detecting interaction effects requires designing for them and powering for them. If the study has not done that, the model is estimating main effects and the synergy argument does not apply.
Building platforms from elements
The point of estimating element-level values is to evaluate candidate platforms. Two design decisions govern whether this works.
The first is the “none” level. If every platform in the design draws exactly one element from each of five buckets, every platform has five elements and the model has no basis for evaluating a three-element platform. Including a none-of-these level in each bucket is what allows the design to represent platforms of varying size. Without it, the study cannot produce the shorter-platform recommendations that are usually the point of the exercise.
The second is how platforms are compared, and this is where the dis-aggregative family gives something up.
Preference models are additive - any element with a positive coefficient increases predicted preference, and nothing in the model represents the fact that the fourth efficacy claim is talking to a physician the first one already convinced. There is no deduplication. Left alone, the model recommends using every message you tested, which is exactly the more-is-better outcome the research was meant to prevent. TURF, by contrast, gets a ceiling for free from the structure of the calculation. The choice model does not.
One response is to build the ceiling in, by specifying the model to account for the disutility of carrying more than some number of messages. In practice this means adding a term that penalizes platform size beyond a threshold and constraining it to be negative. I have tried this. It is clunky, the threshold is arbitrary, and it rarely produces anything statistically reliable.
The response that works better is to sidestep the problem by evaluating platforms within size classes: the best three-element platform, the best four, the best five, and then looking at what each additional element actually buys. This is a judgment call rather than a model output, and it should be presented as one. What the choice model gives you in exchange is the ability to enforce balance across message categories while making that judgment, which TURF cannot do.
A worked example
Consider a study with four buckets: efficacy, safety, dosing convenience, and support services. Each bucket contains four message elements plus a “none” level, which yields platforms ranging from zero to four elements. Suppose the estimated coefficients produce the following top platforms by size, expressed as predicted preference share against a fixed competitive platform:
The move from two elements to three buys seven points. The move from three to four buys two. On these numbers the three-element platform is the sensible recommendation, and the fourth element is worth including only if there is a reason beyond the model to want it, such as a sales force that needs something to say about support services.
Note what is and is not happening here. The flattening between three and four elements is not the model telling you it has reached a ceiling, because the model has no ceiling. Support 2 has a positive coefficient and adding it increases predicted preference, as would a fifth element and a sixth. The flattening is you looking at a small number and deciding it is not worth much. That is a legitimate way to make the decision, but it is a different thing from what TURF gives you, and it is worth being clear with a client about which one they are getting.
What the choice model does give you here is balance. Because the design draws at most one element per bucket, the recommended platform speaks to efficacy, safety, and dosing rather than offering three efficacy claims that happened to score well on their own.
Design heuristics
Sample feasibility, rather than statistical elegance, usually determines what a message study can support. As a rough guide, a choice-based design tends to work well with six or eight buckets containing four to six elements each, which allows testing of up to forty-eight individual messages. Beyond that, the design grows too large, and the sample needed to support it becomes difficult to justify against what the study will actually tell you. MaxDiff designs, in general, can handle more messages, and this might suggest that they are the correct alternative in that scenario.
However, while it is tempting to reduce the choice between the two families to a question of capacity, it hides most of what actually determines the answer. Depending on the “bucket structure” of messages, their individual length, and complexity, a fractional-factorial design for a pair-wise choice exercise might be a better choice than a balanced incomplete block design (BIBD) required for a MaxDiff exercise. Push the message count high enough in a BIBD and each respondent sees a small fraction of the set, which starts to work against the individual-level utilities that TURF depends on. The dis-aggregative side has its own version of this. Capacity there is a function of the number of buckets, the number of elements within them, whether the task is pairwise or presents three or more platforms at a time, and how many tasks a respondent can reasonably complete before the answers stop meaning anything. Neither method has a fixed ceiling, and both have a point past which the design is technically feasible and practically useless. Working out where that point falls for a given study requires careful consideration – which, not surprisingly, is true of all good research design.
Close
The methods discussed here differ in what they can and cannot recover, and the choice between them should follow from the structure of the message set and the decision the brand team is trying to make. What they share is a dependence on the quality of the messages going in. A well-specified choice model applied to a set of poorly framed, occasion-irrelevant messages will return a precise answer to a question that was not worth asking.
Which returns us to where we started. The purpose of all of this is to arrive at a small number of things worth saying, said in a way that a physician can hold onto between one patient and the next. Every method described above is a way of deciding what to leave out. It is worth remembering that the decision to leave things out is the point, not a regrettable consequence of limited sample.