Pricing Gurus and 5 Circles Research Blog Posts

LinkedIn B2B sample looks promising

One of the interesting presentations at yesterday’s Puget Sound Research Forum conference was from LinkedIn, covering their recently introduced sample services.

Key advantages for sample from LinkedIn as I see it:

Profiling information is entered by the LinkedIn user for reasons unconnected with survey taking. Regardless of of how much of a problem you think lying on sample panel profiling or screening questionnaires might be, a LinkedIn user’s description of themselves is likely to be fairly accurate – and useful to a survey researcher. LinkedIn claims that their users inflate career history less than resumes on job seeking sites such as Monster because the information is visible to colleagues.
This isn’t a panel. The primary reason for LinkedIn membership isn’t to take surveys. While response rates may be lower from LinkedIn than from panels, I really care about quality. Response rate figures are meaningless if you are talking to the wrong people, as long as there isn’t a non-response bias. Surveys using LinkedIn sample still have the potential for response bias, of course, but the reasons are less to do with sample than with questionnaire and invitation design.
LinkedIn says that they will minimize the number of invitations sent to users, perhaps with a limit of no more than 1 or 2 per month. Although I’m skeptical about the actual numbers, I accept the point that LinkedIn’s focus isn’t sample and that frequent invitations would annoy members so I am optimistic that the LinkedIn sample will continue to be lightly surveyed.

Results shared with the audience seem to bear out the truth of the LinkedIn sample promise. A small telephone study validated the accuracy of status, title and start dates for LinkedIn members. Results from LinkedIn sample and a B2B panel for online study of U.S. IT decision makers (a notoriously over-surveyed group) showed some interesting differences. In particular, the panel delivered a high percentage of completes between the hours of 3am and 7am. Other data supported the suspicion that many of the responses were from India and China, not from the U.S. Additionally, the panel respondents were more likely than the LinkedIn sample to complete the survey very quickly, meaning that these were probably not the target audience. Of course, LinkedIn presented information that showed them in the best light, but it was convincing.

I’ll be looking at LinkedIn sample for B2B projects in future, both for my self-service(SurveysAlaCarte) and full-service clients.

Idiosyncratically,

Mike Pritchard

Time to cool it? (your tea that is)

As a tea-drinking Brit I was fascinated by a study about tea drinking in Northern Iran concluding that drinking very hot tea is strongly associated with higher risk of oesophageal cancer.

Digging in further, I was struck by a number of points:

The article I first noticed, by Karen Kaplan of the Los Angeles Times, was very clearly written and didn’t mangle the facts or interpretations. Such clarity is unusual and deserves a commendation. Read the article for the details – I don’t need to repeat.
The scale of the study was unusually large compared with many medical studies, including some that draw dubious conclusions from a very small data set. The research team (from England, France, Sweden and the U.S.) matched 300 cancer patients with 571 healthy controls who had similar demographics. These groups are only a small fraction of the entire database of nearly 50,000 people in Golestan province whose tea drinking habits have been studied, so we can expect future refinement and expansion of results.
The original article in the BMJ (formerly the British Medical Journal), BMJ 2009;338:b929, is a well-written source document, complete with properly explained tables and a video.

This is a good example of a well researched and reported project. The results are made available under an open access Collective Commons License, that doubtless encourages completeness.

After evaluating the details, I decided to review my own tea and coffee rituals. The study concluded that the most likely causal mechanism is the temperature, so regardless of what hot liquid you drink it might be a good to be cautious about temperature. I don’t drink anywhere near the quantity of hot liquids that the study participants imbibe daily (nearly 1.2 liters on average – that’s over 2.1 British pints or 2.5 U.S. pints), but the damage may be cumulative and I want to be a tea drinker for many more years. It seems that my latte drinks are cool enough, but I should probably wait for a few minutes after brewing to drink my tea at around 140 degrees Fahrenheit. Perhaps I’ll start to put the tea cosy on after the first cup, but I don’t think I can bring myself to stop warming the teapot. My wife is the smart one – she’s always preferred to cool down her tea with water from the tap.

Idiosyncratically,
Mike Pritchard

Profiting from customer satisfaction and loyalty research

Business people generally believe that satisfying customers is a good thing, but they don’t necessarily understand the link between satisfaction and profits. This is partly because much of the original work was done so long ago that contradictory cases and nuances have allowed confusion to build up. Additionally, some companies have appeared successful for a time despite poor satisfaction, generally in industries where there is limited or no competition such as airlines.

Statistics hero Nate Silver predicts Oscar results

It shouldn’t be a surprise that Nate Silver (statistics whizkid behind FiveThirtyEight.com which did an outstanding job of predicting the results of the 2008 election) is turning his attention to matters other than politics. After all, before he became a political pundit his forte was baseball statistics.

If you want to read the published predictions head over to the New York Magazine’s movie section. Some are calling them spoilers – I guess that’s showing total faith in Mr. Silver’s forecasting ability. The article includes some close calls (including a 51% prediction for supporting actress). The only prophecy I’ll share here (partly because I have to admit that its the only one of this year’s Oscar nominated movies I’ve seen so far – shame on me) is that Nate is convinced that Slumdog Millionaire will win Best Picture and Best Director).

Read the comments on the New York Magazine article for a little discussion about prediction methodology versus quality perceptions (or wishful thinking if you prefer). I hope to see some more comments before the Academy Awards, that will perhaps shed some light on general understanding of forecasting. Or not…

Idiosyncratically,
Mike Pritchard

When Validation Backfires

I just came across an interesting issue with validation in an online survey using a Van Westendorp pricing model. Van Westendorp is one of the common ways to test pricing by directly questioning prospective purchasers. This post isn’t about Van Westendorp, also known as the Price Sensitivity Meter (you can find plenty of references online, including a starting point on Wikipedia) but you need to know a little to understand the issue. Survey respondents are asked a series of questions about price perceptions, as follows:

At what price would you consider the product starting to get expensive, so that it is not out of the question, but you would still consider buying it? (Expensive/High Side)

At what price would you consider the product to be so expensive that you would not consider buying it? (Too expensive)

At what price would you consider the product to be priced so low that you would feel the quality couldn’t be very good? (Too cheap)

At what price would you consider the product to be a bargain—a great buy for the money? (Cheap/Good Value)

There is some debate about the order of questions, but in this example the questions were asked in the order shown. The wording was slightly different. Researchers are sometimes concerned about whether the respondents understand the questions correctly, especially since the wording is so similar (the Expensive, Cheap etc. designations are usually not inclined in the question as seen by a survey taker). One way to address this concern is to highlight the differences. Or you might point out that the questions are slightly different and encourage the respondent to read carefully.

The other approach is to apply validation that tests the numerical relationship. Correctly entered numbers should be Too Cheap < Good Value < Expensive < Too Expensive. (We usually ask these questions on separate pages so as to get independent thoughts from the respondents as far as possible, rather than letting them see the group of questions as one and making them consistent or nicely spaced).

In this case, the research vendor chose to validate, but messed up big-time. When I entered a value for ‘Too Expensive’ that was higher than the value for ‘Expensive’, I was told to make sure your answer is smaller or equal to the previous answer. Yes, they forced me to provide an invalid response! I hope they caught the problem before the survey had gathered all the completes, but maybe they didn’t – given how fast online surveys often fill. They probably had to field the survey again because the pricing questions were integral to the research objectives.

Why did this happen, and how can you prevent a similar problem in your surveys?

My guess is that the underlying cause was that debate about question order that I mentioned earlier. The vendor probably had the questions switched when the validation was tested, and then changed the order before the survey was launched.

But the real message is that proper testing could have identified the issue in time to correct a very expensive error. There is no excuse for what happened. This doesn’t even fall into the class of problems that the pilot or soft-launch would be needed to catch.

So, test, test, and test again. In particular, test using people who aren’t research professionals or experienced survey takers.

If you are creating your own surveys, don’t let this kind of problem stop you. You can do just as good a job of testing as the big companies, and big companies aren’t immune. This survey was delivered by one of the top 10 U.S. market research firms. I won’t publish the company name here, but I’ll probably tell you if you catch me at one of my workshops (coming soon).

Idiosyncratically,

Mike Pritchard

Reporting on reported results may increase confusion

I had decided not to comment on political polling during this election season. Although this post concerns the election, it isn’t about polling. It isn’t directly about research, but it does show the problems that can happen when numeric results are reported imperfectly, and, even more important, when reporting on the report confuses instead of clarifying.

Today (10/7/08), I read a snippet from the Puget Sound Business Journal “Washington State Voter Registration hits 3.5 million” The article stated that Voter registration totals in Washington state hit 3,515,393. Reading further, I see that the previous record was 3,514,078 in 2004, and that number was reduced “significantly” after duplicate registrations and deceased voters were removed.

What do these statements mean in absolute terms? And what do they mean relative to the changing state population?

The original release from the Washington Secretary of State has the same figures – naturally. But there we find that the 3,515,393 number is merely the latest tally, and that it is expected to grow as applications continue to be processed. There is also some more information about the total of voters registered in 2004, and the processes involved in reducing the numbers. I find the writing somewhat unclear, but it appears that the earlier number of 3,514,078 was the high water mark reached during the previous registration process – which wasn’t clear from the Puget Sound Business Journal.

I’m not going to comment on why the Secretary of State decided to publish a release today (the application deadline was last Saturday), but let’s compare the tallies from 2008 and 2004. The current registration count is 0.04% more than last time. Given that the Washington State population increased by 8.5% from 2000 to 2006 according to the US Census Bureau, the 0.04% seems like a so-what. It may very well be that the final tally increases by 5% or more (even before trimming), but why not wait and report on more solid numbers? And (for the Business Journal), why not add some clarity about the process and the meaning?

For more information about reporting on reporting and the way that mistakes have a way of taking on a life of their own, I recommend Damned Lies and Statistics (untangling numbers from the media, politicians, and activists) by Joel Best. It’s a great way to learn to read critically, whether or not you are a researcher. And, of course, to learn what you can do to prevent your own results being abused (answer – not much).

Idiosyncratically,
Mike Pritchard