Measuring user engagement: Lessons from BusinessWeek

Think about the traffic statistics you refer to when you look at Omniture or Google Analytics data for your site. Unique visitors? Pageviews? What do they actually tell you about your audience? The ubiquitous unique visitor metric treats your most passionate and thorough users exactly the same as those of the one-hit scan-and-scram variety. And pageview tallies are so apples-to-oranges in these days of Flash and AJAX that they’re rendered almost meaningless. If you really want to describe your audience, it’s time for some new metrics.

But what else is there? The folks at BusinessWeek think they have an answer, and it’s not about how much content users consume but rather what they do with it. I asked BusinessWeek’s online editor, John Byrne, about his team’s efforts to go beyond pageviews and visits to quantify something more inscrutable: user engagement.

What is BusinessWeek’s definition of user engagement and why is it important?

User engagement is how we nurture and build a community. Our reader engagement index is a comments-to-postings measure for a given month: So we will tally how many comments on X number of stories/blog posts that BusinessWeek.com published that month. This gives us a ratio figure that we track to determine our monthly reader engagement index and growth. In February of this year, we received from our community 28.2 perspectives and insights for every story or blog post we published. A year earlier, we received 23.7. So we know we’re moving in the right direction.

It’s important because we value, and so measure and gauge, all our interactions with our readers on BusinessWeek.com — including commenting on a story or blog post. The next level is how our writers and editors engage our readers in a conversation, and also welcoming our readers to write longer pieces for us, or to report (at least once a week) a reader-suggested story. We’re also engaging with BW readers on other sites, such our Ning network that served as a forum to generate and debate stimulus spending priorities for the Obama administration, or interactions involving our 50+ staffers on Twitter. If we don’t listen to our readers and interact with them, and then act on the feedback and suggestions they’re giving us, we’re dead in the water. That applies to any media brand today, not just BusinessWeek. We’re just making it more of a priority, including featuring readers on an equal plane with our writers — on our home page, for example, our featured reader is given more prominence than even a Jack & Suzy Welch.

What information sources and tools do you use in measuring engagement?

Our reader engagement index involves Omniture (for stories) and Movable Type to track numbers of blog comments and posts.

Beyond our reader engagement index, other measures include how much you’re retweeted — for instance, one of my tweets on March 25 was retweeted 130 times. We also look at referring traffic from blogs or Twitter on Omniture, or by running a Twitter search: http://search.twitter.com/search?q=businessweek or http://search.twitter.com/search?q=johnabyrne.

And also look at Google BlogSearch or blogpulse (owned by Nielsen Buzzmetrics) for mentions of businessweek.com: http://www.blogpulse.com/search?query=www.businessweek.com

What do engagement metrics tell you that conventional metrics do not?

It shows, in quantifiable/measurable terms, how much our readers care about us. To post a comment or submit a suggestion is a strong indicator of a BW loyalist, someone we need to nurture and engage and reward. It also tells us how much (and how well) our staffers are interacting with readers. The problem with time spent on a site is that it also measures, in the case of a portal, email time, or in the case of a site heavy on video, time spent watching video, which can be like TV. I also argue that simple pageview metrics are heavily influenced by slideshows and email. There is no better sign of commitment or engagement than the act of reading a substantive piece of journalism, thinking about it and then forming a point of view on that story that you’re willing to write and share with others. That is true engagement.

How do you use this information to improve the site?

You can’t manage something if you don’t measure it. So having a point of reference for exactly how we’re doing drives other ideas and initiatives to increase engagement.

Does increased user engagement translate into benefits for advertisers?

Yes. Anecdotally, our sales team is selling our engagement story and using it to differentiate what we do versus our competition. It also helps to better position our Business Exchange, a new Web 2.0 product we launched last September, as a key component of our engagement efforts.

So, let’s look at some potential engagement metrics and what they might tell us. This is by no means a comprehensive list; it’s just what came to my mind. If you have other thoughts on ways to measure engagement or how you might use this data, please, um, “engage” in the comments below.

• Internal metrics: Statistics about engagement that takes place on your site
• Comments posted: Shows how much users are inclined to react to a topic, or supply insights of their own.
• Return commenters: In other words, how many people comment multiple times on the same item? This is a measure of conversation around a topic. (Kudos to the Guardian’s Kevin Anderson for this idea.)
• Times e-mailed: Reveals how often users are sharing this information with friends. This metric probably skews toward neophyte users, as more experienced users are presumably less likely to use an “e-mail this” feature.
• Average time spent on page: Shows how thoroughly users are consuming the content, perhaps? Lots of asterisks, though, as John points out.
• External metrics: Statistics about how people share and discuss your content elsewhere
• Tweets/retweets: Measures how “viral” this content is in a social network. There’s also geographic information embedded in these tweets that could tell you where a topic resonates particularly strongly.
• Diggs: Another measure of the viral nature of a topic. Given Digg’s audience, this metric might favor content that appeals to a techie crowd.
• Delicious saves: Shows how many users stored this page with an eye toward returning to it. This metric could be particularly useful for ongoing features that you want to build a regular user base for.
• Inbound links from blogs: Quantifies the discussion taking place in the blogosphere. This could help you identify the blogs that are most attuned to the content you produce — as opposed to just the ones that send you the most visitors (which are not necessarily the most engaged users).

Each of these metrics is easily available for a given URL at a given moment, but keeping track of all your stories over time would be impossible without some automated assistance — particularly with regard to the external metrics.

Here’s what I’d like to see: A web service that will track a URL across several services (Technorati, Delicious, Digg, and maybe internal analytics packages too) to see how it’s being referenced in each medium, then tabulate all those metrics into a single “engagement score”. (And I’d love to hear from any programmers who want to take a stab at building this!)

Meanwhile, anybody have any engagement metrics tips they’d like to share?