Curation questions and the start of some answers

Information curation, like data visualization, is one of the buzzwords being used by those trying to guide, and goad, news organizations into thinking about new content models. Jeff Jarvis talks about “curation” as the activities of sorting, choosing, and display.

Mike Shatzkin on the Idea Logical blog said

“Curation is a term that has always referred to the careful selection and pruning of aggregates, such as for a museum or an art exhibition. But the concept in the digital content world means the selection and presentation of these disparate items to help a browser or consumer navigate and select from them. Aggregation without curation is, normally, not very helpful. Curation creates the brand.”

There have been some forays into news site curation. LJWorld created a Kansas Legislature page in 2005 that aggregated links to general news coverage of the state Legislature. But they took the next step of selecting and organizing stories by specific issues like Death Penalty, Concealed Weapons and Sunday Liquor Sales. The page served as a “one-stop shopping” resource by anticipating the kinds of information someone interested in the Legislature might want by including such resources as bios of legislators, legislative calendars and bill finders, and copies of the State of the Union addresses going back several years. LJWorld still has an aggregated page of Legislative coverage, but it is not longer curated – it is just a list of links to news stories.

Losing the topic focus switched the LJWorld’s page from curation to aggregation because an essential step in curation is organization, as they did with the issues, not just listing. Just as a well-curated museum has the Early Asian art area separate from the Surrealist collection, so should news sites provide some subject organization within large news topics.

Curation can also entail finding and providing resources from all over, not just aggregating your own content.

The New York Times Topic Pages are an example of this kind of curation. They have thousands of subject / event / personality specific pages which provide an overview article on the topic, links to all the NYT past coverage (with a searchable database specific to those articles), and, here’s where the curation comes in, sections on “Headlines from around the web” (a listing of articles found using the NYTimes’ news aggregation program Blogrunner which has been sent to selected sites) and “A list of resources from around the Web as selected by researchers and editors of The New York Times.”

We were interested in observing how this kind of curated content was used by people on an information seeking quest. We conducted eyetracking sessions with 37 undergraduate students at the University of Minnesota. The students were told that they were going to write a research report for a class and were given a selection of 10 New York Times Topic Pages from which they could choose the topic of their report: the U.S. Dollar, Earthquakes, the U.S. Federal Budget, Foreclosures, Mortgages and the Markets, Piracy at Sea, Stem Cells, Tornadoes, and Unemployment.

The students sat at a Tobii Technology eye tracking device in our research lab and were told to go to the Topic page of their choice and do whatever they wanted (read, click). After 10 minutes we stopped the session. All of the participants answered a short online questionnaire about the website and their information seeking experience after their eye tracking session.

In analyzing the eyetrack videos we designated each section of the Topic Pages (e.g., “Summary,” “Multimedia,” “Navigator,” etc.) and we coded the participant’s “attention” on the page using a construct describing the “Path to a Click” by researchers at Yahoo which characterizes activity on a site by the frequency and duration of a person’s attention to a particular part of the page. These levels are:

  1. Saw: when the participant’s gaze passed across a section
  2. Noticed: when a section the participant “saw,” then glanced away from, was returned to
  3. Parsed: when the participant “fixated” on a section, clearly taking in the text / image

Other things we analyzed in reviewing the videos were:

  • the sections that contained URLs that were clicked by the participants to figure out whether the placement of the content influenced their clicking behaviors
  • the content of the items that were clicked (headline only, headline plus abstract, headline plus image, etc.)

Among the findings are the following:

ATTENTION TO DIFFERENT PARTS OF THE PAGE. The percent indicates the number of participants.

  • Most parsed sections (areas where most attention and time was spent)
    • Summary (100%)
    • Articles about (70.4%)
    • Multimedia (58%)
  • Most unparsed sections (virtually no attention paid)
    • top-right search box and top global navigation (both 97.4%)
    • Related Topics (83.3%)
    • left-column advertisement (80.3%)
  • Zones in which the most links were clicked: (based on a 3×3 grid placed on the Topic Page)
    • left column middle row
    • middle column top row
    • left column bottom row
  • Statistically, contents in the left column were seen, parsed, and clicked the most (Note: the left column was where the archived stories were displayed)

WHAT WAS CLICKED
Among the 37 students, a total of 138 items were clicked (an average of 3.7 clicks per participant) Of those items that were clicked:

  • 42% were headline only
  • 24.6% were a headline and story abstract
  • 14.4% were a headline, abstract and photo

Of all the items clicked 85.5% were story links. Of the other items clicked, 65% of them were from the “Related Topics” box.

In the post-eyetracking survey, participants were asked a number of questions about the features of the page and their importance to them.

  • 72% of the participants rated the Navigator (links to other websites) as Useful or Very Useful. Very Useful or Useful ratings of the other key areas of content on the page: Articles from the archive (67%), Headlines from around the Web (59%), Overview of the topic (64%).
  • Of the existing or potential functions on the page that could aid a researcher, the following is the ranking by those considered “somewhat important” or “very important”:
    • 100% Bookmark or save an article (81% said “very important”)
    • 70% See article ratings from others
    • 64% Rate an article
    • 56% Sort articles by your rating
  • When asked what it was about an item they clicked on that prompted them to click (multiple responses were possible):
    • 81% cited information in the headline
    • 27% cited a photo
    • 21% cited information the abstract
  • The preference for display of stories is, by far, most recent to oldest. Articles rated highest by others or most read / emailed articles about the topic was preferred over oldest to most recent stories.
  • Students indicated their preference for finding information (if they did not access something like the NYTimes topic page) as major search engines (59.4%), the school library website (35,1%), and online encyclopedias (5.5%).
  • Compared to their alternate sources of information, 52.8% of the participants perceived NYT Topic Pages as being equally credible, 37.8% perceived NYT Topic Pages as more credible, and 9.4% perceived NYT Topic Pages as less credible.
  • Compared to their alternate sources of information, 54% perceived the information on the NYT Topic Pages as having about the same level of completeness, 30% saw it as more complete, and 16% saw it as less complete.

Students were asked an open-ended question about what they thought of the organization of the NY Times Topic page they used. Here is an analysis of their comments:

  • 49% mentioned the site was well organized and made it easy to find information
  • 40% mentioned something about the site being busy / visually cluttered
  • 35% mentioned the appealing design of the site

Some of the comments might serve as suggestions for other news sites looking into creating similarly aggregated / curated topic pages:

  • One student said, “Without a search option, it was kind of hard to find an article that would benefit my ‘research paper’ just from looking at headlines.” Of course there was a search option for the story archive, but it is below the scroll towards the bottom of the page and, apparently, easily overlooked.
  • “It was hard to distinguish between articles and opinion pieces.” This should be an important curatorial distinction.
  • “Have a way to mark an article you’ve already read.” A followed link is just a slightly lighter color; maybe clicked story links could be made more distinct or could “grey out” for easy recognition.

Honing “curatorial” skills in news organizations is one area that holds potential for creating high value resources for both casual and more motivated information seekers. However, there remain many questions about how best to design and organize these pages rich in both internal and external information. It is a research area we intend to continue to pursue and we welcome your feedback about this study and suggestions for future studies.

If you would like to see “hot spot” images from the study go to: http://picasaweb.google.com/norapaul/EyetrackingHotSpotsNYTTopicsPageProject#. The areas that are red indicate the longest “fixation”, green the next longest, and yellow after that. Areas of the page with no color were not viewed by the participant.

Taking a ride with carousels

Are rotating displays of Web content an effective way to promote news stories? This is the second in a series of articles about findings from the studies conducted for the member of the DiSEL’s Eyetracking Research Consortium.

One challenge that faces all of us who have a wealth of content on our Web sites is how to best promote it. Unlike a print magazine or newspaper there is no big stack of paper to provide a clear physical indicator that there is much to read and experience beyond the front page.

Although we can debate the effectiveness of using a Web site’s front page when it comes to promoting content1, our industry is trying a variety of methods to tackle this challenge. One method is the “carousel” – or a rotating display of a site’s content that appears in a dominant spot on the front of the page.

You can see a variety of carousel styles on sites such as MarthaStewart.com, aol.com, msn.com and even the newly redesigned whitehouse.gov. This past October, the Yahoo! Developer network launched the “carousel control” in their user interface library. They describe it as a widget that provides a means for “browsing among a set of like objects arrayed vertically or horizontally in an overloaded page region.”

So, the obvious question from the DiSEL research consortium was: Do carousels work on news sites?  Also, is there is a preferred design style that is most effective? With the help of page prototypes created by USAToday.com2 we put some carousels to the test. Here’s an overview of some of our findings.

What we did

In July of 2008 we tested 54 people with a mean age of 31. About three-fourths of the participants were women and one-fourth were men. Most had some college education.

This study was conducted in conjunction with two others that explored the ideal number of links and images on home pages. (More on these studies in future columns.) Test subjects were solicited via Craigslist, local newspaper sites and blogs. Each participant was given a $20 Target gift card for participating. The entire test took about 40 minutes per participant.

The third test conducted was the carousel test. Users were directed to one of three home pages and asked to “browse the site as you normally would” and to “tell the experimenter when you have seen what you would like of the site.”  If test subjects browsed beyond five minutes, they were asked to stop.

While users browsed the site, we tracked their eye movements using the Tobii eyetracker.  

After the browse time, users completed a questionnaire that assessed their thoughts about the site’s effectiveness and collected demographic information.

Here are the three different home pages viewed, with links to the actual test sites.

Arrow version (Link to test site)

This site’s home page contains an automatically changing carousel that allows the user to control the rotation of the stories by clicking on navigational arrows in the upper right of the carousel element. Seventeen of the 54 people tested saw this version of the carousel.

Dot version (Link to test site)

This site’s home page contains an automatically changing carousel that allows the user to control the rotation of the stories by clicking on navigational dots in the upper left of the carousel element. Nineteen of the 54 people tested saw this version of the carousel.

Thumbnail version (Link to test site)

This site’s home page contains an automatically changing carousel that allows the user to control the rotation of the stories by clicking on thumbnail images to the left of the main image in the carousel element. Eighteen of the 54 people tested saw this version of the carousel.

What we found

To make these results useful to those thinking about carousel use as a promotional tool, we examined the users’ eyetracking and also asked them survey questions to provide us with some of their overall impressions.

Survey says….

As the chart below indicates, we found that the type of carousel used did not seem affect how engaging users found the home page. Interestingly, though, we did see that users who viewed the home page with the Thumbnail version of the carousel, felt more strongly that the Web site was easier to navigate.

Those viewing the Arrows version also had a stronger desire for more stories on the home page. This could be due to the fact that the arrows navigation did not clearly indicated the number of stories available in the carousel.

Another result worth noting is that users viewing the Thumbnail version of the carousel seemed to indicate more than the other two groups that the overall site was easy to navigate.

All three groups were fairly neutral when it came determining which stories were most important.   The carousel style did not seem to provide them with a clear indication of story hierarchy.

In light of that, we also wanted to know how did users would respond to this question:  What was the headline for the main news story on the site?

As the chart above indicates, is seems that users who viewed the Arrow version of the carousel were more apt to say the first story appearing in the carousel was the “main” story. Those with the other two versions were more apt to say that it either all or some of the stories that appeared in the carousel element.

All these survey results – while purely observational – may suggest that editors ask themselves what their goals are with one type of carousel-style presentation over another. Obvious navigation – such as thumbnails – seems to encourage users to view the site navigation as easy. It also may be an indicator to users that all stories within the carousel are of equal importance.

But, this is what users SAID. Let’s take a look at the hotspots to see what they DID.

 Eyetracking shows…

Eyetracking offers researchers valuable information about where users actually look on a site. It is accomplished by calibrating a user’s pupils with a small camera hidden in the base of the computer monitor. Calibration takes only a few seconds, but the tracking results allow researchers to know where users’ eyes went within a centimeter of accuracy. (For more on how eyetracking works, click here.)

We can get a variety of data forms from eyetracking. An aggregate view of what viewers saw is displayed in a “hotspot” or “heatmap” of the Web page studied. The chart to the left details how to read this data.

We generated hotspots for users’ fixations on each version of the carousel. Here are hotspots from the top portions of the page, which seemed to show us the most interesting data about the pages. See the actual hotspots from the pages below.

Past results ring true

As eyetracking researchers, we always will spend some time observing what trends are similar to results we found in past studies. In this case, we found two things worth mentioning:

  • The areas of the photos that got the most fixations were faces.
  • Top navigation gets the most use.

    Past studies have shown that faces in photos are the areas where eyes tend to fixate. From observation during this study (and the heatmaps below) you can see that this is true again. No matter what carousel navigation style was used – and regardless of the size of the photo or the size of the person in the photo – users tended to look for and find that human element to relate to. (Stay tuned: In a future column, we will go results of a study conducted with prototype pages from the DesMoinesRegister.com that experimented with size and quantity of images on a home page.)

    We also saw that the carousels that employed top navigation elements (Arrows version and Dots version) got more clicks than the Thumbnail version. Other factors (discussed below) contributed to this observation, but it is interesting we consistently find users viewing and using navigation more if it runs along the top of a page or page element.

    Attention to change

    One clear observation is that the Dots version had a higher percentage of user fixations on all elements on the top part of the page than the other two versions (Arrows and Thumbnails).

    Why? Well, one clear difference here is that the navigational dots changed color when story changed. The change had high color contrast as well – from a blue dot to a white dot. The moving dots seemed to draw attention to the page overall.

    The Arrows version had no visible change in when stories changed in the carousel. In the Thumbnail version the box around the image changed when the story changed, but the movement and the change in color contrast was not as marked or clear as the dot changes.

    So it seems that the moving navigational dots in the Dots version encouraged a higher percentage of users to view all aspects of the top portion of the page. For example, only about half of the users looked at the page headlines on the arrows and thumbnail pages, but closer to 70 percent looked at this list when the dot navigation existed.

    Interestingly, though, even though there was a higher percentage of eyes on the headlines in the Dots version, there were less total clicks in this area of the page than on the other two. In addition there were more “dead” areas – or areas with zero eye fixations – on the tops of the pages with the Arrows version and the Thumbnail version than on the page with the Dots version.

    The type of carousel used did not seem to affect the number of clicks on the carousel photo and the corresponding headline and blurb.

    These observations seem to suggest that if you are going to use a carousel element it may be best to place navigational elements above the display elements and to utilize a navigational design that clearly indicates the change from one story to the next.

    But – what do you think?  Below are the hotspots from each carousel version. We look forward to your thoughts and comments about what you see – and why.   Post your questions here. We’ll respond.

    Arrows version 

    Automatically changing carousel that allows the user to control the rotation of the stories with navigational arrows in the upper right of the carousel element.

    Dots version

    Automatically changing carousel that allows the user to control the rotation of the stories with navigational dots in the upper left of the carousel element.

    Thumbnails version

    Automatically changing carousel that allows the user to control the rotation of the stories with the use of thumbnail images to the left of the main image in the carousel element.


    1 Some news sites editors have mentioned “off-the-record” that less than 20 percent of their story traffic comes from people linking to pieces from the home page. The rest comes from blog posts, Google searches, Delicious links and other forms of online promotion.

    2 Many thanks to DiSEL Research Consortium members Josh Hatch and Kristin Novak at USAToday.com for creating the testing materials. Also, thanks to UNC-CH Ph.D. candidate Bart Wojdynski for his assistance with designing this study and running test subjects.

  • Eyetracking research shows how younger readers view news websites

    In January 2008 a group of interactive producers from news websites gathered at the University of Minnesota for the first Eyetracking Research Consortium, part of the Digital Story Effects Lab project run by Nora Paul and Laura Ruel. Following is the first in a series of articles about findings from the studies conducted for the Consortium members. For more information about the Eyetracking Research Consortium, go to www.disel-project.org

    San Jose Mercury News

    Some of the eyetracking studies conducted with the consortium members were comparisons of different design approaches or navigational schemes and their impact on user behavior. Other members just asked for feedback from users about their experience on the website. With the eyetracking we could record not just what someone said about what they did on the site, but to actually see what they did.

    The San Jose Mercury News wanted to see how their site compared with the Contra Costa Times site – a sister site with a slightly different emphasis on visuals on the home page. They wanted an unstructured study of how people engage with the two sites and then to hear reactions.

    Between April 29 and May 1, 2008 fifteen undergraduate journalism and mass communication students were brought in for the eyetracking session. All were between the ages of 19 and 22. All self-identified as being very comfortable with the web. They were set-up with the following scenario: “You are considering moving to northern California for a job and decide to look at two regional news websites. Look at the sites as if you were sitting in your own room. Go where you want to on the site, stay for as long as you would normally. Let us know when you are done.” Half of the participants saw the Contra Costa site first then were sent to the San Jose site, the other half did it in reverse order.

    San Jose was also interested in the usability of their calendar function. We asked a few of the participants specifically to go to the Things to Do function and search for particular items: a concert on Friday night, the movie “Forgetting Sarah Marshall,” sushi restaurants. We also asked them to look at the My News / My Blogs function. This video shows one of the participants’ use of those two functions:

    Comments / Observations: In the post-exposure discussion about the sites we asked participants if they remembered a “Things To Do” feature. Seventy-two percent did not recall anything like that. Some of that can be attributed to the fact that the features is below the scroll and people didn’t scroll down the page:

    Other times they did scroll but they simply didn’t fixate on it.

    Additional studies / observations: Contra Costa and San Jose use a left hand embedded ad style on their story level pages. We looked at all of the stories that were linked to by the participants to see if the left-embed ad was fixated on. Ninety-nine stories were clicked on by participants within the two sites, an average of 6.6 stories per participant or 3,3 stories per site. The most number of stories clicked by one participant was 11, the least was three.

    We looked at the information that was available about the stories that were clicked on (headline only, headline and a blurb, headline and photo and blurb, or headline and photo only.) Fifty-nine of the 99 stories clicked on were headline only, with 17% headline / blurb and 21% headline / blurb / photo. (Two were headline / photo links – they were ARA “stories”, essentially advertisements.)

    Of the 99 news stories clicked on 63 were stories with the left-embed ad. We looked at the gaze plot for each of these participants and saw that for 23 of the 63 embedded ads was there some level of “fixation” or about 36%. Sixty-four percent of the time embedded ads were ignored. You could see that the people read right around the ad.

    We also looked at whether people went to the right rail ads on the page. Only 22% of the story level viewings indicated any looking at the right rail advertising.

    By contrast, though, 56% looked at the left rail. In the left rail was the “most viewed / most blogged” listing and a “top classifieds” box. Sixty-nine percent of the viewings of left rail content was for the “most viewed”, 30% looked at both of the content features in the left rail.


    Click to see full image

    The other content on story level pages is sidebar or related material boxes. Fifty-two of the stories looked at contained sidebar material. When people went to those stories with sidebars, 75% of the time the material was looked at – mostly when there was a box in the upper right with a photo or slide show.


    Click to see full image

    The participants were interviewed after they went through both of the sites. Following is the transcript of one of the interviews, some of the insights into how young adults use news sites and different features and functions are interesting.

    Interviewer: You looked at two websites. The first one was the San Jose Mercury News. Let’s start with that. Was there anything that stood out?

    Interviewee: I noticed that there were columns. I paid more attention to the middle column.

    Interviewer: Which was?

    Interviewee: I don’t know. I think it was headlines. There was a most viewed section too. I think I clicked on something there. I kind of skipped over all the political stuff because I blocked those out. That’s not a good attitude.

    And I think there was some kind of moving ad that caught my attention because there were women in it moving and laughing.

    The weather at the top was nice.

    Interviewer: There are different ways you can navigate the site. So what did you prefer and what did you use?

    Interviewee: I just remember clicking the titles of stories in all three columns. But they were kind of up at the top a little bit more. I tend to think of stuff down at the bottom as just like useless. There’s not usually very many links down there.

    Interviewer: The second website was the Contra Costa site. And the same question here. What kind of features do you remember?

    Interviewee: I remember that the most viewed / most sent was way down at the bottom. And it’s usually like way up at the top. But they had celebrity news. Like where I usually look right in the center column with all the most important, I don’t know, bad celebrity stuff in there which I usually don’t see on anything other than AOL. And I thought that was a little funny. Although I never heard of the Contra Costa before. Is that like a location?

    Interviewer: Yep, a city in northern California.

    Interviewee: Oh, okay. Well, I thought they were both very jumbled. Both websites were very busy. They had lots of columns. And text wasn’t very large. It was very small. And not a very readable typeface. So I didn’t really like the layout of either one.

    I’m used to – I spend a lot of time on like AOL. So they’ve got like a box with images and then text. And it kind of switches, too. I don’t know. That’s not really a news website though. It’s kind of like an entertainment thing. I don’t know. I could have done with a little less pictures, a little more text.

    Interviewer: In both of them or-

    Interviewee: Mostly in the second one, the Contra Costa.

    Interviewer: So which of the two do you prefer?

    Interviewee: The first one. The San Jose Times.

    Interviewer: Okay. What was in it that-

    Interviewee: I don’t know it just seemed more informative. But I saw some of the same stories echo between both websites. So I thought the first one was a little more interesting because by the time I got to the second, the Contra Costa one, I had already seen the story. San Jose Times, I just liked it because it seemed more professional.

    Interviewer: In what sense? What made you feel it was more professional?

    Interviewee: I guess just the layout. There’s a little less jumble. Like the headlines were larger. I don’t think it buried the stories. And they started with topics that were a little more sober. They didn’t go right off the bat with like – stars. I can’t remember.

    I kind of look at – the movies were good. Local news was good. And stuff that affects me, I guess.

    Interviewer: Okay. Now there were a few places that you could do things – for example, you had an option to create a section for my news. Did you notice those?

    Interviewee: No. I noticed that I could personalize the weather.

    Interviewer: The weather is important.

    Interviewee: I like the weather.

    Interviewer: In which of the news websites was that?

    Interviewee: San Jose Times had a weather thing up at the top. No, I didn’t really notice any of the interactive stuff I guess.

    Interviewer: Did you notice the photo slideshows?

    Interviewee: No, just that one ad with the moving, laughing women.

    Interviewer: Which website was that on?

    Interviewee: I think it was the first one. The San Jose Times. It was over on the right.

    Interviewer: Did you notice the web polls or web vote? You know they ask you questions-

    Interviewee: No, I never vote on those. They always try to set you up.

    Interviewer: In what way?

    Interviewee: Like on AOL it’ll ask you the stupidest questions. Like I spend most of the time on AOL. And every single news story that you view, it’ll have some kind of – I think it’s some kind of trick to get people to interact more with the story. But they’ll have a little question at the bottom like, do you agree? Do you think this is right? It could be like a – like the kid who was like drowning the swimming pool. It would be like, do you think the conditions could have been stricter? And I just get so tired of those things.

    Interviewer: Now did you notice a bar with Digg and Delicious and Facebook on it?

    Interviewee: No, but I don’t share stories. I don’t email stuff out unless it’s about dogs. ‘Cause my mom really likes dogs. And she’s pretty much the only one I’d send that to.

    Even without eyetracking, this kind of session with casual users of the site can give valuable feedback into what works, and what doesn’t, about your site’s design and features.