Stories were randomly selected over 11 months (October ’99 through August, 2000) based on a system guaranteeing equal numbers of each day of the week. That balances traditionally lean news days with fuller ones.
In almost all cases, stations and newspapers were compared every 10 days on the same news-gathering cycle¾so all had access to the same set of events. In other words, all stations were sampled on the same evening and all papers on the following morning. Random samples are good estimates of routine coverage but often miss the exceptional¾whether it’s awesome or awful.
We skipped extremely brief stories (under 20 seconds or 10 square inches) because to include them would have watered down quality. In fact, we weighted every story by the percent of the newspaper or newscast it consumed. A four-minute think piece got eight times the credit of a 30-second “reader.” Because television often breaks a large story into related parts, we put them together in the analysis to maximize the station’s score. We did the same for newspaper sidebars.
Two analysts¾both experienced journalists¾agreed on their assessments of the same stories an average of 79 percent of the time after subtracting for chance agreements. For a chance-corrected measure, this falls within accepted social science standards. (The two graders were John McManus and Alan Goldstein, former news director at KRON.)
Top editors in every newsroom assessed were given the opportunity to correct, respond or suggest improvements before this analysis was posted. So were several members of the Grade the News Advisory Board with significant journalism experience. The analysis received only positive feedback.
Even though two independent coders graded the stories approximately the same way, the scoring conventions¾such as making the newsworthiness index worth twice the value of any other index¾are subjective. These were developed with the help of Grade the News’ advisory board.
We based our yardstick on a consensus of journalism’s codes of ethics. Most of these are derived from a “social responsibility” theory of journalism’s purpose. This is the idea that journalism receives special privileges from government¾such as low postal rates, freedom from content regulation, exemptions from anti-trust and child labor laws, and free access to air waves¾which are unavailable to other businesses, in exchange for doing its best to build an informed citizenry. (There are alternative quality measures, e.g., whichever station or newspaper attracts the largest audience, which others might consider superior.)
As with most social science, there were occasional problems with data gathering. Equipment sometimes failed, stations would change broadcast schedules at the last minute (usually for sports playoff games), and paper deliveries were missed. Sometimes we just plain forgot to tape. Still, we believe the analysis presents a basic, but fair, portrait of quality.