Content Analysis Standards
Creating a valid and reliable measure of the quality of a product as complex and varied as a newspaper or newscast is very difficult. The news is as varied as life.
Evaluation can’t be so complicated and subjective that two judges would grade the same content differently. And it can’t be so simple that it lumps good with bad. Into the chasm between these two poles, we thrust the following plan.
I. General Rules
To provide the fairest representation of news, we will sample randomly, but set up the survey so that each day of the week—Sunday through Saturday has an equal chance of selection. That way, certain weekdays when city councils and county boards of supervisors meet will be included equally with weekends when some news organizations may attempt longer stories with greater perspective.
We will sample all news organizations on the same day, so the raw material of events available is the same. The sample will be drawn approximately one day out of ten.Readership and viewership studies show that the front page and section fronts and the first segments of newscasts attract the largest audience. Stories appearing in these places are the most important to analyze since they are being offered to the largest audience. Extremely brief stories generally do not make as great an impression on readers and viewers as full-size ones. In addition, most newsrooms allocate more space or time to stories they judge more important.
So we will include all stories in premier evening newscast’s first half-hour longer than 20 seconds and all stories longer than 10 square inches on the front and local/metro front pages of the newspaper. (Extremely brief stories are also excluded so as not to artificially lower measures of quality such as number of sources.) Stations will be invited to nominate which newscast they would prefer sampled.Stories will be weighted by time or space, respectively. It wouldn’t be fair to average as equals a story that runs 5 minutes with a short one only 30 seconds long.
Sidebars—contiguous stories so closely related to one another that they could have been included in the main story—will be treated as part of one larger story. This avoids lower context and fairness scores that might result from judging stories individually.
We will only count news time or space. So banter among anchors, advertisements and promotions will not be considered. All indices will be based on the percentage of news time or space used a certain way. Absolute volume of news will not be considered so as to be fair to both newspapers and television.
Newspaper photographs that are related to a particular story on the relevant pages shall be included as part of that story, both in cutline information and space. Photos or graphics unrelated to a particular story on the target pages will be treated as separate stories. Informational graphics will be included with the relevant story.
We will evaluate news providers on seven basic criteria: newsworthiness, context, local relevance, accuracy, fairness, civic contribution, and enterprise. All count equally except the first, which counts double because of its fundamental importance. All fractions and ties are rounded upward, in favor of the news organizations. The seven evaluative criteria are inferred from journalism’s principal codes of ethics (SPJ, APME, ASNE, RTNDA).
What counts as “A” work is necessarily subjective. We take into account that all seven news sources are parts of large corporations with great resources. All operate in one of the nation’s largest and wealthiest media markets. Finally, we are only grading top stories—where we can expect the best work.
II. Newsworthiness Index (all stories)
This measure will examine the selection priorities of news providers. It will be judged against the primary ethical mandate of journalism: To help the community served make sense of current issues and events. Events and issues that shape the quality of life are most important. The more people for whom the information has this quality, the greater its value. (Events exciting short-term emotional response, say over a sports team winning or losing a game, or even a championship, are important, but less so.)
Each story will be judged on two criteria: whether its topic is core news or marginal, and the level of the reporting—basically episodic and affecting few residents of the area, or thematic and affecting many residents. Core topics rate 2 points; non-core 1 point. Micro level reporting rates 1 point; big picture reporting rates 3 points, to reflect its far greater usefulness to consumers. The two are multiplied to arrive at the score for the story. Thus no story can earn fewer than 1 point nor more than 6.
Putting a premium on thematic “big picture” reporting in the top stories of the day is appropriate for media striving to serve an area as large as the nine-county Bay Area, just part of which constitutes the nation’s fifth largest media market. The size of the market also provides the revenue needed for reporting broadly rather than chronicling isolated incidents.
Level of reporting
We are introducing a distinction based on two elements: 1) number of people directly affected over a significant period of time; and 2) a reporting style that focuses on issues or trends in events vs. a focus on isolated events.
Micro stories: More than half of the story describes a particular discrete action, or perhaps several actions unconnected, except by category, e.g. fires, police actions, criminal acts, etc. It is episodic, event-oriented, occurring at a particular place and time. It emphasizes the “what.” Examples include stories about a particular shooting, rape, accident, fire, or trial. Generally, those whose lives are directly affected are few relative to the size of Bay Area audience. Even if reporting takes place in several places, e.g. the homes of local victims of an airliner crash, if the principal focus is the crash, it is episodic.
Exception: Some events, such as natural disasters and wars, also affect life conditions for many people. Also, the actions of politicians and candidates for office at all levels, and some courts affect all of the people living in a certain jurisdiction. Even if these are distinct events, such as votes or judicial rulings, if they directly affect many people, reports about them will be considered big picture stories.
Big picture stories: More than half the story examines an issue or pattern of events, or an event directly affecting many people (in a metro area this large, >10,000). Generally thematic, these stories usually involve a variety of people in more than one specific location coping with a common issue. Reports focus more on “why,” “who” or “how” than “what.” Examples include trend stories about educational practice, or transportation or housing conditions, or systematic reporting on the criminal justice system, the environment or economic conditions.
Note that the distinction between micro and big picture stories is determined by their breadth of focus, not their popularity or length. The trials of O.J. Simpson, though very popular and frequently lengthy, were micro stories. A 90-second story about why spousal abuse happens, however, was a big picture story.
1. Politics, including any news about candidates, their positions, political issues, and ballot initiatives or fund-raising or the political system, U.S. or foreign. Also include political careers and leaders’ obituaries and memorials for social and political leaders. Profiles of leaders of political, or reform, or social movements go here. Any news about the process of people taking political power, losing it, protesting it, or the process of transition fits in this category. Consider action of incumbent politicians in this category only if they are clearly political in nature. Otherwise, place them in category #2.
Unless stories bear on fitness for office or are related to public policy, articles about personal lives should be excluded (other than obits).
2. Government (executive and legislative branches) new government policies, actions, deliberation and decision-making—from the level of city councils and mayors to county boards of supervisors to state legislatures, directors of state bureaucracies and governors to the president, directors of federal agencies and Congress, and other nation's governments. But if half or more of a story is about some issue listed below, choose that topic. If more than half is about the governmental decision-making, it stays here. Thus a city council discussion about police misconduct would go here, but were half or more of the story about the misconduct itself, and non-council discussion, it would move to the criminal justice category.
Deliberations and decision-making of school superintendents and school boards and supervisors or commissions for police or other public services that are below the level of city council, go in the more specific categories below.
Profiles of government leaders that relate to their fitness for office or actions in office go here, but exclude stories that concern only their private lives.
If a proposal by politicians is introduced as legislation, it goes here. But if it’s merely a political proposal, not yet submitted to become law, place it in number 1.
3. Natural disasters, earthquakes, floods, landslides, avalanches, etc. To qualify, it must include severe damage--destruction of life, homes or major buildings.
4. Education, private or public, from pre-school to high school to college to post-graduate. Also corporate training, recruiting, retaining teachers, etc. Stories about sports as part of educational or personal development belong here, but not accounts of varsity games.
5. Economic events and issues, including business conditions, how a company is doing (except health care), labor, employment, stock markets, trade agreements and disputes, civil suits in which one business sues another (criminal and government regulatory business suits, however, go in the next category), etc. Promotional stories--those about a single company that are written from a positive, rather than objective point of view, and contain no independent expert sources--go in category 19, Other Stories.
6. Crimes, criminal justice system, criminal and civil trials involving individuals, police actions (including Coast Guard patrols--but not rescues unless criminal action occurs first). Also investigations of wrongdoing. Aftermath of crimes and courts including funerals, memorials, hospital reports, investigations, imprisonment, parole, cash settlements, etc.
Supreme Court, and other court decisions go here. So do regulatory trials, e.g. Microsoft.
7. Medical/health/fitness events and issues, including scientific research and discoveries in these areas and threats to public health or physical safety. Include fund-raising for medicine--such as AIDS bike rides. Also medical ethics. Include the business of medicine/health/ nutrition.
8. Environmental events and issues. Include government or private actions to clean toxic waste sites, limit housing density, protect open space or environment, etc. Also natural phenomena such as eclipses, global warming, El Nino, etc.
9. Other important social events, issues or trends, e.g., race-, sexual orientation-, ethnic-, disability-, class- relations or tensions or programs; transportation/commuting; agriculture, housing, land use, parenting styles; divorce or marriage rates; lifestyle changes, religion, historical info, civic initiatives and projects such as building or stocking museums, art galleries, zoos, etc.
10. Technology/science issues and events. Non-medical scientific discovery--physical and social science, e.g. space exploration. History of technology. New social theories. Business aspects of technology companies/services go in #5.
11. Major fires, accidents--auto, air, rail and otherwise--or other mishaps, their investigation or their repair, e.g. sewer line rupture, etc.—that involve more than $25 million in damage or the loss of three or more lives of Bay Area residents, or five or more lives elsewhere. Also the results of such incidents—hospital reports, funerals, damage reports, etc. (Accidents involving criminal charges, however, go in category 6.)
12. Weather (short of natural disasters) and direct effects of weather, record heat or cold or wetness, heavy surf.
13. Military actions, deployments, attacks, threats, military peace-keeping efforts. Also military policies, studies of readiness, budgets and weapons. Include US and foreign.
14. Consumer reporting about commercial goods and services (but excluding medicine/fitness/ health). Consumer reporting must include explicit comparisons of products, service or selection with other retailer or manufacturer. The tone must be objective, or critical, rather than promotional. Promotion goes in category 19.
15. Human interest, seasonal celebrations/festivals and entertainment stories, including stories about TV programs, movies, and popular culture. Also fashion trends, style, auto and boat shows, and fads. Stories in this category often focus primarily on amusement, or other emotions: curiosity, surprise, adventure, joy, love, loyalty, sorrow, sexiness, anger, disgust, rivalry--e.g. cute animal or other stories, re-unions, what people are doing on a holiday, features on children, tales of oddities, etc.. These are often light stories meant to bring a smile or tug a heartstring, but can also be sad—such as obits--or serious reminiscences about tragedy or the human condition.
16. Information about celebrities and other widely known persons both famous and notorious. Generally concerns the lives/deaths of actors, actresses, athletes, authors, etc. Exclude politicians unless story un-related to public's business.
17. Sports and sports-related (including recreation and hobbies), amateur and professional, including equipment and facilities. However, exclude stories about sports facilities that include public money or a vote, including facilities in public schools or parks, and the role of sports in education. These go in categories for politics, government or education.
18. Minor fires, accidents, mishaps, traffic jams—those in which fewer than three Bay Area residents (fewer than five from elsewhere) lost their lives or less than $25 million in damage was done. Results of such incidents, e.g., hospital reports, damage, funerals, etc. Misadventures—people hurt or missing or killed in non-criminal or uncertain circumstances--individual suicides.
19. Other stories.
Grades are based on a weighted average. Depending on topic and level of reporting, stories earn between 1 and 6 points. A score of 100% would indicate the top stories were all core topics that inherently affect many (>5,000) in the Bay Area or that are reported in a way that makes them applicable to many. Averages of 80% or more earn an A; 77-79 =A-; 74-76%=B+; 71-73%=B; 68-70%=B-; 65-67%=C+; 62-64%=C; 59-61%=C-; 56-58%=D+; 53-55%=D; 50-52%=D-; < 50%=F.
The grading scale is less stringent than an academic evaluation to give news media the benefit of the doubt and to avoid penalizing them for variation among judges applying the standards below. While we believe that skilled journalists with sufficient resources can render the consequential compelling, we won’t penalize the judicious display of merely interesting stories.
This measure is based on the number and quality of sources in a story. Up to 150 points are assigned as follows:
· 100 for a random sample of more than 100 persons conducted by the newspaper or station under normal social science requirements.
· 40 for each named independent expert source, up to a total of 3 sources.
· 20 for each specific source, up to a total of 5 sources.
· 10 per story for 1 or more un-named sources.
Sources may be persons or documents. They may be quoted directly or paraphrased. They need not be on air. Generalized sources, such as “police,” “educators,” or “critics” count as unnamed sources. A specific source must be a particular person, never plural. He or she may be anonymous only if expressly granted anonymity in the report, e.g., a source who spoke only on condition of anonymity. A document’s specific title is unnecessary, but the authoring institution, department or individual must be named, e.g., a report prepared by the Public Utilities Commission.
Because stories vary in how many sources they need, stories with more than 100 points can make up for those with less. We estimated that the top stories of the day should average more than two identified non-expert sources and one non-specified source. Simple event stories might appropriately have fewer and issue stories more.
Grades range from A, 85 points or above to F, less than 55. Minus and plus grades are assigned at 3 point intervals, i.e., 82-84=A-; 79-81=B+, etc.
This measures the percentage of all stories that either take place in the nine-county Bay Area or have a direct relationship to the region’s citizens. Such directly related stories would include all stories affecting the entire state, especially those emanating from state government. Thus a story about a train wreck in Southern California would not be included (unless the story focused on stricken residents from the Bay Area), but a story about how Africanized bees are making their way north in the state, would. Weighted by space or time, stories would be sorted into Bay Area/Bay Area-related and outside stories.
While the Bay Area has large numbers of immigrants and foreign nationals and what happens outside the U.S. is important to them, they do not represent a majority of residents, nor is what happens abroad as important as often as what happens in their present locale.
The index would be graded differently for newscasts with (usually adjacent) national/international news supplements (network newscasts) than for stand-alone newscasts and newspapers. The former would be expected to concentrate more on local news than the latter. Channel 2 cannot rely on a network newscast to present news outside the Bay Area. Channels 4, 5 and 7, however, also carry network news.
The grading standard for Channel 2 and the three newspapers is at left; at right is the standard for Channels 4, 5 and 7:
This measure will be based on telephone interviews with identifiable and reachable Bay Area sources. Unlike other indices, this survey will occur infrequently (because it will consume so much time). The unit of analysis also changes, from the story to the source. Accuracy assessment will be from the source’s perspective. Points are assigned as follows:
· 20 for correctly spelled name. (In TV, 10 for spelling and 10 for pronunciation.)
· 10 for correct title or identifier.
· 40 for accurate, or mostly accurate quotation; 20 for moderately accurate; 0 for substantially inaccurate.
· 30 for accurate or mostly accurate representation of source’s main point during interview; 15 points for moderately accurate; 0 points for substantially inaccurate.
However, if the interview was adversarial and the source reports substantial inaccuracy, we’ll deduct 15 points from this category and add it to the previous category. (In adversarial interviews, sources may have a self-interest in disparaging reporter’s accuracy, particularly about which parts of the interview were most newsworthy.)
Unlike other indices where the unit of analysis is the story, here scores are based on the average degree of accuracy per source. Grading standards are the same as on the context index.
This measures the percentage of time/space consumed by controversial stories in which more than one side is given the opportunity to make its case (even if that opportunity is rejected). Stories in which a single perspective is acceptable are excluded from the analysis. Such stories include uncontested decisions of boards and other groups, authoritative accounts of events such as fires, accidents, and the weather. Also excluded from analysis are stories in which an opposing view would not be available, e.g., a not-yet-apprehended criminal suspect. Finally, opinions offered by columnists are not counted.
The fairest stories offer competing sides the same opportunities for comment. If one side is quoted directly, so are the others if they choose (and can be reached for comment). If one side is on air, so are the others (unless there is some obvious reason why they cannot be.) Neither time nor space must be equal, however. (We seek objectivity of method, but not of result).
Less fair are stories in which all sides are represented, but not given equal opportunity to respond. Unfair stories don’t give all obvious sides a chance to comment.
Fairness grades are based on the percentage of news time or space in stories judged completely fair plus one-half the time or space in stories judged partly fair. Grades range from A, 85% or above to F, less than 55%. Minus and plus grades are assigned at 3 point intervals, i.e., 82-84=A-; 79-81=B+, etc.
This question seeks to measure reporting on the actions of those at the controls of our government at all levels, but not those who carry out their decisions. So we count the percentage of time or space taken by all stories in which 1/4 or more of the story describes politics, actions (related to public business) or deliberations of government supervisory or regulatory or lawmaking boards or bodies, elected officials or the heads of any agencies of the U.S., state or local governments. We include studies (conducted by government or outsiders) of effectiveness or problems of arms of domestic government and their policies. Likewise, we include court cases involving civil liberties, and government vs. corporate defendants (e.g. Microsoft), but not prosecutions of individuals unless they are government officials.
We exclude routine actions of police, schools, fire departments and other arms of government, unless expressly political. Also excluded are actions of governments outside the U.S. One exception: Include any investigative report about government or about a private company that touches on its relationship with government.
Journalism ethics asks for constant scrutiny of how our government is behaving. So such stories ought to be a mainstay of coverage. However, they need not consume the majority of news space or time.
The grading standard is as follows: 40% or more=A; 37-39=A-; 34-36=B+; 31-33=B; 28-30=B-; 25-27=C+; 22-24=C; 19-21=C-; 16-18=D+; 13-15=D; 10-12=D-; <10=F.
VIII. Enterprise Index (locally-originated stories and parts of stories only)
This measures the proactivity of the newsroom, its willingness to seek our answers to the public’s questions rather than simply react to events or other’s agendas. The most demanding of such stories on newsroom resources is the investigative story.
An enterprise story differs from spot news in time and focus. It may begin with a recent event--although not usually something that happened that day--but the view is more informational and broader. It generally answers a question the public may have about some issue or event. Enterprise stories are originated by journalists usually trying to explain or contextualize important happenings or issues.
Investigative stories are enterprise stories that focus on mis- or malfeasance usually by government, but sometimes also by private industry or others. The investigation is conducted by the journalist, not law enforcement authorities.
Because they are more demanding, time/space spent on investigative reports is weighted by a factor of 4. In other words, a 3-minute investigative story is treated as if it lasted 12 minutes.
Grading follows the same standard as the Civic Contribution Index: 40% or more=A; 37-39=A-; 34-36=B+; 31-33=B; 28-30=B-; 25-27=C+; 22-24=C; 19-21=C-; 16-18=D+; 13-15=D; 10-12=D-; <10=F.
IX. Overall Grade
This is a mean, computed as if the constituent letter grades were part of an academic grade: An A is worth 4.0, A- is 3.7, B+ is 3.3, B is 3.0, etc. The first index, newsworthiness, is counted twice. The grade assigned is whichever lies closest to the mean grade with ties going to the news organization, e.g., a 2.5 average lies equidistant between 2.3, a C+ and 2.7, a B-; we would assign a B-.
Standards revised on 4/12/00. They apply to content analyses beginning March 5, 2000.