Evaluating print and broadcast news in the San Francisco Bay Area from A to F.

When a poll isn't really a poll

Experts agree voluntary Internet surveys are flawed, even when demographically adjusted. Yet KGO Channel 7 insists its surveys are "scientific."

Dan Ashley narrates the ABC7 Listens poll.

For more than a year, KGO Channel 7 in San Francisco has been reporting what it calls "scientific" and "statistically valid" polls of Bay Area residents on such weighty topics as the economy, gay rights, war in Iraq and the popularity of candidates for governor.

But the station's polls are neither scientific nor statistically valid, polling experts say. In fact, they fail to meet the standards of the network to which KGO belongs. ABC News advises against even reporting on such informal "convenience" surveys.

Those polled by "ABC7 Listens" are not selected randomly from all residents of the nine-county Bay Area, but are limited to visitors to KGO's Web site, who volunteer to participate.

KGO may well present an accurate reflection of this group's opinions. But reporting their responses as truly representative of the views of all Bay Area residents breaks the cardinal rule of probability, or scientific, polling -- that every person in the population have an equal chance of being selected.

"If they are doing that they should be ashamed of themselves," said Warren Mitofsky, president of Mitofsky International in New York, who 33 years ago invented the practice of random-digit dialing for telephone polls, and for most of that time was director of elections and surveys at CBS News.

"They're misleading their viewers," Mr. Mitofsky said. "Sure, and you can make diamonds out of coal, and silk purses out of sows' ears. It's right up there with all those other theories. Who do they think they're kidding?"

Kevin Keeshan, KGO's news director, defended his station's survey. "We wouldn't be doing it if we didn't think the results were on par with what you get in telephone polls," he said. Mr. Keeshan said his station does not consult with ABC News on polling.

ABC News opposes convenience surveys

Gary Langer, head of polling for ABC News in New York, has written extensively about the standards the network uses, and which types of polls are to be avoided.

"Methodologically," Mr. Langer explains on ABC News' Web site, "in all or nearly all cases we require a probability sample, with high levels of coverage of a credible sampling frame. Self-selected or so-called 'convenience' samples, including Internet, e-mail, 'blast-fax' call-in, street intercept, and non-probability mail-in samples do not meet our standards for validity and reliability, and we recommend against reporting them."

Reached at his office in New York, Mr. Langer declined to comment on the practices of KGO, except to say that ABC-owned-and-operated stations are "editorially independent."

Mr. Langer's views coincide with those of the American Association for Public Opinion Research, an organization of more than 1,500 professional pollsters in business, government and academia. As Internet polls have proliferated over the last decade, AAPOR has taken a public stand on its Web site:

"Only when a Web-based survey adheres to established principles of scientific data collection can it be characterized as representing the population from which the sample was drawn. But if it uses volunteer respondents, allows respondents to participate in the survey more than once, or excludes portions of the population from participation, it must be characterized as unscientific and is unrepresentative of any population."

The elevation of self-selected Internet polling to the status of headline news in this country and abroad has angered scientific pollsters, who say the practice is cheap but in the long term will erode public trust in polls.

"I believe it would not be too strong to say that these Internet based panel surveys will destroy the progress of work that has been done with sound methods during the last 60 years," Mr. Mitofsky said in a recent debate about the issue. "These surveys will eventually destroy whatever confidence consumers of survey research have in the field."

KGO defends Web surveys

KGO is one of the first news organizations in the country to give volunteer Internet surveys the imprimatur of journalistic respectability. The station says it abandoned the expensive and time-intensive practice of randomly dialed telephone polls because response rates have continued to fall, many below 50%, making their results suspect.

Meanwhile, more than three-fourths of households in the Bay Area now have Internet access, Mr. Keeshan said, allowing cheap distribution of questionnaires with production values that are impossible over the phone. The station permits text responses to questions, and sometimes requires respondents to play embedded video clips showing candidate debates before making their choices.

The station's pollster, Richard Hertz of Petaluma, said random sampling isn't necessary, since the demographic profile of KGO's 5,000 Internet volunteers happens to resemble that of the whole Bay Area in terms of age, gender, race and political party. "It's almost a perfect mirror of the population," Mr. Hertz said.

When asked whether one respondent in a particular demographic category could speak for another, Mr. Hertz responded that no poll is perfect regardless of method.

"Anyone can tell you a poll where someone doesn't have to take it is self-selected," he explained. "But when you have half or three quarters of all the people in phone polling hanging up on you, we think that self-selection bias is worse. When you add up all the pieces, to us the Internet makes more sense."

Polling for less

Mr. Keeshan said the low cost of Mr. Hertz's Talking Polls system -- which Mr. Hertz put at between one-quarter and one-half of a telephone poll -- allowed him to do many more surveys. Quality telephone polls can cost up to $50,000. Since the Internet poll was introduced in October 2002, the station has sent out 21 questionnaires, and is planning to do even more next year. When Mr. Hertz started working for KGO in 1996, the station was able to afford only about four telephone polls a year.

"We're able to do more polling for the buck," Mr. Keeshan said.

On KGO's Web site, anchor Dan Ashley narrates the video introduction to the ABC7 Listens poll by saying demographic weighting makes up for any discrepancies: "Unlike most Internet surveys, our ABC7 Listens polls are scientific. This is important. With results statistically balanced to reflect the views of all Bay Area residents." On-air reports by a variety of reporters claim to know what "Bay Area residents" believe about Proposition 13, the California budget or SARS, and by what percentage.

Scientific pollsters scoff at this, however. No amount of weighting can make up for opinions excluded from the sample.

Mr. Keeshan said local news organizations are left with few options, asserting that the validity of telephone polls has declined. Some pollsters report the results of polls when 70% of those reached hang up the phone, assuming it's a telemarketer, he said. Or they don't pick up because they have caller-ID. Or they are not home when researchers call.

Mark DiCamillo, director of the respected Field Poll, acknowledged that this decade has seen less participation in phone polls. But, he said, "There's still no evidence from our own polls to show there is a breakdown in the system. The numbers are still as accurate, if not more so, than they were 20 or 30 years ago." The average discrepancy between the election polls and vote results in the last 20 years was 2.1%, he said. In that time, he said, the Field Poll has not gotten an election prediction wrong.

Evaluating Web polls

Few of the questions KGO asks its volunteers via e-mail are comparable with those asked by other California pollsters, and often the latter's polls do not single out the Bay Area. But Jon Krosnick, a communication professor at Stanford and author of numerous papers on polling science, has examined the results of the largest Internet polling company, Harris Interactive of Rochester, N.Y.

Harris Interactive claims to have recruited millions of Web users, and produces an opinion survey every week. It has had some good results and some pretty embarrassingly bad results predicting elections, Prof. Krosnick said. The company also does rankings of business schools and HMOs for The Wall Street Journal's Web site.

In a recent study, Mr. Krosnick compared Harris Interactive's results with those of Knowledge Networks, a market research company that distributes free Internet devices to randomly selected participants in order to avoid self-selection bias.

Mr. Krosnick found that Harris Interactive produces very accurate reports of the opinions of those surveyed, but those surveyed are unrepresentative of the larger population, whose opinions the surveys are intended to measure.

"What we found was a tremendous bias toward the people interested in the topic they're surveying," he said.

Other Web pollsters don't claim scientific validity

Visit www.surveys4money.com and you'll see some of the thousands of sites asking visitors to weigh in on their favorite soft drink or perfume. Pop culture sites ask whether a pop star is going to jail, and news sites sometimes take a whimsical stab at interactivity.

SFGate.com, the portal for the San Francisco Chronicle, asked this week, "Is it worth it to send spacecraft to Jupiter's moons?" The site's disclaimer says the polls are "strictly surveys of those who choose to participate and are therefore not valid statistical samples." The most careful voter-response Web tallies, such as Harris Interactive's, are labeled as specifically unscientific, "for entertainment purposes only."

Humphrey Taylor, chairman of the Harris Poll and Harris Interactive, said: "I never use the words myself, 'scientifically valid.' I only use it for telephone or in-person surveys."

In contrast, not only do KGO's polls claim scientific validity, but they report margins of error between 3 and 4.5 percentage points. Scientific pollsters argue that it is meaningless to report a margin of error because the underlying sample is flawed. Mr. Mitofsky said that if the station wants to report the results, it should explicitly say the survey is informal and unscientific.

"In the purest sense," Mr. Hertz conceded, "no, it is not perfectly scientific. But neither is phone polling. This is a serious attempt to communicate with people in a way that is convenient for them and will create a representative sampling of what the Bay Area is thinking."

In many ways, KGO's survey is a large, high-tech focus group. Respondents are given the chance to send in detailed narratives along with their votes. Mr. Keeshan said one of the most exciting applications of the system is a feedback mechanism that allows viewers to provide a detailed critique of newscasts.

"It's just very rich and incredibly useful to us in terms of attitudes, and feedback, but also story ideas -- more than I can imagine getting to in a year," he said.

What do you think? Discuss it in The Coffeehouse.

WEEKLY UPDATES

More...
A project of the School of Journalism and Mass Communications at San Jose State University, Grade the News is affiliated with the Graduate Program in Journalism at Stanford University and KTEH, public television in Silicon Valley.

Monitoring the Bay Area's most popular news media:

Contra Costa Times

Knight Ridder

San Francisco Chronicle

Hearst

San Jose Mercury News

Knight Ridder

KTVU, Oakland (FOX)

KTVU, Oakland (FOX)

KRON, San Francisco

KRON, San Francisco

KPIX, San Francisco (CBS)

KPIX, San Francisco (CBS)

KGO, San Francisco (ABC)

KGO, San Francisco (ABC)

KNTV, San Jose (NBC)

KNTV, San Jose (NBC)

 

Bay Area media advocates:

Media Alliance
Center for the Integration and Improvement of Journalism at SFSU
Maynard Institute
Youth Media Council
Project Censored
New California Media
Society of Professional Journalists, Northern California chapter
National Writers Union Bay Area chapter

Site highlights

THE GROWTH OF FREE NEWSPAPERS

The three-part series follows the rise of three Bay Area handouts:
• Part 1: At free dailies, advertisers sometimes call the shots
• Part 2: Free daily papers: more local but often superficial
• Part 3: Free papers' growth threatens traditional news
• See also: SF Examiner and Independent agree to end payola restaurant reviews
• And: The free tabloid that wasn't: East Bay's aborted Daily Flash

FATE OF KNIGHT RIDDER NEWSPAPERS

Lou Alexander started a firestorm with his original guest commentary predicting the company would be sold. Several other experts on newspapers have weighed in:
Newspapers can't cut their way back into Wall Street investors' hearts, by Stephen R. Lacy; Alexander responds
Humbler profits won't encourage buyouts, by John Morton; Alexander responds
Newspapers can't maintain monopoly profits because they've lost their monopolies, by Philip Meyer
Knight Ridder in grave jeopardy, by Lou Alexander...

KQED-FM AUDIO PERSPECTIVES BY JOHN MCMANUS

Leakers and plumbers: There's no difference between a good leak and a bad leak? Journalists need a shield law. 11/22/05
Unintended consequences: How Craigslist and similar services are sucking revenue from faltering newspapers. 9/13/05
Is CPB irrelevant? As Congress moves to cut public broadcasting funds, has CPB become obsolete in the modern marketplace. 6/26/05
The paradox of news: There's more news available and its cheaper than ever before, but fewer young people are interested. 5/12/05

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Most recent updatesHow the Bay Area's most popular media stack up.Talk about Bay Area journalism in our on-line discussion forum. A printable news scorecard you can use at home or in school. Raves and rants aimed at the local media. What would you do if you were the editor? Upcoming happenings and calls for public action. Let 'em know! Contact a local newsroom.Codes of ethics, local media advocates and journalism tools. Tip us off about the local media, or tell us how we're doing.Oops.A comprehensive list of past GTN exclusives.