Please, stop doing this nonsense
Friday, Jan 6, 2023 - Posted by Rich Miller
* Crain’s…
This report has been corrected to indicate the Polco research is a survey, not a poll.
* Block Club Chicago…
Editor’s note: This story has been updated to reflect the results were from a survey rather than a poll.
Right off the bat you know we got heaps of trouble.
* Daily Line…
The Chicago Index, a collaboration between The Daily Line and Crain’s Chicago Business, conducted its second administration of the 2023 Race for Mayor survey from November 29 through December 14, 2022. A total of 1,757 Chicago residents were surveyed from a variety of sources, including subscribers to The Chicago Index’s panel, members of The Daily Line Chicago, subscribers to Crain’s Chicago Business, and via digital engagement on social media platforms
Results were weighted on age, region, housing tenure, race, ethnicity, and gender, so the sample was consistent with the demographics of Chicago. The margin of error was +/- 3%.
I mean, that’s basically just one step above the little online “polls” I run here. The difference is, I don’t claim that my little polls are scientific and they rightfully don’t generate news headlines or create talk show fodder. They’re for fun, not info. If a reporter tried to do a story on one of those silly polls, I’d laugh in their face.
If you want to see the results, by all means click those links, but they’re worthless and the media outlets need to stop publishing this silly nonsense.
* The pushback has been justifiably fierce…
[Chicago mayoral candidate Willie Wilson] is criticizing the survey and Crain’s decision to publish it, and says that as a result he will not participate in Crain’s mayoral forum scheduled for Feb. 1.
“Crain’s published a poll that seeks to mislead the voters of Chicago,” Wilson said. “The poll Crain’s published has me at 3%. This is an insult to my supporters and Crain’s has zero journalistic integrity. I will not be a pawn in their charade; thus, I will not participate in their upcoming mayoral candidate forum.”
He added, “The news media has an obligation to demonstrate integrity and thoughtfulness when deciding which polls to publish. If they know it is misleading or inaccurate it should not be published.”
Lightfoot spokeswoman Christina Freundlich also is ripping the survey: “This is about as scientific and reliable as an Elon Musk poll,” she said. “No amount of weighting can save a survey of Chicagoans where 74% of the respondents are white. Any media organization that gives this poll any air time needs to examine their standard for poll credibility.”
One quibble: The survey sample was 53 percent white. But that’s still way, way out of line with reality.
* What Bowen said…
* But don’t take our word for it. Jason McGrath with GBAO Strategies and Brian Stryker at ALG Research both took strong exception to this goofy survey way back in April of 2021…
Good public opinion research should be as representative as possible. A representative sample attempts to provide every potential respondent with the same chance of entering the sample. A representative poll of Chicago-area residents must give people in every community the same opportunity to take the survey.
But the data collection method in the Crain’s/TDL/ABC survey fails to meet that standard. The survey’s sponsors reached out to their personal networks—especially political operatives and insiders—and those reading their stories directly. That gave some well-connected people (like us) a higher likelihood of entering the sample than millions of others. This meant their disproportionately white and male network of political insiders and business executives were more likely to take the poll.
There are certainly deficiencies in modern polling. The areas where pollsters are working to address problems, including achieving full coverage of the electorate, declining response rates, omitted variables and the incorporation of multiple modes of data collection, have been discussed in detail elsewhere. But the Crain’s/TDL/ABC survey fails to correct for any of the problems present in public opinion research today—if anything, it makes them worse and creates new ones.
The massive design flaws in the survey are only part of the problem. The widespread media coverage of the index treated this instrument with the same seriousness as it would one from Pew, Gallup or another academic institution with a long record of measuring public opinion.
And now they’re doing it again.
- Candy Dogood - Friday, Jan 6, 23 @ 11:53 am:
Science is expensive.
Expertise is expensive.
Refusing to pay for those expenses can be even more expensive.
- Teve Demotte - Friday, Jan 6, 23 @ 11:55 am:
I have seen four polls, all showing Lightfoot in trouble. Granted, polling has become less accurate, but maybe instead of attacking people for reporting the Daily Line poll, Lightfoot should make public her internal polling.
- Rich Miller - Friday, Jan 6, 23 @ 11:57 am:
=== but maybe instead of attacking people for reporting the Daily Line poll===
Stop.
The poll isn’t a poll. It’s complete garbage data and it’s been irresponsibly flung into the discourse. And if you want to call for that, fine. Get your own blog.
- Oswego Willy - Friday, Jan 6, 23 @ 12:04 pm:
Still “recogning” it as a poll, that’s peddling in alternative facts predicated on calling a phony measure the same as a measure that has scientific givens and measures.
It’s been a tiring few years standing and pushing down the phony propping up of “stats”
- Jerry - Friday, Jan 6, 23 @ 12:07 pm:
Just a reminder: 4 out of 5 dentists surveyed recommend sugarless gum for those patients who chew gum. This is scientific fact and not some marketing slogan for gum (or a poll for that matter!)
- Back to the Future - Friday, Jan 6, 23 @ 12:13 pm:
Actually participated in this survey.
Not certain how I got on the list, but clearly this should not be viewed as a poll as “poll” is generally currently defined.
I subscribe to the Crain’s Business publication so maybe that is how they contacted me.
- Michelle Flaherty - Friday, Jan 6, 23 @ 12:19 pm:
– If you are a journalism outfit and you are covering this poll, you are engaging in misinformation. –
Has anyone checked to see if Paddock Publication is printing and distributing this poll/survey?
- Three Dimensional Checkers - Friday, Jan 6, 23 @ 12:33 pm:
Total selection bias in addition to not representing the actual electorate.
It might represent Chicago government insiders who read the Daily Line and Crain’s. Unsurprisingly, this population does not seem to like Mayor Lightfoot. /snark.
- Rich Miller - Friday, Jan 6, 23 @ 12:38 pm:
===Unsurprisingly, this population does not seem to like Mayor Lightfoot===
Real polling shows that the rest of the city isn’t all that high on her either. That’s beside the point. The data is garbage.
- TheInvisibleMan - Friday, Jan 6, 23 @ 12:39 pm:
The definitions of words have been horribly blurred in media, and it does seem to be getting worse.
Here, the problem is treating a survey as if it was a poll, and then calling it a poll because that’s what the story demanded after it was written based on the output of the survey.
The other popular blurring is population, and estimated population. The same process takes place where a story is written based on the estimates and then taking that conclusion as the reasoning to declare what actual population numbers are.
It’s terrible journalism, and I question even calling it journalism. How badly does someone need to do the very basic tenants of their supposed job, before we stop calling them by that title?
- Left of what - Friday, Jan 6, 23 @ 12:47 pm:
==variety of sources, including subscribers to The Chicago Index’s panel, members of The Daily Line Chicago, subscribers to Crain’s Chicago Business, and via digital engagement on social media platforms==
Sounds like someone needs a history lesson about this type of sampling. Maybe the 1936 presidential election is a good place to start.
- H-W - Friday, Jan 6, 23 @ 1:23 pm:
== The survey’s sponsors reached out to their personal networks—especially political operatives and insiders—and those reading their stories directly. ==
I suppose you could call if Facebook on steroids sampling. But to infer the respondents thoughts and beliefs represent the thoughts and beliefs of even some segment of the Illinois population is foolishness.
Survey Methodology 101: Sampling frames matter. Representativeness matters. Margins of error. Generalizability matters. Interpretation of results matter.
Unless you are selling something.
- Moe Berg - Friday, Jan 6, 23 @ 1:28 pm:
This is a fine follow up to Rich’s fisking, aided by Rep. Krishnamoorthi’s letter, yesterday of all the sensationalist Census reporting about Illinois population loss.
In a different vein, also brings to mind the transparently bogus Bailey poll that Fox 32 promoted back in September that showed Bailey 7 points behind Pritzker and the Libertarian getting 8%. Also debunked by Rich.
“No lessons learned” seems to be the operating principle for a lot of our media outlets. They don’t hold themselves accountable to doing anything besides feeding the beast and getting clicks. Everything goes down the memory hole each day and the world is born anew the next.
- Rich Miller - Friday, Jan 6, 23 @ 1:35 pm:
===and the world is born anew the next===
I am constantly amazed that so many people can’t seem to use the Google machine.
- Anon 1:50 - Friday, Jan 6, 23 @ 1:50 pm:
Regarding Brandon Johnson’s incredible 25% showing, clearly this “survey” had an over-sample of Oak Park residents, you know, the 51st Ward.
- Pundent - Friday, Jan 6, 23 @ 3:29 pm:
=people can’t seem to use the Google machine.=
Laziness abounds. I’m amazed by the number of stories that attach significance to what someone has tweeted. But yet these “opinions” continue to get amplified as if they are reflective of how the majority of us feel.