Capitol - Your Illinois News Radar » More on the strengths - and weaknesses - of ChatGPT
SUBSCRIBE to Capitol Fax      Advertise Here      About     Exclusive Subscriber Content     Updated Posts    Contact Rich Miller
To subscribe to Capitol Fax, click here.
More on the strengths - and weaknesses - of ChatGPT

Friday, Jan 27, 2023 - Posted by Rich Miller

* Background is here if you need it. Daily Mail

A group of University of Minnesota Law School instructors administered popular artificial intelligence bot ChatGPT four law school exams that it took alongside real students, and discovered that the bot was a C+ student on paper.

The ChatGPT bot is currently considered the world’s most advanced example of generative intelligence.

The instructors were curious about ways the bot might be used both to help students cheats and professors teach, so they decided to test its abilities.

The bot ended up with a ‘low but passing grade’ in Constitutional Law, Employee Benefits, Taxation, and Torts. All told, it wound up with a C+ average.

According to the paper, the bot came in 36th out of 40 students in Con Law, 18th out of 19 in Employee Benefits, 66th out of 67 in Tax, and dead-last out of 75 in Torts.

* A few days ago, I asked ChatGPT for some help with the blog…

Q: Summarize today’s news about Illinois government and politics.

A: I’m sorry, I don’t have the capability to provide you with today’s news as my training data only goes up to 2021 and my knowledge cut off date is 2021.

To which a friend replied in a group chat…

Oh my God, is ChatGPT Mike Madigan?!?


* Yesterday, I thought I’d ask ChatGPT a question that might be able to provide us some humorous blog content…

What would Abraham Lincoln say to Illinois Gov. JB Pritzker?

The answer was really weird. Here’s a brief excerpt…

The process of creating a podcast involves several steps:

1. Idea generation: The first step in creating a podcast is to come up with an idea for the show. This could be a topic you’re passionate about, a hobby you enjoy, or a problem you want to solve.

2. Research: Once you have an idea for your podcast, you’ll need to research the topic to ensure that it’s something people will be interested in listening to. This could involve looking at the competition, identifying gaps in the market, and talking to potential listeners.

Um, what?

* But I went back last night and that answer had mysteriously disappeared and was replaced by another one…

It is impossible to know exactly what Abraham Lincoln would say to JB Pritzker as they lived in different times and contexts. However, based on Lincoln’s speeches and writings, he likely would have emphasized the importance of preserving the Union, promoting equality and justice, and working for the betterment of all citizens.

Not great, but at least it’s on-topic.

* A suburban legislator I talked with uses ChatGPT to draft some letters and says it’s like having a world-class chef helping in the kitchen…

It’s the biggest technological innovation since at least the Internet. Will reorient almost every sector of the economy over the next 5-10 years. I don’t know what the policy implications are yet but I’m certain it will raise novel legislative issues we will need to deal with.

* Not long after that conversation, I asked ChatGPT to write a letter to an Illinois legislator’s constituent defending the state’s new assault weapons ban…

Dear Constituent,

I am writing to address your concerns about the recently passed assault weapons ban law in Illinois. I understand that this is a highly debated issue and one that many people feel strongly about. However, I firmly believe that this law is necessary for the safety and well-being of our community.

First and foremost, it is important to understand that this law does not prohibit the ownership of all firearms. It specifically targets the possession and sale of assault weapons and high-capacity magazines, which have been used in numerous mass shootings in our country. By limiting access to these types of weapons, we can reduce the risk of mass shootings and other gun-related violence in our state.

Additionally, it is important to note that this law includes a provision for current owners of assault weapons to register their firearms with the state, ensuring that they can continue to legally possess them. This is a fair and reasonable solution that allows individuals to continue to exercise their Second Amendment rights while also taking steps to protect public safety.

I understand that this law may not be popular with everyone, but I believe it is a necessary step in addressing the ongoing issue of gun violence in our country. As your representative, I am committed to working towards solutions that keep our communities safe and I believe this law is a step in the right direction.

Thank you for your understanding and please do not hesitate to reach out if you have any further questions or concerns.


Your thoughts?


  1. - Kelly Cassidy - Friday, Jan 27, 23 @ 2:36 pm:

    I’ve played with it to create fact sheets. I’ve fed a few bills into it and asked for bullet points and it’s surprisingly helpful.

  2. - Rich Miller - Friday, Jan 27, 23 @ 2:36 pm:

    ===I’ve fed a few bills into it and asked for bullet points===

    Fascinating. That could change a whole lot of things.

  3. - Michelle Flaherty - Friday, Jan 27, 23 @ 2:41 pm:

    Per the AI letter, people contacting a lawmaker are most likely upset with the assault weapons ban and specifically the require to register existing weapons with the state police.

    It’s not a poorly crafted letter, but it reads like a form letter from a bureaucrat or computer and it likely serves to only further stoke the anger of the writer.

  4. - DuPage Saint - Friday, Jan 27, 23 @ 2:44 pm:

    I foresee another cut back amendment. Senate down to one Chat GBT and house down to two. No salaries no pensions no relative on payrolls. A win for Illinois

  5. - cover - Friday, Jan 27, 23 @ 2:46 pm:

    = the bot came in 36th out of 40 students in Con Law =

    Did it place ahead of Tom Devore? /s

  6. - G'Kar - Friday, Jan 27, 23 @ 2:48 pm:

    I keep trying to access the ChatGPT website and I’m constantly unable connect. It must be popular.

  7. - G'Kar - Friday, Jan 27, 23 @ 2:49 pm:

    My last comment reads like it was written by a bot. I meant to write “I’m constantly unable to connect to the site.”

  8. - levivotedforjudy - Friday, Jan 27, 23 @ 2:51 pm:

    I’ve been doing my own little surveying. People who write essays have said they view it as a sort of editorial assistant. A lot of people I know who are more journalists (including me a Medill grad) view it as a tool for plagiarism. Is this really progress?

  9. - Wensicia - Friday, Jan 27, 23 @ 2:52 pm:

    The letter is pragmatic and concise, which is beyond the capability of the far right to understand.

  10. - Jaded - Friday, Jan 27, 23 @ 2:58 pm:

    I’ve seen how this story ends and it involves Neo, Morpheous, Trinity and/or John Connor trying to save the world…

  11. - H-W - Friday, Jan 27, 23 @ 2:59 pm:

    All of the ideas represented (as in, re-presented) in an AI essay are taken from existing sources. In the absence of sourcing, these are plagiarized essays.

    That the user does not know the actual sources that are used to frame the essay can be problematic for people presenting these ideas as their own.

  12. - OneMan - Friday, Jan 27, 23 @ 3:01 pm:

    I shared on the earlier post its description of CapFax.

    I asked it about a charitable organization I am involved with and it was on target until it added several countries to places the group is involved with.

    That being said, when I asked it to describe symbolism in Moby Dick, it came up with an answer that would have made my HS English teacher happy.

  13. - Dysfunction Junction - Friday, Jan 27, 23 @ 3:02 pm:

    =The letter is pragmatic and concise, which is beyond the capability of the far right to understand.=

    Not to mention NO CAPITAL LETTERS (banned punctuation). It’s not gonna fool anyone in the Eastern Bloc as having been from their legislator.

  14. - Amalia - Friday, Jan 27, 23 @ 3:02 pm:

    I’m most concerned with how to recognize that work comes from it. Don’t like it. Letter was meh.

  15. - Sonny - Friday, Jan 27, 23 @ 3:06 pm:

    Letter is vastly better than drafts of letters that I’ve had to rewrite or edit over the years

  16. - very old soil - Friday, Jan 27, 23 @ 3:10 pm:

    additional thoughts on the topic

  17. - Jerry - Friday, Jan 27, 23 @ 3:22 pm:

    Does it order a take out pizza correctly? I’ll hangup and listen for my answer. (humor intended)

  18. - clec dcn - Friday, Jan 27, 23 @ 3:22 pm:

    Very impressive letter in many respects and far above what some might craft themselves. I have a friend who works in the AI world as a creator and designer. I believe he is paid handsomely for sure. It has potential for many things however not truly creative in the human sense. It also could depend on the slant of the creator or system. I wonder if I could give me a good Homily for Sunday Mass. Be interesting to find out.

  19. - /s - Friday, Jan 27, 23 @ 3:23 pm:

    Maybe I’m biased having written a number of these letters in my day but as an above comment stated, it reads like a form letter, with no tie-in to the district/community. This is less like having a world class chef in your kitchen and far more like having a PR firm based in NYC having their intern just out of college writing a letter for you, some representative in some town they’ve never heard of.

  20. - Dysfunction Junction - Friday, Jan 27, 23 @ 3:24 pm:

    Strange. I asked it to read me Doctor Memory and all it said was >SYSTAT UPTIME 9:01 unhappy READ MAKNAM”

    In other words, still placed ahead of Tom DeVore.

  21. - OneMan - Friday, Jan 27, 23 @ 3:27 pm:

    Subject area AI will create challenges for the government sooner rather than later. One thing will be the requirements to retain and disclose training data for some systems, either for regulatory or litigation purposes.
    If you feed training data that have biases present in them (obvious or not), those biases will tend to make their way into the AI and be harder to identify. Who is going to be liable for issues that become present in AI due to issues with the training data?

    You are also looking at potential “ownership” issues down the road. Will vendors try to retain some ownership of results produced by their AIs for other entities? You can use the results for homework, but not a commercial product.

    If some bot gives you bad investment advice, who is going to be liable? The person who used the bot, the folks who built the bot, the folks who provided training data? What if your data is used for training without your knowledge or permission? Can you still be liable for how it’s used? What if you give permission? What if you work someplace for five years, and they use what you have done as a work product to train a bot? What rights do you have?

    Down the road, AI law is going to be a real thing and is going to be interesting.

  22. - 47th Ward - Friday, Jan 27, 23 @ 3:30 pm:

    For faculty worried about cheating, the absence of typos and use of excellent grammar should be a hint that the student used a chat bot.

    You can tell my writing is original by all of the mistakes I make.

  23. - Hannibal Lecter - Friday, Jan 27, 23 @ 3:35 pm:

    === I’m most concerned with how to recognize that work comes from it. Don’t like it. Letter was meh. ===

    Unfortunately, many people would kill for “meh” nowadays. Gen Z should be concerned that they will not be needed because they will be replaced by robots.

  24. - XonXoff - Friday, Jan 27, 23 @ 3:39 pm:

    I ran a couple basic requests to generate python code a while back and was pretty impressed with the results.

  25. - walker - Friday, Jan 27, 23 @ 3:55 pm:

    Proft newspapers have an alternative source.

  26. - don the legend - Friday, Jan 27, 23 @ 3:55 pm:

    An experiment: A candidate for office should use it for every question asked throughout the campaign. It would be like Peter Sellers character, Chance, in Being There.

    The answers would be so much better than most candidates provide. Think Darren Bailey.

  27. - ZC - Friday, Jan 27, 23 @ 4:02 pm:

    The most interesting thing I read today about ChatGPT, and how it “thinks,” was an exploration of why it still cannot generate academic citations and winds up making up stuff.

    But it will get better, I’m sure.

  28. - JS Mill - Friday, Jan 27, 23 @ 4:14 pm:

    Bailey and devore should consider using the bot for their communication. I too think the bot is bland and uninspiring, but unlike bailey and devore it is intelligible and makes sense most of the time.

  29. - PF Flyer - Friday, Jan 27, 23 @ 4:22 pm:

    It may have been a mediocre C+ but it passed the Bar exam, on the first try. It would result in a law degree.

  30. - TheInvisibleMan - Friday, Jan 27, 23 @ 4:24 pm:

    And this should be exactly how it needs to be integrated into learning.

    Give students an assignment to knowingly use ChatGPT to generate your assignment, then have the students become their own teachers and check the output themselves for anything that is wrong and sign off on the final product. After handing in an assignment, it is then given to another student to do the same checking on the assignment of someone else, to sign off on the final product.

    Which means they have to check every sentence themselves, and the same for questions someone else asks to be generated.

    It integrates the new tool into education, with how it will most likely be used in their adult lives, while simultaneously showing it has weaknesses that can be found with additional checking and teamwork.

    But, there’s no way that’s going to happen on a large scale in the US. Here, it will only be those who take it upon themselves to learn using this method, and it won’t take long before they leave the traditionally educated in the dust.

    The old guy down the street who still talks about ’squaring the circle’ never did go on to be very good at math as an adult.

  31. - Huh? - Friday, Jan 27, 23 @ 4:29 pm:

    “If you feed training data that have biases present in them (obvious or not), those biases will tend to make their way into the AI”

    A few years ago Amazon wrote an AI program in an effort to increase diversity in their hiring. The program kept hiring men because the AI program was trained by scanning the resumes of people that had been hired in the previous 10 years. After a few years, Amazon scrapped the system.

    Just goes to show GIGO - garbage in, garbage out.

  32. - Banish Misfortune - Friday, Jan 27, 23 @ 4:33 pm:

    Constituent letter reads about like every letter I have ever received from a legislator.

Sorry, comments for this post are now closed.

* Reader comments closed for the weekend
* New laws (Updated)
* SUBSCRIBERS ONLY - Campaign update: Circling the wagons
* Isabel's afternoon roundup
* Southern Illinoisan newspaper journalists say goodbye to their community after new owner fires all employees
* Report examines Illinois women’s political power
* Proposal to save suburban, Chicago public transit transit includes tax on services, parking and raising RTA sales tax
* AG Raoul, others file antitrust lawsuit against NCAA
* Question of the day: 2023 Golden Horseshoe Awards
* SUBSCRIBERS ONLY - Supplement to today's edition
* SUBSCRIBERS ONLY - Today's edition of Capitol Fax (use all CAPS in password)
* Open thread
* Isabel’s morning briefing
* Live Ed Burke Trial Coverage
* Live coverage
* Yesterday's stories

Visit our advertisers...



Main Menu
Pundit rankings
Subscriber Content
Blagojevich Trial
Updated Posts

December 2023
November 2023
October 2023
September 2023
August 2023
July 2023
June 2023
May 2023
April 2023
March 2023
February 2023
January 2023
December 2022
November 2022
October 2022
September 2022
August 2022
July 2022
June 2022
May 2022
April 2022
March 2022
February 2022
January 2022
December 2021
November 2021
October 2021
September 2021
August 2021
July 2021
June 2021
May 2021
April 2021
March 2021
February 2021
January 2021
December 2020
November 2020
October 2020
September 2020
August 2020
July 2020
June 2020
May 2020
April 2020
March 2020
February 2020
January 2020
December 2019
November 2019
October 2019
September 2019
August 2019
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004

Blog*Spot Archives
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005


RSS Feed 2.0
Comments RSS 2.0

Hosted by MCS SUBSCRIBE to Capitol Fax Advertise Here Mobile Version Contact Rich Miller