Capitol Fax.com - Your Illinois News Radar » Human error? Try abject stupidity, Amtrak
SUBSCRIBE to Capitol Fax      Advertise Here      Mobile Version     Exclusive Subscriber Content     Updated Posts    Contact
CapitolFax.com
To subscribe to Capitol Fax, click here.
Human error? Try abject stupidity, Amtrak

Friday, Mar 1, 2019

* Tribune

Human error caused the switching system issue that brought rail traffic to an almost standstill at Union Station Thursday, affecting tens of thousands of Amtrak and Metra passengers, Amtrak said Friday.

“The root cause was human error in the process of deploying a server upgrade in our technology facility that supports our dispatch control system at Chicago Union Station,” Amtrak said in a statement. “We failed to provide the service that Amtrak customers, Metra commuters and the general public expect of us. We own the system. We will fix this problem.”

Metra and Amtrak service returned to normal for Friday morning’s rush but it was an entirely different scene at Union Station on Thursday, as more than 60,000 Chicago-area commuters either faced crowds and long delays.

* From Sen. Richard Durbin…

I talked to [Amtrak CEO and President Richard Anderson] this morning and in blunt terms, asked him what happened in Chicago yesterday? Why did thousands of commuters see their service interrupted? He was honest and direct, and admitted that Amtrak made a series of errors.

The most important error they made was to decide to do a server upgrade to their computers during peak hours of service. This should be done in the middle of the night when only a handful of trains are running. Along with that, a worker fell on a circuit board, which turned off the computers and lead to the interruption of service that went on all day long.

Today, Amtrak issued a public apology to the people who were inconvenienced in Chicago yesterday. Mr. Anderson also told me there will be changes made when it comes to computer programming and upgrades in the future. But my job in Washington, along with Senator Tammy Duckworth, is to ensure that these federal agencies are held accountable. Amtrak failed yesterday, but I appreciate their honesty.

Emphasis added because… holy moly, how irresponsibly stupid can you get?

- Posted by Rich Miller        

36 Comments
  1. - Anon E Moose - Friday, Mar 1, 19 @ 12:06 pm:

    Um, “a worker fell on a circuit board,” is something I have NEVER heard.


  2. - Ron Burgundy - Friday, Mar 1, 19 @ 12:10 pm:

    These rush hour mass debacles seem to happen at least ten times a year for one reason or another, and the frequency is increasing. Perhaps running station traffic out of Michigan remotely is a bad idea? Combine this with the wholly inadequate concourses at Union and one of these days someone is gonna get trampled. Is that what it takes to get some real solutions to the overcrowding and disrepair in and outside Union?


  3. - wordslinger - Friday, Mar 1, 19 @ 12:12 pm:

    –The most important error they made was to decide to do a server upgrade to their computers during peak hours of service.–

    No one involved questioned the timing? That’s flabbergasting, and no way to run a railroad.

    Perhaps those responsible in the decision-making loop need to be freed to pursue other opportunities.


  4. - Regular democrat - Friday, Mar 1, 19 @ 12:12 pm:

    I know u normally dont allow swearing on the blog but maybe today u can make an exception. Its mind numbing how dumb this was


  5. - City Zen - Friday, Mar 1, 19 @ 12:22 pm:

    ==decide to do a server upgrade to their computers during peak hours of service==

    Having to manage similar system upgrades from software to infrastructure, I wouldn’t be able to do a daytime upgrade of a production system no matter how hard I tried. It would never pass change management.


  6. - wordslinger - Friday, Mar 1, 19 @ 12:22 pm:

    –Combine this with the wholly inadequate concourses at Union and one of these days someone is gonna get trampled.–

    That’s happened, more than once, that I’ve witnessed over the years during rush hours at Union.

    Someone trips and falls and gets kicked around pretty good before the wildebeest herd can accommodate it.

    The escalators at rush hours are particularly dangerous. Just too many people in a hurry all at once in very tight spaces. I go out of my way to enter and exit through the Great Hall. Room to maneuver.


  7. - Blue Dog Dem - Friday, Mar 1, 19 @ 12:24 pm:

    If we paid those servers the new minimum wage of $15/ hr now, instead of waiting til 2025, I bet this never would have happened.


  8. - Ron Burgundy - Friday, Mar 1, 19 @ 12:29 pm:

    — enter and exit through the Great Hall. Room to maneuver.—

    Except when half of it is taken up by a squash court for no apparent reason. Also to continue my rant, communication both at the station as to what is going on and what people should do is still disjointed and insufficient. Communication to the public before they arrive is improving.


  9. - Skeptic - Friday, Mar 1, 19 @ 12:32 pm:

    ” I wouldn’t be able to do a daytime upgrade of a production system no matter how hard I tried.” That would be the sane thing to do. Beancounters, however, when presented with a potential bill for overtime are frequently not sane.


  10. - ike - Friday, Mar 1, 19 @ 12:32 pm:

    I work by the metra station by Western Avenue and lucked out that my afternoon train was only delayed 30 mins (though i had to board a completely filled train), but the signal/switch issues has been an on going problem that Metra never seems to try and fix.


  11. - Skeptic - Friday, Mar 1, 19 @ 12:32 pm:

    “a worker fell on a circuit board,” Is that what they’re calling it these days?


  12. - PublicServant - Friday, Mar 1, 19 @ 12:33 pm:

    Somebody ought to tell the server infrastructure group that they’re 24×7 when it comes to infrastructure upgrades. While general maintenance might lend itself to a regular 9-5 schedule, when large software migrations and/or server upgrades occur Saturday and Sunday evenings are no-brainers for an entity that has such clearly defined heavy periods of use.


  13. - anon - Friday, Mar 1, 19 @ 12:37 pm:

    But did they try rebooting?


  14. - Not It - Friday, Mar 1, 19 @ 12:38 pm:

    Someone should be fired, but it is government, so they’ll probably be promoted up and out.


  15. - Da Big Bad Wolf - Friday, Mar 1, 19 @ 12:38 pm:

    “A worker fell on a circuit board”. Sometimes it’s a good idea to have a spare circuit board laying around.


  16. - PublicServant - Friday, Mar 1, 19 @ 12:47 pm:

    ===Sometimes it’s a good idea to have a spare circuit board laying around.===

    That’s why someone fell on the first one…just sayin.


  17. - What's in a name? - Friday, Mar 1, 19 @ 12:53 pm:

    Once again Wordslinger is my hero.

    “Perhaps those responsible in the decision-making loop need to be freed to pursue other opportunities.”

    Perfect.


  18. - Stuff Happens - Friday, Mar 1, 19 @ 12:54 pm:

    Something doesn’t add up here, and feels like it was easier to share this narrative than to explain what really happened.

    Was there a zero-day exploit on the server? Was it being actively hacked and they had to address it immediately? I have a hard time believing that someone was so irresponsible as to pick that time without having a good reason. But if it’s a reason like that, maybe the story being shared is better…


  19. - Perrid - Friday, Mar 1, 19 @ 1:02 pm:

    Stuff Happens, I am a firm believer in Hanlon’s Razor: “Never attribute to malice that which is adequately explained by stupidity.”

    The odds that they would lie about this to hide a security breach are much, much smaller than a bunch of people making stupid decisions.


  20. - Tommydanger - Friday, Mar 1, 19 @ 1:03 pm:

    Little did we know that Homer Simpson is now working at Amtrak.


  21. - Anonymous - Friday, Mar 1, 19 @ 1:12 pm:

    Give them time, they will show you.


  22. - Lester Holt’s Mustache - Friday, Mar 1, 19 @ 1:12 pm:

    ==Um, “a worker fell on a circuit board,” is something I have NEVER heard.==

    I’m going to assume this is just Durbin misspeaking, and what he was actually told was “control” board. If someone fell on a circuit board, that means Metra has people working around equipment with the outside panels removed.


  23. - Huh? - Friday, Mar 1, 19 @ 1:12 pm:

    So the three stooges were in charge of the computer upgrade? Or was it Laurel and Hardy?


  24. - FormerParatrooper - Friday, Mar 1, 19 @ 1:13 pm:

    1:12 was me


  25. - Independent - Friday, Mar 1, 19 @ 1:14 pm:

    While falling on the circuit board did the worker take out a vacuum tube as well?


  26. - GA Watcher - Friday, Mar 1, 19 @ 1:17 pm:

    “Communication to the public before they arrive is improving.”

    The Transit Tracker on the Ventra App was worthless yesterday. It reported that all BNSF trains were running “As Scheduled” even as the media was reporting BNSF was implementing load and go.


  27. - Anon - Friday, Mar 1, 19 @ 1:20 pm:

    Survey says: Entity was trying to avoid paying overtime or shift differential so they did it during the middle of a week day


  28. - Terry Salad - Friday, Mar 1, 19 @ 1:31 pm:

    Did they try turning it off and on again?


  29. - Sonny - Friday, Mar 1, 19 @ 1:57 pm:

    What’s wild is they just did a free pass weekend as some kind of please forgive us campaign.


  30. - ChrisB - Friday, Mar 1, 19 @ 1:59 pm:

    So fun story: I was on the first train affected by the shutdown, scheduled to be in the station at 8:35. We sat outside of Union, just south of the post office for over an hour. We were already late/slow because BNSF switches went down earlier. Finally got off the train at 10:10.

    What kills me about this explanation is that Amtrak didn’t even have switchmen on site to fix the problem. The conductor was trying to keep up updated, but there was literally nothing to say. No updates from the radios, no news from Union. He was audibly frustrated and couldn’t see any workers to manually flip the switches for a good hour. That tells me that there was absolutely no redundancy or failsafes in place for this server upgrade.

    I didn’t even try to go home on the BNSF. The excuses are unacceptable.

    This is all going to get really fun when the current contract is up on April 30th, and Metra and Amtrak have to negotiate a new one. Metra is paying relative peanuts ($244Mil total since 1984), and word is that Amtrak wants more. Meanwhile the concourses don’t have ceiling tiles, and the concrete above the tracks is literally falling on people’s heads, but hey, Ye Olde Fashioned Barbershop is top notch. Keep fiddling, Amtrak/Metra.


  31. - Sir Reel - Friday, Mar 1, 19 @ 2:03 pm:

    The second explanation was, my dog ate the server upgrade.


  32. - OneMan - Friday, Mar 1, 19 @ 2:15 pm:

    “holy moly, how irresponsibly stupid can you get”

    As a long time BNSF commuter, I have every confidence that someone in this situation is going to get stupider as time progresses.

    It would be interesting to see exactly what sort of “server upgrade” they were doing, was this an OS upgrade, and upgrade to a software system or a hardware upgrade? All of those have different levels of risk and different risks involved.

    The ‘circuit board’ getting fallen upon adds a whole different set of risk questions, did someone fall into a server (it can sort of happen) was the board normally covered? Was the board normally exposed without a cover? What was it’s height above floor level? Did Amtrak have an additional board handy?

    But the Amtrak part of this, while comical and annoying is a relatively small piece of the risk.

    The risk in this whole thing is when we have the station full and the vestibules full and something happens. Lets use the least ’sinister’ option of someone has a can of mace accidentally discharge in an enclosed area with hundreds of people right next to each other jostling for position. There are a half dozen other things that could cause a crowd like that to panic without it being a malicious act that have happened at Union station. A fist fight, ceiling tile coming down (that risk is mitigated by the station not having had ceiling tiles in most spaces in the drop ceiling for a at least a year), pipe burst, platform ceiling collapse, etc. Don’t even think what a malicious actor could do in that situation.

    You are going to have hundreds of people trying to get out a small space very quickly, that isn’t going to end well. Tbh that is why I avoid Union Station in these situations, because short of a ceiling collapse on the 2-4 platform I think that is the most likely way my commute might kill me someday, in a stampede of panicked people out of Union Station.

    But why do people congregate in Union Station when the trains are having issues?

    Because Metra does a terrible and I mean terrible, job of communicating how there is a problem, how long they think the problem is going is last and what the cause of the issue is. I think at some level Governor Rod has more credibility than Metra.

    Since Metra has 0 credibility and offers these sort of wishy washy alerts, people ignore them and head to the station.

    I am a commuter who gets alerts from Metra and I found out the issues were going to impact the afternoon rush via tweets from Chicago media outlets. No alert email from them, no tweet, those came later.

    With Information I can make my decision to, wait it out (go to a bar), take the UP-W line and plan how to get from Geneva to my station or Uber to my station.

    Fundamentally Metra and Amtrak need to make the ‘we are not letting people into the station at all’ call earlier and with more feeling. Their crowd control plans (relatively new), are a joke.

    I have said this before, but Metra as an organization views itself as a victim, a victim of low funding, a victim of Amtrak and victim of the rail carriers (they don’t seem to hold the BNSF accountable for anything) and with that attitude they are destined to fail.


  33. - Ron Burgundy - Friday, Mar 1, 19 @ 2:59 pm:

    One thing they could do pretty quickly is to put up some electronic signage at the entrances to the north and south concourses for how they want traffic to go. For instance Southwest and Heritage Corridor here, BNSF that way, etc. When BNSF only has an issue, people on those other lines can’t get through the mob of people standing around. Ideally what they need is one method of internal communication.Announcements plus people with bullhorns who sometimes say two conflicting things is not working.


  34. - OneMan - Friday, Mar 1, 19 @ 3:11 pm:

    Ron

    True that. The amount of times during an issue where Metra and the BNSF contradict each other is disappointingly high


  35. - a drop in - Friday, Mar 1, 19 @ 3:18 pm:

    Someone had to call in for a fix, “this is the AMTRAK Technical Support. All line are busy helping other people. Please stay on the line…..”


  36. - Keyrock - Friday, Mar 1, 19 @ 4:12 pm:

    “A worker fell on a circuit board”.
    They might as well have blamed the Pepsi Syndrome,


Sorry, comments for this post are now closed.


* Question of the day
* Learn something new every day
* Ives could kick off congressional bid soon as Sanguinetti announces lots of establishment endorsements
* Pick a lane
* Elections have consequences
* Still waiting on the fact check
* City hires Florida man to trap Chance the Snapper
* The return of the circular firing squad
* Zalewski raid/ComEd probe roundup
* *** LIVE COVERAGE ***
* Yesterday's stories

Support CapitolFax.com
Visit our advertisers...

...............

...............

...............


Loading


Main Menu
Home
Illinois
YouTube
Pundit rankings
Obama
Subscriber Content
Durbin
Burris
Blagojevich Trial
Advertising
Updated Posts
Polls

Archives
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004

Blog*Spot Archives
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005

Syndication

RSS Feed 2.0
Comments RSS 2.0
WordPress




Hosted by MCS SUBSCRIBE to Capitol Fax Advertise Here Mobile Version Contact Rich Miller