Capitol Fax.com - Your Illinois News Radar » State’s unemployment system felled by computer problem
SUBSCRIBE to Capitol Fax      Advertise Here      Mobile Version     Exclusive Subscriber Content     Updated Posts    Contact
CapitolFax.com
To subscribe to Capitol Fax, click here.
State’s unemployment system felled by computer problem

Wednesday, Jul 10, 2019

* IDES website

Temporary System Problems

​7/9/19 - The agency is temporarily experiencing system problems. Our system is currently unable to process claims, certifications or access claim information. We are working diligently to resolve the issue. You may contact our call center at 800-244-5631 or visit an office near you for general information.

* Sun-Times

Illinois is temporarily unable to pay unemployment benefits and accept any new unemployment claims or mandatory certifications through its website or mobile site.

“The department expects that the issue will be resolved by the end of the week and that most recipients of unemployment benefits will receive their payments by Friday,” the Illinois Department of Employment Security told the Chicago Sun-Times Wednesday afternoon.

The state estimates payments will be delayed for about 29,000 people.

Technology troubleshooters have been working through the night and will continue to work around the clock to fix a database error causing the problems with the website and the agency’s ability to access claim information, according to an IDES statement.

“At this time, the cause of the malfunction is not fully known,” it said.

- Posted by Rich Miller        

44 Comments »
  1. - Yosemite Sam - Wednesday, Jul 10, 19 @ 2:57 pm:

    Someone should pay the IT people..


  2. - Skeptic - Wednesday, Jul 10, 19 @ 3:02 pm:

    “At this time, the cause of the malfunction is not fully known,” it said. Did someone unplug the server to vacuum the floor?


  3. - RNUG - Wednesday, Jul 10, 19 @ 3:24 pm:

    Whatever happened to the backup plan to just run the previous set of payments and accept there will be some mistakes that will have to be corrected later?

    For that matter, where are the nightly backups and the full-time mirroring of the database that used to be SOP?


  4. - Huh? - Wednesday, Jul 10, 19 @ 3:34 pm:

    All State mainframe computers are down, not just DES. IDOT has been down for 2 days.


  5. - ExpressPersonal - Wednesday, Jul 10, 19 @ 3:41 pm:

    This is what happens when contractors are given god rights to critical applications with zero to little oversight. Any comment from DOIT senior management?


  6. - SilenceDoGood - Wednesday, Jul 10, 19 @ 3:44 pm:

    The last part of that statement is a flat lie….they know exactly what caused the error ….they just don’t want to admit it.


  7. - Just Me 2 - Wednesday, Jul 10, 19 @ 3:48 pm:

    Just turn it off and turn it back on again. Duh!


  8. - PublicServant - Wednesday, Jul 10, 19 @ 3:58 pm:

    Janitor probably unplugged a switch, to plug in his new coffeemaker.


  9. - Nieva - Wednesday, Jul 10, 19 @ 4:03 pm:

    Russians trying to cause problems…


  10. - It guy - Wednesday, Jul 10, 19 @ 4:04 pm:

    ##This is what happens when contractors are given god rights to critical applications with zero to little oversight. ##

    Lol, if it weren’t for contractors state IT projects would never get done, correctly anyway.


  11. - Honeybear - Wednesday, Jul 10, 19 @ 4:05 pm:

    DHS went down for most of the day yesterday.


  12. - Better IT guy - Wednesday, Jul 10, 19 @ 4:17 pm:

    ####Lol, if it weren’t for contractors state IT projects would never get done, correctly anyway.####

    I have been on both sides and that statement isn’t true.


  13. - RNUG - Wednesday, Jul 10, 19 @ 4:27 pm:

    == All State mainframe computers are down, not just DES. IDOT has been down for 2 days. ==

    1) Comptroller operation is / used to be totally seperate. Should still be able to run the previous set of checks again.

    2) Since they are / were seperate operations, both IDOR and SOS should be unaffected.

    3) The one building where most the mainframes for most agencies under the Governor are housed was built with all duplicate physical support equipment in terms of a UPS system, redundant (and oversize) A/C units, etc. The one glaring weak spot was the A/C cooling tower on the roof; building design called for a redundant pair, budget cutters eliminated one tower. It has been a known vulnerability for 40 years, and was partially addressed at one time by adding a water holding tank.

    4) The other building with mainframes also had a backup UPS … and should have been able to take over the critical applications if it was a physical building or hardware problem.

    Revealing a bit more about myself, I know all this because I had a hand in designing and building a lot of it.

    So if all the mainframes are down, I have to assume it is more likely a software or operating system problem. If I was guessing, and it was implied in their statement, someone messed up one of more of the back-end database engines.


  14. - Bothanspy - Wednesday, Jul 10, 19 @ 4:42 pm:

    I was just be of the PMs On this IDES system before I went SPSA. I can tell you in no uncertain terms could this system have been built without contractors. We were smart and hired them on as state employees after they proved their abilities.


  15. - Bothanspy - Wednesday, Jul 10, 19 @ 4:45 pm:

    Fixed typo
    I was one of the PMs on…


  16. - Second Hand Knowledge - Wednesday, Jul 10, 19 @ 5:00 pm:

    A friend who works for IDES has told me in the past that the software that runs the system is ancient. No one has ever had the capacity to upgrade all the records and the system in its entirety from the 80s to the 21st century. Lack of modern communications causes all kinds of delays due to interoperability issues. This is all second hand, so take it with a grain of salt, but I trust my source.


  17. - Bothanspy - Wednesday, Jul 10, 19 @ 5:12 pm:

    ==A friend who works for IDES has told me in the ==past that the software that runs the system is ==ancient. No one has ever had the capacity to ==upgrade all the records and the system in its ==entirety from the 80s to the 21st century

    It certainly was prior to implementation of the IBIS system in 2010 and nearly a new enhancement/debugging release every three to six months after that.


  18. - MyTwoCents - Wednesday, Jul 10, 19 @ 5:37 pm:

    It guy, considering the absolute disasters that are IES and ERP I think State employees couldn’t do any worse that the contractors.


  19. - IT Guru - Wednesday, Jul 10, 19 @ 5:44 pm:

    This was caused by human error. On Monday night, someone (wouldn’t want to be this person) accidentally deleted a few disk volumes from the mainframe. It has been a scramble all week to restore the DB2 databases that were corrupted. It has affected many systems across the board. Thankfully we are on SQL.


  20. - Dorf - Wednesday, Jul 10, 19 @ 5:50 pm:

    The contractor was from EMC and has rights to everything. Homeland security should get involved because a majority of the contractors are not from this country and oops this happened? Sounds fishy. Who hired these people?


  21. - revvedup - Wednesday, Jul 10, 19 @ 5:53 pm:

    DB2??? As in dBase II?? In the name of HAL, that dates back to the 1980’s (banned punctuation). It’s probably running on a screaming 386DX2 with 1 Mb memory and a 20Mb hard drive. Guess they had to download a copy of Norton Utilities from an old web site before they could start fixing the problem….


  22. - Anonymous - Wednesday, Jul 10, 19 @ 5:56 pm:

    ###Homeland security should get involved because a majority of the contractors are not from this country and oops this happened? Sounds fishy.
    ####
    Yeah, um, no. What an awful thing to say.


  23. - Me Again - Wednesday, Jul 10, 19 @ 6:00 pm:

    I was told by a friend in DOIT that around 80 disk volumes were accidentally deleted. No better time to test your disaster recovery procedures than after you have accidentally triggered a major disaster.


  24. - Union thug - Wednesday, Jul 10, 19 @ 6:02 pm:

    It is a state wide outage to one of the two main frame systems . Both went down monday but one was able to be recovered monday. The other will be down “a few more days”. Part of most agencies are affected. Thankfully most of DCFS is not on the broken system. I believe most of medicaid is on the working system.


  25. - Just A Dude - Wednesday, Jul 10, 19 @ 6:17 pm:

    revvedup; No, DB2 on a mainframe platform/operating system. DBase II is an old PC based database package. The States mainframe facilities are pretty much up to snuff but running a lot of older COBOL applications with with DB2 and still some IMS Databases. A lot of the problem is new staff are trained and want to work in a Web based server environment and find themselves in a crash course when it comes to dealing with the mainframe environment. The State is in transition but the mainframe won’t go away as fast as some hopefuls think. Mainframe folks are older/retiring and few and far between. Could be hearing more of this type of thing and DoIT will face some new scrutiny and make adjustments, (hopefully).


  26. - Union thug - Wednesday, Jul 10, 19 @ 6:27 pm:

    IMS sytem is running. Brought that up monday. VMI is what id down.


  27. - RNUG - Wednesday, Jul 10, 19 @ 6:45 pm:

    == I was told by a friend in DOIT that around 80 disk volumes were accidentally deleted. ==

    How the @#$&¢€£%@ did someone ACCIDENTLY deleted 80 disc volumes? Where are the testing, change control / approval process that used to be in place? And why the h*ll does a contractor have that level of authority?


  28. - Unauthorized - Wednesday, Jul 10, 19 @ 6:48 pm:

    Whatever happened to configuration change management? Who authorized the contractor to write code and wipe out vollumns of data affecting millions of people, costing millions in repair/restore? Disaster recovery failure. Looks like no testing was even done.


  29. - Union thug - Wednesday, Jul 10, 19 @ 6:50 pm:

    Hey RUNG. Don’t know about the main part of DoIT. But where I am at. Mostly contractors working on mainframe. Excuse given is they don’t want to hire for a system that will be phased out. Course tje been phasing them out forever…. Most state workers don’t have near that access.


  30. - SilenceDoGood - Wednesday, Jul 10, 19 @ 6:57 pm:

    @RNUG….you know how this happens? In this case DELL /EMC probably sold DOIT senior management all sorts of pie in the skybideas …like standing up entire environments of storage…but now they can’t deliver on those promises


  31. - RNUG - Wednesday, Jul 10, 19 @ 7:01 pm:

    == Most state workers don’t have near that access. ==

    Yep, about 8 employees and a few engineers had that access level back in the day … and we all had to follow the change control process.


  32. - Union thug - Wednesday, Jul 10, 19 @ 7:05 pm:

    I’d be surprised if that many non contractors had that level of access


  33. - RNUG - Wednesday, Jul 10, 19 @ 7:16 pm:

    == I’d be surprised if that many non contractors had that level of access ==

    We had a dedicated staff just managing storage and customer issues. At total of 10 people but normally 4 people handled it. There were also a couple of DR specialists.

    Heck, back before the automated tools, I remember manually rebuilding directories bit by bit one time to recover critical data which had no backup. We had to verify and cleanse the data afterward using the transaction logs, but it beat the agency re-entering the entire database from scratch.


  34. - IT Guy - Wednesday, Jul 10, 19 @ 7:20 pm:

    As noted by others, this was self-inflicted by DoIT. They’ll learn from this and put better controls in place. But it’s also a warning sign that the State is dangerously lacking in IT skills and resources.


  35. - RNUG - Wednesday, Jul 10, 19 @ 7:25 pm:

    == They’ll learn from this and put better controls in place. ==

    There USED to be controls in place.


  36. - Union thug - Wednesday, Jul 10, 19 @ 7:25 pm:

    I wana say thing that are not public knowledge yet. So I’ll just say they really broke it and gonna be “days” till it’s fixed.
    I can say the mainframe people like to mess with them and not tell people ahead of time. Then play dumb when they break it.


  37. - Blue Dog Dem - Wednesday, Jul 10, 19 @ 8:05 pm:

    Darn union operating systems. Sounds like a work slow down to me.


  38. - Lack of control - Wednesday, Jul 10, 19 @ 8:24 pm:

    We had controls and security when I was on the mainframe. The fact that a single person can have alter access to both test and production says it all. Something like this command should require 3 separate isolated authorization. Contractor’s have been taking union IT jobs for years, results in a lack of ownership.


  39. - Old COBOL guy - Wednesday, Jul 10, 19 @ 8:41 pm:

    The Mainframe still rules…..Younger IT people should be kept away until they know what they’re doing. They hate mainframes cause most don’t know anything about them.


  40. - RNUG - Wednesday, Jul 10, 19 @ 8:54 pm:

    == They hate mainframes cause most don’t know anything about them. ==

    They also don’t seem to know anything about manual processes, process control, designing systems, and then testing them. From what I have seen, all they know is fast prototyping with minimal specs and quality control … let alone optimizing them for the specified workloads.

    Yeah, I sound like a cranky old guy … which I am … but I also know how to gather specs, design, write code in at least 8 mainframe or cross-platform languages, do load testing, and a phased implementation with full documentation and roll-back plans … all of which seems to have been forgotten.


  41. - Union thug - Wednesday, Jul 10, 19 @ 9:05 pm:

    RUNG I agree no ownership with the contractors and to many of them. Also no single person should have been able to break the mainframe that much. Personally I thing some of them like to play around in the system.


  42. - Leatherneck - Thursday, Jul 11, 19 @ 4:40 am:

    Secretary of State computers never went down. (At least not mine while at work).


  43. - Another Mainframer - Thursday, Jul 11, 19 @ 6:38 am:

    I can also confirm the deleted disk volumes incident, became I was called in late that evening to deal with the fallout in my agency. This problem affected several agencies and multiple systems and we’re still trying to get back to normal operations.

    Biggest problem is that over the last decade or so the bulk of experienced senior mainframe technical staff retired out and either weren’t replaced at all or with very junior staff with insufficient experience.

    Consider it yet another example of the hollowing out of critical operations.

    The sad reality is a well staffed and managed mainframe shop with the appropriate configuration change controls in place will have a level of reliability, performance and security that Wintel/Microsoft server technology can still only aspire to.

    Unfortunately, the management of the recruitment/hiring and procurement processes are so resistant to change in the face of documented deficiencies, that all one hears is whining that they just can’t find quality replacement staff.


  44. - It Doesn’t Have to Be This Way - Thursday, Jul 11, 19 @ 9:19 pm:

    Curious to all who know the behind-the-scenes of this: how is the level of access to K-12 Computer Science education in your neck of the woods in the state? How about anywhere in the state?


TrackBack URI

This is not Facebook, so uncivil comments, profanity of any kind, rumors and anonymous commenters will not be tolerated and will likely result in banishment.



* SUBSCRIBERS ONLY - Today's edition of Capitol Fax (use all CAPS in password)
* *** LIVE COVERAGE ***
* Yesterday's stories

Support CapitolFax.com
Visit our advertisers...

...............

...............

...............


Loading


Main Menu
Home
Illinois
YouTube
Pundit rankings
Obama
Subscriber Content
Durbin
Burris
Blagojevich Trial
Advertising
Updated Posts
Polls

Archives
July 2019
June 2019
May 2019
April 2019
March 2019
February 2019
January 2019
December 2018
November 2018
October 2018
September 2018
August 2018
July 2018
June 2018
May 2018
April 2018
March 2018
February 2018
January 2018
December 2017
November 2017
October 2017
September 2017
August 2017
July 2017
June 2017
May 2017
April 2017
March 2017
February 2017
January 2017
December 2016
November 2016
October 2016
September 2016
August 2016
July 2016
June 2016
May 2016
April 2016
March 2016
February 2016
January 2016
December 2015
November 2015
October 2015
September 2015
August 2015
July 2015
June 2015
May 2015
April 2015
March 2015
February 2015
January 2015
December 2014
November 2014
October 2014
September 2014
August 2014
July 2014
June 2014
May 2014
April 2014
March 2014
February 2014
January 2014
December 2013
November 2013
October 2013
September 2013
August 2013
July 2013
June 2013
May 2013
April 2013
March 2013
February 2013
January 2013
December 2012
November 2012
October 2012
September 2012
August 2012
July 2012
June 2012
May 2012
April 2012
March 2012
February 2012
January 2012
December 2011
November 2011
October 2011
September 2011
August 2011
July 2011
June 2011
May 2011
April 2011
March 2011
February 2011
January 2011
December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
May 2010
April 2010
March 2010
February 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009
July 2009
June 2009
May 2009
April 2009
March 2009
February 2009
January 2009
December 2008
November 2008
October 2008
September 2008
August 2008
July 2008
June 2008
May 2008
April 2008
March 2008
February 2008
January 2008
December 2007
November 2007
October 2007
September 2007
August 2007
July 2007
June 2007
May 2007
April 2007
March 2007
February 2007
January 2007
December 2006
November 2006
October 2006
September 2006
August 2006
July 2006
June 2006
May 2006
April 2006
March 2006
February 2006
January 2006
December 2005
April 2005
March 2005
February 2005
January 2005
December 2004
November 2004
October 2004

Blog*Spot Archives
November 2005
October 2005
September 2005
August 2005
July 2005
June 2005
May 2005

Syndication

RSS Feed 2.0
Comments RSS 2.0
WordPress




Hosted by MCS SUBSCRIBE to Capitol Fax Advertise Here Mobile Version Contact Rich Miller