Capitol Fax.com - Your Illinois News Radar

State’s unemployment system felled by computer problem

Posted in:

Illinois

Temporary System Problems

7/9/19 - The agency is temporarily experiencing system problems. Our system is currently unable to process claims, certifications or access claim information. We are working diligently to resolve the issue. You may contact our call center at 800-244-5631 or visit an office near you for general information.

* Sun-Times…

Illinois is temporarily unable to pay unemployment benefits and accept any new unemployment claims or mandatory certifications through its website or mobile site.

“The department expects that the issue will be resolved by the end of the week and that most recipients of unemployment benefits will receive their payments by Friday,” the Illinois Department of Employment Security told the Chicago Sun-Times Wednesday afternoon.

The state estimates payments will be delayed for about 29,000 people.

Technology troubleshooters have been working through the night and will continue to work around the clock to fix a database error causing the problems with the website and the agency’s ability to access claim information, according to an IDES statement.

“At this time, the cause of the malfunction is not fully known,” it said.

posted by Rich Miller
Wednesday, Jul 10, 19 @ 2:52 pm

Comments

Someone should pay the IT people..

Comment by Yosemite Sam Wednesday, Jul 10, 19 @ 2:57 pm
“At this time, the cause of the malfunction is not fully known,” it said. Did someone unplug the server to vacuum the floor?

Comment by Skeptic Wednesday, Jul 10, 19 @ 3:02 pm
Whatever happened to the backup plan to just run the previous set of payments and accept there will be some mistakes that will have to be corrected later?

For that matter, where are the nightly backups and the full-time mirroring of the database that used to be SOP?

Comment by RNUG Wednesday, Jul 10, 19 @ 3:24 pm
All State mainframe computers are down, not just DES. IDOT has been down for 2 days.

Comment by Huh? Wednesday, Jul 10, 19 @ 3:34 pm
This is what happens when contractors are given god rights to critical applications with zero to little oversight. Any comment from DOIT senior management?

Comment by ExpressPersonal Wednesday, Jul 10, 19 @ 3:41 pm
The last part of that statement is a flat lie….they know exactly what caused the error ….they just don’t want to admit it.

Comment by SilenceDoGood Wednesday, Jul 10, 19 @ 3:44 pm
Just turn it off and turn it back on again. Duh!

Comment by Just Me 2 Wednesday, Jul 10, 19 @ 3:48 pm
Janitor probably unplugged a switch, to plug in his new coffeemaker.

Comment by PublicServant Wednesday, Jul 10, 19 @ 3:58 pm
Russians trying to cause problems…

Comment by Nieva Wednesday, Jul 10, 19 @ 4:03 pm
##This is what happens when contractors are given god rights to critical applications with zero to little oversight. ##

Lol, if it weren’t for contractors state IT projects would never get done, correctly anyway.

Comment by It guy Wednesday, Jul 10, 19 @ 4:04 pm
DHS went down for most of the day yesterday.

Comment by Honeybear Wednesday, Jul 10, 19 @ 4:05 pm
####Lol, if it weren’t for contractors state IT projects would never get done, correctly anyway.####

I have been on both sides and that statement isn’t true.

Comment by Better IT guy Wednesday, Jul 10, 19 @ 4:17 pm
== All State mainframe computers are down, not just DES. IDOT has been down for 2 days. ==

1) Comptroller operation is / used to be totally seperate. Should still be able to run the previous set of checks again.

2) Since they are / were seperate operations, both IDOR and SOS should be unaffected.

3) The one building where most the mainframes for most agencies under the Governor are housed was built with all duplicate physical support equipment in terms of a UPS system, redundant (and oversize) A/C units, etc. The one glaring weak spot was the A/C cooling tower on the roof; building design called for a redundant pair, budget cutters eliminated one tower. It has been a known vulnerability for 40 years, and was partially addressed at one time by adding a water holding tank.

4) The other building with mainframes also had a backup UPS … and should have been able to take over the critical applications if it was a physical building or hardware problem.

Revealing a bit more about myself, I know all this because I had a hand in designing and building a lot of it.

So if all the mainframes are down, I have to assume it is more likely a software or operating system problem. If I was guessing, and it was implied in their statement, someone messed up one of more of the back-end database engines.

Comment by RNUG Wednesday, Jul 10, 19 @ 4:27 pm
I was just be of the PMs On this IDES system before I went SPSA. I can tell you in no uncertain terms could this system have been built without contractors. We were smart and hired them on as state employees after they proved their abilities.

Comment by Bothanspy Wednesday, Jul 10, 19 @ 4:42 pm
Fixed typo
I was one of the PMs on…

Comment by Bothanspy Wednesday, Jul 10, 19 @ 4:45 pm
A friend who works for IDES has told me in the past that the software that runs the system is ancient. No one has ever had the capacity to upgrade all the records and the system in its entirety from the 80s to the 21st century. Lack of modern communications causes all kinds of delays due to interoperability issues. This is all second hand, so take it with a grain of salt, but I trust my source.

Comment by Second Hand Knowledge Wednesday, Jul 10, 19 @ 5:00 pm
==A friend who works for IDES has told me in the ==past that the software that runs the system is ==ancient. No one has ever had the capacity to ==upgrade all the records and the system in its ==entirety from the 80s to the 21st century

It certainly was prior to implementation of the IBIS system in 2010 and nearly a new enhancement/debugging release every three to six months after that.

Comment by Bothanspy Wednesday, Jul 10, 19 @ 5:12 pm
It guy, considering the absolute disasters that are IES and ERP I think State employees couldn’t do any worse that the contractors.

Comment by MyTwoCents Wednesday, Jul 10, 19 @ 5:37 pm
This was caused by human error. On Monday night, someone (wouldn’t want to be this person) accidentally deleted a few disk volumes from the mainframe. It has been a scramble all week to restore the DB2 databases that were corrupted. It has affected many systems across the board. Thankfully we are on SQL.

Comment by IT Guru Wednesday, Jul 10, 19 @ 5:44 pm
The contractor was from EMC and has rights to everything. Homeland security should get involved because a majority of the contractors are not from this country and oops this happened? Sounds fishy. Who hired these people?

Comment by Dorf Wednesday, Jul 10, 19 @ 5:50 pm
DB2??? As in dBase II?? In the name of HAL, that dates back to the 1980’s (banned punctuation). It’s probably running on a screaming 386DX2 with 1 Mb memory and a 20Mb hard drive. Guess they had to download a copy of Norton Utilities from an old web site before they could start fixing the problem….

Comment by revvedup Wednesday, Jul 10, 19 @ 5:53 pm
###Homeland security should get involved because a majority of the contractors are not from this country and oops this happened? Sounds fishy.
####
Yeah, um, no. What an awful thing to say.

Comment by Anonymous Wednesday, Jul 10, 19 @ 5:56 pm
I was told by a friend in DOIT that around 80 disk volumes were accidentally deleted. No better time to test your disaster recovery procedures than after you have accidentally triggered a major disaster.

Comment by Me Again Wednesday, Jul 10, 19 @ 6:00 pm
It is a state wide outage to one of the two main frame systems . Both went down monday but one was able to be recovered monday. The other will be down “a few more days”. Part of most agencies are affected. Thankfully most of DCFS is not on the broken system. I believe most of medicaid is on the working system.

Comment by Union thug Wednesday, Jul 10, 19 @ 6:02 pm
revvedup; No, DB2 on a mainframe platform/operating system. DBase II is an old PC based database package. The States mainframe facilities are pretty much up to snuff but running a lot of older COBOL applications with with DB2 and still some IMS Databases. A lot of the problem is new staff are trained and want to work in a Web based server environment and find themselves in a crash course when it comes to dealing with the mainframe environment. The State is in transition but the mainframe won’t go away as fast as some hopefuls think. Mainframe folks are older/retiring and few and far between. Could be hearing more of this type of thing and DoIT will face some new scrutiny and make adjustments, (hopefully).

Comment by Just A Dude Wednesday, Jul 10, 19 @ 6:17 pm
IMS sytem is running. Brought that up monday. VMI is what id down.

Comment by Union thug Wednesday, Jul 10, 19 @ 6:27 pm
== I was told by a friend in DOIT that around 80 disk volumes were accidentally deleted. ==

How the @#$&¢€£%@ did someone ACCIDENTLY deleted 80 disc volumes? Where are the testing, change control / approval process that used to be in place? And why the h*ll does a contractor have that level of authority?

Comment by RNUG Wednesday, Jul 10, 19 @ 6:45 pm
Whatever happened to configuration change management? Who authorized the contractor to write code and wipe out vollumns of data affecting millions of people, costing millions in repair/restore? Disaster recovery failure. Looks like no testing was even done.

Comment by Unauthorized Wednesday, Jul 10, 19 @ 6:48 pm
Hey RUNG. Don’t know about the main part of DoIT. But where I am at. Mostly contractors working on mainframe. Excuse given is they don’t want to hire for a system that will be phased out. Course tje been phasing them out forever…. Most state workers don’t have near that access.

Comment by Union thug Wednesday, Jul 10, 19 @ 6:50 pm
@RNUG….you know how this happens? In this case DELL /EMC probably sold DOIT senior management all sorts of pie in the skybideas …like standing up entire environments of storage…but now they can’t deliver on those promises

Comment by SilenceDoGood Wednesday, Jul 10, 19 @ 6:57 pm
== Most state workers don’t have near that access. ==

Yep, about 8 employees and a few engineers had that access level back in the day … and we all had to follow the change control process.

Comment by RNUG Wednesday, Jul 10, 19 @ 7:01 pm
I’d be surprised if that many non contractors had that level of access

Comment by Union thug Wednesday, Jul 10, 19 @ 7:05 pm
== I’d be surprised if that many non contractors had that level of access ==

We had a dedicated staff just managing storage and customer issues. At total of 10 people but normally 4 people handled it. There were also a couple of DR specialists.

Heck, back before the automated tools, I remember manually rebuilding directories bit by bit one time to recover critical data which had no backup. We had to verify and cleanse the data afterward using the transaction logs, but it beat the agency re-entering the entire database from scratch.

Comment by RNUG Wednesday, Jul 10, 19 @ 7:16 pm
As noted by others, this was self-inflicted by DoIT. They’ll learn from this and put better controls in place. But it’s also a warning sign that the State is dangerously lacking in IT skills and resources.

Comment by IT Guy Wednesday, Jul 10, 19 @ 7:20 pm
== They’ll learn from this and put better controls in place. ==

There USED to be controls in place.

Comment by RNUG Wednesday, Jul 10, 19 @ 7:25 pm
I wana say thing that are not public knowledge yet. So I’ll just say they really broke it and gonna be “days” till it’s fixed.
I can say the mainframe people like to mess with them and not tell people ahead of time. Then play dumb when they break it.

Comment by Union thug Wednesday, Jul 10, 19 @ 7:25 pm
Darn union operating systems. Sounds like a work slow down to me.

Comment by Blue Dog Dem Wednesday, Jul 10, 19 @ 8:05 pm
We had controls and security when I was on the mainframe. The fact that a single person can have alter access to both test and production says it all. Something like this command should require 3 separate isolated authorization. Contractor’s have been taking union IT jobs for years, results in a lack of ownership.

Comment by Lack of control Wednesday, Jul 10, 19 @ 8:24 pm
The Mainframe still rules…..Younger IT people should be kept away until they know what they’re doing. They hate mainframes cause most don’t know anything about them.

Comment by Old COBOL guy Wednesday, Jul 10, 19 @ 8:41 pm
== They hate mainframes cause most don’t know anything about them. ==

They also don’t seem to know anything about manual processes, process control, designing systems, and then testing them. From what I have seen, all they know is fast prototyping with minimal specs and quality control … let alone optimizing them for the specified workloads.

Yeah, I sound like a cranky old guy … which I am … but I also know how to gather specs, design, write code in at least 8 mainframe or cross-platform languages, do load testing, and a phased implementation with full documentation and roll-back plans … all of which seems to have been forgotten.

Comment by RNUG Wednesday, Jul 10, 19 @ 8:54 pm
RUNG I agree no ownership with the contractors and to many of them. Also no single person should have been able to break the mainframe that much. Personally I thing some of them like to play around in the system.

Comment by Union thug Wednesday, Jul 10, 19 @ 9:05 pm
Secretary of State computers never went down. (At least not mine while at work).

Comment by Leatherneck Thursday, Jul 11, 19 @ 4:40 am
I can also confirm the deleted disk volumes incident, became I was called in late that evening to deal with the fallout in my agency. This problem affected several agencies and multiple systems and we’re still trying to get back to normal operations.

Biggest problem is that over the last decade or so the bulk of experienced senior mainframe technical staff retired out and either weren’t replaced at all or with very junior staff with insufficient experience.

Consider it yet another example of the hollowing out of critical operations.

The sad reality is a well staffed and managed mainframe shop with the appropriate configuration change controls in place will have a level of reliability, performance and security that Wintel/Microsoft server technology can still only aspire to.

Unfortunately, the management of the recruitment/hiring and procurement processes are so resistant to change in the face of documented deficiencies, that all one hears is whining that they just can’t find quality replacement staff.

Comment by Another Mainframer Thursday, Jul 11, 19 @ 6:38 am
Curious to all who know the behind-the-scenes of this: how is the level of access to K-12 Computer Science education in your neck of the woods in the state? How about anywhere in the state?

Comment by It Doesn’t Have to Be This Way Thursday, Jul 11, 19 @ 9:19 pm