44 Comments - Yosemite Sam - Wednesday, Jul 10, 19 @ 2:57 pm: Someone should pay the IT people.. - Skeptic - Wednesday, Jul 10, 19 @ 3:02 pm: “At this time, the cause of the malfunction is not fully known,” it said. Did someone unplug the server to vacuum the floor? - RNUG - Wednesday, Jul 10, 19 @ 3:24 pm: Whatever happened to the backup plan to just run the previous set of payments and accept there will be some mistakes that will have to be corrected later? For that matter, where are the nightly backups and the full-time mirroring of the database that used to be SOP? - Huh? - Wednesday, Jul 10, 19 @ 3:34 pm: All State mainframe computers are down, not just DES. IDOT has been down for 2 days. - ExpressPersonal - Wednesday, Jul 10, 19 @ 3:41 pm: This is what happens when contractors are given god rights to critical applications with zero to little oversight. Any comment from DOIT senior management? - SilenceDoGood - Wednesday, Jul 10, 19 @ 3:44 pm: The last part of that statement is a flat lie….they know exactly what caused the error ….they just don’t want to admit it. - Just Me 2 - Wednesday, Jul 10, 19 @ 3:48 pm: Just turn it off and turn it back on again. Duh! - PublicServant - Wednesday, Jul 10, 19 @ 3:58 pm: Janitor probably unplugged a switch, to plug in his new coffeemaker. - Nieva - Wednesday, Jul 10, 19 @ 4:03 pm: Russians trying to cause problems… - It guy - Wednesday, Jul 10, 19 @ 4:04 pm: ##This is what happens when contractors are given god rights to critical applications with zero to little oversight. ## Lol, if it weren’t for contractors state IT projects would never get done, correctly anyway. - Honeybear - Wednesday, Jul 10, 19 @ 4:05 pm: DHS went down for most of the day yesterday. - Better IT guy - Wednesday, Jul 10, 19 @ 4:17 pm: ####Lol, if it weren’t for contractors state IT projects would never get done, correctly anyway.#### I have been on both sides and that statement isn’t true. - RNUG - Wednesday, Jul 10, 19 @ 4:27 pm: == All State mainframe computers are down, not just DES. IDOT has been down for 2 days. == 1) Comptroller operation is / used to be totally seperate. Should still be able to run the previous set of checks again. 2) Since they are / were seperate operations, both IDOR and SOS should be unaffected. 3) The one building where most the mainframes for most agencies under the Governor are housed was built with all duplicate physical support equipment in terms of a UPS system, redundant (and oversize) A/C units, etc. The one glaring weak spot was the A/C cooling tower on the roof; building design called for a redundant pair, budget cutters eliminated one tower. It has been a known vulnerability for 40 years, and was partially addressed at one time by adding a water holding tank. 4) The other building with mainframes also had a backup UPS … and should have been able to take over the critical applications if it was a physical building or hardware problem. Revealing a bit more about myself, I know all this because I had a hand in designing and building a lot of it. So if all the mainframes are down, I have to assume it is more likely a software or operating system problem. If I was guessing, and it was implied in their statement, someone messed up one of more of the back-end database engines. - Bothanspy - Wednesday, Jul 10, 19 @ 4:42 pm: I was just be of the PMs On this IDES system before I went SPSA. I can tell you in no uncertain terms could this system have been built without contractors. We were smart and hired them on as state employees after they proved their abilities. - Bothanspy - Wednesday, Jul 10, 19 @ 4:45 pm: Fixed typo I was one of the PMs on… - Second Hand Knowledge - Wednesday, Jul 10, 19 @ 5:00 pm: A friend who works for IDES has told me in the past that the software that runs the system is ancient. No one has ever had the capacity to upgrade all the records and the system in its entirety from the 80s to the 21st century. Lack of modern communications causes all kinds of delays due to interoperability issues. This is all second hand, so take it with a grain of salt, but I trust my source. - Bothanspy - Wednesday, Jul 10, 19 @ 5:12 pm: ==A friend who works for IDES has told me in the ==past that the software that runs the system is ==ancient. No one has ever had the capacity to ==upgrade all the records and the system in its ==entirety from the 80s to the 21st century It certainly was prior to implementation of the IBIS system in 2010 and nearly a new enhancement/debugging release every three to six months after that. - MyTwoCents - Wednesday, Jul 10, 19 @ 5:37 pm: It guy, considering the absolute disasters that are IES and ERP I think State employees couldn’t do any worse that the contractors. - IT Guru - Wednesday, Jul 10, 19 @ 5:44 pm: This was caused by human error. On Monday night, someone (wouldn’t want to be this person) accidentally deleted a few disk volumes from the mainframe. It has been a scramble all week to restore the DB2 databases that were corrupted. It has affected many systems across the board. Thankfully we are on SQL. - Dorf - Wednesday, Jul 10, 19 @ 5:50 pm: The contractor was from EMC and has rights to everything. Homeland security should get involved because a majority of the contractors are not from this country and oops this happened? Sounds fishy. Who hired these people? - revvedup - Wednesday, Jul 10, 19 @ 5:53 pm: DB2??? As in dBase II?? In the name of HAL, that dates back to the 1980’s (banned punctuation). It’s probably running on a screaming 386DX2 with 1 Mb memory and a 20Mb hard drive. Guess they had to download a copy of Norton Utilities from an old web site before they could start fixing the problem…. - Anonymous - Wednesday, Jul 10, 19 @ 5:56 pm: ###Homeland security should get involved because a majority of the contractors are not from this country and oops this happened? Sounds fishy. #### Yeah, um, no. What an awful thing to say. - Me Again - Wednesday, Jul 10, 19 @ 6:00 pm: I was told by a friend in DOIT that around 80 disk volumes were accidentally deleted. No better time to test your disaster recovery procedures than after you have accidentally triggered a major disaster. - Union thug - Wednesday, Jul 10, 19 @ 6:02 pm: It is a state wide outage to one of the two main frame systems . Both went down monday but one was able to be recovered monday. The other will be down “a few more days”. Part of most agencies are affected. Thankfully most of DCFS is not on the broken system. I believe most of medicaid is on the working system. - Just A Dude - Wednesday, Jul 10, 19 @ 6:17 pm: revvedup; No, DB2 on a mainframe platform/operating system. DBase II is an old PC based database package. The States mainframe facilities are pretty much up to snuff but running a lot of older COBOL applications with with DB2 and still some IMS Databases. A lot of the problem is new staff are trained and want to work in a Web based server environment and find themselves in a crash course when it comes to dealing with the mainframe environment. The State is in transition but the mainframe won’t go away as fast as some hopefuls think. Mainframe folks are older/retiring and few and far between. Could be hearing more of this type of thing and DoIT will face some new scrutiny and make adjustments, (hopefully). - Union thug - Wednesday, Jul 10, 19 @ 6:27 pm: IMS sytem is running. Brought that up monday. VMI is what id down. - RNUG - Wednesday, Jul 10, 19 @ 6:45 pm: == I was told by a friend in DOIT that around 80 disk volumes were accidentally deleted. == How the @#$&¢€£%@ did someone ACCIDENTLY deleted 80 disc volumes? Where are the testing, change control / approval process that used to be in place? And why the h*ll does a contractor have that level of authority? - Unauthorized - Wednesday, Jul 10, 19 @ 6:48 pm: Whatever happened to configuration change management? Who authorized the contractor to write code and wipe out vollumns of data affecting millions of people, costing millions in repair/restore? Disaster recovery failure. Looks like no testing was even done. - Union thug - Wednesday, Jul 10, 19 @ 6:50 pm: Hey RUNG. Don’t know about the main part of DoIT. But where I am at. Mostly contractors working on mainframe. Excuse given is they don’t want to hire for a system that will be phased out. Course tje been phasing them out forever…. Most state workers don’t have near that access. - SilenceDoGood - Wednesday, Jul 10, 19 @ 6:57 pm: @RNUG….you know how this happens? In this case DELL /EMC probably sold DOIT senior management all sorts of pie in the skybideas …like standing up entire environments of storage…but now they can’t deliver on those promises - RNUG - Wednesday, Jul 10, 19 @ 7:01 pm: == Most state workers don’t have near that access. == Yep, about 8 employees and a few engineers had that access level back in the day … and we all had to follow the change control process. - Union thug - Wednesday, Jul 10, 19 @ 7:05 pm: I’d be surprised if that many non contractors had that level of access - RNUG - Wednesday, Jul 10, 19 @ 7:16 pm: == I’d be surprised if that many non contractors had that level of access == We had a dedicated staff just managing storage and customer issues. At total of 10 people but normally 4 people handled it. There were also a couple of DR specialists. Heck, back before the automated tools, I remember manually rebuilding directories bit by bit one time to recover critical data which had no backup. We had to verify and cleanse the data afterward using the transaction logs, but it beat the agency re-entering the entire database from scratch. - IT Guy - Wednesday, Jul 10, 19 @ 7:20 pm: As noted by others, this was self-inflicted by DoIT. They’ll learn from this and put better controls in place. But it’s also a warning sign that the State is dangerously lacking in IT skills and resources. - RNUG - Wednesday, Jul 10, 19 @ 7:25 pm: == They’ll learn from this and put better controls in place. == There USED to be controls in place. - Union thug - Wednesday, Jul 10, 19 @ 7:25 pm: I wana say thing that are not public knowledge yet. So I’ll just say they really broke it and gonna be “days” till it’s fixed. I can say the mainframe people like to mess with them and not tell people ahead of time. Then play dumb when they break it. - Blue Dog Dem - Wednesday, Jul 10, 19 @ 8:05 pm: Darn union operating systems. Sounds like a work slow down to me. - Lack of control - Wednesday, Jul 10, 19 @ 8:24 pm: We had controls and security when I was on the mainframe. The fact that a single person can have alter access to both test and production says it all. Something like this command should require 3 separate isolated authorization. Contractor’s have been taking union IT jobs for years, results in a lack of ownership. - Old COBOL guy - Wednesday, Jul 10, 19 @ 8:41 pm: The Mainframe still rules…..Younger IT people should be kept away until they know what they’re doing. They hate mainframes cause most don’t know anything about them. - RNUG - Wednesday, Jul 10, 19 @ 8:54 pm: == They hate mainframes cause most don’t know anything about them. == They also don’t seem to know anything about manual processes, process control, designing systems, and then testing them. From what I have seen, all they know is fast prototyping with minimal specs and quality control … let alone optimizing them for the specified workloads. Yeah, I sound like a cranky old guy … which I am … but I also know how to gather specs, design, write code in at least 8 mainframe or cross-platform languages, do load testing, and a phased implementation with full documentation and roll-back plans … all of which seems to have been forgotten. - Union thug - Wednesday, Jul 10, 19 @ 9:05 pm: RUNG I agree no ownership with the contractors and to many of them. Also no single person should have been able to break the mainframe that much. Personally I thing some of them like to play around in the system. - Leatherneck - Thursday, Jul 11, 19 @ 4:40 am: Secretary of State computers never went down. (At least not mine while at work). - Another Mainframer - Thursday, Jul 11, 19 @ 6:38 am: I can also confirm the deleted disk volumes incident, became I was called in late that evening to deal with the fallout in my agency. This problem affected several agencies and multiple systems and we’re still trying to get back to normal operations. Biggest problem is that over the last decade or so the bulk of experienced senior mainframe technical staff retired out and either weren’t replaced at all or with very junior staff with insufficient experience. Consider it yet another example of the hollowing out of critical operations. The sad reality is a well staffed and managed mainframe shop with the appropriate configuration change controls in place will have a level of reliability, performance and security that Wintel/Microsoft server technology can still only aspire to. Unfortunately, the management of the recruitment/hiring and procurement processes are so resistant to change in the face of documented deficiencies, that all one hears is whining that they just can’t find quality replacement staff. - It Doesn’t Have to Be This Way - Thursday, Jul 11, 19 @ 9:19 pm: Curious to all who know the behind-the-scenes of this: how is the level of access to K-12 Computer Science education in your neck of the woods in the state? How about anywhere in the state? Sorry, comments for this post are now closed.