DoIT needs to get its act together
Thursday, Mar 26, 2020 - Posted by Rich Miller
* On Monday, Gov. JB Pritzker was asked about the crush of people trying to file for unemployment insurance who couldn’t get through…
First of all, this is just an unprecedented number of people that are seeking unemployment claims during this time period. I don’t think we’ve ever seen this before, even in periods during the 2008-2009 crisis. Having said that, we found over the weekend and over the last you know couple days of the week that the systems that we had that were supporting the online applications wasn’t robust enough to take all of the applications that were coming in at one time. And so we’re porting that system entirely over to a much more expansive foundational software system and server. So we won’t have any of those problems going forward. So we’ve taken care of, I think, the online problem.
The phone line problem is another issue. We had to take non essential staff and push them up to the front lines to answer phones to make sure we answered any questions we would like anybody who needs to apply for unemployment though, to go to the website it really will be. It’s now a, you know, as I say it’s on a new platform, we’ll be able to take many more at the same time, we want to fulfill peoples’ needs during this crisis and we’re going to.
He’s right about the unprecedented nature of this crisis. It’s just never happened before. Every state is having these sorts of problems. But he’s the one who raised expectations.
* When a reporter pointed out the ongoing problems with the IDES website and problems getting through by phone, the governor said the reporter was 100 percent right, and added…
As I’ve said, this is an unprecedented number of people who are applying at the same time. And our DoIT, our Department of Innovation Technology, our state CIO, have been on this every day. They’re trying very hard to expand availability, they are expanding the availability. But it is true that we remain overloaded.
People are going to have to be patient at least for now. Over the course of this week those changes are coming online. I can’t guarantee that it’s going to be easy for everybody who gets there, especially if people show up all at the same time during work hours for example. But you can go online, any time of day or night. And so I would suggest to people that perhaps finding off hours to go online to make that filing will be much easier for you and easier on the system.
So, you know, hang with us here we’re going to make changes that are making it better. But it is true, it’s not working the way that I want it to, either.
He needs to seriously kick some DoIT tail and perhaps call in some of his homies from the tech world to help us out.
Yes, nobody ever expected this. I’m not blaming DoIT for its initial failure. I’m faulting DoIT because its honchos told the governor they had fixed the problem when they hadn’t. That’s unforgivable.
- Take a breath - Thursday, Mar 26, 20 @ 10:43 am:
As an aside. Maybe if the Governor’s Office understood the complexity of technology push outs, they’d understand when things are glitchy. Making general demands to push things out without being tested is bad policy and will cause this type of situation. Just saying. And maybe, just maybe, DoIT gives the Governor’s office lip service cause that’s what they want to hear. But bring it, cause it’s the internet and y’all will have a Facebook/hide behind your keyboard response.
- NIU Grad - Thursday, Mar 26, 20 @ 10:46 am:
Some things never change…
DoIT is a buzzword factory of people making big plans about connectivity without actually coming from the IT world or from the respective agencies that have diverse technological needs. They’ve probably been spinning their capabilities to the Gov’s office for weeks about this…
Time to hire even more consultants, I guess…
- Insideguy - Thursday, Mar 26, 20 @ 10:50 am:
People that don’t know how to spell IT shouldn’t have an opinion about it. You people don’t know what or where the failures are but sure like passing judgment. Any system that gets overloaded by an unprecedented situation is not a failure on the people running it. We have contractors and technicians from around the world working to keep up with the current demand. We’re just too busy to tell you all about it. Btw, I have things to do. Giving 30,000 people The ability to work from home when they can’t click a mouse has us a little busy.
- SSL - Thursday, Mar 26, 20 @ 10:51 am:
For anyone impacted this is an absolutely critical need, and they are probably feeling a great deal of anxiety as they try to get through.
Unfortunately there weren’t many idle techies before this crisis started, so building a fix that was never anticipated simply isn’t going to happen that quickly. It’s just not realistic.
The governor is being honest and upfront about the challenges. Keep reinforcing that they are working to address this issue and tell people they remain a top priority.
- Ok - Thursday, Mar 26, 20 @ 10:52 am:
They bought all these fancy servers with millions to run their systems, when they should have just run it on AWS or other cloud service that can (almost instantly) scale up when traffic spikes.
- State Engineer - Thursday, Mar 26, 20 @ 10:53 am:
in defense of DoIT they caught a lot of state employees spun up to work from home in a relatively short time. I wonder how much of their time was used on that.
they do need to get this spun up
- Lefty Righty - Thursday, Mar 26, 20 @ 11:08 am:
Not surprised. Times are unprecedented. J.B. however expects the federal government to run smoothly. Maybe now he’ll tone down his rhetoric since his house is not completely in order. Even before this crisis DoIT was short staffed and well behind. Can only imagine what a cluster it is now.
- Anonymous - Thursday, Mar 26, 20 @ 11:13 am:
Short memories. The Gov’s Office was on notice of problems with this decades-old system over the summer. https://chicago.suntimes.com/metro-state/2019/7/10/20689089/illinois-unemployment-claims-delayed-computer-failure
- Winderweezle - Thursday, Mar 26, 20 @ 11:16 am:
My wife is an IT pro. Her company provides a “work at home” solution and was solicited to bid it.
It was decided that they would not bid it. I don’t think it was even briefly considered. They knew that while their company provides “value add” support to help buy the right product and make it work, they could not compete with companies that would sell the state a general solution, that may o may not be a good solution, deliver it and walk away.
There are many reasons the state fails in IT. One is that they have little control to buy the best product from a company that will become a true partner- they are instead forced to buy from a low bid, low support provider.
Price is what you pay. Value is what you get.
- Anyone Remember - Thursday, Mar 26, 20 @ 11:18 am:
ok -
Don’t want State of Illinois confidential data / files with AWS / the cloud, which is how things were headed under Rauner. And yes, also disagree with the Quinn era decision to mandate agencies put their confidential data / files at the State Data Center, a building that has had convicted felons working in it. (You should see the chain link fence around the ISP server … on the typical data center elevated floor.)
- Mr. K. - Thursday, Mar 26, 20 @ 11:19 am:
DoIT has always, always been behind the curve.
Who still uses Sharepoint 2013 for websites across the state?
Yep.
- Winderweezle - Thursday, Mar 26, 20 @ 11:19 am:
If someone knows, did the state use CDW or a similar type of supplier?
- Mr. K. - Thursday, Mar 26, 20 @ 11:21 am:
BTW –
DoIT refuses to use open-source solutions. Instead, they rely on Microsoft, Adobe, and Oracle.
That says a lot. Especially when open-source is where app and web development is these days.
Disappointed.
- Mr. K. - Thursday, Mar 26, 20 @ 11:23 am:
And nevermind the mess that is — and has always been — DoIT’s data.illinois.gov.
A ghost town.
- Annonin - Thursday, Mar 26, 20 @ 11:29 am:
Let’s not blame DoIt without remembering it is GovJunk program that never did anything right when folks did not need them. We remember the state agency that dealt with the crash of their main frame by sending unusable law tops to serve as replacement.
Hopefully this gets worked out quickly. Anyone who expected more must have been in a coma.
- Reddevil1 - Thursday, Mar 26, 20 @ 11:31 am:
So there is no way to magically snap your fingers and boom system upgraded to unprecedented levels ….it’s not as simple as the generally public thinks …as for moving to the cloud…what happens if the bills not paid and the company holds the data access?…also DOIT has a lot of capabilities….but due to agency staff not having been able to hire developers to upgrade out dated applications
- RNUG - Thursday, Mar 26, 20 @ 11:33 am:
I’m no fan of the extra layer of bureaucracy they add, but I’m actually going to defend DoIT a bit.
At one time, part of my job was forecasting computer workloads like this, and planning equipment acquizations to match. While we always left some headroom for spikes, this truly is unprecedented. About the only way a peak this big could have been handled in the past would have been to reduce service to a bunch of the other State agencies … and for all I know, that may be what they are doing now, diverting resources.
- Rich Miller - Thursday, Mar 26, 20 @ 11:36 am:
===but sure like passing judgment===
1) You screwed up;
2) You lied to the governor about fixing it;
3) Get off my blog and get back to work.
- D - Thursday, Mar 26, 20 @ 12:06 pm:
Inside Guy - Thanks for what you do. I can’t even imagine how crazy things are at DoIT right now. Unprecedented stuff.
- Just Me 2 - Thursday, Mar 26, 20 @ 12:13 pm:
It’s maddening. IDES says my website is expired, but when I try to update it, it wants me to enter my existing password, and then I get the error message again that my password is expired. Been like this all week.
- Mr. K. - Thursday, Mar 26, 20 @ 12:25 pm:
Just me — try doing a ‘Empty Cache and Hard Reload’ in Google Chrome:
- Press F11 to open up dev console (half window at bottom of screen)
- Then right click on ‘refresh’ button (top of browser window).
- Then select ‘Empty Cache and Hard Reload’
This will take a minute or so — but it should clear out any expired certs or other errors that you might be seeing.
BTW — and make sure you’re using Google Chrome or Firefox — not Internet Explorer 11 or MS Edge.
- Annonin - Thursday, Mar 26, 20 @ 12:27 pm:
Hey capt Fax my records suggest today might be an important day. If accurate should we plan a virtual shot in place say right after JB finishes his homily?
- A Jack - Thursday, Mar 26, 20 @ 12:30 pm:
Added levels of bureaucracy have never solved any problems. Before DoIT, agencies kept the state running with their individual IT departments. And when resources were scarce, interagency agreements were available. Blagojevich did his own version of DoIT. It eventually failed as well.
- Pelonski - Thursday, Mar 26, 20 @ 12:42 pm:
Agree with A Jack.
DoIT tries to treat every agency as if they operate the same, but the reality is that many of the agencies have unique needs. Since the agencies are largely stuck with going to them to meet their IT needs, DoIT has no incentive to try to make things better. The result is an outdated and poorly managed State IT infrastructure that is ill-equipped to respond to issues.
- Huh? - Thursday, Mar 26, 20 @ 12:42 pm:
The problem with DoIT is they are trying to handle all of the computing statewide when they don’t understand what the individual departments do or their needs.
- Mr. K. - Thursday, Mar 26, 20 @ 5:19 pm:
On the other side of this, I hope — really, really hope — more agencies say “NO” to DoIT.
- RNUG - Friday, Mar 27, 20 @ 3:22 am:
== On the other side of this, I hope — really, really hope — more agencies say “NO” to DoIT. ==
I don’t think they can.
Back in the late 1960’s, maybe 1968, at State agency was given authority to consolidate all computing under the Governor’s Office through the creation of the Management Information Division of the Department of Finance … which eventually became part of CMS … and now, after about 50 years, DoIT. The consolidation was any off and on priority, gradually absorbing parts of the various agency’s computer operations.
I don’t remember all of the players on the blue ribbon committee that came up with that, but two of them were Jim Norris and Dale Brown. Later on Dale Brown ended up the head of the Legislative Reference Bureau for a number of years. And Jim Norris ended up as a Commissioner on the Springfield City Council.
You have to be pretty old to be able to remember it.
- peanut - Friday, Mar 27, 20 @ 8:12 am:
this is Friday- and I know some that still cant get the system to allow them to set up and apply for unemployment. How much longer is acceptable? People are waiting and counting on this. The longer before they can apply , the longer before they receive their benefits.
- On the way out - Friday, Mar 27, 20 @ 10:00 am:
The real problem here is the reduction in qualified staff in IT positions. The state is currently unable to attract, hire or retain qualified staff. The state has completely failed to address professional development with the little staff there is. DoIT is far short of staffing goals and nobody wants the job. Those that are leaving are allowed to do so without a single care given to knowledge transfer. The state and I mean all agencies and DoIT have completely failed to manage their workforce and this is the result.