Monday, August 21, 2006

Are We Ready?

Are we, punk? Right now Scotland Yard is looking like the varsity squad, while the FBI may be experiencing some equipment issues.

That link really set me off.

During the Katrina Week that is upon us, so that we remember what happens when all levels of government wash their hands and say "it ain't my job" until and after the poo hits the fan, this link is just more of the same. What's wrong with this country today isn't ideology, what's wrong is competency, and no one currently in charge seems willing to step in and move some of this modernization along.

Edit: 10:31am -The Alter link did set me off, but that was more of an op ed piece that linked to this extensive article by the Washington Post. That was the one that really set me off.


dadvocate said...

The marketing research company I work for is heavily technology based. We're constantly upgrading something, networks, PC's, Internet servers, telephones, etc. Of course, as the article points out, we have to make money to survive.

I would find the FBI situation humorous if it weren't so tragic. Since Katrina, I keep a months worth of food and plenty of fuel for my Coleman stove on hand.

It should be quite clear to all that we can't depend on the government whether it's protecting our borders, fighting terrorism, etc.

Dante said...

I missed the comma and thought you asked "Are we punk?" Based on that, I thought this post might be about the Misfits or the Sex Pistils or even the Ramones. Instead its about the government throwing money down a rat hole. And it's a very one-sided article about such an issue.

Since Mr. Alter can't be bothered to talk to both parties, I'll lay down a few tidbits here about what they were trying to do and what could go wrong. Keep in mind that I've not been involved with the FBI project in particular but I am aware of what they were trying to accomplish and how government-contractor relationships work at least at the state level.

First of all, there's nothing wrong with the current featureset of the FBI's current computer setup. "Its rusty mainframes used text-only “green screens”" work just fine for their originally intended purposes. The real issue here is computer imagining. They want to take documents, photos, etc, and scan them in a way that makes them accessible to just about anyone at the FBI.

"Instead of adapting off-the-shelf software, which any computer consultant with half a brain (make that any person with half a brain ) knows is the only way to go, the company decided to design a software system from scratch."

This is where Mr. Alter particularly shows off his ignorance. For small to mid-sized companies this is very true but there is a threshold at which it is more cost effective to implement your own solution than it is to get off the shelf programs to behave together on a large scale. Given the size of the FBI and the need for a very distributed computer architecture, I imagine they far exceed that threshold.

"If you’ve read even one of the 500,000 articles in the popular press about software development, it’s obvious that the first try never works."

Only because it doesn't make news when software does work the first time around. Do you think we'd even have heard of this software program if it weren't for the problems it is having?

"The consultants would, naturally, have us believe that the FBI’s requirements were so complex and security-related that they needed their own system. This is a crock. Many businesses are larger and more complex than the FBI and they, too, have major security considerations. But this is how SAIC ballooned its contract from $14 million annually to $170 million."

What exactly were the FBI's stated requirements? Maybe the FBI stated they needed more than they really did. Maybe they did have outlandish requirements. Maybe it was "a crock" but I'm not very clear on how Mr. Alter knows this. If he is aware of the FBI's original specifications, then he's awfully quiet about how he knows the specs.

"The review team found that three years after 9/11, the system still didn’t allow agents to take their cases into the field on laptops. The program did not have bookmarking or archiving features that are standard in the least expensive commercial software. And the system could not properly sort data."

Were these things even part of the original specs? I don't know that even after reading an article about it.

"Then, when it had clearly failed, the contractor blamed the supervisors within the FBI. Natch."

The thing here that makes me think the contractor might be telling the truth is the plural form of "supervisor." Why are multiple supervisors directing the development team? Being a liason between a contractor and a company or organization is a one man job. Only one person should be in charged of making sure the contractor properly implements the specifications and only one person should be in charge of asking for changes to the original specification once the project is past its planning stages. Sure than one person may have a team to assist them but there should still only be one person in charge. Furthermore, that one person should have the full authority to pull the plug on the project if the contractor isn't living up to their end of the deal. Maybe this did happen but we don't know that because the author of this piece doesn't bother to find out for us.

"Worst of all, the researchers found that the whole system could easily crash and the FBI had no backup plan."

Now that piece in particular is not in the realm of the software development team. Whoever is in charge of disaster recovery at the FBI should've had a plan ready and furthermore should've had any backup features the software would need spelled out explicitly in the original specifications. If the software developer delivered on this, then you really can't blame them. Of course, we don't know anything about the original specifications.

"The sad truth is that accountability in Washington is dead. We won’t get it back until we throw out those responsible. Let’s hope it’s not too late."

You can replace the word "Washington" with "journalism" and be equally correct. There are so many very real stories about government waste. Why do a one-sided hack job to try and make yet another story where it may or may not exist? I'm not saying SAIC is blameless here, but I am saying that I don't know that for sure. And from what I've read in this article, Jonathan Alter doesn't know that for sure either.

Dante said...

"I'll lay down a few tidbits here about what they were trying to do"

After reviewing what I wrote I realized I left the part out about what a pain in the rump implementing an imaging system actually is. The biggest hurdle is that in any imaging solution there are both pictures and text data that need to be accounted for. You could try to kill two birds with one stone by implementing some sort of text recognition but even the best text recognition only has about a 98% accuracy. That sounds pretty good but if you think about it that means one out of every fifty letters/symbols is messed up in some way. That's going to make searching based off that text data a hit or miss proposition at best.

Add that to the fact that good large scale off the shelf imaging solutions simply don't exist and even the best small scale off the shelf solutions leave something to be desired. Imaging is no Holy Grail and there are plenty of companies that have a workable solution in place, but it's not something you can just go buy the box for and install it on your machines.

Patrick Armstrong said...

"Why are multiple supervisors directing the development team? Being a liason between a contractor and a company or organization is a one man job."

This being one of the indicators of bureaucracy not working.

"Whoever is in charge of disaster recovery at the FBI should've had a plan ready"

Doesn't that sum up just about everything wrong with government today?

[Re: accountability] "You can replace the word "Washington" with "journalism" and be equally correct."

I have updated the post re: the actual link I wanted you to read as opposed to the op ed piece.

I guess we can file that under 'blogger accountability.'

Patrick Armstrong said...

"It should be quite clear to all that we can't depend on the government whether it's protecting our borders, fighting terrorism, etc."

I know the government can't do everything, and I know that we must do for ourselves in many instances, but these last few years have just been a new low. We cannot just let ourselves slip into anarchy that comes from government breakdown on the scale we are looking at.

Last night I was watching Spike Lee's When the Levees Broke on HBO last night (review coming in another thread), and one of the things that astounded me was how many folks had to break the rules to get stuff done. For example: one report claims that Coast Guard pilots can only fly for 8 hours a day. That's the regulation. But during Katrina response, someone threw that out and kept helicopters in the air as much as they could. How many lives do you think they saved by saying 'No' to a stupid rule, and applying pressure at the point of attack?

I mean, could you imagine Churchill in 1941 keeping RAF planes on the ground because of regulations?

No, you can't. Because Churchill was a leader, and knew that when the crap hit the fan, leaders must emerge.

Dante said...

The new article is much better than the oped but it's still a little light on what specifications were asked for and what was delivered in the VCF system. It is important (and largely glossed over in the oped) that the failure rate of VCF was high. It's one thing to deliver a nebulous product based on nebulous design specs. It's another entirely to not deliver on the concrete specs you do have. I do wonder what exactly constitutes an error for the error rate reporting. I'm assuming that to qualify as an error, implemented functionality must not work but sometimes user error has a nasty tendency to mess up those numbers.

The other thing the oped left out that was important is that VCF had an organization-wide simultaneous rollout. This is almost never a good thing. It's generally always worth the extra time and effort to design a system that can be run simultaneously with the old system so an incremental rollout is possible.

"But Azmi and other FBI officials say Sentinel is designed to be everything VCF was not, with specific requirements, regular milestones and aggressive oversight."

That would be a good start. If they have reasonable requirements and don't change those requirents in any significant way during production, they'll probably get it right this time around. As with any high turnover organization, the real test will be how well the project fares a change in management. New management always likes to make changes and fervently tries to fix the system when first taking control. If that new management can keep their hands off this high profile project and its specifications, then there is hope.

I do wonder how aggreive the oversight will be and whether or not that will do more harm than good. The original project seemed to suffer from having too many chiefs and not enough braves. An overly-aggressive oversight may compound that issue.

"one of the things that astounded me was how many folks had to break the rules to get stuff done."

That reminds me of the Loma Prieta earthquake in California back in the late 80's. They repaired the SF-Oakland bridge in a month by cutting through a lot of the red tape involved. After everyone saw this, they started wondering aloud why other roads couldn't be built/repaired as quickly. Criticism of road crews and their work ethic skyrocketed during this time. How many lazy road crew jokes did you hear from 1988 to about 1994. This has led to some pretty big changes in the way road construction works across the country, including the prevalence of nighttime crews in heavily populated areas and a reduction in paperwork on repair and rebuilding of pre-existing roads.

dadvocate said...

Patrick - I agree with your assessment that something needs to be done and the government needs to be held accountable. Disaster relief capability and Homeland Security is a joke. We continue to be in a situation where thousands are fighting in Iraq, etc. but crossing the southern borders into our country is ridicously easy.

But, also, I'm doing everything I reasonably can to have family protected and prepared. I don't live on the coast so hurricanes (or their radios) aren't a problem. Tornadoes, earthquakes, floods, influenza, terrorism could be.

petallic said...

Wow. It sounds like my place of employment is more technologically-savvy than the FBI. Not bad for a Title I Georgia high school with only 1300 students. I just might take my techies some doughnuts tomorrow.

petallic said...

Ooh, and we have a retired Marine colonel on staff who just finished his doctorate in Homeland Security. He coordinates our Emergency Response plans. I'd say we're 2 for 2.

Dadvocate, you could always come stay with us. The chicken nuggets aren't that bad.

petallic said...

2 slight amendments to my former post:

He's retired from The Air Force, not the Marines.

And the doctorate is in Health Sciences specializing in quantitative research of all hazard emergency planning, capabilities, performance and evaluation of public school systems.

My bad. Will fact-check in the future.