Full text search on big databases really is tricky. Of course Lockmart will rip them off at every turn too.
MikeJ |
08.22.08 - 9:25 am | #
Maybe they shouldn't be running it on an Apple IIe.
Jubilation T. Cornpone |
08.22.08 - 9:25 am | #
Queering the database? Don't ask, don't word process?
Doofus |
08.22.08 - 9:25 am | #
My fave story is the pilot who's authorized to carry a gun in the cockpit and fly commercial jets who can't get on a plane someone else is piloting.
megisi |
08.22.08 - 9:26 am | #
And just yesterday Droopy was encouraging sales of Military hardware to Georgia....
plantsman |
08.22.08 - 9:26 am | #
Good grief.
$500 million? I smell a boondoggle.
Billy B |
Homepage |
08.22.08 - 9:26 am | #
what a ripoff. i'da built the fucker for $300 million.
fokowi |
Homepage |
08.22.08 - 9:26 am | #
That's OK, the real one is buried under the Pentagon, and is capable of reading your mind from 200 miles away.
Doc |
08.22.08 - 9:26 am | #
the owl has landed yet again
bill |
08.22.08 - 9:27 am | #
And just yesterday Droopy was encouraging sales of Military hardware to Georgia....
plantsman
Just in time for deer-huntin' season.
Doc |
08.22.08 - 9:27 am | #
More: When tested, the new system failed to find matches for terrorist-suspect names that were spelled slightly different from the name entered into the system, a common challenge when translating names from Arabic to English. It also could not perform basic searches of multiple words connected with terms such as "and" and "or."
You know, Lockheed-Martin isn't an amateur at this stuff. This is not an accident.
Halfdan |
08.22.08 - 9:28 am | #
You know, for $500 million, they could hire a shitload of interns to look stuff up. (If "making it work right in the first place" isn't an option.)
Jubilation T. Cornpone |
08.22.08 - 9:28 am | #
The people like Google who know how to build stuff like this are mostly Democrats, thus it was built by clueless but Republican defense contractors. It's actually just a copy of ACT for Windows 3.1 running on a 133 MHz Pentium.
wtfwjd? |
Homepage |
08.22.08 - 9:31 am | #
Microsoft Office includes Access which is capable of doing keyword searches. If they know any students they could get a copy for around $100.
The Fool |
08.22.08 - 9:35 am | #
Hell I still have the installation disks and codes. I'm wiling to do my patriotic duty and let them pirate a copy of Access from me for free.
The Fool |
08.22.08 - 9:36 am | #
DB search isn't THAT hard. I can do it from scratch and I didn't go to a fancy-ass Computer Science school.
Plus, there are only a million names in it? At my work we deal with millions of records per day...this problem has been solved many times over.
I'm sure NSA has expertise in this area.
What they're talking about sounds like an excel spreadsheet or something.
Monkey |
08.22.08 - 9:36 am | #
...heck, I could 'prolly figure a way to do it on an Apple IIe...sounds like something from the IBM 360 punch-card era....
Don Drennon |
Homepage |
08.22.08 - 9:39 am | #
The Fool wrote, If they know any students they could get a copy for around $100.
Access sucks; it doesn't have a server-client architecture. Major problems with data integrity as a result.
They could go with Oracle or SQL server, but I myself heart PostgreSQL, which is free to boot.
liberal |
08.22.08 - 9:40 am | #
I think the word "program" in this context does not mean a single computer program, it means the whole effort over almost 7 years, including compiling all those names.
At least I hope that is the case.
winner |
08.22.08 - 9:45 am | #
$500, hell that's chump change in terms of Federal dollars. If they'd spent at least 3 bil then the system would have at least partially worked.
Windowdog |
08.22.08 - 9:47 am | #
Boondoggle: noun
• work or activity that is wasteful or pointless but gives the appearance of having value
• a public project of questionable merit that typically involves political patronage and graft:
Chris |
08.22.08 - 9:48 am | #
The good news is when this thing collapses and gets pitched, we get rid of all the false positives it has in it. I doubt you can find terrorists with a database anyway.
Republicans are incapable of building anything. They're good at cheating, lying and stealing, but actually contributing in a positive way to governance? They're absolutely at sea.
jimBOB |
08.22.08 - 9:52 am | #
"I like to pleasure myself with an Atari 2600 joystick while looking at pictures of John McCain"-Rick Warren
jr |
08.22.08 - 9:56 am | #
In related news, 3 by 5 card companies have experienced a boon through this program.
PSoTD |
Homepage |
08.22.08 - 9:58 am | #
You know, Lockheed-Martin isn't an amateur at this stuff. This is not an accident.
Color my hat tinfoil. They really, really want another big terrorist attack to happen, remember. Many republicans come darn close to flat out saying so on TV. That way they can remind everyone why all liberals should be rounded up into camps.
So they're just following the same rule of thumb they always do when governing. Put people in charge of programs who are hostile to the goals of said program. The people in charge of the War on Terror do not want to catch terrorists. Terrorism is good for republican politics. In fact, at this point, its necessary.
David Eoll |
08.22.08 - 10:14 am | #
They could just put the damn thing on a Linux box running PostgreSQL.
liberal
Yep. I know the Defense department required Ada as the programming language of choice not many years ago. Lockheed is probably hobbled by that and the inherently mediocre programmers you get on these kind of government projects (defense/intel agency stuff). Beyond a certain point of complexity, these guys simply cannot cut it.
LittlePig |
Homepage |
08.22.08 - 10:16 am | #
Lockheed Martin is a great company if you want a new fighter. There are better places to go for this.
Gus |
08.22.08 - 10:23 am | #
Lockheed Martin was probably required to purchase the software from a Bush cronie.
wysiwyg |
08.22.08 - 10:29 am | #
i have a google box... i could help them out...
Hubris Sonic |
Homepage |
08.22.08 - 10:37 am | #
"Access sucks; it doesn't have a server-client architecture. Major problems with data integrity as a result."
Non sequitur.
Dude in Princeton |
08.22.08 - 10:42 am | #
HAHAHAHA!
it can't do keyword searches! HAHAHA!
dude, I can barely count my own fingers anmd evenr I could design somethign that does keyword searches.
.
.
..
ok I can't do that either.
brendancalling |
Homepage |
08.22.08 - 10:46 am | #
Dude in Princeton wrote, Non sequitur.
LOL!
If you don't have client-server architecture, you're moving all sorts of data over the network all the time, because every Access "client" has its own DB engine. (The central location holding the files is just a file server.)
If the local client/server app goofs up, the files on the remote file server can get corrupted.
liberal |
08.22.08 - 11:04 am | #
LittlePig wrote, Lockheed is probably hobbled by that and the inherently mediocre programmers you get on these kind of government projects (defense/intel agency stuff).
That's true about Ada, but I doubt that that's a requirement on many projects these days.
But your general point is correct.
Though (related to your point) typically the most disastrous thing on these big government IT projects (well, also things like projects to build ships for the Navy and Coast Guard) is constantly changing specs.
liberal |
08.22.08 - 11:06 am | #
What? Did they use a state-of-the-art dot matrix printer to put the list out?
Dr. Squid |
08.22.08 - 11:07 am | #
And in the HOURS following 9-11 volunteers built a searchable database for families and survivors...
Typical of this administration. Hand the job to inept cronies who then charge $500 million for what any off-the-shelf PC from Office Depot could do for $800 and then wail about "technology challenges."
goldfive |
08.22.08 - 11:35 am | #
Folks isn't this cronyism at its best? Really, Lockheed Martin Corp is the vendor of choice here. Let me guess, this was a no-bid contract.
You certainly wouldn't want the folks at, oh let me just throw out some names here: Google, Yahoo, Microsoft, IBM, Oracle, etc. to have any expertise at throwing a database together with a search engine front end.
Oh My GOD. This is why i blame Nancy Pelosi for not impeaching the entire lot of Bush-Cult-Members. Everyday these folks leave our nation at risk, every single day. Yet that's okay with Nancy.
ScrewBush |
08.22.08 - 11:40 am | #
Don't criticize them because they're not doing eyewash right.
Criticize them because they're doing eyewash at all.
The very idea of a terrorist watch list is eyewash to begin with. Implementing it poorly doesn't subtract from our security because even a sound implementation wouldn't enhance our security. It's a question of garbage in, garbage out. An easily searchable database of info based on a faulty assumption, and gathered with no standards or sound criteria, is no more useful than an unsearchable database of the same crap. And it's crap because the very concept is stupid. We don't live in a Marvel Comics universe, or a 24 universe. There isn't any terrorist command structure out there for us to penetrate and foil. We keep killing the 3d ranking al Qaeda leader over and over again, not because they keep putting someoen else into any real al Qaeda third spot, but because we have no frickin clue what if any organization there actually is to anything reasonably answering to the description al Qaeda, but our leaders definitely find it useful to always be killing al Qaeda's notional third spot.
It's important not to criticize them for the wrong reasons. Yes, if their database is not readily searchable, that does tend to show that they are incompetent. But criticize them over that, and you tend to validate the underlying mistake, that such a database, however well-organized, makes any sense at all. And with that, you let them continue to use a Reichstag Fire even they don't believe in to continue to terrorize the electorate.
Glen Tomkins |
08.22.08 - 11:44 am | #
I don't understand either. Computers are magic.
Mr Blifil |
08.22.08 - 11:46 am | #
Actually I think Glen Tomkins has it right. It's not so much the IT failure, as the failure to properly define the problem, derive a set of requirements for data points needed to solve that problem, data gathering methods and finally, algorithms that crunch that data.
Can you find correlations between what are ostensibly all independent, uncoordinated cells? Maybe. Can you do that via a database search? Not likely. More infiltration and human intelligence is likely the way to discover actionable information, but of course that information may be limited to stopping one cell, vs. piecing together some sort of "nefarious master plan" which likely doesn't exist at all.
And that's the real kicker. Our Western minds want and need there to be a Big Master Plan that explains everything so we can make sense of it. Thus the need to label someone as "Osama's number 3" when such hierarchy may not even exist. It's for our benefit.
AJ |
08.22.08 - 12:04 pm | #
O BTW, will investigating this project and the motives and actions of the people involved with it, be part of the Truth and Recounciliation phase of the "War on Terror"?
Mooser |
08.22.08 - 12:05 pm | #
It's because the $ystem was never designed to really catch terrorists, but 1) to enrich the right companies and 2) to allow monitoring of all political opposition.
They will eavesdrop on Ted Kennedy even in his grave, but those Turkish heroin smugglers with the CIA connections? No way!
kelley b. |
Homepage |
08.22.08 - 12:15 pm | #
prob. written in dBase II.
Cynthia |
Homepage |
08.22.08 - 12:22 pm | #
Data management, storage, and warehousing with business intelligence (fancy name for complex searching, sorting, and reporting on a database) is my company's bread and butter. We do this kind of stuff all the time for HUGE companies. And yes, we do fuzzy matching too.
The google search appliance suggestion is an amusing joke, but it's probably not quite the right solution since it's more of a generic search appliance (albeit a very cool one).
Despiet that, we typically charge between 1 to 3 million for a 3 or 4 year commitment to our technology (which usually includes a heckova lot of web development in addition to data management).
It's amazing how much Lockheed effectively stole from the government to make a system that doesn't, well, work. $500 million?!?! Unbelievable.
Parrotlover77 |
Homepage |
08.22.08 - 12:25 pm | #
Hmm, I wonder what the $500 million dollars was spent on. It couldn't have just been taken by greed-happy corporate officers looking at a no-bid contract from a desperate government, could it? Not in this administration, certainly.
Todd Alcott |
Homepage |
08.22.08 - 12:25 pm | #
Lockheed should be fired.
This thing should be rebuilt from scratch.
It could be done so at a tiny fraction of the price. Give me $5 million and I'll get three guys who worked on databases at NCBI to do it.
"Full text searches on data bases are hard". And yet NCBI has been doing this quickly and with ease while serving the international medical community, 24/7 for the past decade.
The "terrorist watch list" is just a way for Lockheed to make money and for police staters to act like they are serious. And spending more money means that they are even more serious, right??
Whispers |
Homepage |
08.22.08 - 12:39 pm | #
What was the $500 million spent on? Well if, as the WSJ reported, the two major contractors were feuding, and there were 842 contractors from "dozens of companies", a whole shitload of it went to "administration".
Meetings, memos, white papers, conferences, paper and e-messages flying from desk to desk, management node to management node, up the chain and down in a fashion that's the next best thing to random. Formal reporting lines get blind-sided by informal channels. Real information gets harder to come by even as the level of noise in the system increases exponentially. Panicked managers attempt hasty reorganizations, re-prioritizing, re-engineering, re-thinking. Specs keep changing, as someone above noted. Programming begins before the design phase, (perhaps even before the analysis phase) is complete.
More panic. Pressure for results mounts. Perfectly sound development principles get abandoned in the haste to roll out a product. The only answer seems to be: throw more bodies at the problem, giving us that huge swarm of contractors. The bodies (I mean the grunts, not their companies) look at the situation, shrug, mutter "There's never time to do it right, but there's always time to do it over", and toil away. It's a gig, after all. Morale plummets. Middle managers hit the wall and start quitting or going on stress leave. New bodies come in, slow things down while they get up to speed, but since nobody's really in charge, everything stalls.
And all that's just from ordinary incompetence, that happens even when you really, truly, want to accomplish the objective. If there's a hidden objective that's at odds with the stated goal (i.e. deliberately building a system that doesn't work), then all these troubles are compounded.
So that's where most of the half-bil went. And yes, I've seen it from both the inside and the outside.
North of 49 |
08.22.08 - 1:02 pm | #
I don't understand.
It's difficult to describe just how fucked up the federal government's civilian technology systems are (I dunno know about the Pentagon's -- I assume they're somewhat better because of many multibllions more thrown at them, but I don't know if that's actually true)
But, for the rest of the government, 1975-level systems would be an improvement. That's literally true in some cases: The IRS is still using old Sperry-Univac vacuum tube mainframes for most of its data processing. They tried to upgrade about 10 years ago, but that project, too, collapsed in shambles. Ditto the Social Security Administration. Ditto the Postal Service.
It weren't for cheap, ubiquitious desktop PCs that can be wired together to create half-assed semi-integrated networkss, the entire government would have shuddered to a halt by now.
I know, I know: That would be different how? But really, when the Social Security checks stop coming and the IRS starts dunning people for taxes they paid five years ago, maybe people will finally notice.
Peter Principle |
08.22.08 - 2:22 pm | #
Jesus. Even with hightened security you could pull this off for less than a million dollars for the data side.
Client and testing add a million more.
This is just grand larceny. The product I'm working on now is a piece of shit, and we can handle these requirements with out of the box SQL Server.
Parvis |
08.22.08 - 2:23 pm | #
Astonishing! Can you say war profiteering? Isn't that treasonous? Oh, I'm sorry, this is a most favored defense contractor so it's not important.
It seems everytime I hear about a new government computer system, six months or so later the 'junk' stories come out. Having worked with and for these organizations, I can assure you the grunts really want to do a good job but are restrained by management.
Obviously, the contract stated that searching a freakin' database wasn't a requirement. Your average 10th grader knows better. That's how it works in contracting. The entire process is one long cluster f*ck.
numi |
08.22.08 - 3:07 pm | #
"What was the $500 million spent on? Well if, as the WSJ reported, the two major contractors were feuding, and there were 842 contractors from "dozens of companies", a whole shitload of it went to "administration". "
i.e. the project was badly managed and there was no transparency or accountability.
Politicians should be competent managers, not act like they can pull results out of their ass. The Enron execs acted like that. Bush acted like that. And the media is building up Obama to be like that.
Well it's bullshit. It's sound policy which comes as a result of a transparent, accountable process and which is competently implemented that gets things done. Markets don't do things well: they do things profitably. Only regulation makes them do things well consistently. Companies are only transparent and accountable as far as the people, by the implementation of their laws, require them to be. Lying, stealing, bullying, neglect and deceit are all highly profitable if there is no comeback and no-one knows.
me |
08.23.08 - 3:05 pm | #
"But, for the rest of the government, 1975-level systems would be an improvement. That's literally true in some cases: The IRS is still using old Sperry-Univac vacuum tube mainframes for most of its data processing. They tried to upgrade about 10 years ago, but that project, too, collapsed in shambles. Ditto the Social Security Administration. Ditto the Postal Service."
Sorry, bullshit. Can you give a reference for even one of those claims?
me |
08.23.08 - 3:08 pm | #