So those in the know in the cryptography world realize that MD5 is dead for anything other than simple file hashes, and SHA1 is not far behind it. MD5 is broken, and SHA1 is garnering new attacks on a regular basis making it a poorer and poorer choice. They also perform pretty lousy by today's standards.
NIST realized this and had an open competition a while back (which as of this writing is still in Round 3) and the front runners are looking really good, both in performance and security. Being a long-time recipient of the Crypto-Gram Newsletter of one Mr. Bruce Schneier (considered the Chuck Norris of the security world), when he announced that he was part of the team that submitted Skein, which is a flexible hash algorithm with a tweakable-block-cipher at its core, my curiosity peaked.
First, a little primer for those not in the know or who haven't had to work with such matters (consider yourself lucky, this field can make even the most paranoid feel unprepared), what does a Cryptographic Hash Algorithm do and why is it important. *COUGH* Ok, now that we've covered that, why do we need a new one?
Simply put, computers are becoming faster (if not through base mathematical power or speed, then in the ability to do multiple things at once and to hold more things in memory and do more complex things to that memory). The developers are getting smarter. Crypto analysts are also getting smarter. The ways in which the older algorithms used to scramble data are becoming less secure, not because the algorithms changed; once they become standardized quite the opposite, they remain perfectly static save for maybe a patch or two. The way people have looked at the data coming out of the algorithms has changed.
The data and algorithms have been picked over and scrutinized and churned under close watch by mathematicians, cryptographers, and statisticians for many years now. SHA1, for example has been around since 1993. It's 2011 (18 years later). MD5 has been around even longer, and was the basis for SHA1, which improved on MD5 but still suffers from some of the same internal flaws.
With that kind of scrutiny, the cracks and flaws in any crypto system only get wider, not smaller, that's just how it works. So now that the first real government-standard algorithm has aged to its breaking point, it needs a successor.
They did try with the SHA2 family 10 years back (and there's talk of even more variants of the SHA2 family to make them direct drop-ins for SHA1 to speed up the adoption rate). These use more data and change the structure of the algorithm, but that only goes so far, and they still perform pretty slowly for today's needs. That's where SHA-3 comes in.