Welcome to this issue of Dr. Dobb's Data Compression Newsletter! This newsletter keeps you up to date with news items, product information, and the occasional editorial aside about the world of Data Compression. Your input is always welcome in the form of comments, submissions, or original material.
Windows Media Player Puts Out Welcome Mat for Crackers
May 15, 2003
This morning when I booted my system at work I saw the familiar Windows Update icon in my system tray. This tells me that Microsoft has yet another urgent download they recommend I install. Figuring that Bill's boys know best, I just about always accept the update, and today's was no exception.
But I couldn't help notice that the security breach of the day was pinned on Windows Media Player 7.1. Turns out that a malicious operator could use the skinning interface of the player to place executable code on my system without my knowledge. No doubt a bad thing.
Microsoft of course owned up to the problem, issued a security alert, and spread the news through all the traditional outlets, allowing the common folk such as you and I to quickly patch up. I've done so, and I hope you have as well.
Yahoo! news article
Microsoft's mea culpa
DataCompression.info Addition: Compression Researchers
May 15, 2003
Michael Dinolfo submitted this link to DataCompression.info a few days back, and it looks like a pretty nice list. If I'm not mistaken, Google uses the same database as dmoz.org, the Open Directory project, but it appears Google dresses it up a bit with their page ranking system.
Of course, the Google page ranking system isn't all it's cracked up to be. The highest ranking on this page belongs to yours truly, which is somewhat bogus, as I am not a compression researcher, I just happen to have lots of links to my page. But it does appear to me that most of the folks listed on this page are the real cheese of the compression pizza, and it might be worth your time to visit their pages from time to time.
Krakow: AAC Sucks
May 14, 2003
Gary Krakow, writing for MSNBC, has a major problem with the AAC format used by Apple's new iTunes music service. Basically, Gary thinks that the sound quality is not so hot. And considering that he paid roughly the same for his iTunes downloads as he would have for the CD at BestBuy, and has to contend with DRM as well, he's not thrilled.
Judging the quality of audio codecs is a nasty problem. We have to forget about objective testing -- modern high-performance audio codecs model their compression on the perceptual qualities of the human ear. How well or how poorly they work has a lot to do with their ability to capture the processes that take place inside our ears and brains when we listen to music.
To complicate matters, different codecs also show big differences in their ability to handle different types of music. Compressing dance music with a driving bass line is quite a bit different from processing classic ABBA tunes. What works for one might not work so well for another.
So right now we're stuck with caveat emptor. If at all possible, sample before you buy, and see how it sounds to you. That's all that's really important. Better yet, only listen to music in your car or on airplanes -- the heavy duty background noise will level the playing field and save you a lot of money!
GraphicsMagick 1.0.0 Ships
May 14, 2003
I mentioned back in April that the GraphicsMagick image library was shipping its 1.0 beta. Things must have gone pretty well, because the 1.0.0 release is shipping less than a month later. You can get a few more details on the library by following the above link to the April archives, then paging down to the 25th.
Video Codec Comparison
May 13, 2003
Nothing we love more than subjective comparisons of audio and video codecs! And here's a good one on the doom9.net Web site, which pulls in all the high-performance candidates, including Windows Media 9, RealNetworks, various forms of MPEG-4, and a few others.
The test was done against a few good samples: The Matrix, Saving Private Ryan, and Futurama Season 2. These films provide a good range of input data designed to stress any codec.
I won't give away the ending, but this is a really detailed review that includes copious data, comments, and captures designed to illustrate various flaws. Much to learn from the results. And if you like to kibitz, follow the links to the Slashdot page and see what the hoi polloi have to say about it.
Doom9.net video codec review
Slashdot discussion of the article
60G Nomad Dissected
May 13, 2003
Just a few days back I ran an article on the new 60GB Nomad Zen MP3 player from Creative. I'm still pretty excited about this device, (and would love a review unit from Creative, is anyone listening in Singapore?) and with luck there may be a few in the hands of the public by now.
The folks at Slashdot are haggling about whether this is a good purchase or not, and you may find the conversation illuminating. Or not: much of it seems to be nothing more than chest-thumping and brand-loyalty. Yes, I'm absolutely convinced that you won't let go of your iPod until they pry it from your cold, dead hands.
RealNetworks Streaming to Cell Phones
May 12, 2003
RealNetworks made a big splash with the news that they were going to start streaming info to cell phone users on 3G phone systems.
The theory is that users are going to get these streams for free from RealNetworks, and then Real will get a kickback for connect-time charges. Lots of skepticism out there, as it's a completely untested service model. Mabye we're going to find out that people really like seeing Derek Jeter field hot grounders on their Nokia phone. Or not.
Yahoo! story on RealNetworks announcement
SiliconValley.com store on RealNetworks announcement
Compress Your PHP Pages
May 12, 2003
Lately I seem to have written a lot of stories about compression of Web pages. One of the important facts here is that all modern browsers are capable of handling gzip compression of Web pages. So if you can provide your pages in this format, you're going to get the benefit of some pretty good compression.
It turns out that this is quite easy to do for PHP4 pages, which represents a pretty large slice of the Web. This article on The Learning Journal gives you the details on how to make it happen.
Learning Journal article on compression under PHP4
Microsoft To Set Hook On High Definition Enthusiasts
May 11, 2003
A big part of Microsoft's strategy for Total World Domination requires that all transmission of High Definition video of any sort be done in Microsoft's proprietary format.
This idea isn't completely out in left field. If DVD manufacturers support Windows Media 9, we could have HD videos of feature films shipping today, a fact which will be demonstrated when the new Terminator 2 DVD ships this summer.
But just because something is technically possible doesn't mean it's going to happen. Bill knows that getting the hearts and minds of the consumers of video entertainment on his side might help with the manufacturers of DVD players and settop boxes. So as part of a PR blitz that has been rolling for a few months already, Microsoft is going to give you a demo DVD that contains a big batch of High Definition content.
If things go as planned, you're going to love watching the three shorts from the BMW Film Series, and you'll be ready to watch T2 in WM9. If that goes well, Bill can convince the Hollywood studios to start including HD versions of all their films on DVDs, and then, well, you get the idea.
Redmond Press Release
Microsoft's Do Amazing Things order page
Java ID3 Tag Library
May 11, 2003
This project supports a nice Java library to read and write ID3 tags from MP3 files. It supports what appears to me to be every version of ID3 tags, plus the Lyrics3 1 and 2 formats, which aren't as widely used.
Does this look like something you might like to use? Then now is the perfect time to join in. The library just moved out of alpha into beta, so get in, try it out, and get involved with the process of refining it into something really good!
Java ID3 Tag Library home page
Conversations With Unisys
May 10, 2003
It's not always easy to communicate with a big corporation. And unfortunately, the LZW data compression patent is owned by Unisys, which indeed seems to be a big company.
A recent poster on comp.compression, Mitch Grunes, wanted to find out if Unisys was after royalties for the use of freeware or software he developed on his own, should said programs use GIF or TIFF file formats. The link below shows that the answer is most decidedly yes, they do expect you to have a license for anything and everything that uses LZW compression.
Mitch Grunes transcript of his dealings with Unisys
CXImage
May 10, 2003
CXImage is a free Win32 library that you can download from the CodeProject. It supports a really nice variety of compressed image formats, and has a pretty good list of how-to sample code you can cut and paste into your project.
So many free software packages get off to a great start and then stagnate, but CXImage doesn't seem to be going this way. Davide Pizzolato cranks out a new release with bug fixes and new features every month or so. The latest adds JPEG2000 support, along with a batch of nice smaller features.
CXImage page on The CodeProject
RealNetworks Losing Money, Making It Up In Volume
May 9, 2003
RealNetworks, being a public company, has to post those pesky quarterly results just like clockwork. Turns out that for the first quarter of 2003 they lost $2.8 million.
Bad news? Maybe so, but they're happy to also announce that they've signed their millionth subscriber for the online entertainment service. So presumably we're in the stage where we are sacrificing profit for growth, ala Jeff Bezos at Amazon.com.
Reading between the lines, it sounds as though Real is not too excited about the growth prospects for their server software, and instead is betting on content delivery for all future growth. Good luck, Real!
LZOP Waits Five Years, Ships 1.01!
May 9, 2003
LZOP is a GPLed library that is designed for very high-performance data compression. Markus F.X.J. Oberhumer wrote this app using his LZO library, which is lean, mean, and reputedly quite fast.
Over the years I've seen quite a lot of interest in LZO, particularly from those working in embedded systems. Unfortunately, many folks are daunted by the fact that the library is released under GPL, which means it will spread the GPL to systems that it is linked into. Most commercial concerns feel this is a bad thing.
In any case, LZOP sat quietly for five years on release 1.0, apparently not giving anybody any trouble. I was quite suprised to see a 1.01 release of this program show up on the radar last week. It appears that the biggest change is that Markus has redone the build system -- other than that just a few bug fixes.
Check back here in five more years and we'll see what Markus unveils for release 1.02.
iTunes DRM: The Downside
May 8, 2003
In the midst of the nearly universal press praise of Apple's iTunes music service, The Register dares point out that the rollout is not without a problem or two, and wonders if the long term is going to be as rosy as the first week.
I guess the biggest problem with Apple's DRM is the lack of backup protection. It's as simple as this: Trash your hard drive, and your collection is lost. Despite the fact that Apple has your name, a record of all transactions, and a guarantee that only you can play the downloaded music, they're going to insist that you purchase new copies of every song you acquired through iTunes.
Another potential glitch awaits those who migrate their tunes to a new computer. Be very sure you go through the deauthorize/authorize sequence before you discard the old Mac! Failure to do so may leave you in tears.
Personally, I suspect that most of the enthusiastic users of iTunes are blissfully unaware of the limitations placed on their purchased tunes. One of my chums at work told me he was quite pleased with the two albums he purchased on opening day. When I asked him about DRM, he told me there was none. "I'm able to copy the files anywhere I please," was his comment. I didn't have the heart to suggest he might want to try playing them anywhere he liked.
The Register looks at Apple's DRM
Nomad 60G: An iPod Killer?
May 8, 2003
You have to give Creative their props for staying alive in the dog-eat-dog world of PC peripherals. Since their birth in the gum-free country of Singapore, they've managed to hang in there through massive shifts in hardware and software markets. Back in the 80s we were putting together systems using Hercules graphics cards, Hayes modems, and Creative SoundBlaster cards. Of those three and many other early innovators, few survived, and Creative is one of them. (If that doesn't get me a Nomad demo unit, what will?)
In any case, Creative is going head to head with Apple's iPod juggernaut with the release of their new 60G Nomad Jukebox Zen media player. This unit has twice as much storage as Apple's new top iPod, features longer battery life, support for MP3 and Windows Media format files, a USB 2.0 connection, internal Mic, FM receiver, and retails for $399, a hefty $100 discount from the equivalent iPod. Best of all, it has an optional wireless remote.
I think that Creative could have a real winner with this product, assuming the initial rollout goes well. I'll hold off on any strong predictions though -- just a few glitches can really ruin word of mouth on cutting-edge consumer gadgets, and there are any number of things that can go wrong in this space: long boot times, erratic battery performance, crummy host software, firmware glitches, you name it. We'll see how the next few months treat this new Nomad Zen.
Slashdot on AAC
May 7, 2003
Apple is getting a ton of press for their iTunes music service. Never mind that Rhapsody, PressPlay, et. al. have been doing the same thing for the better part of a year, and have far better catalogs. When Steve does something, we pay attention.
Anyway, in order to implement DRM, Apple chose to use AAC as their format for distribution of music under iTunes. AAC is the audio layer from MPEG-4, and is undoubtedly much better than MP3 at the same bit rates. But how much better? And more importantly, how does it sound at the bit rates being used by Apple?
For the answer to these questions, and many more, we turn to the audio mavens at Slashdot, and eavesdrop on their sophisticated analysis.
Slashdot thread on AAC vs. MP3 vs. Ogg
Source Coding Matlab Toolbox
May 7, 2003
John Kieffer from the University of Minnesota (home of the Golden Gophers) has posted a nice library of Matlab code to be used for data compression. Matlab is a really useful platform for doing experimentation, and is probably a far more productive use of your time than writing C++ code. Check out John's code and see if you find yourself getting the whole week's work done by Monday afternoon.
John apparently uses this Matlab source in a couple of different courses:
EE 5581: Information Theory and Coding
EE 5585: Data Compressoin
Information Theory Research Group page
Linus on DRM
May 6, 2003
Digital Rights Management. Like it or not, it's coming to a computer near you soon. This latter-day version of copy protection is essential for creation of an active market in digital media. Without copy protection, the record moguls aren't going to be willing to let you buy MP3s online, and the movie studios are certainly not going to let you download The Matrix Reloaded.
So apparently there is some serious debate within the Linux community as to whether DRM has any place in upcoming releases of the kernel. You can see why there might be a bit of a philosophical problem here. DRM represents serious commercialization, and Linux is decidedly averse to excessive commercialization. And the notion of freely copying is a big plus to Linux users, but a big minus in the world of DRM.
So where am I going with all this? It turns out that Linus Torvalds, the honorary chairman of all things Linux, recently came out in public with a decidedly pro-DRM statement! Actually, by pro-DRM, what I mean to say is that he refused to take an anti-DRM statement. Linus basically said that Linux should be agnostic towards DRM.
Finally, I should add that Linus was speaking specifically about signed images, which is not exactly the same thing as supporting DRM for digital media, but I think we're definitely in the same ballpark.
Slashdot thread on Linus and DRM
Original post from Linus -- follow the thread for responses
CDex 1.50 release
May 6, 2003
CDex is a free CD ripping program for Win32. This has little to do with data compression, but I happily used (and continue to use) CDex to rip my entire CD collection to my local server, where I enjoy it daily.
CDex does everything I want. It reads the audio data from my CDs and converts it to MP3 on the fly. It looks up my CD track info from CDDB and writes the data into ID3 tags. It handles various encoding options in an easy manner. And it's completely free.
According to the CDex home page, CDex is the number one download at SourceForge, and that's no surprise to me. Worth having on your system!
Sony RoomLink Not Ready For Prime Time
May 5, 2003
Yes, there are quite a few companies out there that would really love to make a bundle off this whole notion of digital convergence. Sony is just the latest with their RoomLink product.
The RoomLink box hooks up to your TV and uses a standard network connection to get back to your PC. It then is supposed to let you play music and view photos on your TV, and more importantly, act as a Digital Video Recorder, performing TiVo-like functions with the help of your PC's hard drive.
The folks at the SilconValley.com got their hands on one of these units and didn't have much in the way of positive comments. First among their complaints was that Sony, as usual, attempted to toe the company line by insisting that RoomLink only work with Sony-branded hardware and software. Things went downhill from there.
Kalashnikov MP3 Player
May 5, 2003
Admit it, you've long been wishing that you had an MP3 player that fit in a standard AK-47 magazine. Once you were off the front lines, you could remove your live ammo and plug in your player, at which point your rifle could feed you a steady diet of compressed tunes.
With 20GB of storage and a 10 hour battery life, it looks like AudioBooksForFree.com can make your wish come true. If you're willing to plunk down $500, you can get the device with 200 audio books preloaded. That's a lot of literature for the buck, and it's nice to see a great peacetime use for what is undboubtedly one of the world's most popular automatic rifles.
AudioBooksForFree.com Web site
DataCompression.info Addition: FreeVo
May 5, 2003
This is an open source Digital Video Recorder that aims to turn your Linux system into a TiVo equivalent. This is an ambitious project which has already built up a nice feature list. A huge wish list of features-in-waiting is ready for any developers willing to step into the breach.
If you step back into the April archives of this news feed, you'll find an article on ExtremeTech.com that goes through the basics of setting up a home DVR based on Freevo.
Please Don't Feed the Troll
May 4, 2003
The term trolling on Internet newsgroups refers to the practice of deliberately posting provocative statements on a newsgroup to get a reaction out of the locals.
A fellow named Tim Bernard was recently accused of trolling on comp.compression owing to some very provocative posts:
- Test results ultimate compression
- Help license my code and get a % !
- Anyone here work for a compression company?
- Software Demo 100 Bytes Down to 7 Bytes Losslessly
- Any 100 bytes down to 7 Bytes Losselssly!
- New Improvements to Ultimate Compression
Anyway, Bernard is naturally castigated as a troll, but by my reading, he's just another guy who has deluded himself into thinking he has captured lightning in a bottle. I truly believe that he thinks he is on to something, and will make a fortune if he can just get the world to believe in him. Worth a read!
C Library to Search Over Compressed Texts
May 3, 2003
One question I receive quite frequently: "Mark, how do I develop routines to search through my compressed data?" Until now, the answer was limited to just one choice, zgrep, which is a fairly poor compromise.
Last week I received an email from Paolo Ferragina at the University of Pisa describing a set of code that he and some fellow researchers had developed: the Huffword Library and the CGrep Library.
Their first library is a compressor/decompressor package, but it contains some secret sauce that helps enable the second package to efficiently sort through compressed data.
I'd really be interested to see if this code could be adapted to create what seems to be the holy grail for people who send me the previously mentioned question: the compressed database. Even if this isn't possible, being able to perform simple text searches on compressed data is a valuable piece of work.
Project page at University of Pisa
DVD Boy Player 2.1 Ships
May 3, 2003
I love product names that sound like the output from a not-so-good translation program. How else could you explain DVD Boy Player? My first thought was "Where do you insert the boy?"
In any case, this is your basic Windows media player. DVD Boy Player plays DVDs as well as SVCD, DivX, and MPEG-4 files. Best of all, this page says that it has "a new arithmetic and sound effect database." What's that supposed to mean?
HiShare Inc. and GhostLab Web site
ABC Source Code Released
May 2, 2003
ABC is a free data compression program written by Jürgen Abel and based on the Burrows-Wheeler transformation. Jürgen has finally made the source code available to the public with version 2.4. The Delphi source is available under the Jürgen's own Abel Public License.
Jürgen advertises ABC as achieving 2.238 bits per character on the Calgary Corpus, which is quite respectable. This puts it roughly on a par with Bzip2, and of course quite a bit ahead of LZSS/HUF programs such as WinZip.
ABC page on data-compression.info
Random Numbers
May 2, 2003
Random numbers are useful in data compression, if for nothing else than for debunking claims of incredible compressors. But one big problem is that random numbers generated using functions such as those in the C standard library are not really random.
In point of fact, if you know how a random number is actually being generated, it is anything but random! And it is well-known that many popular random number generators suffer from some bad deficiencies.
Somebody at Sunny Beach Technologies has addressed this with the first professional product I've seen that uses your sound card to generate random numbers. It apparently does this by picking bits out of a white noise stream coming out of your sound card! This may qualify as truly random, but it would be nice to see some histograms and other tests.
Sunny Beach Technologies product page
Digital Media Units of SonicBlue Sold
May 1, 2003
SonicBlue's recent bankruptcy proceedings resulted in the auction of their business units that sell Replay TV and Rio MP3 players. The purchaser was Japanese consumer electronics company D&M Holdings, which you may know as the manufacturer of Denon and Marantz audio and video equipment.
Both units were only recently purchased by SonicBlue. Rio was original owned by Diamond Multimedia, and was sold to SonicBlue in 1999. ReplayTV was purchased in 2001.
It's tough to make a living in the cutthroat world of consumer electronics, particularly when you're trying to sell into the undifferentiated sweet spot of broadest popularity.
New Listing: HTML2Zip 2.0
May 1, 2003
This product compresses your Web pages using the gzip format, which is readable by virtually every modern browser. Compressing the pages this way can do a lot to improve your page loading times.
The Web page makes a nice list of comparison points describing why using gzip compression is so much better than the javascript-based algorithms that others are selling. All this is true, but it's also true that you can accomplish the same thing quite easily without the help of this program!
A Few Questions for Igor Pavlov
April 30, 2003
Igor Pavlov is the man behind 7-Zip, a compressor that features excellent performance by using the latest lossless compression algorithms. Igor was nice enough to answer a few questions about 7-Zip and his work.
Mark Nelson: There are plenty of archiving programs out there. What made you decide to create another one?
Igor Pavlov: I created my first compressing program when there were not
so many archivers. And it was really interesting for me to create my own
program that was better than existing programs. First time it seemed like
magic, that files become smaller. And when you create compression programs,
you always can fairly judge your work with objective marks: compression
ratio and decompression/compression speed. So you always know whether your
algorithm is good.
MN: Do you feel like 7-Zip is at the top of the heap when it
comes to compressed file size? If not, who is your most serious
competition?
IP: We must differentiate between popularity and performance.
There are some archivers with good performance, but they are not popular.
I think LZMA compression algorithm (which is main method in 7-Zip)
has pretty good results among all archivers, if we consider overall ranking
by decompression speed / compression ratio. The main competitor for
7-Zip now is WinRAR.
MN: What formats are you evaluating to include in future releases?
IP: I plan to add new methods for multimedia compressing to 7-Zip.
MN: Are you getting any help from other programmers? Any financial help to speak of?
IP: 7-Zip uses some open source libraries from other programmers.
For example, BZip2 and PPMd.
MN: Have you ever considered a commercial product based on 7-Zip?
IP: Some time ago 7-Zip was a shareware program. But now it is free
software with open source. There are some commercial products that
use 7-Zip's code, but I plan to keep 7-Zip free and open-source.
Are We Ready for Video-To-Go?
April 30, 2003
As you know, Microsoft has been pitching their Media2Go reference design as a possible next-big-thing for the consumer electronics world. Bill Gates has been really trying to pump up the notion at various industry gatherings.
Well, I like the idea of a Walkman-sized video player with a hefty hard drive. I don't know if I'd watch video on it or not, but there would be plenty of room for music, still photographs, and text. But maybe I'm not a good representative of the market.
David Morgenstern of ZiffDavis talks about this in a recent article on the Ziff Davis Media StorageSupersite. He doesn't come right out and say it, but clearly Morgenstern thinks the Media2Go platform is at most half-baked. See if you agree with his logic.
ZiffDavis StorageSupersite article
Video on GameBoy Advance
April 30, 2003
In the above article, David Morgenstern compares the small screen on the Media2Go reference platform with something he considers to be a lot nicer: the average notebook computer.
Well if he doesn't like Media2Go, he's really going to dislike this latest news. It turns out that a company named Pocket PC Films has created a plugin that is going to let you watch video on the teeny tiny GameBoy Advance screen. Of course, when something like this happens, we all ooh and ah even if the end result is not all that hot.
We're not going to see this new cartridge until E3 in Los Angeles in May, but word is right now that it will play 10 fps in 15 bit color using unspecified video compression. It will also reportedly play MP3 files. Yes, I want one.
Photo Archive Converted to JPEG 2000
April 29, 2003
EE Times reports that Fratelli Alinari, owners of the world's largest photo archive, will be converting all of its stock photos to the JPEG 2000 image format. Compression firm Algo Vision LuraTech is providing the expertise (and presumably the software) to make this conversion happen.
This conversion will undoubtedly lend some legitimacy to the format, which is acknowledged as being technically superior in every way to old-school JPEG, but is not yet taking the world by storm. Those of you who are interested in checking on the conversion might want to check the corporate Web site in the future. The article says that
Fratelli Alinari will add the new image files to its Web site for demonstration and marketing purposesbut I don't see anything there yet. This could be the typical case of the press release beating reality to the punch.
DataCompression.info Addition: CoolPlayer
April 29, 2003
CoolPlayer is a nice, fast MP3 player targeted to Windows. This Open Source project is a nice piece of work, and the team has created a player that has the features to compete with good commercial players.
The team admits that they're using old-school C Programming to create this thing, which is not the easiest way to code today, but it makes for a nice fast, compact player. Why not download a copy and give it a try?
Easy Way to Read/Write Zip-Compatible Files Under MFC
April 29, 2003
Larry Leonard has created this CodeProject article with the goal of making it super easy to read and write Zip files from your MFC project. There are some limitations to this project, such as a limit of one file per archive, no disk spanning, and no encryption, but he has definitely made it easy to use. Three lines of code and you're done!
Good Times Ahead for MP3 Players
April 28, 2003
Market Research firm Instat/MDR analyzed the outlook over the next five years for digital audio players. They predict a compound annual growth rate of 39.7%, enough to put a smile on the face of any manufacturer, no doubt even Steve Jobs.
At that crisp rate, total sales per year are expected to rise from 6.8M in 2002 to 36M in 2007. The highest growth rate is expected to be found in devices that ship with hard drives. The entire market is going to be driven by expanding capacities and cheaper components, but apparently the capacity afforded by a rotating disk is going to be a big sell.
Jesper Larsson Returns!
April 28, 2003
Jesper Larsson spent a fair amount of his years in academia studying Suffix Trees. He published some nice papers during that time, and I saw quite a few click-throughs to his papers from DataCompression.info users.
As is often the case, Jesper had to leave the friendly confines of his university and apply himself in the real world. Soon after this happened, Jesper's Web page disappeared, much to my dismay.
I'm happy to report that Jesper has now found on new home for his page, and has spent enough time to make sure his papers are back on line. Follow the links and you will be up to speed on Suffix Trees in no time.
CodeProject: Compress Data
April 28, 2003
A new article on The CodeProject describing code to compress/decompress to/from an ISequentialStream interface. Code is supplied to implement this for Cfile and CByteArray. The compression itself is done via zlib. The rationale for this project is that the author needed cookbook code that worked with MFC objects.
CodeProject Archiver -- Targets Files and/or Win32 Resources
April 26, 2003
This CodeProject article presents an archiver that moves files in and out of an archive, and will extract from resources as well. It doesn't support the standard Zip format, and in a blinding flash of frankness, the author says The code is crap but it works and I couldn't find it done anywhere else.
Andrei Nikolaevich Kolmogorov
April 25, 2003
We celebrate the 100th anniversary of the birth of Andrei Nikolaevich Kolmogorov today. Kolmogorov developed many of the fundamental concepts of complexity, which provides the mathematical foundations to answer some very basic questions that are critical in Information Theory.
In a nutshell, classical Information Theory doesn't really give an absolute answer to the question of how compressible a stream of data is. We can only speculate on how compressible it is with respect to a given model. So the notion of compressibility is very wishy-washy. A file might be completely incompressible with one program but highly compressible with another. This state of affairs is unsatisfying, because we intuitively feel that there are fundamental differences between files filled with white noise and others filled with something like English text.
Kolmogorov lets us address the problem in a more general way, by determing what the absolute complexity of a given data stream is. In a huge oversimplification, I can sum up this idea by saying that the complexity of a given stream is given by the size of the smallest program that can create that stream.
To celebrate the centennial of Kolmogorov, a conference is being held in Moscow this June, which will concentrate on work in the areas he helped define. If you're interested in complexity, probability, topology, or any of the other areas his genius touched, you might want to drop in.
The Andrei Nikolaevich Kolmogorov Centennial site
Fine-tuning Mplayer
April 25, 2003
You've probably seen a mention or two of the MPlayer media player in this space before now. It's a well respected renderer of audio and video that runs on multiple platforms, although I suspect most users are employing it to watch video under Linux.
Getting MPlayer to do a good job on Linux isn't always easy. Contending with video and audio drivers, getting those Windows DLLs loaded, etc., can be quite a task. Fortunately, A'rpi, the helmsman of MPlayer, has written a tutorial that is published on the Freshmeat Web site. He'll walk you through some of the major issues, point you to various resources, and hopefully provide you with enough information to make you a happy MPlayer user.
GraphicsMagick 1.0 Beta
April 25, 2003
GraphicsMagic is a branch off of the ImageMagick project. I suppose there are some interesting political machinations behind all that, but for right now you should just know that GraphicsMagick is attempting to provide a stable set of code that can be used in other free and proprietary products.
The current release of GraphicsMagic consists of a big batch of image processing library functions, as well as a command line tool that lets you easily exercise some of those functions. Functions include the mundane, such as resizing, resampling, characterizing, and move up to the more interesting, such as creating thumbnails, compositing images, and more.
http://www.graphicsmagick.org/
Windows Media 9 to rule HDVD World?
April 24, 2003
Using standard MPEG-2 encoding technology, it's simply not possible to encode High Definition versions of feature films on current DVDs. The industry is currently wrangling about how to deal with this. Possible solutions include a conversion to MPEG-4 encoding or adoption of higher-capacity blue laser systems.
All this discussion will be moot if Microsoft has its way, however. The video codecs used in Windows Media 9 are probably going to be able to use existing DVDs as storage for High Definition movies. And we're all going to get a chance to see this when a new edition of Terminator 2 is re-released with an HD version of the movie compressed using Microsoft technology. Naturally, you won't be able to watch this on TV using your standard DVD player, but a properly equippped PC will do the job.
I guess that the Redmond Army is figuring that if enough people see a beautifully encoded version of the film on their PCs, it will provide enough of a proof-of-concept to justify a blitz on DVD manufacturers. In fact, Microsoft might just decide to sidestep the DVD consortium and start convincing individual player manufacturers to support this non-standard format. What do they have to lose?
Cringely Weighs In
April 24, 2003
I love reading Robert X. Cringely's column—I just wish it was a daily instead of a weekly. This week Robert got started talking about his encounter with a pet psychic, which didn't really seem germane to any of the topics that I'm usually interested in.
But fortunately, after a couple of hundred words Robert started in on TiVo, streaming media, and how Microsoft intends to rule the world with Windows Media 9 in every DVD player in the world. Now you've heard it from me, and you've heard it from Robert, it's as good as certain.
Robert's column also mentions some nice new features in the latest versions of the TiVo box. Their new network capability means the TiVo box can stream MP3 files from your home server to your TV. If you dropped a bunch of money on a nice TV, there's a good chance it has decent sounding speakers, which means it might not make a bad player for all those MP3 files you've got stashed on the server.
But of course, what everyone is really going to be interested in is the ability to store streamed audio on the server, and just as importantly, the ability to store DVDs on a server and then play them back from the TiVo unit. This capability will have to be hacked in by the vast army of TiVo savants that are busy reverse-engineering TiVo internals. If only my satellite system offered a TiVo receiver. Sigh...
Philips Adds Streaming to DVD Recorder
April 24, 2003
It looks like Philips Semiconductor is setting themselves up to be a player in the Microsoft's new world of DVD technologies. The giant Dutch technology company has a reference design for a DVD+RW recorder, and has added a multimedia chip that will be capable of processing multiple streaming formats, which presumably will include MPEG-4 and Windows Media. The design will be able to connect to networks and dynamically stream from network sources.
Reference designs of this sort are sold to manufacturers of consumer products, and often end up under American Christmas trees with few changes from the original manufacturer's specs. If this design proves popular, it could signal the true beginning of a streaming video industry. Wonder what the cable and satellite TV providers think of all this?
Apple Pushing AAC
April 23, 2003
QuickTime 6 was an early adopter of MPEG-4 technology, and they seem to be big believers in the standard. This is in contrast to Microsoft's Windows Media 9, which uses proprietary codecs to compete with the open standards of MPEG-4.
The MPEG-4 standard uses AAC for encoding audio, and Apple has a site with a few links and information about this audio encoder. Alas, they don't have a date when AAC will be included in the iPod!
Update!It turns out that AAC is an integral part of Apple's iTunes strategy. In order to be able to license music for download and storage on Macs and iPods, Apple needed a compressed format that support Digital Rights Management (DRM). Windows Media might have worked for another company, but definitely not for Apple.
In any case, as of May 1 iPod users can download version 1.3 of their firmware, which plays AAC files purchased from the iTunes Web site.
Rob Glaser Speaks For Real Networks
April 22, 2003
Rob Glaser has quite a bit to say in this interview with PC Magazine. Naturally he takes quite a few swipes at Microsoft, particularly their attempts to strong-arm the rest of the industry into using Windows Media 9. (Rob happily trumpets Real's adherence to standards such as MPEG-4.)
The PC Magazine article must have hit the Web only hours before the next big revelation from Real, which was that they are purchasing Listen.com and the Rhapsody online music service. This continues to drive home the message that Real can't survive by just being a technology company -- they need to keep their fingers in the content pie as well.
Bill's Dirty Tricks Department
April 22, 2003
Looks like the folks in Redmond are doing their best to discourage users of "non-standard" browsers, such as Mozilla, Opera, or others. Online scandal sheet "The Inquirer" reports that users attempting to view High Definition videos from this browser are getting a message stating that said content can only be viewed from the friendly confines of Internet Explorer.
If you were thinking that there is special code in IE that is needed to enable viewing these videos, think again! Testers whose browsers can masquerade as IE have reported that said browsers are able to render the HD video just fine.
I hope the folks on the IE software team know what they're doing. This kind of behavior can lead to one of those unpleasant consent decrees when followed to their logical conclusion. And we don't want that, particularly after the Softies have sworn they are going to be be on their best behavior from now on.
Zip Identifies Unknown Composers
April 21, 2003
This story is a variation on a theme that has been popping up regularly over the past year. In this particular instance, a research team is using the Zip format to identify the composers of anonymous musical pieces.
How do you do this? By a simple mechanical procedure involving the concatenation of musical scores, you can determine which preconditioning text gives the best compression of an anonymous piece. In a nutshell, you can use Zip to perform a massive comparison of a musical work against Brahms, Beethoven, and Mozart. The composer with the most matches will presumably be the likeliest composer of the mystery score.
Previously we've seen the Zip format used to identify mysterly languages and to search for artificial structures on Mars. With the addition of the ability to identify lost composers, we can only wonder what's next. My guess is that that Zip will be used to finally identify the true author of the works of Shakespeare.
VideoPhone for the Masses
April 21, 2003
By now there's nothing original in carping about the fact that AT&T promised us videophones back in 1965 and we have yet to see a practical one show up in our homes. The folks at D-Link seem to think that perhaps now is the time to foist this invention upon the masses -- at least the members of the masses fortunate enough to have a broadband connection.
The result of this thinking is the D-Link DVC-1000 i2eye, a standalone videophone that plugs into any TV and feeds an H.323 A/V stream out to the Internet. The H.323 standard is pretty well known in the world of videoconferencing, so there's actually a good chance that you can get this guy to connect to another endpoint out there. In particular, Microsoft's NetMeeting can probably be used to strike up a useful conversation.
So is this device any good? I'm not the one to speculate on that, but you can follow this thread on Slashdot to pick up the general direction of public opinion. This review might help too.
All I know is that as long as Comcast limits my uplink to 128Kbps I'm not likely to make a very good impression on whoever has to watch me at 5fps.
Windows Media 9 to Linux
April 20, 2003
You've heard me say it over and over here: Microsoft wants Windows Media 9 to rule the world. What could be more of a sure sign of that than the fact that they are porting their codecs, protocols, and DRM to Linux?
Oddly, Microsoft is farming out this project to InterVideo, Inc., rather than doing the work themselves. My guess is that the idea of having a cadre of of Linux programmers deep in the bowels of the Evil Empire would be equivalent to letting the camel's nose in the tent. And Linux is a camel that they definitely don't want to share the tent with!
Although the details aren't available yet, it's my guess that you aren't going to see these Windows Media 9 components shipping with popular Linux distros any time soon. Much more likely that the libraries will be offered as value-added packages that developers can use in embedded applications, consumer products, and the like.
This story showed up as a PR feed in a few different places, but didn't seem to draw any real reporting.
Oddly enough, I wasn't able to find a Microsoft press release about this deal. Could it be that not everyone in Redmond is entirely lined up behind this project?
Aladdin Ships Stuffit/Mac
7.0.3
April 20, 2003
Normally I probably wouldn't run an article on a point.point release, but Aladdin gets a little bit of slack for a couple of reasons. Mostly because they are nearly alone when it comes to Mac-centric archiving programs, and they've kept it real for Apple for a long time. Stuffit ships Windows versions of their products now, but I have a feeling the entire staff at Aladdin still wears Jobsian black to work every day.
So what's new in Stuffit for Mac? According to the press release, new features include:
- Ability to browse Tar, Bzip2, and Gzip archives.
- Support for RAR.30.
- Improved Expert Mode dialog for compression settings.
- Optimized for tighter compression.
- The usual list of bug fixes.
Dolby Picks Up Compression Technology
April 20, 2003
Dolby Laboratories is in the process of buying their way into a position of compression leadership. Earlier this month they announced their purchase of DemoGraFX, developer of various codecs and related products.
If you're like me, you associate Dolby with sound processing and sound processing only. And it's true that their most recent news flash was related to AOL picking up Dolby AAC technology to replace RealNetworks as the streaming format for their user base. But with this purchase, Dolby is gaining expertise in the H.264 compression standard, which you might also know as MPEG-4 Part 10. Sounds like Dolby is looking to muscle its way into some new markets!
Slashdot Toasts/Roasts mplayer 0.90
April 19, 2003
mplayer is a very popular media player for Linux. It has a huge variety of supported audio and video formats, and is the choice of movie players for a ferociously loyal cadre of Linux loyalists.
The big news with release 0.90 of mplayer is that long-time project leader A'rpi is leaving the project the second the binaries hit the Net. And much like Iraq, mplayer does not have a plan in place for replacement of a strong leader, so what happens next is anyone's guess.
One of the big controversies about mplayer is that it cleverly uses Microsoft codecs to provide support for several of the formats it supports. To some people, this is a nifty way to get the job done, while to others it represents tarring with the brush of the Evil Empire. Follow the link here to see what the folks at Slashdot have to say about it all. And after that, download the 0.90 version of mplayer and see what all the fuss is about.
Another 7-Zip Release
April 19, 2003
7-Zip comes out with a new release every six weeks or so on average. My last notice of this event was at the very end of February, so I don't have too much to say about 7-Zip 2.30 Beta 29, other than to say this is a really cool Open Source project -- worth a look.
Freedom Audio
April 19, 2003
FreedomAudio is a free Open Source package that lets you stream MP3 and Ogg Vorbis files from your Web site. It renders audio using a small Java plugin. The nice part about using a plugin is that you only have to download and install this once. Freedom Audio also now supports a plugin-free mode, but apparently this results in reduced fidelity.
Some of the more interesting features of FreedomAudio include:
- Skinnable/customizable interface
- Controllable through Javascript
- Supports MP3 and Ogg Vorbis
- Multiple-domain file access
DevIL Releases Linux SDK
April 18, 2003
DevIL provides the code needed to load a wide variety of images into your program. A nice long list of file types that can be loaded are listed here, and a smaller but still quite nice list of formats that can be written to. Displaying images is currently supported under OpenGL, Windows GDI, SDL, DirectX, and Allegro.
I hadn't seen any activity out of DevIL for months and months, and assumed it had passed into the classic Open Source doldrums. But suddenly a Linux SDK release of the current library version showed up, so perhaps there is some life left in this guy after all.
JPEG 2000 Browser Plugin
April 18, 2003
Elysium says that this product is the first publicly released JPEG 2000 plugin for the Windows platform. It works with Netscape, Opera, and IE browsers, and is free.
That's certainly worth a look, although it's not easy to get a good feel for how well it works while there is a scarcity of JPEG 2000 content on the Web. But the price is right, and at least with this plugin you can view the odd file that does come your way.
Unfortunately, no release information on the Elysium Web site.
New Archiver: Zipbit 1.05
April 18, 2003
Zipbit is yet another inexpensive Win32 archiver with support for a wide range of archive types. Creates SFX files, integrates with Windows Explorer, and has localized versions for English, German, and Russian. And of course, you can get a limited-time evaluation before you purchase Zipbit.
Zipbit 1.0 shipped at the start of 2003, and has already been through a few minor releases. We wish Zipbit the best of luck; it can't be easy selling a product in what has become a very crowded market segment.
ISMA Cooks Up MPEG-4 Encryption
April 17, 2003
One of the biggest problems with industry acceptance of MPEG-4 video compression is the lack of Digital Rights Management. A group called the Internet Streaming Media Alliance is working on correcting that, with a proposal for a complete package for the compression technology that would be built around the NIST 128-bit AES encryption standard.
Ideally, the knowledge that content could be protected against easy copying will help encourage content providers to distribute their media in this format. For example, if Hollywood studios felt secure against bootlegging of their movies, they might be willing to distribute their films in MPEG-4 format, which might allow HD versions of the films to fit on conventional DVDs.
VideoTele.com Transcodes MPEG-2 to MPEG-4 In
Real Time
April 17, 2003
The National Association of Broadcasters convention brings out quite a few video-related press releases every year, and this is yet another. In this case, VideoTele.com added real-time transcoding from MPEG-2 to MPEG-4 for their content processor.
This is big news for broadcasters, because it makes it easier to start sending video down their various paths in MPEG-4 format. Now all they have to do is refit all their customers with set-top boxes capable of rendering the new format.
Sony Loses Preliminary JPEG Patent Fight
April 16, 2003
Last year I reported on the fact that Forgent Networks was busily beating the bushes in an attempt to squeeze royalties out of digital camera companies, ostensibly for infringement on a patent Forgent claims covers JPEG compression.
At the time there was much excitement about the case, including many who claimed that there was no way this particular patent covered JPEG compression. But as always, the proof is in the rulings, and it appears that Sony just lost a big one.
The Detroit News Online Web site reports that Sony lost a $25M judgment in this case, with the jury ruling that Sony's hardware and software did indeed infringe on Forgent's purchased patent. The fact that a jury ruled on the case may highlight the difficulties companies have arguing highly technical cases in front of laymen.
Still, things could have been worse. Forgent was asking for $171M in damages, and only received $25M. And it's always possible that this could be reduced upon appeal.
For more comment and discussion on this story, you can catch up with the Slashdot coverage of the story.
Dmitry Shkarin Surfaces!
April 16, 2003
Last week when reporting on a new release of PPMD, I indicated that Dmitry Shkarin's home page had disappeared from the Web. A few days later I heard from Dmitry himself, who provided a link to his current home.
Dmitry's home page has links to current and past versions of his PPMD compressor, and to the BMF image compressor, as well as a few other papers and Web sites.
English speakers will be a bit disappointed to find that Dmitry's page is in his native Russian language, but I find that running it through BabelFish produces a usable translation, albeit with the usual amusing goofs.
The Windows Media Digital Cinema Process
April 16, 2003
As you have read here a few times in the past week, Microsoft would like to have Windows Media 9 be an integral part of the digital production in the movie business.
In this article on the company Web site, they have published a white paper discussing how they used Windows Media 9 to bring the BMW Films Digital Cinema Series to theaters, and they talk about the system that actually did the projection of the digital film.
This demonstration project is obviously a long way from making this a commercial reality, but right now I don't see any serious competition that is going to keep Microsoft from owning this market. If they can convince the film industry to use their codecs and their production software, they are going to lock in a nice revenue stream that will flow as long as people keep going to the movies. With luck, it will spill over into the home as well, if consumer devices start using the WMA 9 codecs.
Remember, you heard it here first!
That's all until next month. Write me with your feedback at [email protected].
Thanks! -- mrn
