All Your Drop Are Belong to Us

In fact, for the money these folks were talking, I could buy, use, and discard a moderately fancy printer on this one job. I figured that I would spend about the same amount of time and aggravation either doing the job myself or chasing back and forth getting them to do it right.

I wound up buying an HP 2000C Professional Series inkjet printer that featured fast printing, decent photographic image quality, high duty cycle, and four large ink tanks. It was billed as having nearly the throughput of a color laser printer at a fraction of the price. Although my average use would be rather low, the peak for this project was about right.

I did not fully appreciate the significance of one item buried in the spec sheet, touting "smart chips embedded both on the print heads and ink cartridges that communicate with the printer firmware."

As it turned out, those chips form a perfect example of how embedded systems firmware can have a major effect on other business decisions — and vice versa. Let's peek under the lid to see what happened.

Dated Drops

At first glance, inkjet printing seems to be a fairly simple process: Sweep a print head across a sheet of paper while spraying ink spots, move the paper, and repeat until done. The more you look into it, though, the messier and more difficult it gets. From what I've seen, it's amazing that the things work at all, let alone as well as they do.

Inkjet ink must satisfy several simultaneous and mutually incompatible requirements. It must dry rapidly when applied to paper, yet remain ready to use for months or years in the cartridge. It must produce vivid colors without pigments that could clog the nozzles or require stirring (although some inks now include micro-fine pigments). The carrier fluid must behave predictably when vaporized by heat or shocked by piezoelectric ejectors, while being nonflammable, relatively nontoxic, and transparent.

It's not surprising, then, that printer inks deteriorate over time. Manufacturers provide a "Best Used By" date in various ways, either based on the manufacturing date or with an actual expiration date printed on the cartridge or its packaging. In general, the dates are just recommendations.

The HP 2000C printer takes those recommendations to a different level. Each ink cartridge and print head includes an IC with four gold contacts; see Figures 1 and 2. The chip carries a unique ID number, the device's expiration date, the amount of ink in the cartridge, and, I'm sure, various other useful and interesting tidbits.

Relaying that data back to the printer driver through the ubiquitous PC parallel port becomes a simple matter of firmware. The PC driver controlling the printer can perform all manner of useful functions, including verifying and enforcing the "Use By" dates.

Unfortunately, this didn't work quite as nicely in the real world as HP thought it would.

Deployment Date

Because the HP 2000C can handle high- duty cycles and large print runs, it's typically found in departmental printing situations, connected to a network and serving many workstations. It can be net- worked through an HP JetDirect brick or a standard PC, with a variety of different functions available in each case.

Recall, however, that I bought the printer in 1999. What else do you recall about 1999? Yup, that was the year before The Dreaded Y2K Rollover when companies around the world were verifying that their software wouldn't disintegrate when the clock struck zero.

Some companies using HP 2000C printers ran into a peculiar date-related problem: The printers sporadically complained that their ink was outdated and refused to print. It seems that the driver included today's date along with the print data, whereupon the printer's firmware compared it with the expiration dates stored in the cartridge chips and shut the printer down.

After all, as far as the printer was concerned, today's date was beyond the cartridge's expiration date. QED.

Evidently, the driver didn't enforce the print head date code quite so rigidly, as I didn't find any evidence of print head expiration problems.

HP had this to say about the problem on its web site:

The system date has been set incorrectly, usually to a date in the future, and the expiration date of the ink cartridges is exceeded. If printing across the network, any workstation on the network that has the incorrect date set could cause the false expiration message.

Ooops!

Worse, however, is their warning that:

Ink cartridges have limited shelf life and should be installed before the "install by" date listed on the ink cartridge packaging. If the chip in the ink cartridge is set to "expired," the cartridge cannot be restored.

This is one of those rare situations where software can cause hardware damage. Well, not exactly damage, but I suspect relatively few HP customers appreciated trashing handfuls of $40 ink cartridges while debugging the problem.

Various support forums reported that the results differed depending on which OS generated the jobs, which OS printed them, and which driver levels were installed on which systems. It may well be that different printers included different firmware revisions, which would confuse the results beyond recognition.

Now it gets interesting.

Demand Drops

HP maintains a strong position in the inkjet printer market, selling nearly half the total units throughout the world. In fact, judging from its annual report, HP's Imaging and Printing Systems group accounts for nearly all of the company's earnings — this division earned $768 million in the second quarter alone — while the other business units are a net wash. King C. Gillette pioneered the business model now known as "give away the razor and sell the blades" in the early 1900s. It worked well then, making him a millionaire, and works well enough now that the legacy of Hewlett and Packard's instrumentation genius is a business that largely designs and sells printers, ink, and paper.

Although inventing a printer and bringing it to market requires considerable skill and investment, duplicating its consumables isn't nearly as challenging. In fact, refilling ink cartridges can be a kitchen-table operation, and providing ink to those tables doesn't require much investment at all.

Which leads to a major problem for printer manufacturers: Competition drives printer prices downward while, at the same time, ink cartridge refillers eat away at the exceedingly profitable consumables market. Wouldn't it be nice to lock consumers into using only approved commodities, ideally without violating fair trade laws, so that paying a kilobuck per liter for printing ink seems reasonable?

Which is essentially what those embedded chips in the 2000C's ink cartridges and print heads enable. The printer knows how much ink is in each cartridge and how much goes out with each drop, so it can calculate the remaining amount of ink very exactly. When a tank hits "empty," the printer stops printing, the driver displays a message, and you must install a full cartridge to continue.

This sounds like a difficult problem, but it's just a matter of adding thus-and-so many picoliters for each squirt to get the total ink consumption, which you subtract from the cartridge's remaining contents. If I were doing it, I'd update the cartridge chips after printing each line or page.

Similar calculations apply to the print heads. The embedded chip and the PC's driver code can ensure that the printer delivers the expected performance by monitoring the nozzles and deducting the total flow from the recommended maximum.

So, although refilling a cartridge using a syringe and a bottle of ink is trivial, the printer still considers it empty because its chip has recorded the maximum number of squirts available from that unique cartridge ID.

As with all copy protection technology, ink cartridge chips spawned an arms race between printer manufacturers and refillers. Epson uses a similar system for which you can find reprogrammers that set the chip's counter to zero (or whatever represents the full state) that cost about the same as two or three OEM tanks. The legal issues around this aren't clear to me, but the practice obviously supports many thriving businesses.

The HP 2000C was offered primarily to businesses and didn't sell in mass-market quantities, which probably explains the lack of cartridge reprogrammers. But perhaps there's another reason.

Driving Drops

The HP 2000C software provided with the printer included a Toolbox program that configured the Windows printer options, displayed printer status information, showed ink cartridge contents and expiration dates, printed test pages, and performed various other diagnostic functions. Many of these functions depended on reading data back through the parallel printer port from the printer, which was easy to do under Windows 3.x and only slightly more difficult in Windows 9x.

When Windows 2000 came out, the printer software update on the HP web site didn't include the Toolbox. Although some of its functions now appeared in the driver itself, the printer status displays were conspicuous by their absence.

It turned out that the driver did not support reading data back through the parallel port, most likely because Windows 2000 does a better job at enforcing hardware protection than its predecessors. Without being able to read data from the printer, the driver had no way to display ink levels or expiration dates.

The new driver did add the very useful ability to save and recall configurations by name, greatly simplifying the task of printing duplexed two-up pages or envelopes. It also took away the ink-level page, which now required a Vulcan Nerve Pinch on the printer's buttons. However, you had to yank the cartridges out of the printer to find their expiration date.

A vestigial 2000C driver came with Windows XP, but the HP web site doesn't list an update for that OS. The printer lost essentially all its unique functions under Windows XP — no named configurations, status displays, or Toolkit. The driver can print a simple ink-level page, albeit one much less informative than the full printer status display of yore.

Printer drivers under Windows 2000 and XP need not be output-only, as my Canon S630 printer demonstrates by displaying the ink levels from an optical sensor (no chip!). Evidently, the 2000C user base isn't big enough to justify much driver maintenance.

Now, for the $160 (that's 4×$40) question: Do the new drivers enforce the ink cartridge expiration dates? I do not have any hard knowledge and, for what should be obvious reasons, don't plan to fiddle with my system date to find out, but I suspect that they do not.

Various refillers offer 2000C ink in bulk quantities and their instructions mention that some users have no trouble reusing expired tanks while others simply can't get them to work. Given the decline in driver functionality, it seems that the printer firmware does not handle the date comparison and, without the ability to read data back from the printer, the latest drivers don't do it either.

The original 2000C design depended on a complex interaction between cartridge hardware, printer firmware, Windows driver architecture, and user-level software. A problem with any link in that chain caused the entire system to fail and triggered what seems like considerable customer dissatisfaction.

It may well be that HP decided to reduce the number of support calls by dumbing down the drivers and eliminating the date checks. There's a fairly obvious trade-off between revenue from consumables and expenses for support: One support call probably wipes out any profit from that particular printer.

As we saw last month, system design must take into account not only expected events but things that seem completely implausible. After all, who would imagine users deliberately setting the system date ahead by a few years?

Getting just the embedded code right isn't the entire battle, nor does applying a pretty user interface mean the whole system will operate beautifully. In fact, concentrating on the details probably means you're missing the big picture.

We can't afford that any longer. Imagine if this problem were something serious, like facial recognition?

Reentry Checklist

Although I like my 2000C, I'm obviously underwhelmed by HP's attempt to hard-code expiration dates and usage limits into the machinery. As with any copy protection technology, it seems to work really well where it's least needed while causing nasty problems for legitimate users.

In the interest of full disclosure, I must relate that I own a bunch of HP and Agilent doodads. Mostly, I like all of them, too.

I bought the Canon S630 when the 2000C developed a mechanical problem that I fixed with the liberal application of bent aluminum and epoxy, as sending it in for repair made no sense at all. The S630 has its own quirks, of course.

Perhaps I'm the only person who didn't see the All Your Base video a few minutes after it hit the Web some years ago, as a 2-MB Flash presentation didn't attract my dial-up attention. If you also missed it the first time around and have broadband access now, make up for lost time at http://www.planettribes.com/allyourbase/ or Google for any of the slightly different ayb.swf, ayb2.swf, ayb3.swf, or ayb4.swf files.

HP 2000C date issues are covered at http://www.hp.com/cposupport/printers/ support_doc/bpd50007.html. General troubleshooting hints can be found at http://www.hp.com/cposupport/printers/ support_doc/bpd06201.html.

Repeat-O-Type has a lockout chip resetter for Epson printers at http://www .repeatotype.com/resetter.html. There are some tricks for HP cartridges at http://www.rundletech.com/hp2000tip.htm.

See Gillette's vision for the future of cityscapes at http://www.library.cornell.edu/Reps/DOCS/gillette.htm.

DDJ