An operating system for digital cameras
Carlos is a senior software engineer and Eugene is a developer for FlashPoint Technology. They can be contacted at [email protected] and farrelly@ flashpoint.com, respectively.
Digital cameras are becoming ubiquitous. The increased sophistication of some of these relatively inexpensive units enables operational modes and applications formerly possible only in expensive, professional models. The new devices, featuring image resolutions in excess of 2M pixels, are built around state-of-the-art, 32-bit RISC processors running at 50 MHz or higher, and may include upwards of 16 MB of DRAM, most of which is used to implement multiple capture buffers.
The array of equipment and power packed into the new devices creates opportunities to extend the functionality of the camera. In addition to the set of functions associated and expected from consumer devices, features usually available only with custom professional equipment can easily be enabled, such as customized capture applications and sophisticated, autonomous end-to-end systems.
The Digita operating environment (OE) from FlashPoint (the company we work for) is one such enabler. Digita implements the required, basic camera features (including a GUI), then extends the functionality of the device through scripts and applications. On the scripting side, DigitaScript is a simple, text-based scripting language that is interpreted by Digita OE. DigitaScript lets users manipulate, with a limited UI, the functionality that exists in a Digita-enabled digital camera as well as write out any kind of ASCII-based file. (A free DigitaScript kit and examples of DigitaScript can be found at http://www .digitaphoto.com/.)
In-camera applications, however, are compiled outside the device using the Digita Application SDK and loaded through the same storage medium used to save the pictures. In this article, we examine the main features of the Digita OE and explain how third-party in-camera applications can be written using the SDK and run in a Digita-enabled device. Our main purpose is to explain the implementation details of a specific end-to-end application that Digita enables.
The Digita Structure
Figure 1 shows the basic software structure in a Digita-enabled device. The OE abstracts hardware platform differences by providing a uniform set of APIs. You invoke these APIs to run otherwise complex operations, such as taking pictures, varying exposure conditions, managing the database of images, communicating with external devices, accessing the filesystem directly, modifying selected picture file metadata, and other miscellaneous functions. The real-time, multithreaded kernel supports a sufficient number of ANSI and Posix functions, and the application framework library facilitates the creation of sophisticated GUIs. The framework APIs are linked into the application to form the final run-time image. In the worst case, a simple recompilation enables the same application to run on a different hardware platform.
With the help of the APIs available from the toolbox, the basic structure of a camera application reduces to Listing One. Applications can be initialized with a single call to a library function named AMAppInit(); this API in turn initializes all the required core subsystems. Since in most cases you will need to initialize your proprietary code, the main app initialization API will likely be encapsulated into a proprietary initialization routine. A typical application will likely require you to initialize the camera's preview display to implement a user interface.
Next, it is necessary to register the application to receive event notifications. These events may result from direct user interactions with the camera (such as pressing the shutter button), from removal/insertion of storage media (typically a Compact Flash Card or CFC), or from the completion of functions triggered by previous user input, such as an event notifying the application that the OE has completed spooling a picture to the storage medium. Once the application has been properly initialized, a while loop is entered where the two main activities consist of waiting for events and dispatching the events received for proper handling. In this manner, you can write a variety of applications that use the same basic structures, but can link in different modules to implement a distinct behavior. Assuming that you include functionality to exit the while loop, processing will reach the exit routine that typically cleans up the system and turns the camera power off.
HandleEvent() is, of course, where the hard work is typically done. It is up to this function to implement code to discriminate among the received events and to dispatch the routines that implement specific functionality.
Callback functions are supported by various core subsystems to aid you in assembling multithreaded applications. For example, burst captures are possible to implement, but since most devices will have a finite number of capture buffers, callback functions can be used to trigger notifications when these buffers become available for reuse. In the meantime, you can implement graceful behavior to deal with the momentary inability to take additional pictures.
Specific Functionality
The Digita platform has been designed such that different modules (subsystems) control the features of the camera. Each subsystem has a unique set of APIs in the SDK that can be used in application development. Because a detailed look at each subsystem and its corresponding APIs is beyond the scope of this article, we will present a high-level discussion of the subsystems. We will then examine a few of the APIs with more detail in conjunction with the sample code. More information about the application SDK, including an online reference, FAQ, and updates, is available at http://www.digitadev.com/.
The primary subsystems in the Digita platform include the application, capture, image, communication, and utilities subsystems. The application subsystem controls the execution of programs that run in the camera, launching each application in its own separate thread. While the Digita platform currently allows only one application to run at a time, the application may be multithreaded for added control and robustness. The application subsystem is used not only for the execution of applications developed by third parties, but also for the camera control application, which is the default application that loads when the camera is powered on.
The capture subsystem controls the features used to take photographs. This includes such actions as zooming to a specified level of magnification, charging the strobe light, checking for available memory and disk space, activating live view, and taking the photo. The capture subsystem also controls watermarks, allowing date, time, and/or logos to be integrated into the photo as each is processed and saved.
The image subsystem provides the means to track and access information about images. It maintains a database of images on the removable media, and it updates this database according to customer use. It allows images to be filtered and grouped for better organization, and it provides access to file tags in images (the image metadata). Digita supports the Exchangeable Image File (EXIF) format for digital still cameras. This format contains the actual JPEG image as well as tags that contain data about settings on the camera when the picture was taken, such as the shutter speed, orientation, compression, white balance, focus distance, and image size.
The communications subsystem handles message exchanges between the camera and other devices, such as cell phones, computers, printers, and other cameras. While the communications subsystem is necessarily limited to supporting the hardware provided with the camera, it does have the means for supporting Normal Speed Serial (NSS), Infrared Data (IrDA), and Universal Serial Bus (USB) data transmissions. The same software mechanism supporting these means of communications can be extended to support Bluetooth when it becomes a camera feature.
The utilities subsystem gives access to a variety of additional camera functions, including the LCD, camera parameters, sound, memory, and filesystem. For the LCD, the utilities subsystem offers APIs to turn the screen on/off, change the background colors, and render text to the screen. This subsystem also includes functions to access camera parameters. For cameras that support sound, the utilities subsystem provides means for recording, saving, and playing sound files with an image. It also provides dynamic memory allocation functions and basic filesystem utilities.
A Sample Application
We will now describe an application that was created with the Digita platform and SDK. This sample is written in C and behaves in a similar manner to the standard time-lapse feature that is available on most cameras. However, it is an enhanced version of time-lapse, as the user can specify a longer duration between shots, and the app can be run without user control.
The target user of this application is, say, a bird watcher who wants to take photos of a particular event in a remote location at regular time intervals during the year. This use-case scenario dictates that the application:
- Be configurable to a desired workflow. (For instance, specifying how many shots to take, when to capture the images, what zoom setting should be used, and so on.)
- Operate without user guidance (because the image capture is in a remote location).
- Support basic camera operations (such as capture, zoom, and the like).
- Conserve power when not actively taking pictures.
To accommodate these requirements, we first determined a core set of commands that users need to automate the camera operation in the absence of human control. We then defined a set of functions that grouped together the necessary Digita APIs to mimic the behavior described by these commands.
The set of commands we ultimately identified included:
- The LOG command, which enables the creation of a log file for error checking during application execution.
- The WAKE command, which puts the camera into a sleep mode with reduced power consumption until the specified time of the day when it wakes up to process subsequent commands.
- The CAPTURE command, which indicates the number of images to take.
- The ZOOM command, which moves the lens to achieve a desired level of magnification.
- The LABEL and GOTO commands, which are used for flow control; the GOTO command moves the command execution sequence to the LABEL command with the same string argument. Listing Two is a sample configuration file implementing these commands.
The application was designed to run in the following manner: When the application is launched, it reads the configuration file and stores the commands in a doubly linked list. The commands are mapped to the correct functions via function pointers in the nodes of the linked list, as in Listing Three. The links in the list are modified to reflect any desired flow control in the command execution. A single button event from the user triggers the execution of the first command and the subsequent traversal of the list.
An inspection of the code reveals some of the major features of Digita applications. First, many of the normal C data types in Digita have been mapped to new data types with typedefs. This is done to make Digita SDK code distinguishable and consistent. Also, the starting point for a Digita application is the entry() function -- not the main() routine that is common in traditional C programming. The entry() function for this application is given in Listing Four.
ProcessInit() in the entry() routine starts the various Digita subsystems that are needed by the application. The entry() function then calls ProcessBeforeEventLoop(), which executes a series of routines before enabling the user to control the camera with different button events. In this particular sample, ProcessBeforeEventLoop() reads in the configuration file, verifies the commands, stores them in the linked list, and resolves the flow-control statements.
With the initialization and verification completed, the application gives the user control of the camera and waits for button events. In this sample, the user begins the execution of the commands from the configuration file by hitting the center button (or soft key) under the LCD on the camera. Listing Five illustrates how this button is used to start the command execution. Digita defines IDs for each of the buttons on the camera, and these IDs can be used in a switch statement to invoke certain functions in response to certain events.
Listing Six outlines the function that is invoked to start the command execution in the application. Since it is undesirable for users to start the command execution after it has already been initiated, this function first checks a flag before executing the rest of the routines. CommandGoToCmdStart() then moves the list pointer to the beginning of the linked list, and CommandDispatch() is used to execute the command via the function pointer, as in Listing Seven.
Listing Eight contains a snippet of code from the function used to implement the ZOOM command from the configuration file. This code reveals how camera parameters can be used effectively in an application. In this case, we need only to set the value of the zpos (zoom position) camera parameter to reach a desired level of magnification. Digita automatically waits until the lens is positioned correctly before returning from PARMGetCurrentSetting().
Some of the other features in the camera, such as capture, cannot be set in this manner. For features like these, callback routines are used. In this application, we do not want to execute the next command until we are sure that the photo has been completely spooled to the removable medium. We use the callback routine from Listing Nine to dispatch an event to let us know when it has been invoked. When this particular event is received, as indicated by the fData1_Spooled argument, then we know the image has been completely written to the disk, and it is safe to evaluate the next command from the configuration file.
The only issue that the application does not address is that of storage. Clearly, users will have finite disk space in which to store the images. To make a completely autonomous solution, we would need to provide a means for users to upload the images to a back-end system so that disk space can be reused. The communications subsystem in Digita gives us the means to transmit data via NSS, IrDA, and USB; however, the discussion on the implementation of this is beyond the scope of this article.
This application, like the Digita OE itself, has been designed for extensibility. Giving users new commands for the configuration file involves simply adding a new corresponding function to the code and a simple recompile.
The Application Framework
The application framework available with Digita-enabled cameras consists of an object-oriented library of APIs. While applications need not be built with the framework, these APIs provide efficient means to build attractive, functional GUIs. The library supports a set of UI elements and controls that are practical for in-camera applications, including title bars, simple menus, and buttons. An object enabled within the UI accepts the input focus of user actions, and the user is able to move an on-screen cursor to select or enable objects. Objects alter their visual appearance (or generate a sound in audio-capable cameras) when enabled. Because this library is based on an audio-visual framework, components can be easily exchanged to customize the appearance of the UI while preserving functionality.
Listing Ten is a basic framework application structure. The library provides two important classes called CAFShell, which manages the views for all application components, and CAFView, an object that provides a title bar, command bar, and a menu screen that you can easily populate with your proprietary work. It is also within the particular instance of the CAFView object that you build the functionality of your application by adding components to the default CAFWorkspace object, or by replacing this object with one that may have been built separately.
A framework application requires its own event handler, and the CAFShell member method RegisterSystemHandler() is available for this. It is usually invoked before the application view is opened.
Listing Eleven shows the operations that are typically implemented with the constructor, the TargetSwitchEnter() method invoked when the view gets focus, and the HandleEvent() method invoked when buttons are pressed.
In the constructor, a set of default objects is created that will be useful for the entire life of the object, and the soft buttons are assigned labels to invoke particular operations. The construction of the menus that become available when a view is entered is performed within TargetSwitchEnter(). Note that in a desktop environment this work might have been done elsewhere, but in the in-camera situation when memory resources are limited, recycling of objects is advantageous.
The final method illustrated is HandleEvent(), which must be implemented by every view doing useful work. When an event such as a button press notification is received, functions within HandleEvent() are invoked. A final, useful CAFView method is Draw(), which you may wish to override with your own implementation. Draw() is called by the framework whenever it determines that the component has become visible and needs to be redrawn.
The current application framework implementation consists of some 50 C++ classes that include a rich set of methods, giving you the ability to completely customize your application's GUI with text labels, icons, and colors. Figure 2 is an example view screen built with these APIs. Since the core OE also includes localization facilities, it is not difficult to customize applications for languages supported by the camera's manufacturer.
Conclusion
The Digita OE has been designed and implemented to take advantage of the increased capabilities of digital cameras. The architecture of the OE not only facilitates the implementation of the usual functionalities users expect of consumer-grade devices, but it also enables the development of specific applications that would otherwise be difficult (if not impossible) to obtain in a consumer product.
Both commercial and freeware third-party developers have extended the Digita OE. Vertical applications include industries such as insurance, e-commerce, law enforcement, tourism, and others. In particular, some interesting applications have been made with the SDK; see for instance, http:// www.bidpath.com/, http://www.activephoto .com/, and http://www.veripic.com/.
Freeware applications written with the SDK can be found at http://www .digitaphoto.com/ and http://www.denpix .com/. DENPiX can be used to analyze pictures (histogram) and process out the image noise.
Finally, Digita OE is, like any technology, evolving with frequent updates and extended functionality in progress. Updates and information can be found at the Digita Developer community website at http://www.digitadev.com/.
DDJ
Listing One
#include <FPApplicationManager.h>
#include <FPEventManager.h>
TVoidPtr entry(void)
{
TAMAppID appId ;
TEMEventRecord * theEvent ;
TBoolean doExit ;
AppInit() ;
RegisterForEvents() ;
while( !doExit )
{
EMGetEvent( appId, &theEvent ) ;
doExit = HandleEvent( &theEvent ) ;
}
AppExit() ;
return(NULL);
}
Listing Two
# Sample configuration file for extended time-lapse application # Turn logging on to catch any errors during command execution Log on # Label statement for flow control with goto statements Label repeat # Wake at 1:30:00 pm wake 13 30 00 # Zoom to 2x magnification zoom 200 # Capture 1 image capture 1 # Zoom out Zoom out # Capture 1 image capture 1 # Repeat the sequence goto repeat
Listing Three
/* structure of a node in the doubly linked list */
struct cmd_list
{
TErrCo (* cmdFunc) ();
TShort argc;
TChar ** argv;
TShort cmdMinNumOfArgs;
TShort fileLineNum;
TBoolean isResolved;
TShort cmdExecutionCounter;
struct cmd_list * nextCmd;
struct cmd_list * prevCmd;
} ;
Listing Four
/* entry point for Digita apps */
TInt entry(TInt argc, TChar **argv)
{
TErrCo err = kNoErr,
appErr = kNoErr;
TAMAppID appID = 0;
sigset_t pendingSet;
TBoolean alarmSignal;
TEMEventRecord *theEvent;
TChar text[kMaxTextLength];
TBoolean doExit;
// Start the necessary subsystems according to app content.
err = ProcessInit();
// Start the graphics subsystem and display a global banner....
if (kNoErr == (err = SDKDInit(TRUE)))
{
LMEnableLCDController() ;
SDKDDrawTextandScrollUp(" Digita Sample Application ");
}
else
{
return(NULL);
}
// Process functions before enabling button events.
err = ProcessBeforeEventLoop();
// Get the application ID and ensure it's valid
if (0 == (appID = RegisterWithEventManager()))
err = !kNoErr; // Set to not no-error
gAppID = appID ;
SDKDDrawTextandScrollUp("Press Menu button for options");
// Process while there are no system event errors
doExit = FALSE ;
while ( (err == kNoErr) && !doExit )
{
// Block & wait for next event. Event subsystem allocated mem for event.
err = EMGetEvent( appID, &theEvent);
if (err != kNoErr)
{
//Handle a signal that is posted.
if (0 == sigpending(&pendingSet))
{
alarmSignal = sigismember(&pendingSet, SIGALRM);
err = kNoErr;
}
}
else
{
// Got an event, now Handle it.
appErr = HandleEvent(theEvent, &doExit);
// Release the event, specifically the mem allocated
EMReleaseEventRecord(theEvent);
// Now check the application error.
if ((err == kNoErr) && (appErr != kNoErr))
{
err = appErr;
}
}
}
// Exit the application.
SDKDScrollUp();
sprintf( text, "Exiting sample, rc = %ld", FPERROR(err) );
SDKDDrawTextandScrollUp(text);
DelayTime( 30 ) ; // let me see the message ...
err = SDKDTerminate(); //Terminate the graphic manager
err = ProcessTerm(); // ProcessTerm() will turn off the camera.
return(NULL);
} // entry
Listing Five
TErrCo ProcessButtonEvent(const TBTNDigitaButtonId buttonID,
const TBTNDigitaButtonPosition buttonState)
{
TErrCo err = kNoErr;
switch (buttonID)
{
case kDigitaSoftKey2:
CenterSoftkeyButton(buttonState);
break;
default:
break;
}
return(err);
} // ProcessButtonEvent
Listing Six
TVoid CenterSoftkeyButton(TBTNDigitaButtonPosition buttonState)
{
TErrCo err = kNoErr;
TChar msg[kMaxTextLength];
if (!sCommandProcessorStarted)
{
if (buttonState == kButtonUp)
{
if (sCommandInitSuccess)
{
// set a flag disable button events (except power down)
sCommandProcessorStarted = TRUE;
// Start with the first command
CommandGoToCmdStart();
if (kNoErr != (err = CommandDispatch(gCmdCur,FALSE)))
{
sprintf(msg,sDispatchEventError, FPERROR (err));
SDKDDrawTextandScrollUp(msg);
}
}
else
{
sprintf(msg,"Commands contained errors.");
SDKDDrawTextandScrollUp(msg);
}
}
}
return;
} // CenterSoftkeyButton
Listing Seven
TErrCo CommandDispatch(TCmdData * pCmd, TBoolean validate)
{
TErrCo err = kNoErr;
// if arg is null, use current command
if (pCmd == NULL)
{
pCmd = gCmdCur;
}
// Record the time when the command is dispatched
CommandGetTime(&sCurSystemTime,&sCurTmTime);
// Dispatch the command
if (pCmd != NULL)
{
err = (*(pCmd->cmdFunc)) (pCmd,validate);
}
else
{
// Current command is not set
err = kCurrentCmdNotSetErr;
}
if (sLoggingEnabled)
{
// Log the command
CommandLogCmd(FPERROR (err));
}
return(err);
} // CommandDispatch
Listing Eight
// Convert the zoom value to an integer
zoomValue = atoi(pCmd->argv[zoomIndex]);
// Retrieve the current zoom position setting
if (kNoErr == (err = PARMGetCurrentSetting( kFTAGzpos, &nameTypeValue )))
{
// Set the zoom position
nameTypeValue.fTypeValue.fValue.fUint = zoomValue ;
// write the structure back to make it take effect.
err = PARMSetCurrentSetting( kFTAGzpos, &(nameTypeValue.fTypeValue) );
}
Listing Nine
TInt LocalCaptureSpoolingComplete(TVoidPtr data)
{
TErrCo err = kNoErr;
DispatchEvent(fData0_Group,fData1_Spooled,"Capture spooling complete");
return(err);
} // LocalCaptureSpoolingComplete
Listing Ten
#include <FPTypes.h>
#include <CAFShell.h>
#include <MyView.h>
// forward declaration
TBoolean myHandler(TEMEventRecord pEvent, TVoidPtr pData) ;
TBoolean myHandler(TEMEventRecord pEvent, TVoidPtr pData)
{
TBoolean handled = TRUE ;
// sample implementation
return(handled) ;
} // end of myHandler
TVoidPtr entry(void)
{
TErrCo err ;
CAFShell *app = new CAFShell(kAMImagingApp) ;
MyView *view = new MyView() ;
err = app->OpenFW() ;
app->RegisterSystemHandler(myHandler, NULL) ;
if(err == kNoErr)
{
// miscellaneous initialization
view = new MyView() ;
app->OpenView(view) ;
app->DoEventLoop() ;
app->CloseFW();
delete view ;
}
delete app ;
return(NULL) ;
}
Listing Eleven
#define kMyDoThis "MyDoThis"
#define kMyDoThat "MyDoThat"
#define kMyAbout "MyAbout"
MyView::MyView()
{
// Could create own workspace (container) but using the one supplied.
// mWorkspace = new CAFWorkSpace() ;
mItemBar = new CAFItemBar() ;
mOptionListBox = new CAFListBox();
mAboutView = new MyAboutView() ;
SetCommandButton(kCBLeftButton, kMyDoThis ) ;
SetCommandButton(kCBMiddleButton, kMyDoThat ) ;
SetCommandButton(kCBRightButton, kMyAbout ) ;
} // end of constructor
MyView::TargetSwitchEnter(TVoid *pData)
{
int n, options ;
CAFActionListItem *nextItem ;
CAFView * pView ;
// create a menu
nextItem = new CAFActionListItem();
nextItem->SetItemLabel(kMyMenu);
nextItem->SetActionLabel(kMyDoThis);
pView = new MyMenuView();
nextItem->SetView(pView);
mViewList.AddItem(pView);
nextItem->SetHelpText((TChar *)NULL, kMyMenuIcon);
mOptionListBox->AddOption(nextItem);
// create another menu
nextItem = new CAFActionListItem();
nextItem->SetItemLabel(kMyNextMenu);
nextItem->SetActionLabel(kMyDoThat);
pView = new MyNextMenuView();
nextItem->SetView(pView);
mViewList.AddItem(pView);
nextItem->SetHelpText((TChar *)NULL, kMyNextMenuIcon);
mOptionListBox->AddOption(nextItem);
// more menus
// Initialize the item bar for this view
SetItemBar( mItemBar, FALSE) ;
// set the view parameters
options = mOptionListBox->GetOptionCount() ;
for(n = 0 ; n < options ; n++ )
{
mOptionListBox->GetOption(n)->SetViewParm(mItemBar) ;
}
} // end of TargetSwitchEnter()
TBoolean MyView:HandleEvent( TEMEventRecord *pEvent )
{
TBoolean handled = FALSE ;
TBTNDigitaButtonID buttonIndex ;
if( pEvent->fEvClass == kButtonClassEvent &&
pEvent->fParams.fButtonEvent.fPosition == kButtonUp )
{
// will handle this event
buttonIndex = pEvent->fParams.fButtonEvent.fButtonIndex ;
switch(buttonIndex)
{
case kDigitaSoftKey1:
// perform specific operation
// might be appropriate to refresh views
Refresh() ;
FWRefreshView() ;
handled = TRUE ;
break ;
// other cases . . .
} // end of switch
} // end of handling event
if(!handled)
{
// let the parent class handle this event
handle = CAFBaseView::HandleEvent(pEvent);
}
return(handled) ;
} // end of HandleEvent()
