Multithreaded Asynchronous I/O & I/O Completion Ports

When developing server applications, it is important to consider scalability, which usually boils down to two issues. First, work must be distributed across threads or processes to take advantage of today's multiprocessor hosts. Second, I/O operations must be scheduled efficiently to maximize responsiveness and throughput. In this article, I examine I/O completion ports—an elegant innovation available on Windows that helps you accomplish both of these goals.

I/O completion ports provide a mechanism that facilitates efficient handling of multiple asynchronous I/O requests in a program. The basic steps for using them are:

1. Create a new I/O completion port object.
2. Associate one or more file descriptors with the port.
3. Issue asynchronous read/write operations on the file descriptor(s).
4. Retrieve completion notifications from the port and handle accordingly.

Multiple threads may monitor a single I/O completion port and retrieve completion events—the operating system effectively manages the thread pool, ensuring that the completion events are distributed efficiently across threads in the pool.

A new I/O completion port is created with the CreateIoCompletionPort API. The same function, when called in a slightly different way, is used to associate file descriptors with an existing completion port. The prototype for the function looks like this:

HANDLE CreateIoCompletionPort(
   HANDLE FileHandle,
   HANDLEExistingCompletionPort,
   ULONG_PTR  CompletionKey,
   DWORD NumberOfConcurrentThreads
   );

When creating a new port object, the caller simply passes INVALID_HANDLE_VALUE for the first parameter, NULL for the second and third parameters, and either zero or a positive number for the ConcurrentThreads parameter. The last parameter specifies the maximum number of threads Windows schedules to concurrently process I/O completion events. Passing zero tells the operating system to allow at least as many threads as processors, which is a reasonable default. For a discussion of why you might want to schedule more threads than available processors, see Programming Server-Side Applications for Windows 2000 by Jeffrey Richter and Jason D. Clark.

Associating File Descriptors with a Port

Once a port has been created, file descriptors opened with the FILE_FLAG_OVERLAPPED (or WSA_FLAG_OVERLAPPED for sockets) may be associated with the port via another call to the same function. To associate an open file descriptor (or socket) with an I/O completion port, the caller passes the descriptor as the first parameter, the handle of the existing completion port as the second parameter, and a value to be used as the "completion key" for the third parameter. The completion key value is passed back when removing completed I/O requests from the port. The fourth parameter is ignored when associating files to completion ports; a good idea is to set this to zero.

Initiating Asynchronous I/O Requests: OVERLAPPED Explained

Once a descriptor is associated with a port, (and you may associate many file descriptors with a single I/O Completion Port), an asynchronous I/O operation on any of the descriptor(s) results in a completion event being posted to the port by the operating system. The same Windows APIs that let callers perform standard synchronous I/O have a provision for issuing asynchronous I/O requests. This is accomplished by passing a valid OVERLAPPED pointer to one of the standard functions. For example, take a look at ReadFile:

BOOL ReadFile(
  HANDLE    File,
  LPVOID    pBuffer,
  DWORD     NumberOfBytesToRead,
  LPDWORD   pNumberOfBytesRead,
  LPOVERLAPPED  pOverlapped
  );

For typical (synchronous) I/O operations, you've always passed NULL for the last parameter, but when doing asynchronous I/O, you need to pass the address of an OVERLAPPED structure in order to specify certain parameters as well as to receive the results of the operation. Asynchronous calls to ReadFile are likely to return FALSE, but GetLastError returns ERROR_IO_PENDING, indicating to the caller that the operation is expected to complete in the future.

A common mistake when using OVERLAPPED structures is to pass the address of an OVERLAPPED structure declared on the stack:

OVERLAPPED Gone;
// Set up 'Gone'..
ReadFile ( hFile, pBuf, Count,
          &NumRead, &Gone );

This just won't work because ReadFile returns immediately, and when the function containing the call to ReadFile exits, the stack will be unwound and the data pointed to by &Gone will become invalid. Thus, you should ensure that your program manages its OVERLAPPED structures (and any buffers you're using) carefully. The example employs a fairly common strategy that involves having a C++ class representing a connection derive from OVERLAPPED—which may offend some C++ purists, but is a practical solution to the problem. The connections are allocated on the heap, and when I/O operations are initiated, the connections' pointer is passed as the pointer to OVERLAPPED.