Toby Opferman is a Senior Software Engineer in the Advanced Products research group at Citrix Systems. He can be contacted at [email protected].
Virtual machines let you run an operating system inside an operating system while emulating certain aspects of hardware. Microsoft Windows 95, for example, emulated MS-DOS, leveraging the CPU's native "Virtual 86" mode. With systems such as VMWare [1], you can run Windows in VMware on Linux, or Linux in VMware on Windows.
Of course, running operating systems on virtual hardware is not a new concept. So why all of the recent interest in virtual machines? The answer to this question is simple—usability and power. The x86 currently has more power than average users can completely utilize. This is one reason virtual machines have become more usable and run almost as fast as host operating systems in which they are running. Moreover, things will likely get better with the 64-bit platforms and dual-core CPUs coming out next year. Intel, for instance, is also attempting to push this industry by providing hardware support for virtualization with their VMX instruction set codenamed "Vanderpool."
So how do you make applications "virtual-machine aware" or otherwise extend application functionality to virtual machines? That is, how can applications communicate between the guest and the host? The simple solution is to create a network connection between them. While this approach is fine if you need a streaming IPC implementation, what if you want to do, say, shared memory?
You'd think the VMWare SDK would provide you with an API that lets guest and host operating systems share a memory-mapped file. However, when I searched the VMWare SDK, Googled the Internet, and even posted a message on VMWare's community forums, I was surprised that there wasn't any information on sharing memory. The only information I could find was how VMWare can share memory between guest operating systems as an optimization. Consequently, I decided to find out if there was any method of sharing memory. To do so, I needed to find a link between the guest and host memory locations.
The Wrong Address
The first approach I took was simply to use the kernel debugger to view the physical addresses of the guest operating system. The idea was that perhaps VMWare used the true physical addressing and I would be able to use this information in the host to map my own virtual addresses.
The host and guest operating system that I used to do this experiment was Windows XP Professional. I downloaded the debugging tools from Microsoft and set up the kernel debugger for VMWare. This required setting one of the COM ports in the guest to be mapped to a named pipe in the host, a standard option in VMWare.
The physical address can be obtained by finding the Page Table Entry of a virtual address. The debugger command to use is !pte, the debugger command to dump a virtual address is dc, and the comand to dump a physical address is !dc". Figure 1 illustrates these steps in WINDBG.
WINDBG supports a local kernel debugging mode in which you may view various aspects of the system but cannot set break points or actively debug. I used this mode of the debugger for its simplicity. This let me quickly paste addresses between debuggers because I could work on the same machine; Figure 2 is the outcome. Unfortunately, the data at the physical address does not match in the host and guest. I tried several addresses and methods but, as you can see in Figure 2, I didn't have any success.
The guest operating system is allowed to manage the physical addresses and would assume they are linear starting from zero. After all, the guest operating system is meant to think it's running on physical hardware. You can then determine that the debug interfaces being used in the guest by the kernel debugger would also be limited to viewing physical addresses in the same way as VMWare portrays them to be.
The Search for Bits
I knew that this memory must have some type of representation in the host operating system. The question is, can you find where it is in the host, and if you do, can you access it easily? The simplest method of finding this memory would be to brand it in the guest with a unique byte sequence, then search the host for it; see Figure 3.
The logical starting point for this method would be to search in a VMWare process. If you pay any attention to what is occurring on your system, you notice that "vmware-vmx.exe" is launched for each Virtual Machine you have running. I used CDB to break into this process and search the memory for my unique data string, as in Figure 4.
The first attempt found two locations that contained this unique data string. This may be caused by caching or perhaps the buffer for the named pipe was found. I tried clearing this memory in a simple test by reusing the kernel debugger (Figure 5) and doing another search, as in Figure 6. There was now only one location that contained the unique data string. But this still leaves unanswered questions, such as, is this the correct location and, if it is, how can I access it easily?
My next idea was to verify random addresses in the guest with the host by finding the physical address of the data set in the guest. I would use this address as a starting point and subtract or add it with other physical addresses. The difference between the two would then be used against the virtual address found in the host. The memory contained at both locations would then simply be displayed and compared.
This is definitely the memory location, but you need to know how to access it. This is where knowing how Windows applications work helps provide direction. This can either be a memory-mapped file or a large heap allocation. A large heap allocation would require a hack to be able to use the memory; however, a memory-mapped file would possibly make it simple to access.
There are different methods that you could try to find out how this memory is mapped into the process.
- One way involves debugging the application, setting breakpoints on the APIs that map files, and allocating virtual memory.
- Another approach is to find handles that represent sections that could be used to map this memory. The Windows Object Viewer [2] could be used to find names of mapped files in the hopes that they are descriptive; see Figure 7.
There is neither a right nor wrong approach. The method of debugging would find all memory-mapped files and their starting addresses. These could be used again in combination with the memory search method. The size could also be compared to the physical RAM allocated in the guest.
The opening of mapped files could also be used to determine and compare the size. A mapping could also be created and, again, the memory-search method could be used to determine which was correct.
The end result is that "VMwareMem<PID>Memory" (case sensitive) is the memory-mapped file containing only the physical RAM of the guest operating system. The "PID" is the process identifier of the "vmware-vmx.exe" associated with a particular guest. This information can then be used to create a model for sharing memory between the guest and host operating systems.
The recently released VMWare 5.0 no longer provides a named memory-mapped file. The unnamed memory-mapped file can be retrieved by enumerating section handles in the "vmware-vmx" process, duplicating them, and then finding the section handle whose maximum size matches the physical memory of the guest.
The Shared-Memory Idea
At this point, what the guest thinks is that the physical RAM has been located in the host operating system. The implementation of this as a memory-mapped file makes shared memory straightforward. This means that any application running in the host operating system with correct privileges would be able to open and share the physical memory being used in the guest.
There are limitations that are inherent to this method of shared memory.
- Nonpaged Memory. The application or driver in the guest operating system would need to allocate nonpaged memory. The memory in what is thought of as the physical RAM would be subject to paging implemented by the guest. The guest could swap memory out and move memory around to where it would not be possible to keep the host application in sync.
- Linear Physical Memory. The physical addresses of the nonpaged memory allocated in the guest operating system would need to be linear. This is similar to the requirement imposed on allocating memory for DMA. The DMA controller is separate from the CPU and, as such, only interfaces with physical addresses. The DMA controller requires a starting address and a length that implies the physical addresses to be linear.
- This problem would only surface if your memory crossed a 4k boundary. The application would either need to know all the nonlinear addresses so the host could handle it appropriately or the guest would need to ensure it allocates virtual memory with linear physical addresses.
The application running in the host maps a view of the memory-mapped file using the physical address of the guest memory as the starting offset.
Nonpaged memory is a limited resource on the system and should definitely be used sparingly. This technique of memory sharing should be considered where an IPC would prove to be less effective.
The good news is that you are not truly allocating physical memory. There are tweaks that you can do to counter the effects, such as expand the nonpaged pool or even expand the virtual machine's perceived physical memory.
The Virtual Display
The virtual display is an implementation of a display driver that renders only to a buffer in system memory. I refer to the display as "virtual" because architecture does not use a physical monitor or graphics card. This can be done simply using a display driver, memory-mapped file, and user-mode application as the display monitor.
The idea quickly grew to include virtual machines. This technique could be used to easily implement a second monitor if you were able to share the Local Video Buffer (LVB).
Figure 8 illustrates the architecture of the Windows display. The graphics display driver links against win32k.sys and is only allowed to use Eng* APIs. The display miniport driver generally links against videoprt.sys and uses the VideoPort* APIs.
The graphics display driver is responsible for rendering while the display miniport is responsible for enumerating devices and managing device resources. Notice the lines from both drivers lead to the graphics display card hardware in Figure 8. To increase performance, both drivers communicate directly with hardware to perform their various operations.
Figure 9 shows what the source code acccompanying this article does (see http:// www.cuj.com/code/). The display driver will be installed into VMWare as a second monitor. The display driver allocates memory to share with applications in the host operating system. An application running on the host then uses the memory-mapped file provided by VMWare to share and display the LVB to act as the second monitor.
Allocating The Shared Region
The shared-memory region can easily be allocated using the MmAllocateContiguousMemory API from most standard kernel drivers. This interface will successfully allocate physically continuous memory that can then be accessed sequentially by an application in the host operating system.
There are alternative methods of allocating memory, such as ExAllocatePool, which are not guaranteed to be physically continuous. This is still an option; however, the segmentation would need to be provided to the host application in order to properly access the memory.
Getting the Physical Address
The MMGetPhysicalAddress method presented here can be used to get the physical address from a virtual base address. The physical address will then need to be provided to the host using any variety of methods, such as using a TCP connection. This is where the architecture usually requires an IPC type of communication.
Using the Shared Region
The host application needs to open the file mapping to the case-sensitive memory-mapped file VMwareMem<PID>Memory. The host application would need the location of the physical memory address and the PID of the vmware application. These can be provided through any means necessary, either manually, using IPC, or other operating-system features.
To map a view of a file you need to be aligned on a 64k boundary. This means you essentially have two options: fix the address length and start address so that the math works out or simply view the entire mapped file.
The prototype views the entire mapped file and simply uses the physical address as the offset into this view by adding it to the base address. This is also a perfectly usable and simple solution.
Compiling the Prototype
The prototype (available at http://www .cuj.com/code/) uses standard makefile and was compiled using Visual Studio. This should work fine for anyone who has the correct environment setup already.
Building the prototype is straightforward. The first thing you do in the "video" root directory is type "nmake dirs /i". This command creates the binaries and symbols directories. You can then type "nmake" to build the entire project.
Installing the Driver
To install the driver, copy the DLL, SYS, and INF files to the guest operating system. The Add Hardware Wizard can then be used to manually install the files by using the Add Disk feature. The device to install is the Toby Opferman VMWare Graphics.
Using the Second Monitor
The prototype comes with a user-mode application to be run in the host operating system. This application asks you for the PID and the physical address of the LVB in the guest operating system. The PID can be found using the task manager. The video driver, however, puts the physical address into the current users' registry key "\Software\ Opferman". If you manually enable the second monitor, this will be under "HKEY_CURRENT_USER"; however, if the driver starts on boot, this will be "HKEY_USERS\ .Default".
Enhancements
The code provided is a proof of concept, not a fully functional product. There are some suggested enhancements that could be made to make it more suitable for use.
- Artifacts and optimization. You may notice that the video display driver contains little code. This is because most of the code is simply calling the software emulation provided by Microsoft's GDI driver. There are a few bugs here that should be fixed, as well as possible optimizations, but this is the slimmest implementation that could be implemented.
- Multiple display modes. The source currently only supports one video mode. This could be expanded to support as many display modes as you want. However, this would then require a secondary control channel between the application in the host and the driver in the guest for these types of control communications.
- Mouse support. The mouse is currently not supported on the second monitor due to limitations, possibly by VMWare's software. This should be researched and the proper implementation and hooking done to achieve the ability to move the mouse beyond the first screen. Keyboard support is obviously not an issue.
- Automatic discovery. The automatic discovery of VMWare sessions and locations, which include the operating system that is running, would be a nice addition. The guest operating system does not even need to be Windows in this case. This may require the ability to associate the IPC (TCP Network) with a VWare application PID.
Conclusion
The use of a shared-memory region can boost performance and provide ease of implementation in some architectures. This is not intended for all solutions but should be considered as a possible option.
References
- http://www.vmware.com/.
- http://www.sysinternals.com/.
CUJ
