System1 and System2
The Windows-based System1 consists of the following programs:
- The Generator, which generates statistical data sets, written in Visual C++.
- The NIST statistical suite, which processes the data sets generated with the Generator, written in C.
- The Extractor, which extracts and summarizes the results generated from the NIST statistical suite, written in Visual Basic 6.
For its part, System2 works as follows:
- Data sets are generated with the Generator on the Windows machine.
- These data sets are then transferred to a shared hard-disk partition on the Linux cluster.
- The NIST statistical suite processes the data sets on the Linux cluster.
- The results are then transferred back to the Windows machine.
- Finally, the data are extracted and summarized by the Extractor on the Windows machine.
Our biggest challenge here was how to get System2 programs to communicate with each other, and how to automate the system as much as possible. To accomplish this, we had to modify the current programs and write new ones. In a nutshell, we decided to divide System2 into three parts:
- Data sets generation and transmission to the Linux cluster.
- Processing data on the Linux cluster.
- Transmitting results to the Windows machine for analysis.
Once we generated the data using the Generator, our next problem was how to transfer it to the Linux cluster. We tried several techniques, including generating all data sets, then transferring them via the Samba client. But the problem here is that we had to wait until all the data is generated (which could take a couple of days), then start the Samba client from a Linux emulator under Windows (cgywin). Clearly, this didn't suit our needs.
In another attempt, during generation of the data sets, we started WinSCP using the command, "Keep the remote directory up to date," and changed the priority of the WinSCP to a higher priority than that of the Generator, to transfer the data once it is created. The problem with this approach was that we ran out of space on the Windows machine, as all the generated data sets are stored on the Windows machine.
Finally, we modified the Generator by adding the function MoveData() (Listing One), which calls a WinSCP script (Listing Two) to copy the data sets from the Windows machine to the Linux cluster. It then deletes these data sets from the Windows machine (to free up space for more generated data sets). MoveData() is then called after each group of data sets is generated.
void MoveData()
{
system("F:\\WinSCP3\\WinSCP3 /console /script=c:\\move.txt");
system("del c:\\test\\* /q");
}
# Automatically answer all prompts negatively not to # stall the script on error # option batch on # Disable overwrite confirmations that # conflict with the previous # option confirm off # Connect a stored connection called mido # (where the password was saved) # open mido # Change remote directory # cd /users/disk2/et/midono1/test # Force binary mode transfer (It is very # important for us to transfer the data in binary # format) # option transfer binary # Upload the file to current working directory put c:\test\* # Disconnect close # Exit WinSCP exit
