Dr. Dobb's | Software Development In the Cloud

Software Development In the Cloud

Cloud management and ALM

September 03, 2011
URL:http://drdobbs.com/tools/software-development-in-the-cloud/212900736

Nick is a Senior Solutions Architect with CollabNet and can be contacted at [email protected]. Darryl is a Senior Systems Architect Consultant with CollabNet and can be contacted at [email protected].

Similar to how virtualization began to take hold in the engineering lab, cloud computing is taking root with the more experimental crowd in software development. The reason for this is obvious: Development teams are quick to jump onto any leading edge technology that solves their challenge of needing dynamic, flexible tools and processes. Development computing environments require more adaptability -- systems specifically must accommodate shorter project sprints, be less static and more configurable-on-the-fly, and support collaborative principles -- in short, on-demand resources that can be shared across teams, managed by development, and have traceability across projects.

Access to flexible, on-demand cloud computing resources, either from a virtual private cloud within the corporate data center or from public clouds, can provide just such a flexible environment. While clouds are interesting on their own (who doesn't want all of Amazon's computing resources available to them?), having tools for managing cloud resources (such as tooling and workflow) is key. 'Cloud management for development' brings teams the level of control and visibility necessary in this new landscape.

With cloud management for development, teams ultimately drive the allocation and provisioning of their systems -- on-demand -- as they need them. They can utilize the pooled physical and virtual machine capacity of a cloud for more flexible automation and reuse across projects. This means less time spent configuring and finding errors when software moves from one stage to the next. The benefits of thinking of servers as 'clouds' or 'pools' of virtual resources and version-controlled configurations include:

Managing and controlling entire development, build, and test processes from one interface means capacity can be monitored and optimized.
Storing and managing profiles (configurations) as reusable assets across a project or across multiple projects, easily accessible by any developer in the project or organization.
Visibility from the project-- and management-team levels as to how the resources are being used, with the ability to charge back resources per project if desired.

In this article, our focus is on how to manage virtual private clouds for Application Lifecycle Management (ALM). We generally define virtual private clouds as groups of public or private server pools from your corporate data center or from public clouds like Amazon EC2. To these server pools, developers would apply some software stack or configuration required for their task at hand. A typical enterprise use case for managing their cloud might be:

To control costs, manage security, or supplement resources during peak use, engineering or project managers set up what clouds a project may have access to.
Individual hardware resources can be assigned to a 'cloud' that now has the capacity of their combined resources. Dell class servers could be allocated to a development cloud, while HP blades with a higher service-level agreement and security could be allocated to a production cloud. The cost associated with a machine in the former is about $0.10 per hour while the latter is $0.25 per hour.
A project manager (or admin) allocates portions of either cloud to projects as the development or production configuration. The manager can limit projects to certain clouds; for instance, services from the development cloud can be used for development purposes only.
Finally, cost accounting data can be derived by tracking usage by project and user, so managers can optimize assets at varying stages and projects.
Amazon EC2 can be used to extend resources temporarily (and at a very low cost).

To complete the concept of cloud management for development, let's also introduce the concept of Development Services or Build and Test Services. Development Services consist of code, build and test tools, applications, and infrastructure stacks that can be stored and managed as configurations or profiles, and applied to an available server. These profiles can be accessed and used globally and version controlled for consistency across the application development lifecycle. So Development Services are simply software configurations or profiles applied to an available server in the cloud, on demand.

Development In the Cloud: A Practical Use Case

One use case for software development in the cloud combines the agile best practice of continuous integration (CI) and some common collaborative development tooling that has been used across companies of varying sizes, locations, and industries. This approach is scalable and flexible around the size of your development team and type of products you are building.

The cloud development use case encompasses the flow of defects/requirements through phases of development/builds/tests and back to submission of new requirements or defects by various stakeholders. Automation at any point possible is a key capability, including the ability to 'turn on' and 'rip down' virtual or physical systems as needed, in a cloud.

Figure 1 illustrates the workflow for development in the cloud through the perspective of the various contributors, along with their collaborative and cloud management tools.

[Click image to view at full size]

Figure 1: Software development in the cloud workflow and stakeholders.

Business Analyst/Quality Engineer (BA/QE) submits defects/requirements.
Project Manager (PM) picks tasks, sets priorities, and assigns them to development.
Developers open their favorite IDE and view their tasks. They begin to work on the defect and write code.
Developers use the cloud-management platform to build and test code.
They merge code and change sets.
They commit code to a source-code management tool (here, Subversion). This triggers a continuous integration that takes place using the cloud management and build automation tool (Figure 2).
CI tool monitors for code changes (Figure 2).
Upon build failure, defect tracker is updated and notification is sent to the development team (Figure 2).
Upon successful build, the defect tracker is updated automatically. If the test succeeds, test results are e-mailed to PM/QE. If the test fails, QE is notified (Figure 2).

Virtual Private Clouds

So what part of this happens in the virtual private cloud?

Step 4 is where the developer may configure his own cloud system on demand for build and test.
Steps 7, 8, and 9--the CI tool and build automation -- run on the cloud systems.
Once a build is successful, the artifacts are uploaded automatically to the Project Build Library on the cloud systems. The Project Build Library (PBL) stores files created and used by the CI process, which can be shared with others who may need to access the build results.
A test system can be dynamically provisioned and build artifacts are downloaded and tested automatically.

Figure 2 illustrates what the CI in your private cloud might look like, with virtual and physical machines and tools for source code, tracking, build, and test.

[Click image to view at full size]

Figure 2: Continuous integration in a virtual prviate cloud.

Setting Up a Cloud Development Environment

We now present an overview of some of the tools and principles to consider when modifying your own environment for development in the cloud. Basic principals include:

All stakeholders should be able to see and make progress on the code and executables in real time without having to replicate data or change ownership.
A version-control system should be integrated with build automation tools.
Simple branching strategy (parallel development) for development, integration, and release work.
Make incremental changes and test often.
Focus on small releases that provide the most business value.
Utilize aggressive milestone management, such as the agile imperative of time boxing (for more details, see the text box entitled 'Time Boxing').

Time Boxing

Time boxing involves splitting projects into separate time periods of 2-6 weeks, each with their own deadline and budget. Deadlines are fixed but deliverables are adjusted. To meet the fast turnaround times and project splitting, engineering must allocate and configure systems quickly. If constrained by access to these systems or incorrect configurations, short iterations are impacted. On-demand cloud management server pools can mitigate this.

Taking advantage of software development in the cloud does not require massive retooling--that's one of the benefits. You may already be using many of the basic collaboration and change-management tools commonly used by software teams worldwide. (As a disclaimer, many of the tools we talk about are those that CollabNet sells.) Table 1 lists the tools.

[Click image to view at full size]

Table 1: Tools for software development in the cloud.

In general, requirements for tools should include:

Support for heterogeneous environments, so teams can have flexibility across the organization, based on specific project requirements.
Adaptive to current workflows and tools. Developers and QA engineers want to stay in familiar environments.
Access to complementary tools and configurations across the development lifecycle.

For instance, CUBiT (CollabNet's Cloud Management for Development Services) can be configured to manage a virtual private cloud with your own corporate computing resources, expanding to a public cloud such as Amazon EC2 if desired. This type of capability is key for automating any development environment. With such automation, overhead for software teams to obtain and manage computing resources is reduced with the Development Build and Test Services previously described. New systems can be provisioned from the predefined profiles within minutes while maintaining corporate security, auditability, and traceability.

Design goals for the cloud environment include:

Teams must have autonomy over their own resources, and must have minimal delay in provisioning their own systems.
Strict access controls so that resources are dedicated to project teams or to a dedicated purpose (such as build or test).
Charge back for resources used. Even if the organization doesn't charge back, teams can begin to understand the resource costs associated with their projects.
Using agile processes such as continuous integration (CI).
Automated build and test.

Figure 3 illustrates how CUBiT manages development clouds of build and test services -- combinations of physical and virtual machines with profiles.

[Click image to view at full size]

Figure 3: Cloud management with CUBIT.

Software developers within their own projects have access to their own dedicated systems that are dynamically sharable among project members. CUBiT management features include visibility of all system resources within cloud, auditing, capacity monitoring, role-based access, and accounting with charge back. Profile management enables version control and traceability for configuring predefined development stacks within minutes.

Step 1: Configuring a virtual private cloud. Using CUBiT, you can configure multiple clouds for different use cases, and have the ability to control the usage of those clouds. For example, we could create clouds for continuous integration build systems and assign them as an exclusive resource to that project, or share it among any number of projects.

Step 2: Giving developers easy access to cloud resources through their IDEs. CollabNet provides a free plug-in to IDEs such as Eclipse and Visual Studio that lets you seamlessly access the source-code management tool, collaboration platform, and development services (Subversion, SourceForge Enter prise, and CUBiT, respectively) without having to leave your IDE. From the IDE, you can browse all system resources within a cloud or CUBiT domain, physical or virtual systems. The management interface can be viewed directly from within the IDE; in addition, the systems can be securely accessed.

Step 3: Creating reusable profiles for on-demand build and test services. By creating reusable predefined configurations explained as profiles earlier, systems can be consistently provisioned on demand, hence the concept of build and test services. Profiles are defined in an XML format and are maintained in a Subversion repository under version control, so systems can be restored exactly to a previous state if needed.

Example 1 shows how a continuous integration build system can be provisioned from CUBiT. This example installs Java SDK, CruiseControl, Apache Beehive, Apache Derby, and Tomcat -- a full development and build stack. Typically, such a system can be provisioned in less than 10 minutes, thus meeting the demands of an agile development team.


<rpms action="install"path=
"pbl://mgr/pbl/cubit_demo1/pub/cruisecontrol-apache/">
<rpm>jdk-1_5_0_15-linux-i586.rpm</rpm>
 <rpm>cruisecontrol-bin-2.7.3-1.i386.rpm</rpm>
<rpm>apache-beehive-1.0.2-1.i386.rpm</rpm>
<rpm>db-derby-10.4.2.0-1.i386.rpm</rpm>
<rpm>apache-tomcat-5.5.27-1.i386.rpm</rpm>
</rpms>

Example 1:A continuous integration build system.

Step 4: Using CUBiT Web Services to automate cloud provisioning for testing services. CUBiT has an extensive set of REST-based APIs that let you automate what would normally be complicated procedures. For example, you can dynamically provision build systems in the cloud, then destroy those systems once testing has completed. This is useful for testing milestone builds that require a 'clean' system in a known good state.

Step 5: Determining how cloud resources are being used and charged back. Finally, tools like CUBiT give teams and managers the ability for each system or profile to be used on an allocated/hour basis. Even if this feature is not used to charge development teams, it is still valuable for determining the usage and allocation patterns of your resources. In Figure 4, CUBiT graphically reports (in either a pie chart or table) who is using the resources in any given project and the associated costs. This ability becomes more important as organizations begin to dynamically allocate computing resources across projects that require managing their own virtual private clouds.

[Click image to view at full size]

Figure 4: CUBIT report.

Conclusion

Software development in the cloud brings adaptability and flexibility to any size project team. By utilizing some common software development tools and a new one that introduces the concept of cloud management and development services, teams can build their own cloud development environment with continuous integration automation and 'roundtrip' feedback to provide on-demand access and visibility across the development lifecycle.