With eight business divisions, more than 370,000 employees, and nearly 300 worldwide major production and services sites, Siemens has many systems and technologies, creating a complex infrastructure. Siemens drives its IT activities in a centrally shared service organization (GS IT) with delivery centers (centers of excellence or CoEs), demand units (business partner organizations), and a governance unit consolidated to create a unified IT infrastructure. Within GS IT under the CoE for divisional ERP systems, the service management organization is specifically responsible for defining process frameworks and guidelines for the operation and maintenance of more than 60 SAP systems worldwide — including roughly 20 systems that run core, mission-critical business processes requiring continuous upgrades and enhancements to keep pace with process changes.

While central IT consolidation helped reduce complexity by simplifying systems management and maintenance, testing still presented major challenges. One imperative issue was regular support package upgrades that required heavy business unit involvement. Another was the company’s shift to an agile development methodology for projects to shorten the innovation cycle, which required more testing than its previous waterfall approach. And Siemens was also moving many small process changes into production which, because of tight SAP process integration, tended to have a downstream effect on later project stages.

Before IT consolidation, Siemens relied mostly on manual testing, but realized that an anticipated heavy volume of technical upgrades for complex SAP systems, support packages, and SAP HANA upgrade projects to include a planned move to SAP S/4HANA would strain resources. The company was devoting the full-time equivalent of roughly 1,000-person days to carry out required testing for complex SAP systems and each project.

In 2013, Siemens completed a small-scale automated testing project for an SAP ERP 6.0 upgrade and, based on its success, adopted Worksoft Certify for a larger project — and ultimately across its entire SAP landscape. Siemens had evaluated and tested several other solutions prior to the upgrade project, but found the speed and ease of use of Worksoft Certify as well as its support and experience for SAP S/4HANA upgrades to be key differentiators. Business analysts could map and create test scenarios after roughly 20 minutes of training without any programming or prior experience with the solution. “After that first project where we saw promising initial results, we moved to the next SAP system, which entailed a much larger consolidation project involving the integration of a separate SAP system into a core divisional system,” says Jens Neuhaus, Director of Test Automation Service at Siemens. “This project was a good candidate for test automation because of its multiple test cycles, and we achieved 60% coverage of the core functionality in the SAP system that was automated using Worksoft Certify tests.”

This consolidation effort was named ATLAS Lincoln, and resulted in the company’s first set of core end-to-end automation for integration and regression testing. The 60% coverage of core functionality represented roughly 350 separate business process transactions and 10,000 test steps specific to the power and gas division, covering locations in Sweden, Germany, and the Netherlands.

A Mindset Change

Today, Siemens is using Worksoft Certify to support end-to-end testing for more than 20 SAP systems. Beginning with a single upgrade project before advancing to ATLAS Lincoln was a helpful strategy in building a business case for expanded test automation, Neuhaus says. Siemens was able to demonstrate gains with each successive project and quantify the benefits and savings achieved with end-to-end automated testing of standardized business processes. “With every test cycle, we gain more from test automation,” he says. “And we found the best candidates for automated testing are those core processes that are standardized across many organizations, like value-added tax determination.”

To reinforce the benefits of automated business process testing, Siemens stressed the importance of moving away from institutional knowledge — unwritten rules or information known to some individuals within Siemens but not common to others — as the lone source of high-level test design. Previously, a tester might take for granted that he or she knew all the downstream process steps if there was a discrepancy in a bill of materials, for example. One of the key benefits of test automation, of course, is to increase the complexity of test scenarios far beyond what an end user would be capable of handling manually.

“Business users who design automated test cases need to be aware of the variables that prevent interaction during testing,” Neuhaus says. “They need to know that these variables are now accounted for in the tests.” This is especially important for a conglomerate like Siemens, according to Neuhaus, because the company is so large that there are very few people with a complete understanding of what end-to-end processes look like — such as purchasing, invoicing, or distribution.

Neuhaus says that bolstering test documentation through Worksoft Certify’s integration with SAP Solution Manager was instrumental in the company’s ability to change the testing mindset and solidify users’ trust in test automation. Further, the documentation of system changes helped satisfy audit and other IT general controls (ITGC) requirements. “When we implement a change, we store the technical specifications and process documentation in SAP Solution Manager,” he says. “Because we can link test scenarios from Worksoft into SAP Solution Manager quite easily, we use it for planning, tracking, and monitoring test execution. And by having process documentation in SAP Solution Manager, we have the foundation for test management because test designs rely on that process and technical information.”

A Short-Cycle Fix

One of the biggest unanticipated benefits of the test automation program at Siemens has been the use of automated business process tests for smaller process changes or bug fixes. If there is an error in the creation of a process order, for example, the legacy process was to limit testing to a spot check of that specific functionality. Siemens discovered that end-to-end business process test automation could also be used for all these small bug fixes.

“If we have a process with 1,000 test steps, on a weekly basis, we can organize that for every small bug fix and run an entire end-to-end test,” Neuhaus says. “That’s the really big benefit, so it’s not just large annual or business-driven projects, but also smaller changes where test automation increases the quality of what we deliver.”

That benefit, Neuhaus says, lines up with the company’s larger strategy of using test automation to support a move to agile development. “The motivation was to align with Siemens’ larger objective of using test automation to support a more strategic transition to agile development, which leads to much shorter innovation cycles or change cycles during a project,” he says. “That dramatically increases the test effort because for every sprint or wave, there’s another test cycle that automation can help cover.”

To handle the increased testing workload, Siemens expanded its GS IT testing requirements, such as developing a collaboration model for every SAP system that is supported with test automation. The new collaboration model calls for a dedicated test coordinator and test architect to ensure adherence to proper testing protocols.

Close collaboration with the business organization represents an evolution from the company’s initial view that using Worksoft Certify was a technical undertaking requiring Basis experts or other developers, according to Neuhaus. And it did not take long to dispel this notion. “At first we thought test automation was very technical, but we came to understand that Worksoft is not a developer tool but rather an intuitive, easy to learn tool for business users,” he says. “That was a big takeaway for us.”