Many companies already take advantage of SAP HANA to accelerate analytics and the decision-making processes. The next wave of SAP HANA adoption has already started by leveraging the speed of the in-memory database of SAP HANA for SAP ERP HCM and other solutions of the SAP Business Suite. As a consequence many SAP companies consider using SAP Business Suite on HANA migration projects. These projects require a professional and elaborate testing approach that addresses technical, functional, and non-functional needs. The goal of such a testing approach is to ensure a smooth introduction and operation of SAP HANA at minimal costs. This report provides guidance about how to assess your company’s future needs and how to make decisions about which SAP test tools (and its partners test tools) are adequate, as well as showing how to complete the required test management setup and execution activities. SAP offers two test options to support all testing-related activities, as shown in Figure 1.

• SAP Test Option 1– SAP Solution Manager Test Workbench: SAP users who select SAP Test Option 1 are using test management capabilities provided by SAP Solution Manager. In this setup, test cases for single business transactions or end-to-end business processes are assigned to the Business Blueprint at the appropriate level that is the basis for all subsequent test management activities.
• SAP Test Option 2 – SAP Quality Center by HP: SAP users who select SAP Test Option 2 are using test management capabilities provided by SAP Quality Center by HP and integration with SAP Solution Manager. In this setup the company transfers the Business Blueprint nodes from SAP Solution Manager to the Requirements module. The test cases are defined in the Test Plan module and assigned to the items of the Requirements module, which is then the basis for all subsequent test management activities.

SAP HANA Testing and Validation

Companies planning to migrate their current SAP Business Suite solution from a traditional database to SAP HANA need to perform test and validation activities. For the purposes of this report, we have divided the required test activities into three distinct project phases (Figure 2):

1. The test management preparation phase.
2. The test execution phase (during the SAP Business Suite on SAP HANA migration project).
3. The after go-live test activities.

The Test Management Preparation Phase

During the preparation phase, as well as making decisions about what approach and tools to use, you should follow these preparation steps to ensure that all the test-management phases are adequately covered:

• Business Blueprint
• Test scope identification
• Functional regression testing
• Performance and load testing
• Service testing

SAP recommends dividing the preparation phase into separate work streams for each of these phases. After the status assessment and the tools decisions are made, test management setup and planning activities should be performed. At the end of the preparation phase the technical steps of the SAP Business Suite on SAP HANA migration project can be started.

The Test Execution Phase

During the SAP Business Suite on SAP HANA migration project phase the test team executes functional tests, performance and load tests, and service and interface tests. Business analysts are involved in integration testing (Figure 3). Incidents that occur are analyzed, problems are resolved, and tests are performed again. At the end of this phase (assuming the tests results are successful) the test team signs off on the test activities.

The After Go-Live Test Activities

The final testing phase is performed after the go-live is complete (Figure 4). SAP HANA maintenance activities are provided via SAP HANA revisions. Regression tests are required at the technical or business process level to validate the maintenance activities. Smart test scope identification techniques are required to avoid having to perform full regression tests, which are not feasible due to time and effort constraints.

Business Blueprint

It is highly recommended by SAP that the test activities to technical objects not be limited, and include business process tests to the overall test scope. Business process experts are required to be involved in functional integration tests. For this reason manual test cases need to be expressed in business language and assigned to the business-process hierarchy. Automated test scripts used for efficient regression tests are assigned to the appropriate business process of the business process hierarchy, which enables risk-based test scope identification (Figure 5).  

Status Assessment of the Current Business Blueprint

In the first step, the existing business process hierarchy needs to be assessed. The following aspects have to be determined:

1. Percentage of business processes by functional area to be covered.
2. Percentage of mission critical (priority one) business processes.
3. End-to-end documentation of business processes that require predecessor and successor process-step information for end-to-end test execution.
4. Which executables (e.g., transaction codes and reports) are to be assigned to each process step.

If the first assessment does not result in a high coverage percentage of all four of these aspects, we recommend you identify business process experts from all the functional areas who should assist with the set-up activities for the Business Blueprint areas that are lacking.

Business Blueprint Setup

For efficient test scope identification and subsequent test execution, SAP recommends defining the Business Blueprint in SAP Solution Manager. In case of a partial or a missing Business Blueprint you have several alternatives for Business Blueprint setup in SAP Solution Manager:

• Manual: Manually setup business processes and executables using transaction code SOLAR01 based on interviews with business process experts.
• Transfer: Transfer the existing Business Blueprint from other sources such as ARIS or SAP Quality Center via the HP test plan module.
• Upload: Upload Microsoft Excel files that contain descriptions of the business processes.
• Reverse Business Process Documentation (RBPD): Use this service from SAP (see below for more detail).
• Solution Documentation Assistant (SDA):  Use this SAP Solution Manager application (see below for more detail).
• Scope and Effort Analyzer: Use this SAP Solution Manager application. This is scheduled to be available sometime in 2014 (see below for more detail).

RBPD and SDA

With RBPD, SAP provides a service to document business processes using execution statistics of your SAP production systems. For this purpose, SAP provides content of business processes and process-specific selection rules that can be downloaded from SAP Service Marketplace. Alternatively, companies can use the SAP Solution Manager application SDA directly to evaluate business process execution from production systems and semi-automatically update the Business Blueprint in SAP Solution Manager (Figure 6).

SAP Solution Manager Application: Scope and Effort Analyzer

This new SAP Solution Manager application, the Scope and Effort Analyzer, is planned for delivery in the first half of 2014 (Figure 7). It provides the functionality to automatically generate an SAP module-oriented blueprint based on executable usage statistics. The resulting blueprint is functional-oriented and does not provide end-to-end business processes. With the help of business process experts you can model end-to-end business processes using the generated blueprint structure as your starting point.

Test Scope Identification

The goal of the test scope identification work stream is to identify the test scope, which results in test plans for the functional tests. Following SAP Best Practices two test plans will be created: Functional Test Plan 1 for critical business processes and Functional Test Plan 2 for modifications and custom code.

Functional Test Plan 1 (for Critical Business Processes)

The database migration from a traditional database to SAP HANA could potentially affect all areas of the business solution. In order to narrow down the test scope to a manageable level, it is recommended by SAP that you first derive metrics for solution areas of high priority and intensive usage. The following metrics might help your company to identify the test scope:

• Business process priority or criticality (ABC classification)
• Number of affected users of business transaction or business process (ABC classification)
• Usage frequency of business transaction or business process (ABC classification)

The ABC classification may use thresholds such as:

• Class A: Top 15 percent
• Class B: Top 50 percent
• Class C: The remaining percentage

As a best practice, SAP recommends that you (Figure 8):

1. Define one to three metrics as custom attributes for the Business Blueprint.
2. Assess the business processes priority and criticality with the respective business process experts.
3. Assess the usage frequency with SAP tools such as the Workload Monitor (transaction code ST03N).
4. Assign the metric with its value (e.g., ABC classification) at the appropriate business-process level.

Functional test plan 1 should include test cases assigned to business processes with class A classifications of the ABC classification assessment of business processes. Manual and automated test cases are assigned to the appropriate business processes and process steps (Figure 9).  

• SAP Test Option 1 – SAP Solution Manager Test Workbench: Manual and automated tests are assigned directly in the Business Blueprint of SAP Solution Manager at the business-process or process-step level. The test plan is generated using the assigned ABC classification as filter criterion.
• SAP Test Option 2 – SAP Quality Center by HP: The Business Blueprint (defined in SAP Solution Manager and limited to Class A processes) is transferred to the SAP Quality Center by HP (SAP Quality Center by HP). The Business Blueprint structure is then available in the SAP Quality Center by HP Requirements module. Manual and automated test cases (created with HP Quick Test Professional [QTP] and SAP Test Acceleration and Optimization [TAO]) that are available in the Test Plan module are linked to the Business Blueprint stored in the Requirements module. In the last step a Test Set is generated based on test cases assigned to class A processes.

Functional Test Plan 2 for Modifications and Custom Code

Besides your important business processes, you need to test business transactions that include modifications and custom code. This aspect is of equal high importance since the migration from a traditional database to SAP HANA might affect transactions that include modifications or custom code.

Static Code Analysis with the ABAP Test Cockpit (ATC) and Usage and Procedure Logging (UPL)

In preparation for your SAP Business Suite on SAP HANA migration project, SAP recommends you assess your modifications and custom code with the ABAP Test Cockpit (ATC) shown in Figure 10. ATC is an ABAP check toolset for performing static checks and unit tests of your ABAP code to identify high-priority quality issues. ATC is fully integrated in the SAP workbench (including transaction codes SE80, SE24, SE38, SE37, and SE11) and allows easy interpretation, navigation, and filtering of ATC findings. ATC is available with enhancement package 2 for SAP NetWeaver 7.0 SPS 12 and enhancement package 3 for SAP NetWeaver 7.0 SPS 05, and is positioned to be the successor to the SAP Code Inspector.   With ATC you can determine the priorities of the quality issues of your modifications and custom code. In the context of the SAP HANA migration project, the ATC code scan, for example, reveals  problematic SELECT statements for pool and cluster tables that are transformed into transparent tables, but do not return the selection results in an ordered fashion. The second section of the test plan 2 includes business transactions with ATC checking the discovered priority 1 issues. Besides the priority ranking of ABAP code quality issues of modifications and custom code, a usage frequency assessment can be used to identify the most executed code objects that require code adjustments to ensure optimal performance after migration. For this purpose, SAP recommends you activate Usage and Procedure Logging (UPL), which logs all called and executed ABAP units, such as programs, function modules (down to classes), methods, and subroutines (Figure 11). UPL is available in any ABAP-based system and is based on the core functionality of the SAP Coverage Analyzer (for system prerequisites see SAP Note 1828848 and for UPL deployment see SAP Note 1683134). You can work with ATC and UPL in combination to identify top priority ABAP code issues via ATC and limit the ATC result via UPL usage above a certain threshold. In the current release of SAP Solution Manager this merge needs to be done manually, but the joined functionality is planned for a future Support Package of SAP Solution Manager 7.1. Customer Use Case: A Germany-based company using SAP Test Option 1 prepared the test plan for its SAP CRM on SAP HANA migration project using the following approach. As preparation for the migration project it activated UPL in the SAP CRM system well in advance to provide a UPL database with several months of history (Figure 12). For this period, UPL revealed the use of more than 20,000 custom code and modification objects such as function modules and methods. The company performed an ABC classification with the following thresholds:

• Class A: More than 100 million uses
• Class B: More than 1 million uses
• Class C: All other uses

  Subsequently it performed an ATC (Code Inspector) code scan for custom code objects of usage class A (approx. 300 objects) and usage class B (approx. 1000 objects). As a result, they found several hundred ATC priority 1 and 2 issues for their custom code that have been analyzed and adjusted to a certain degree. The business transactions and reports that include the adjusted custom code have been included in the test plan for subsequent regression testing. In an ABAP-based SAP system, a large number of different SQL requests are executed by the most diverse processes. To find performance hotspots and also identify potential optimizations in this very broad SQL-load profile, you need specialized SQL monitoring tools that also provide a connection to the running ABAP processes. The new ABAP SQL Monitor (Figure 13) allows the aggregated logging of all ABAP-based SQL executions in a live ABAP system. At this point, aggregated performance key figures (e.g., number of executions, execution time, and records read) are collected per ABAP SQL command. Furthermore, the entry point of the related process (e.g., the transaction code, report name, RFC module, or URL) is stored for each aggregated SQL record. After a certain active SQL monitoring phase (e.g., week or month), the collected SQL Monitor data can be used to answer the following questions:

• Which SELECT statement in custom code has been executed most frequently in the SAP production system?
• Which SELECT statements cause the longest runtime?
• Which SELECT statement in custom code reads the most data?
• What does the SQL profile of the report XYZ or transaction code ABCD look like?

All this performance and usage data can be used to optimize existing ABAP programs for SAP HANA. SAP recommends complementing static code scans of the ATC with dynamic runtime assessments of the SQL Monitor for custom code tables. Business transactions and reports including hot spots identified by ATC and SQL Monitor should lead to custom-code adjustments and therefore need to be included in the second section of the regression-test plan 2. The data collection of the SQL Monitor is implemented directly in the SAP NetWeaver kernel in a highly optimized manner to ensure that productive operations are not disrupted. The SQL Monitor is deployed via the plug-in of SAP Solution Manager (ST-PI 2008_1_700 SP08 and higher) on SAP solutions such as SAP ERP. SAP Note 1885926 describes the system prerequisites such as the SAP kernel upgrade from SAP NetWeaver 7.20 to 7.21. The SQL Monitor can be executed with transaction code /sdf/sqlm.  

Static and Dynamic Code Analysis with the SQL Performance Tuning Worklist (SWLT)

Based on the feedback from SAP HANA migration projects, SAP has combined the applications for static and dynamic code analysis (ATC and SQL monitoring) for the purpose of SQL performance-tuning into one unified application named the SQL Performance Tuning Worklist (transaction code SWLT), shown in Figure 14.     Transaction code SWLT correlates the static code checks covering the golden rules from the ATC with runtime information such as the number of executions, the execution time, and the fetch count of the SQL Monitor. In addition, SWLT creates a weighted worklist of code occurrences that need attention and provides a code adjustment proposal. SWLT is available with SAP Basis 7.02 SP14, SAP Basis 7.38 SP09, and SAP Basis 7.40 SP02.

Business Process Change Analyzer (BPCA)

SAP strongly recommends including a regression-test section into test plan 2 for business process tests with adjusted custom code and modifications. The goal is to have no negative impact on critical business processes and at the same time reduce the regression-testing effort as much as possible. The most efficient way to create a risk-based and effort-optimized test plan is with the BPCA of SAP Solution Manager (Figure 15). Follow these steps:

1. Business Blueprint: As a preparation step, process traces are performed (e.g., a technical bill of material [TBOM]) for each process step or business process. With SAP Solution Manager 7.1 SP10 the user can automatically create semi-dynamic TBOMs via background jobs. In addition, manual or automated regression test cases shall be assigned to business processes or process steps. When using SAP Quality Center by HP, the appropriate test cases are available from SAP Quality Center by HP via SAP–HP interfaces.
2. Transports: Custom code and modifications are adjusted in the development system and assigned to SAP transports
3. BPCA: On the BPCA selection screen, the user enters the transport numbers that include the changed custom code and modifications. The BPCA change-impact analysis then calculates which business processes are impacted by the changed objects based on the trace information (TBOMs) assigned to the blueprint items. This BPCA result can be directly used to generate a test plan. In case you want to reduce the test effort, SAP recommends continuing with a BPCA test-scope optimization which selects only those process steps and test cases that most efficiently test the changed objects.
4. Test Plan: Finally you can generate the test plan directly within BPCA using one of two SAP test options:

• Test Option 1: SAP Solution Manager Test Workbench: Test cases are directly assigned to impacted process steps of the SAP Solution Manager Blueprint and the test plan can be generated automatically.
• Test Option 2: SAP Quality Center by HP: Tests reside in SAP Quality Center by HP and can be selected via new interface which require SAP Solution Manager 7.1 SP05 (or higher) and HP EI 2.7.

 

Functional Regression Tests

Regression testing plays a vital role in SAP Business Suite on SAP HANA migration projects. Besides the standard tests for new functionality including unit, integration, scenario, and user-acceptance tests, companies running SAP need to ensure that all areas operate as expected after migrating SAP Business Suite to SAP HANA. The following is an outline of the required activities—preparation and planning, execution, reporting, and sign-off—for a successful regression test of all the SAP test options. Keep in mind that executing all test activities in either SAP Test Option 1 (SAP Solution Manager Test Workbench) or SAP Test Option 2 (SAP Quality Center by HP) allows full traceability from the Business Blueprint to the test cases and the related defects in all directions, and is the basis for profound reporting, as discussed below.

Test Preparation and Planning

SAP best practices recommend preparing and planning regression tests within SAP Business Suite on SAP HANA migration projects in the following way:

• Test Cycle 1: Manual execution of the functional test plan for modifications and custom code (as defined earlier).
• Test Cycle 2: Regression tests of the functional test plan for critical business processes via automated tests (as defined previously).

Test Preparation and Planning: Manual Regression Testing

To support manual tests in the most efficient way, the following aspects should be considered when setting up manual test cases:

1. Manual test cases need to include guidance on how to execute the single steps of the business transaction and the expected results.
2. Assigning business transactions of the test systems to allow the direct start of test execution.
3. Setting up test sequences in cases of business scenario tests with multiple roles performed by multiple testers.

  As a general rule of thumb, companies running an SAP system should consider defining their manual tests with clear and easy-to-follow step descriptions. The launch of manual tests should be supported with links in the test package to allow for a hassle-free start of the business process in the assigned test system.

SAP Test Option 1– SAP Solution Manager Test Workbench: Manual Test Cases

Manual test cases are assigned directly to its corresponding notes of the Business Blueprint in SAP Solution Manager. The Business Blueprint can be reached easily from the Work Center Test Management > Test Preparation. A new manual test case is created by choosing the Test Document Test Case Type for the process step of the Business Blueprint (Figure 16).   The manual test case (Figure 17) consists of a document that is typically created by a business analyst who knows the business transaction very well. The document can be created in various formats and should include the following details:

• Author and last editor with time stamp
• Prerequisites—what needs to be in place before this test is executable
• Test data for the business transaction or a link where to find the test data
• Information on how to start the business transaction
• Execution steps including user activity and expected results
• The tester needs to understand how the expected test results can be verified

  The test manager selects the appropriate test cases for manual regression tests based on the identified test scope (e.g., the functional test plan for critical business processes). The selection process can be performed semi-automatically by the Test Workbench of SAP Solution Manager. The various selection methods outlined in Figure 18 help the test coordinator to compile a test plan with appropriate test cases.

• Selection method 1 is commonly used in very small SAP Business Suite on SAP HANA projects where a test manager is in possession of all the required information (outlined earlier in this article).
• Selection methods 2 and 3 are highly recommended when working with ABC classification, and whose results have been documented in Business Process Attributes or Test Case Attributes.
• Selection method 4 is frequently used if BPCA is the starting point for the test-plan generation activities.

  In the second step, the overall test plan is divided into test packages with users assigned to act as manual testers (Figure 19).   To enable a smooth start of the business transaction by the manual tester, we recommend that you include the test object (e.g., the transaction code) in the individual test package of each tester (Figure 20). With this approach, the tester can start executing the test by double-clicking the test object in the assigned test system, which not only improves usability but also avoids having to search for the respective systems.

SAP Test Option 2 – SAP Quality Center by HP: Manual Test Cases

Customers using SAP Quality Center by HP (SAP Test Option 2) can use the Test Plan module in order to define manual test cases. Figure 21 shows the recommended procedure if you’re using SAP Test Option 2. The first step is to transfer the Business Blueprint and assigned business requirements from SAP Solution Manager to the Requirements module in SAP Quality Center by HP. In the optional second step, additional test requirements can be defined. In the third step, manual tests are designed, including test steps, activity descriptions, and expected results, as well as attachments to guide the user through the test execution. The Test Plan module is tightly integrated with the Requirements module to assign test cases to business and test requirements, which in turn allow test-coverage analysis by the test coordinator later on. In the fourth step the test manager selects the appropriate test cases for manual regression testing based on the identified test scope (e.g., the functional test plan for critical business processes) and assigns manual testers to the test sets. A manual test case for SAP Test Option 2 (Figure 22) consists of a sequence of steps to guide the tester through the single test activities. For each step it is possible to assign expected results or descriptions about further activities in order to validate results.   Furthermore SAP Test Option 2 provides the HP Sprinter tool (Figure 23), an advanced way of creating manual test cases. During test-case creation the user actions (list) and story board are recorded and added to the test case. During test execution, floating windows provide step guidance for the testers, and test data listed in the test case is automatically injected into the system under test (SUT, SAP GUI only), and even incidents are created semi-automatically.

Test Preparation and Planning: Automated Regression Testing

All critical business processes must be tested before deployment of the SAP Business Suite on SAP HANA migration project in the production landscape. Performing only a manual test is not suitable to achieve this goal. Therefore companies running SAP need to explore their options for automating regression tests. SAP Test Options 1 and 2 allow semi-automatic creation of test cases that can handle complex business transactions without requiring detailed technical expertise. As a result, business process experts as well as outsource providers can handle the creation and maintenance of automated tests to a large degree. To execute automated tests for a set of critical business processes, companies running an SAP system need to identify the relevant data required for data entry during the business-process execution stage:

• Test data planning: Identify data sets with valid test data suitable to run multiple iterations of regression tests to verify the expected system response.
• Test data provisioning: In a second step the test data that is required during test execution needs to be entered in a test data repository which is connected to the selected test automation tool. With this approach companies running SAP systems can provide relevant test data for the core business processes independent from the test cases that consume the test data later on.
• Test execution: During execution of the automated regression test cases, the relevant test data is selected from the repository and used to populate the input fields of the business transaction. With this approach it is possible to verify that the changes deployed in the test system do not damage the business processes included in the scope of the regression test.

As a general rule of thumb, companies running SAP systems should consider automating the regression test cases of their critical business processes, encompassing approximately 100 to 200 process steps. Many companies reported that this can be achieved within a three-month time frame. Companies starting with test automation should consider the following:

1. Avoid long and too complex test cases.
2. Consider scenario tests that consist of single functional tests with parameter handover from one to the next test case.
3. Select meaningful test data.
4. Avoid automating all possible variants (e.g., start with the most important process flows).

SAP Test Option 1– SAP Solution Manager Test Workbench: Automated Test Cases and Corresponding Test Data

With SAP Test Option 1, companies can integrate SAP and third-party test automation tools via the Test Automation Framework in SAP Solution Manager (Figure 24).   With the Test Automation Framework, companies can easily set up test configurations consisting of three essential parts:

1. Test case: Definition of test cases based on SAP test automation applications (Component Based Test Automation [CBTA],extended Computer Aided Test tool [eCATT]), or third-party test automation applications with certified integration into SAP Solution Manager 7.1, like HP Quick Test Professional, Worksoft Certify, or Tricentis Tosca.
2. Test data container: Environment to plan or upload test data consumed by test configurations.
3. System data container: System landscape information controlled by users to allow convenient switches between landscapes that need to be tested (e.g., DEV, TST or QAS).

These test configurations are assigned to process steps or business processes of the business process hierarchy and can be selected for a test plan to allow mass execution (Figure 25). SAP Enterprise Support customers use SAP’s test automation application CBTA for component-based test automation. For UI technologies not supported by CBTA (such as process steps using non-SAP applications), you can use partner test tools from HP, Worksoft, or Tricentis. Table 1 shows functionality included in the Test Automation Framework of SAP Solution Manager 7.1. Furthermore it is highly recommended by SAP that you plan and provide test data through the mature functionality of Test Data Container (TDC) with the following functionality:

• Definition of the test data structure for a set of single fields and structures with reference to the SAP Data Dictionary
• Manual planning of test data as well as Microsoft Excel file uploads
• Test Data Assignment Wizard to map test data stored in TDC with parameters of the test case

The following section provides a step-by-step procedure for automating test cases and corresponding test data in SAP Test Option 1. Step 1. Set up test data containers. In the first step, the test engineer defines the necessary data structure of the TDC. TDCs can be defined for single business transactions or end-to-end business processes such as order-to-cash. Subsequently the business analyst can provide suitable test data within the TDC or upload test data via Excel files (Figure 26).     Step 2. Create test cases. During design time, Test Configurations are created within SAP Solution Manager. After providing header and SUT information the preferred test automation tool is launched to create a test case. It is recommended by SAP that you replace fields that require data input with parameters within the test automation tool. These parameter definitions are sent back from the test automation tool to the Test Configuration in SAP Solution Manager (Figure 27).   Step 3. Test data assignment to Test Configurations. A Test Data Assignment Wizard (Figure 28) supports in finding suitable TDCs, selecting test data variants stored in the TDCs, and assigning them to the Test Configuration. More complex situations can be handled as well, since it is possible to assign data from multiple TDCs that can hold different types of data.

SAP Test Option 2 – SAP Quality Center by HP

Customers using the Quality Center SAP Test Option 2 (Figure 29) for managing the test process can use HP QTP for test automation of heterogeneous business processes including SAP and non-SAP applications. To accelerate the creation of automated test cases and to lower the maintenance effort of automated tests, SAP strongly recommends that you deploy SAP TAO in conjunction with SAP Quality Center by HP and QTP to build automated regression tests (Figure 30). The approach (Figure 30) and advantages of SAP TAO are listed below:

• Business analysts can easily build automated tests via normal execution of business processes – no special technical expertise is needed.
• SAP TAO automatically creates all important test assets in the background.
  • Test components representing sub-screens of the automated business processes with automatically assigned parameters for all fields.
  • Complete composition of test cases based on SAP TAO test components.
  • Test data entered by the business analysts is captured in specially tailored Excel worksheets and are used for later test execution.
  • Validation steps are included automatically into the test case and can easily be added by a test engineer based on customer needs.
• Upload to SAP Quality Center by HP allows customers to use the test management environment of SAP Quality Center by HP to build test plans and test sets based on standard QTP cases and SAP TAO created test cases.
• Maintenance of damaged automated test cases is accelerated by SAP TAO via integration with BPCA, which helps identifying damaged test components. They can be rapidly repaired via SAP TAO (Figure 31).

For companies using Test Option 2 the following advanced capabilities to handle test data in automated test cases are available:

• New test cases are created via SAP TAO by executing a business transaction and entering data for all input fields.
• SAP TAO creates all necessary test assets in the background, including test cases:
  • Test components that represent screens/sub-screens and parameters for all fields.
  • Excel file with above input parameters as column header and a data row that includes the test data from the initial process execution.
• The file with the test data can be placed on a central group server to allow test execution by multiple users.
• Additional test data can be entered directly in the Excel file as additional rows. At runtime, SAP Quality Center by HP executes the test cases as many times as test data rows included in the data file.

The automatic creation of parameters for input fields allows users to build sophisticated test cases in a very fast manner and to assign test data in a very convenient way, since the Excel file already includes the required data structure. The following sections provide step- by-step procedures for automating test cases and corresponding test data in SAP Test Option 2. Step 1. Create test cases (Figure 32). A casual user executes the business transaction. SAP TAO creates test components with parameters for all input fields, test cases using test components in the right order, a file with the test data, and the connection of the test case parameters with the columns of the test data file. Step 2. Test data maintenance. A business analyst or test engineer can enter additional test data in the data file (Figure 33). Users should consider identifying business documents in the production system that reflect standard variations. The data from these business documents can be entered as test data in the test data files.

Recommendations on Selection of Test Automation Tools in SAP Test Options 1 and 2

With respect to SAP Business Suite on SAP HANA migration projects handled with SAP Test Option 1, SAP recommends you use SAP Component Based Test Automation (CBTA) for all SAP GUI (e.g., SAP ERP) and SAP CRM Web user interface (UI)-based (e.g., SAP CRM) test automation activities. Regarding SAP Business Suite on SAP HANA migration projects handled with SAP Test Option 2, SAP recommends you use SAP TAO for all SAP GUI (e.g., SAP ERP) and SAP CRM Web UI-based (e.g., SAP CRM) test automation activities. For systems that are not SAP GUI or SAP CRM Web UI-based (e.g., Portal, BusinessObjects, BW, Crystal Reports) which are often part of SAP Business Suite on SAP HANA Migration projects, SAP recommends you complement CBTA or SAP TAO by using, for example, HP QTP.

Test Execution

Before test execution in SAP Business Suite on SAP HANA migration projects can be started, the required content has to be transferred to SAP HANA. Furthermore, transport requests (e.g., adjustments on interfaces or reports) including implemented changes on the SAP Business Suite on SAP HANA systems have to be transported from DEV to the TST/QAS landscape. As soon as these activities are completed, the test execution can start. The following sections outline how the work of testers during manual and automated test execution can be supported best in SAP Test Options 1 and 2.

Test Execution: Manual Regression Testing

The following criteria should be considered for the execution of manual regression tests:

• Easy access to test cases and business transactions for SAP and non-SAP test systems
• Email notification to inform manual testers to start test execution
• Convenient documentation of test status and test results
• Creation of incidents in the context of the test-case execution with integration to problem resolution

Rule of thumb: Manual testers shall be supported with a convenient test environment that allows efficient and easy-to-learn test handling. All relevant information and activities should be accessible via a push of a button: For test cases, launching the business transaction for test execution, setting the test status, test result documentation, and incident creation.

Manual Test Execution in SAP Test Option 1

SAP Solution Manager Test Workbench offers easy access to test cases and business transactions (Figure 34). To offer a convenient and straightforward access to test cases and business transactions for manual testers, SAP Test Option 1 provides a Tester Worklist, which provides a list of all test cases assigned to the user. With one click, the Tester Worklist allows the user to:

• Open the test cases to read the guidance on how to test the business transaction
• Launch the business transaction in the respective SAP or non-SAP test system

With this approach the tester does not need any other information to access the test system for the execution of the business transaction that is getting tested. When the test execution has finished, you can perform the following actions within the Tester Worklist:

• Test status setting
• Incident creation in case of unexpected results. Tester will be notified after problem resolution to start the retest (see below).
• Documentation of test results (see below)

  Email Notification to Inform Manual Testers to Start Test Execution The test coordinator should inform the involved testers through email notifications to allow a smooth start of test activities. This feature is of high importance in case of scenario tests with the involvement of multiple users, since test activities from individual testers are dependent on each other. As an example, when testing an order-to-cash process, tester #1 might start with the execution of a quotation, sales order creation, and delivery and goods issue followed by tester #2 performing the billing. In this example, tester #2 can receive an email notification when tester #1 has finished his tests. Documentation of Test Status and Test Results The documentation of test-execution results (Figure 35) by the manual tester is an important task to ensure that the overall test is auditable. Testers can spend a lot of time with this task, if not supported in an efficient way by the test tool. SAP Solution Manager Test Workbench provides a convenient way to capture the results. In SAP Test Option 1 the user provides the test status from a drop-down selection and subsequently creates a test note, which uses a copy of the original test case. Here the manual tester enters the actual results next to the expected results (depending on a company-specific template) and the possibility to add screenprints to document the system response. Incident Creation and Integrated Problem Resolution The main motivation to perform tests is the detection of unexpected system behavior. To allow efficient handling of incident creation in the context of test execution, SAP Test Option 1 as well as SAP Test Option 2 enables the creation of incidents directly during test execution (Figure 36). The incidents include information about the environment such as test system, test case, and many more that are important for the service colleague or specialist who evaluates the incident. It is of paramount importance that the test tools provide an integrated incident management capability to allow the smooth exchange of time-critical information and allow the colleague handling the incident to get the entire context automatically. After incident clarification and problem resolution, the tester is notified automatically and prompted to perform retest activities.

Manual Test Execution in SAP Test Option 2

The Quality Center Test Option 2 includes products and capabilities from SAP Solution Manager and SAP Quality Center by HP that are tightly integrated (Figure 37). Email Notification to Inform Manual Testers to Start Test Execution The test activities are started by the test coordinator who notifies the manual testers via email. Access to Test Cases and Business Transactions for SAP and Non-SAP Test Systems The manual tester performs the following test activities:

1. The tester logs on to the Test Lab module of SAP Quality Center by HP and finds his or her specific test set, including all test cases assigned to the tester.
2. The tester reads the first test cases, which include step descriptions and activities as well as expected results from the application being tested.
3. The tester logs on to the SAP test system and executes the business transaction as described in the test cases. The tester returns to the test cases and documents the actual test results with all necessary details. SAP Quality Center by HP provides a screenprint capability, which helps to better document the test results for later auditing purposes.

Incident Creation and Integrated Problem Resolution In case of unexpected test execution results and test system behavior, the tester can create a defect in the integrated defect module (Figure 38). The defect is linked to the test case as well as to the test requirements, which enables full traceability for the test coordinator and support expert. All activities of the tester are performed entirely in SAP Quality Center by HP. When a defect cannot be clarified or solved by the test team, the defects are transferred to SAP Solution Manager via a standard SAP-HP interface (SAP Solution Manager Adapter for SAP Quality Center by HP). Here a support expert analyzes the incident and provides an appropriate resolution, which is transferred back to SAP Quality Center by HP. The tester is notified and performs a retest. The test status is available in SAP Quality Center by HP for test coverage and test progress reporting as well as in SAP Solution Manager for subsequent project reporting (Figure 39).

Test Execution: Automated Regression Testing

Rule of thumb: Execute automated regression tests in unattended mode to receive the test status for all critical business process. This achieves efficiency gains since test teams can focus their time and resources on tasks that need human interaction.  

Automated Test Execution in SAP Test Option 1

Similar to the manual test execution, the automated execution of test cases can be triggered via an email notification. After receiving this email notification, the tester navigates to the Tester Worklist (Figure 40) and starts the execution of the test configuration. As outlined earlier a test configuration consists of the test case, the test data, and the system data. During test execution, the test configuration reads the test data from the TDC and transfers it to the test case of the test automation tool for execution. Each test data record from the TDC triggers an additional test execution. After the execution, test results and logs are transferred into the Tester Worklist. From there the tester can create an incident, if required. Another efficiency gain of automated tests comes from the ability to schedule automated tests and execute them in unattended mode (Figure 41) so that no users are tasked with monitoring them. Following this approach, test engineers can schedule test execution at non-peak times such as night time, which is also known as a “lights-out test.” With this approach, the test and change management team have much more time to concentrate on value-adding activities instead of execution of recurring tasks (i.e., execution of regression tests again and again).  

Automated Test Execution in SAP Test Option 2

SAP Quality Center by HP executes the SAP TAO case and accesses the test data file via the link included in the test case (Figure 42). Test data from the file is retrieved and entered into the input parameter at runtime. For each test data record a separate test execution takes place.

Reporting and Sign-Off

Reporting and sign-off are key not only during test execution but also during test preparation and planning. Figure 43 outlines different reporting possibilities and milestones for sign-off according to SAP Best Practices. All the above-mentioned reports for test planning and preparation as well as for test execution can be accessed centrally and online within SAP Test Options 1 or 2. This means that all required information is available in just one tool and without the need for tedious and error-prone manual download and rework activities.

Reporting During Test Preparation and Planning

During Test Preparation and Planning it is important to know to what extent business processes are covered by test cases. Only if this information is available is it possible to identify gaps in the test scope.

Reporting During Test Preparation and Planning in SAP Test Option 1

Figure 44 explains how a test coverage analysis can be performed in SAP Test Option 1 via menu path Work Center Test Management > Test Preparation > Pushbutton Evaluate. Besides the need to know gaps in the test scope, it also necessary to be informed about inconsistent test plans. Test plans are likely to become inconsistent if the business process structure or test case descriptions were changed after a test plan has been created. If an inconsistency is discovered, a test manager is enabled to ensure that test execution is based on the latest process and test description. Figure 45 outlines the generation of such a report in SAP Test Option 1 via menu path Work Center Test Management > Reports > Inconsistent Test Plans. The scope of this report is to list all test plans of a project in a table and to indicate inconsistent test plans. Icons and specific search options allow easily finding changed elements in the business process structure. In addition to the above-mentioned reports, there is strong need to know if there are test cases that have been created for a SAP Business Suite on SAP HANA migration project but are not used in any test plan. With this report it is possible to ensure the completeness of the Test Planning activities. The scope of this report is to display the business process structure and highlight test cases without assignments to a test plan or a test package. Figure 46 describes the mechanism of such a report. Central access point for this report is via menu path SAP Solution Manager > Work Center Test Management > Reports > Project Status Analyses.

Reporting During Test Preparation and Planning in SAP Test Option 2

Similar to SAP Test Option 1 there is a test coverage report available in SAP Test Option 2. The central access point for this analysis is the Requirements module within SAP Quality Center by HP (Figure 47). Here you can see which requirements have no assignments to test cases or how many test cases are already assigned to a certain requirement. If there are test cases assigned to a certain requirement, the execution status of these test cases is displayed as well.

Reporting During Test Execution

During test execution the main reporting focus is on test status overview, test status analysis, and test progress including defect messages and their status.

Reporting During Test Execution in SAP Test Option 1

Figure 48 gives examples of the reporting features in Test Workbench Test Option 1 that allow you to access the required information. The test effort report allows analyzing the ratio between planned effort, actual effort, and expected total effort. It can support project leads and test coordinators in identifying potential resource bottlenecks. Furthermore, SAP Solution Manager allows detecting deviations between actual effort and expected effort. The messages report offers an efficient and flexible way to analyze message-related data. The progress report can, for example, support project leads and test coordinators in detecting potential delays.

Reporting During Test Execution in SAP Test Option 2

In Quality Center SAP Test Option 2, you can access a test status report directly in the Requirements module of SAP Quality Center by HP (Figure 49). As outlined earlier, the test execution status of test cases assigned to certain requirement is displayed there. In addition, the progress can be, for example, reported from the module Releases within SAP Quality Center by HP (Figure 50).   Reports on incidents detected during test execution can be generated with SAP Test Option 2 (Figure 51).

Test Management Cockpit for SAP Testing Options 1 and 2

More sophisticated reports for both SAP test options are available via the SAP Consulting Solution Test Management Cockpit. Within this SAP Consulting Solution, metrics, reports and graphs are accessed via SAP Business Objects Design Studio (Figure 52). The reporting scope of the Test Management Cockpit includes all information regarding test design, test coverage, test execution, and test incident processing.

Sign-Off

In addition to the testing preparation and planning capabilities outlined above, both SAP test options provide the possibility to work with release status schemes for test plans (Figure 53). This is especially important for SAP Business Suite on SAP HANA migration projects in a validated environment such as pharmacy or banking where legal regulations require electronic sign-offs. Figure 53 outlines a simplified example of such a process. An efficient way to prepare final test documentation and prepare for audits is to use the Test Report functionality in SAP Solution Manager. It is available for SAP Test Option 1 and with usage of the SAP Solution Manager Adapter for SAP Quality Center by HP for SAP Test Option 2. It is possible to integrate all test-related information as listed below in one document:

• Project or solution
• Test plan – Responsible person
• Test plan – Overall results status (percentage and absolute view)
• Related messages
• Systems under test
• Involved processes
• Keywords
• Tester assignment
• Test case description including attributes
• Status history per test case

In SAP projects this document – as shown in Figure 54 – is often used as the basis for an official sign-off.

Performance and Load Tests

With SAP HANA, users expect an extremely fast data analysis with scan times of one billion rows per second for fact tables and a join performance of 10 million rows per second when joining attribute tables to fact tables. With this expectation, high query performance is one of the key features of SAP Business Suite on SAP HANA migration projects.

Preparation and Planning

To ensure a high end-to-end performance in the context of SAP Business Suite on SAP HANA migration projects, a proper end-to-end performance testing approach is required. It fully unleashes the entire SAP HANA performance and the associated technology stack. In particular, the use of customer individual extensions, modifications, different customizing and data models, and Z transactions fortifies comprehensive activities to assure performance stability.

Performance KPIs

Based on SAP Best Practices, the first step when preparing and planning an end-to-end performance test for SAP Business Suite on SAP HANA migration projects should be to define key performance indicators (KPIs) to measure and optimize. Most common KPIs are:

• End-to-end execution time including specific known queries
• Resource consumption of end-to-end scenarios including long running queries
• CPU consumption
• Memory consumption
• Degree of parallelization
• Throughput
• Transactional load
• Volume restrictions

Sometimes it is difficult to define meaningful KPIs. In such situations you can use performance baselines from the current productive environment such as:

• Workload Monitor (transaction code ST03N)
• Business Transaction Analysis (transaction code STAD)
• Performance Analysis (transaction code ST05)

Load Profiles

As soon as you define the KPIs to be measured and optimized in an end-to-end performance test for SAP Business Suite on SAP HANA migration projects, you have to define the corresponding load profiles (average and peak). Typically the definition of load profiles takes the following considerations into account:

• Top transactions, batch jobs, and reports creating 80 percent of system load
• Performance critical transactions
• Throughput of critical transactions
• Peak hours and maximum number of users
• Transactions with high numbers of affected users
• Processes with custom code or modifications
• High number of online users performing  critical transactions in parallel
• Throughput of processes (with third-party interfaces)
• Critical processes including custom code or modifications
• External visible business processes

You also should take into account stakeholder interviews as well as statistical data taken from transactions, including:

• Transaction code STAD: Business Transaction Analysis
• Transaction code ST05: Performance Analysis
• Transaction code SAT: Runtime Analysis
• Transaction code ST03N: Workload Monitor

During the definition of load profiles it is also important to define a scaling factor between the production (PROD) and the testing system (QAS/TST) in order to reflect potential sizing differences. SAP Best Practices recommend calculating this scaling factor as an SAP Application Performance Standard (SAPS) ratio between TST/QAS and PROD.

Generation of Mass Data

The generation of mass data and its proportionate distribution according to the underlying data models and queries is a key success factor for any performance test. Therefore, planning and provisioning of test data is an essential and fundamental preparation step. Data structures and volumes have to be aligned from a content perspective and have to fit into the data models of the testing scope. The test cases must be not only fully functional but also repeatable. This requires a sufficient amount of data. According to SAP Best Practices, one option for data generation is the SAP HANA JavaScript Tool, which is a Java-based engine for the execution of JavaScript files for data processing in SAP HANA DB/NewDB. Another option would be to build a tool based on Eclipse with JavaScript support (JEE or JavaScript versions). With this second option, SAP HANA Java classes are exposed as JavaScript classes for script developers. Several generators for sequences, numbers, strings, dates, and UUIDs are provided. In addition to that, the usage of the Test Data Migration Server (TDMS) can be evaluated. With TDMS, it is possible to derive mass test data from the production system. It is also possible to make this data anonymous. It is always a challenge to get realistic transaction data. With TDMS, usable transaction data can be easily be created.

Execution

In order to analyze the end-to-end performance, an expected average or peak load profile has to be simulated. This average or peak load profile needs to reflect users, background jobs, and various other workloads. The load profiles defined during the preparation and planning phase are typically executed by combining:

• Batch jobs
• Interfaces (e.g., starting and stopping communication channels on SAP NetWeaver Process Integration [PI])
• Automated test cases simulating end-user behavior

SAP recommends using SAP LoadRunner by HP for simulating end-user behavior for the following reasons:

• Easy scripting using SAP LoadRunner by HP as SAP HANA is capsulated by either user clients or SAP applications (e.g., Business Objects/SAP BI/SAP ERP)
• For all SAP applications there is the required scripting know-how available
• Automated test cases can be easily modified and parameterized
• Integration of monitors from the environment under test so that most of the critical resources are correlated to the response times
• Stable behavior in long-running tests (no crashes in the middle of a run)
• Not only the number of data but also the number of users executing queries in different manners is critical to the performance of SAP HANA. Simulating a high number of users is one of the key strengths of SAP LoadRunner by HP.
• Monitoring and Analysis feature included in SAP LoadRunner by HP
• Ability to monitor multiple KPI’s

Figure 55 gives an example of a SAP LoadRunner by HP test script used in a SAP Business Suite on SAP HANA migration project at SAP. This script simulates a so-called Activity Search in a CRM system. In this particular SAP Business Suite on SAP HANA migration project, SAP LoadRunner by HP was embedded in a load profile simulating 1000 users using a certain think time for each user. The realization of the described load profile using SAP LoadRunner by HP is reflected in Figure 56. After executing an end-to-end performance test using SAP LoadRunner by HP, KPIs have been collected by using the monitoring features of SAP LoadRunner by HP, SAP HANA Studio, SAP Solution Manager Diagnostics, or standard transactions for analysis such as transaction codes STAD or ST05. Figures 57 (SQL–Statements) and 58 (Transaction Response Time) show examples of the monitoring features of SAP LoadRunner by HP. If KPIs measured during the end-to-end performance test using SAP LoadRunner by HP have not been met, a detailed performance analysis of SAP HANA DB/NewDB is required as part of the root-cause analysis. The first step of such a root-cause analysis is, for example, to understand the processing time spent in the SAP HANA DB/NewDB when executing a query.  

Measuring the Processing Time of an SQL Statement

You can measure the overall processing time of a SQL statement on SAP HANA DB/NewDB by investigating:

• Expensive SQL statements using the corresponding view within SAP HANA Studio
• SQL traces via the diagnostics feature with SAP HANA Studio

Using the expensive (expensive with regards to resource consumption) SQL statement view within SAP HANA Studio you can gain the insights listed in Table 2. Using SQL traces provides certain time stamps (e.g., start and stop of the query) as indicated by Figure 59. By calculating the difference between stop and start times, you can measure and calculate the processing time of the SQL statement on the SAP HANA server.

Analyzing SQL Statements Using EXPLAIN PLAN

You can analyze SQL statements using the EXPLAIN PLAN statement. It is used to evaluate the execution plan that is followed by SAP HANA DB/NewDB in order to execute a SQL statement. The result of the evaluation is stored into the EXPLAIN_PLAN_TABLE view for later examination by the user. Exactly this result can be used in order to understand the different execution steps and their influence on the processing time of a SQL statement on SAP HANA DB/NewDB.

Analyzing SQL Execution with the Profile Trace

You can also analyze SQL execution with a profile trace. The profile trace is one of the most detailed trace levels available in the SAP HANA Studio. The profile trace is particularly helpful in understanding how the various engines inside HANA work together. For instance, you can examine the number of records that are passed from one engine to another. Using the profile trace helps you understand the SQL statement and its execution procedure, which might enable understanding of the root cause of a very high processing time.

Attribute, Analytic, and Calculation Views

You also need to capture and analyze performance traces for attribute, analytic, and calculation views. The detailed analysis of SQL statements as described above helps to analyze pure SQL statements that are executed by the SQL engine of SAP HANA DB/NewDB. However, all models created with the SAP HANA Studio (i.e., attribute views, analytic views, and calculation views) are executed in the OLAP, JOIN, or Calculation Engine of SAP HANA rather than the SQL Engine. Thus these models require a performance trace to analyze their runtime behavior in detail. To capture performance traces the following options are available:

• A performance trace from the SAP HANA Studio
• A performance trace with the SAP HANA database administration tool

After capturing a performance trace, an appropriate analysis is required. To get the most out of such an analysis, we recommend you execute the following steps:

• Identify performance-relevant SQL queries in the SAP HANA DB administration tool:
  • Select specific known queries
  • Select the top time-consuming queries
• Understand which engine executes which plan operations
• Understand the degree of parallel execution
• Drill down into single plan operations

Service Tests

Companies develop their own applications and use services for the communication with the underlying database tables. Each custom-built application can be used by a variety of use cases. After migration to the SAP HANA database, these custom-built applications, services, and tables work functionally correctly, but SAP recommends that you assess the functional and performance behavior after SAP HANA migration (Figure 60). For custom applications that are under rework and for situations in which a high number of different use cases are in place for the custom-built (Web) services, SAP provides functionality to test the services without the need to use the custom applications with their respective user interfaces. Instead companies can perform service tests entirely without the calling application and respective UIs. In the example shown in Figure 61, two custom applications (A and B) make use of a custom-built API to access three database tables when two of these tables are custom-defined. There are multiple use cases by each of the applications A and B with different paths through the application logic that make use of the API to read and write to the database tables. To set up “headless tests” (i.e., service tests without the application UI layer), you can use the test tool SAP Quality Center by HP – Service Test module. During test definition you can define data and flows for the variety of use cases. In the context of the SAP HANA migration project this allows you to validate that the database access is executed with the expected performance and provides important information to potentially improve the access logic of the custom-built API.

Further Information

Table 3 lists more sources of information about testing, SAP Solution Manager, and HANA.