Data has intrinsic parameters that you can use to determine the optimal archiving strategy for your SAP NetWeaver BW data. Using a three-stage approach, you can choose the appropriate solution from popularly used techniques. When it comes to archiving in SAP NetWeaver BW there are two broad categories:   

• Archiving of data
• Archiving of reports

Archiving reports is like storing a snapshot of data at any given point in time that you can retrieve later. Report archiving has limitations such as the number of rows that can be stored and the report file size restrictions. In contrast, data archiving is more flexible and allows a user to recreate scenarios or snapshots of data that can be retrieved and ultimately viewed in a report. I primarily discuss data archiving and the available approaches and techniques for accomplishing this task. Numerous scenarios call for archiving data in SAP NetWeaver BW. It is important to understand these archiving drivers as these have a tremendous bearing on the ultimate choice of a solution. Here are a few scenarios:

• When there is a need to reduce the data volume in the SAP ERP system. The SAP NetWeaver BW system may be expected to store or archive the data extracted from the SAP ERP system.
• When the SAP NetWeaver BW system growth has to be restricted to minimize hardware and maintenance costs or to keep the system easily manageable
• To improve the load and query execution performance by decreasing the volume of data stored on the database
• Migration from legacy application systems to the SAP system thereby causing a surge in data volume in the SAP environment
• Decommissioning of existing applications with historical data
• Differential between the extent (duration) of legacy data and the extent of the data to be migrated to the SAP system
• Historic data requirements for reports (statutory and otherwise) and analysis

These drivers can be broadly categorized into business and technical drivers. Once the major drivers for data archiving have been determined, you can identify other parameters or qualifying criteria. These parameters or criteria in turn govern the type of archiving solution to be deployed.

Determining the Optimal Archival Solution

There are three stages to determining the drivers for an archiving solution, evaluating those parameters, and ultimately choosing an archiving strategy based on the feasibility of the solution against the drivers. The options for data archiving within SAP NetWeaver BW are covered in detail in the next section. This three-stage approach is generic in nature; it is not specific to archiving in SAP NetWeaver BW. You can apply it to archiving from any SAP R/3 or SAP ERP Central Component (SAP ECC) environment as well.  The stages are:

1. The gathering stage  

2. The evaluating stage 

3. The devising stage

Gathering Information

The gathering stage consists of collecting hardware information as well as identifying the drivers for archiving (as listed in the section above). The technical information that is gathered in this phase includes database information (e.g., the largest tables), the extent of historical data present in these tables/objects, and the frequency at which this data is accessed. You can use a variety of standard SAP tools (e.g., transaction DB02 to determine cube size) as well as third-party tools to do this analysis (Figure 1). In addition to the technical information, it is also important to gauge user comfort and bias. Although technical tools do help in determining which archiving option to use over another, the ultimate choice can only be made after taking users into consideration.

Figure 1

Approach to determine archival strategy

Evaluating Data 

In the evaluating phase, the information collected is analyzed and the drivers for archiving are identified and categorized. The usage analysis tells you how important it is to access certain information and, in turn, has a bearing on how that information is ultimately archived and retrieved for future use. Similarly, the user information gathered can also tell you about the need for archiving this data. For instance, some data may not be accessed frequently but may be required to be stored from a compliance perspective. Other soft parameters gathered from the users, such as readiness for an archival solution, are also analyzed in this phase. Some of the important parameters that govern the archival strategy are:

1. The extent of the need for online retrieval (or extent to which offline retrieval is acceptable)

2. The duration (how old the data is) and the size of the historical data

3. The extent of response time during retrieval for historical reporting/analysis

4. The choice of solution based on factors such as user comfort, cost, and readiness for change

Based on these parameters, the feasibility of the various archiving solution options available needs to be analyzed, then an archival strategy can be recommended.

Devising Solutions 

In this stage, two or three of the most critical parameters or archiving drivers are selected from the categories created in the Evaluating data stage. These parameters then serve as dimensions for categorizing the data to be archived. This dimensional analysis ultimately helps to point toward the appropriate archiving strategy. For example, the dimensions can be cost of implementation, ease of historical analysis, or effort required for implementation.

Depending on the parameters that are of importance to you, you can create a similar two-dimensional graph. Finally, based on the chosen parameters and drivers for the archiving solution, the devising stage results in a recommended strategy and approach for data archiving.

The following section details which archiving strategies are best applied in what scenarios. A list of objects to be archived is part of the deliverables of this phase. I recommend that you provide a document listing the archiving scope against every business process in scope of your implementation. For example, it may be desirable to list the extent of time up to which the orders would be archived for an order-to-cash process implementation.

This list of deliverables given in Figure 1 is indicative and may vary depending on business and technical requirements. Figure 1 also represents the steps carried out in each of these stages, while outlining the tools that are used and the final deliverables and outcome from this approach.

Information Life Cycle and Data Archival Solutions in SAP NetWeaver BW

Information Lifecycle Management (ILM) aims to manage information throughout its life cycle. It makes information management easier and more cheaply available to users by making use of different principles and technologies. These principles guide you in understanding your organization’s data and its intrinsic parameters as well as in managing and governing that data. The technology aspect helps in systematically implementing your data management and governance policies. One of the pillars of information life cycle is data archiving and storage.

The data that needs to be archived has certain intrinsic parameters that can help determine the optimal choice of an archiving solution. The archived data can be split into four quadrants based on how old the data that is being analyzed is (ranging from recent data to old data) and how frequently this data is accessed by users (ranging from frequently used to rarely used). Figure 2 outlines the typical SAP NetWeaver BW information life cycle and details how the data in different stages of the life cycle can be stored.  

Figure 2

Mapping of information life cycle with storage/archiving methods in SAP NetWeaver BW

SAP NetWeaver BI Accelerator

Starting with recent data and data that is frequently accessed, the SAP NetWeaver BI Accelerator provides quick access to data, and is the most appropriate method of storage in the case of new data in the system. This is because the BI Accelerator is a specially configured hardware tool that helps improve query performance through in-memory data compression. This was introduced with the SAP NetWeaver suite.

Toward the other end of the spectrum, as the data becomes older and is accessed rarely, different archiving methods can be mapped to different stages of the data age, taking into account how frequently it is accessed. Note that the timelines shown in Figure 2 for the extent of historical data are guidelines, not hard and fast rules.

Apart from the archiving data’s intrinsic parameters discussed above, the evaluate stage also yields several other parameters. Once the parameters required for data archiving have been gathered and evaluated, a formal archiving approach needs to be formulated. Within SAP NetWeaver BW, there are three possible solutions that can be used for data archival.

1. Traditional Archive Development Kit (ADK)

2. Near-Line Storage (NLS)

3. ADK with NLS

Approach 1: ADK

ADK is the standard SAP NetWeaver BW archiving solution based on the SAP ERP ADK solution. This time-tested method provides reliable and efficient archiving and retrieval of data. ADK-based archive files are compressed. You can write them to a variety of storage media (e.g., file systems and database systems), thus freeing up the memory space on the SAP NetWeaver BW online system (hard drive).

ADK capability for SAP NetWeaver BW InfoProviders was provided from SAP NetWeaver BW 3.0 based on the ERP-tested and proven ADK capability. This approach generates archiving objects for DataStore Objects and InfoCubes. ADK files are generated based on the archiving objects stored. The archived data can be compressed from its original volume for easy storage (usually from 1.5 to 5 times smaller than its original size).

The ADK approach is usually driven by business-oriented needs such as archiving for compliance and the ability to retrieve data at a later date. Although this solution does provide the ability to retrieve data later, it is best used for storing data that is near the end of its life cycle and is not accessed frequently (mostly data stored for compliance purposes). This is because data retrieval is generally slower than with other techniques.

The pros of this option are:

• Stored on the file system of the application server
• Proven ADK method has been around and is a tried and tested solution
• Is platform-independent (can work with different application servers) and can handle structural changes in the archived data

The cons of this option are:

• ADK files are not directly readable by BEx queries
• Need to reload for query access

• Slow retrieval as compared to newer methods

Approach 2: Near-Line Storage (NLS)

The NLS storage technique to archiving was popularized with the SAP NetWeaver suite of products and is a relatively new technique in comparison to ADK. It provides some convenient features:

• New type of data persistency which is similar to archiving. This method basically revolves around taking the data out of the database system and storing it on cheaper storage devices such as file systems
• With NLS, the data resides neither in an SAP NetWeaver BW system nor in a traditional archive system
• Provided for DSOs and InfoCubes and processed using Data Archiving Processes (DAP)
• Data compression rate is usually much higher than ADK (compression rates as high as 90 percent)

The NLS approach is usually driven by technically-oriented needs such as faster query performance or the ability to see recent data without long waits (i.e., without reloading the data from offline storage). This method is best used for information which is in the middle stage of its life cycle (i.e., data that is not used in day-to-day business work anymore but does need to be pulled up occasionally for operational purposes).

The pros of this option are:

• Quick data availability
• Data can be accessed directly for analysis (queries) and data load (DTPs)
• Background data storage invisible to end user
• Platform- and database-independent
• Lower total cost of ownership than other methods

The cons of this option are:

• Needs a third party to implement (additional software license)
• More effort than traditional ADK
• Compliance/legal requirements are typically not the drivers for an NLS solution

Following are the SAP NetWeaver BW-relevant features of using NLS:

• SAP provides an NLS interface (in the form of an ABAP class with several methods) with SAP NetWeaver? BW with which a third-party software add-on can interact
• NLS data is read-only
• Connection between SAP NetWeaver BW and NLS service, modeled by three ABAP object interfaces: connection interface, write interface for packaged data write into the NLS, and cursor interface for packaged data read from NLS
• NLS partitioned portions of an InfoProvider are write protected. This helps in maintaining data consistency and is useful from a data security aspect

Approach 3: Near-Line Storage with Archive Development Kit

NLS with ADK has an information life cycle management- and data requirement-based approach for the business needs of an organization. The data archiving process has to be configured to ADK-based archiving with an additional NLS. Two options for migration and data access (when used to migrate from ADK to NLS) are:

• Creating indexes for ADK files that are accessible through NLS
• Reloading the ADK files from archives into NLS partitions

This method is traditionally used for migration from a classical ADK system. It is also well suited when SAP NetWeaver BW is used as the enterprise data warehouse, as it is low cost and provides quick performance and compliance benefits.

The pros of this option are:

• combines the best features of both ADK and NLS
• Provides direct access of this data to queries
• The entire spectrum of approaches used in conjunction makes use of almost the entire information life cycle as described in Figure 2, thus helping to optimize system performance while keeping costs low

The cons of this option are:

• Needs a third party to implement (additional software license)
• More effort to implement than the individual ADK or NLS method

Figure 3 provides a matrix view of the data life cycle along with the archiving approach at different stages of the life cycle. Figure 4 gives a brief summary of the three available approaches and how to access data with each of these. Again, there is no one best technique when it comes to archiving; however, each of these does provide its own advantages and disadvantages. The right approach to determine the appropriate solution would be to first determine the parameters that are critical to your organization and then follow a methodical approach to determine the best fit for your archival solution.

Figure 3

Information life cycle data and data storage

Figure 4

Summary of archiving approaches in SAP NetWeaver BW