Avoid inconsistent or unexpected results in data loads by ensuring that data that needs to be grouped stays together in data packages during loading.
Key Concept
You can use the semantic key to aid in grouping data during loading to ensure like data is processed and loaded in the same data package.
When loading data into an InfoProvider, an SAP system groups the data into packages in order to load the data into the source. The size and grouping of the packages are dictated by the system at the time of extraction. The extractor uses the packet size and source data structure to assemble the data into coherent loading packets in order to group the source data. The extraction is executed in individual packages rather than in one large full data package because large data volumes can overwhelm system resources if the system tries to extract large data volumes as one unit.
In most data loads, the contents and grouping of the data into packages is not relevant. The system chooses the first package, loads this package, then the next. The contents of the packages are based on the contents of the source data. The system is careful to cut off the data when the package size limit is reached. In order to see the package size, double click on the data transfer process (DTP) and go to the Extraction tab. (Figure 1).
Figure 1
The Extraction tab in the DTP that shows the package size for the process
In some cases, however, the contents of the specific packages are relevant. This is typically because there is some start routine coding involved in the transformation that loops through the source data. Since the system does not bring in the entire data set at once, the start routine is run for each package that is loaded. Therefore, if data is split across packages, the results of the start routine can be inconsistent.
For example, a flag is needed in the source to note if any material on the sales order comes from division 10. This flag requires a looping through all the lines on the order to determine if any lines are division 10. If so, all lines are flagged with a division 10 indicator. Based on the way the system gathers the data from the source, there is no assurance that all the items from the sales order will be loaded into one package. Thus, if the start routine ABAP code loops through all the records to determine if any are division 10, and some of the lines are split into other packages, you could get inconsistent results. This is because not all the data from the sales order line is contained in this package. The sales order lines for one order could be split among multiple packages.
The best way to ensure that all the relevant data is contained in one package is to use the semantic groups in the DTP. This option allows you to choose the key fields for the extraction so the data packages are grouped. Note the semantic key does not need to correspond to the key in either the source or destination InfoProvider.
To set the semantic key, click the Semantic Groups button in the Extraction tab of the DTP (Figure 2). If semantic groups have been chosen, a green checkmark is shown next to the button to indicate the presence of the semantic group.
Figure 2
Select the Semantic Groups button to choose fields
Once you click the Semantic Groups button, the system provides you with the list of characteristics from the source (Figure 3). You can select one or many fields to use. The system uses this grouping any time this DTP is used to load the data. In this case, by choosing the sales document and item, you are assured that the system always has all order lines for one order in the same package. They will not be split among multiple packages.
Figure 3
List of possible fields to select when you click the Semantic Groups button
The system also uses the semantic key to set the key structure of the error log. If error handling is turned on via the option on the update screen of the DTP, the system shows the error log using the semantic key. This means that any errors are grouped and displayed based on the semantic key when looking at the error log.
The error log also works without choosing a semantic key. If no semantic key is chosen, the system shows the error log grouped by the package order. If a semantic group is chosen, it uses that key structure to show the error log.
Performance can be affected with the addition of a semantic key to the DTP. This is because the system must read and gather the data in a specific key set. This can slow performance, especially if no index exists on the selected field grouping in the source InfoProvider. Thus, unless there is a specific reason to use the semantic group (i.e., a start routine that needs the data grouped), do not include it in the DTP because it does not provide any benefit, but slows the loading of the data.
|
Gary Nolan is an SAP NetWeaver BW-certified consultant and author of the SAP Press books Efficient SAP NetWeaver BW Implementation and Upgrade Guide and Efficient SAP NetWeaver BI Implementation and Project Management. Gary specializes in gathering and evaluating requirements, configuring SAP NetWeaver BW, and providing project management, performance management, and data architecture and data modeling expertise. A former platinum consultant with SAP America, Inc, he has more than 15 years of SAP experience, working with SAP NetWeaver BW since version 1.2B.
To contact the author, click here: Gary Nolan. If you have comments about this article or BI Expert, or would like to submit an article idea, please contact the BI Expert editor.
|
