In my recent articles about SAP HANA Cloud Integration (SAP HCI), you learned a lot about SAP HCI’s functionality, the basic concepts behind it, and different modeling techniques for solving typical integration problems such as message enrichment, message mapping, and message routing. However, in the examples so far, I focused on the handling of messages containing just one item, a single order, for example. Now I dive into the details of coping with messages comprising a list of entries, such as how to:

• Split up such a message into individual pieces
• Iterate over each list item
• Handle the resulting single messages in an SAP HCI message processing chain
• Combine the results of each single message handling sequence back into one response message

The Scenario

In real-life scenarios integrators are quite frequently confronted with input messages consisting of several items of the same message structure grouped in a list (e.g., order line items). The integrator wants to iterate over the list. Individual list items have to be separated and individually managed by the integration flow. Finally, the result of each individual message handling procedure needs to be consolidated into one response message being sent to either the sender of the message (in case of a synchronous scenario) or to the final recipient (in case of an asynchronous scenario). To illustrate this functionality using SAP HCI, I’m using the input message shown in Figure 1 throughout this article.

Figure 1

Example message comprising a list of order numbers

It is based on a WSDL file I created using the Enterprise Services Builder of SAP Process Integration (SAP PI). For sure, you can use any XML tool supporting the WSDL standard. For your convenience you can view the WSDL in Figure 2. The file is called SendOrderList_Async.wsdl.

<?xml version="1.0" encoding="UTF-8" ?> 
- <wsdl:definitions name="GetOrderShipDetails_Sync" targetNamespace="https://hci.sap.com/demo" xmlns:p1="https://hci.sap.com/demo" xmlns:wsp="https://schemas.xmlsoap.org/ws/2004/09/policy" xmlns:wsu="https://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" xmlns:wsdl="https://schemas.xmlsoap.org/wsdl/">
  <wsdl:documentation /> 
  <wsp:UsingPolicy wsdl:required="true" /> 
  <wsp:Policy wsu:Id="OP_GetOrderShipDetails_Sync" /> 
- <wsdl:types>
- <xsd:schema targetNamespace="https://hci.sap.com/demo" xmlns:xsd="https://www.w3.org/2001/XMLSchema" xmlns="https://hci.sap.com/demo">
  <xsd:element name="OrderNumber_MT" type="OrderNumber_DT" /> 
  <xsd:element name="OrderShippingDetails_MT" type="OrderShippingDetails_DT" /> 
- <xsd:complexType name="OrderNumber_DT">
- <xsd:sequence>
  <xsd:element name="orderNumber" type="xsd:string" /> 
  </xsd:sequence>
  </xsd:complexType>
- <xsd:complexType name="OrderShippingDetails_DT">
- <xsd:sequence>
  <xsd:element name="orderNumber" type="xsd:string" /> 
  <xsd:element name="customerName" type="xsd:string" /> 
  <xsd:element name="shipCity" type="xsd:string" /> 
  <xsd:element name="shipStreet" type="xsd:string" /> 
  <xsd:element name="shipPostalCode" type="xsd:string" /> 
  <xsd:element name="shipCountry" type="xsd:string" /> 
  <xsd:element name="shipDate" type="xsd:string" /> 
  </xsd:sequence>
  </xsd:complexType>
  </xsd:schema>
  </wsdl:types>
- <wsdl:message name="OrderNumber_MT">
  <wsdl:documentation /> 
  <wsdl:part name="OrderNumber_MT" element="p1:OrderNumber_MT" /> 
  </wsdl:message>
- <wsdl:message name="OrderShippingDetails_MT">
  <wsdl:documentation /> 
  <wsdl:part name="OrderShippingDetails_MT" element="p1:OrderShippingDetails_MT" /> 
  </wsdl:message>
- <wsdl:portType name="GetOrderShipDetails_Sync">
  <wsdl:documentation /> 
- <wsdl:operation name="GetOrderShipDetails_Sync">
  <wsdl:documentation /> 
- <wsp:Policy>
  <wsp:PolicyReference URI="#OP_GetOrderShipDetails_Sync" /> 
  </wsp:Policy>
  <wsdl:input message="p1:OrderNumber_MT" /> 
  <wsdl:output message="p1:OrderShippingDetails_MT" /> 
  </wsdl:operation>
  </wsdl:portType>
- <wsdl:binding name="GetOrderShipDetails_SyncBinding" type="p1:GetOrderShipDetails_Sync">
  <soap:binding style="document" transport="https://schemas.xmlsoap.org/soap/http" xmlns:soap="https://schemas.xmlsoap.org/wsdl/soap/" /> 
- <wsdl:operation name="GetOrderShipDetails_Sync">
  <soap:operation soapAction="https://sap.com/xi/WebService/soap1.1" xmlns:soap="https://schemas.xmlsoap.org/wsdl/soap/" /> 
- <wsdl:input>
  <soap:body use="literal" xmlns:soap="https://schemas.xmlsoap.org/wsdl/soap/" /> 
  </wsdl:input>
- <wsdl:output>
  <soap:body use="literal" xmlns:soap="https://schemas.xmlsoap.org/wsdl/soap/" /> 
  </wsdl:output>
  </wsdl:operation>
  </wsdl:binding>
  </wsdl:definitions>

The message obviously consists of a list of order numbers. The other fields are not relevant for this article. The goal is to split the message into individual order messages, enrich each individual order with order details (e.g., shipping date, shipping city, and shipping address), and send back the enriched message as a reply to the sender in a synchronous scenario.

I solve this problem in two steps. The first step is to understand splitting of one input message into several individual messages and joining the pieces back into one large message. The second step explains how to enrich the individual messages with order details and collect those results into one large message. Once you understand the basic principle, you can apply this knowledge for even more complex scenarios. So let’s get started.

The key for splitting large messages into individual pieces, iterating over them, and joining them back again into one large message is the use of two new steps from SAP HCI’s web-based graphical modeler: the Splitter and Gather steps. You need to position them correctly in your message processing chain (aka route). They are used in the integration process model depicted in Figure 3. It shows the message processing chain for the first step of the implementation plan outlined above.

Figure 3

Splitting a large message into single pieces and collecting them back using the Gather step

You find the Splitter step at the beginning and the Gather step at the end of the processing chain. This is for good reason because the two steps belong together. It is important for the overall understanding of the whole Splitter-Gather construct to recognize the following:

1. The steps in between a Splitter and a Gather step (the two Content Modifier steps in my example) are executed as many times as the Splitter creates individual messages
2. Each step within a Splitter/Gather pair gets the separated individual messages that the Splitter has created as input messages, one after another.
3. It is possible to model a splitter scenario without a Gather step. In those cases all steps following the Splitter are executed repeatedly until an end event is reached. The repeated execution of steps is dependent on the Splitter step, not on the Gather step.

Let’s see how the configuration looks for my scenario. As a reminder I will repeat the settings of the Sender pool and the message flow from the Sender pool to the Start message event. The sender is configured as shown in Figure 4.

Figure 4

Configuration of the Sender pool

The entry of the Authentication Type: drop-down field is important. I’ve set it to Basic Authentication to allow the sender to provide a user name and password credentials while invoking the integration flow. The configuration of the Simple Object Access Protocol (SOAP) adapter is shown in Figure 5. You can set the values by selecting the message flow in the process model (the color of the dashed arrow from the Sender pool to the start event changes to orange when selected).

Figure 5

SOAP Channel configuration

The Address field on the Adapter Specific configuration tab is important. The address entered here will later be part of the URL to actually call the flow.

Let’s continue with the integration flow itself. The first step in the flow after instantiating it is the Splitter activity. As this step is new, I will include more details. The first question to answer is: Where can you find the activity during development? You can find it in the palette on the left of the modeling environment by following menu path Message Routing (diamond shape icon ) > Splitter in Figure 6 > General Splitter (Figure 7).

Figure 6

Open the Splitter sub-menu from the palette

Figure 7

Select the General Splitter from the Splitter sub-menu

Obviously, SAP HCI supports several different splitter types. For this article I’m  focusing on the General Splitter. For more information about the other splitter alternatives, see the sidebar “Splitter Implementations.”

Splitter Implementations

SAP HCI provides several Splitter implementations (Figure 7). In the article I use the General Splitter, which I describe in detail.

In this sidebar I briefly describe the other Splitter implementations.

The Splitter with behavior closest to that of the General Splitter is the Iterating Splitter. It also splits a composite message into a series of individual messages, but doesn’t copy the enveloping parts of a message to the single splitted messages. Enveloping elements are the message parts of the original incoming message above the nodes that are used for splitting. In the documentation for SAP HCI integration at https://cloudintegration.hana.ondemand.com/PI/help, I found a nice visual depiction of the differences between the General Splitter and the Iterative Splitter (Figure A).

Figure A

The difference between the General Splitter and the Iterative Splitter (taken from SAP Online Help for SAP HCI)

Obviously, the Iterative Splitter just copies the parts starting with the splitting tag (C in the example of Figure A) and the sub-nodes, whereas the General splitter also copies the nodes above the splitting tag (A and B). This is especially important if you want to navigate to elements in the tree structure using absolute XPath expressions.

One dedicated splitter takes care of IDoc messages. It is the IDoc Splitter, which is used for composite IDoc messages. It splits the composite IDoc into a series of individual IDoc messages, including the enveloping elements of the composite IDoc message. There are no special configuration settings available for this splitter.

Finally, SAP HCI supports the PKCS#7/CMS Splitter. PKCS stands for Public-Key Cryptography Standard, which is used to sign and encrypt messages. It is useful if a client sends a message that is PKCS#7 signed and that contains both a signature and content. This splitter type breaks down the signature and the content into separate files. As configuration options you can provide names for the files that should contain either the payload or the signature after the splitting step. You can also prescribe which file should be handled first after splitting, the signature or the content file, and you can decide whether the payload/signature should be BASE64 encoded after splitting or not.

After positioning the General Splitter shape in the main pool named Integration Process (Figure 3), you can set its properties. I’ve configured it with the parameters shown in Figure 8.

Figure 8

Configuration of the Splitter step

Let’s walk through the parameters one by one:

1. XPath Expression: It tells the integration engine during run time to search for the given tag in the input message and take it as the split argument. In my example I’ve used the relative path //orders. Relative paths are indicated by the double slash at the beginning and are quite convenient to let the engine search for the tag’s occurrence in your input message. The alternative would have been to use absolute paths starting with a single forward slash. As a consequence, you need to know the exact path from the root to the tag that should be used for splitting.

In case the message is extended later (e.g., by adding tags between the root tag and the splitting tag), the absolute path is not valid anymore as it doesn’t consider the new tag on the way from the root to the splitting tag. Hence, absolute paths are quite static and error prone if it comes to changes on the message’s structure. This error would not happen with a relative path.

Let’s see how the definition influences the execution during run time. Take a look at my example input message in Figure 1. Obviously, there are three <orders>-tags in the message. Consequently, the splitter generates three individual messages. Each individual message that is forwarded to the two Content Modifier steps following the Splitter looks like the message shown in Figure 9 (with different order number values, of course).

<orders>
    <orderNumber>10248</orderNumber>
    <customerName>?</customerName>
    <orderAmount>?</orderAmount>
    <currency>?</currency>
    <taxAmount>?</taxAmount>
</orders>

Hence, the single messages also contain the <orders>-tags. Consider this while processing each of the individual messages.

2. Grouping: With the grouping parameter you define the number of items to be grouped together into one message for individual processing. As I chose 1 as the value, every occurrence of <orders> results in a dedicated individual message for further processing. If I had chosen 2 as the value, the splitter would group the first two items into one single messsage, the next two items in the second message, and so forth. So assume an input message contains 10 items in total. How many messages would the Splitter generate if the Grouping parameter is set to 2? It would produce five messages, each containing two items.

3. Timeout: You have the option to provide a time limit for processing your complete message using the Splitter. If this given time limit is violated, the processing chain is aborted and an error results.

4. Streaming: The Streaming parameter is important in regards to memory consumption. Normally, the Splitter works on messages by loading them completely into main memory. However, messages can become quite large. Think about payments that are sent to a bank once a day. They are collected over the course of the day and forwarded, sometimes at night. These collected messages can be really huge, and it doesn’t make sense to load them entirely into main memory before processing them (and it is sometimes even impossible). So it is useful to let the Splitter start working on the incoming streamed data although it is not entirely loaded. This is called “streaming” as it allows you to read chunks of data, work on it, read another chunk, and so forth until the entire message is processed.

5. Parallel Processing: As the name indicates, it allows you to run the individual message processing tasks in parallel leveraging Java’s concurrency features using thread pools. By parallelizing tasks, you can get more work done in less time. However, the execution sequence can not be guaranteed as the threads are running independently from each other.

6. Stop On Exception: If this check box is set, the processing of the route immediately stops in case of an error and the error is propagated back instantly. Otherwise, the Splitter continues working on the individual messages and reports the error back at the end after handling the complete input message.

Now that you have an understanding of how the Splitter handles the incoming message, I’ll continue with the first Content Modifier. Its behavior has been described several times in my other articles before, so it is enough to just take a quick look at its configuration in Figure 10.

Figure 10

Configuration of the first Content Modifier in the route

The step sets a variable named OrderNo in the message’s header area and by this configuration stores the current order number of the current item for later reference. As the entry for the Value field you could have taken //orderNumber as well as it is a relative path and there is only one order number available in each individual message.

The header value you just stored is retrieved by the second Content Modifier and copied into the result message as the configuration in Figure 11 shows.

Figure 11

Configuration of the second Content Modifier in the route

Again, the second Content Modifier is also invoked for each individual message. It is the Gather step that finally collects all the individual single messages created for each invocation of the Splitter into one bulk message. The Gather step’s configuration is shown in Figure 12.

Figure 12

Configuration of the Gather step

Obviously, all the resulting single messages are of the same format. The Incoming Message Format drop-down field is set to XML (Same Format) and the Gather step should just put each of the individual pieces together into one result message. Consequently, the Aggregation Algorithm’s drop-down field is set to Combine.

Which other configuration options does the Gather step provide? It actually all depends on the entry chosen from the Incoming Message Format drop-down list. The list provides three options:

1. Plain Text: As an Aggregation Algorithm only Concatenate is allowed. Obviously, all the plain text messages generated by the single messages will simply be concatenated to one large string, one after another. This is the result that is propagated back to the caller.

2. XML (Different Format): Aggregation Algorithm only allows Combine, which is similar to the behavior of the Concatenate option in the previous case. The individual XML fragments from the single messages are brought together in one response message using a multi-mapping process known from SAP Process Integration’s mapping engine. (Take a look at my article “Working with Mappings in SAP HANA Cloud Integration” to learn more about the reuse of SAP PI’s mapping engine in SAP HCI.)

3. XML (Same Format): If you choose this option, you can influence the construction of the response message in two ways.

Firstly, you can, for sure, also apply the Combine option of the previously discussed Different Format case. Its behavior is identical to the one just described.

Secondly, there is one more interesting option that allows you to copy parts from the source message into a user-defined envelope consisting of XML tags. The XML node from which parts of the source message should be copied is formulated using an XPath expression. The XML envelope is defined using an absolute path definition such as /root or /level_1/level_2. The associated configuration screen is depicted in Figure 13.

Figure 13

Configuration of Gather step in case the single messages should be combined using XPath expressions

The behavior is actually pretty simple: The XPath expression of the field with the label Combine from Source (XPath) takes the source message (the single message created by the Splitter), navigates within that message to the node specified by the XPath expression, and copies the node, including all nested XML sub-nodes. It takes the copied XML sub-tree snippet and pastes it after the tags that the user has defined as the envelope.

The idea behind the envelope is that every valid XML document requires one root element. This is exactly what you can define in the Combine at Target (Path) field: the root element (e.g., /root) or, if necessary, an absolute root path (e.g., /level_1/level_2). During run time the copied XML sub-tree is pasted after the nodes given in the Combine at Target (Path) field. In other words, the node in the Combine at Target (Path) field is the parent of the snippet that needs to be inserted. The corresponding closing tags of the envelope are automatically added once the gathering has completed (e.g., </root> or </level_2></level_1>).

Let’s look at a concrete example. Assume the Splitter generates two messages like the ones shown in Figure 14 (later they are the SOURCE messages in the Gather step’s configuration).

Message 1:
 <payload>
     <route>
         <multicast>Parallel</multicast>
         <branch>A</branch>
     </route>
 </payload>
 
 Message 2:
 <payload>
     <route>
         <multicast>Parallel</multicast>
         <branch>B</branch>
     </route>
 </payload>

Next, assume the settings shown in Figure 15 for the configuration of the Gather step. For the input field labeled with Combine from Source (XPath), enter /payload/route (Note: Only absolute XPath expressions are allowed for this field.) Finally, enter /xyz/abc for the Combine at Target (Path) field.

Figure 15

Sample configuration of the Gather step

The final resulting message after executing the Gather step looks like this (Figure 16):

<xyz>
     <abc>
         <route>
             <multicast>Parallel</multicast>
             <branch>A</branch>
         </route>
         <route>
             <multicast>Parallel</multicast>
             <branch>B</branch>
         </route>
     </abc>
 </xyz>

You can easily understand now how this result message was created. The engine copied the parts from the source messages starting at the /payload/route node, including the <route>-tag and all nested nodes. The engine pasted it into the target message, which starts with the <xyz><abc> nodes representing the opening part of the envelope.

Remember: The pasting is done for all messages resulting from the Splitter. Hence, you get two <route>-tags in between the envelope. Finally, the closing tags of the envelope are added. They can be derived from the definition of the Combine at Target (Path) field. As the definition for that field was /xyz/abc, the closing tags must be in the opposite sequence resulting in </abc></xyz>.

Note: It is not mandatory to provide an entry for the Combine at Target (Path) field. If you leave it empty, the resulting message will have the same tags that are specified in the Combine from Source (XPath) field. In the example above, the resulting message would start with <payload><route>.

Running the Integration Flow

Now that the scenario is completely configured, you can run it. Save your changes and deploy the integration flow. Don’t forget to retrieve the URL for invoking the integration scenario from the SAP HCI’s Monitor (for details see my first article in this series, “Your First SAP HCI Integration Flow”). Provide the URL in the SOAP tool of your choice (e.g., SoapUI) and invoke the integration flow. The result should look like the one in Figure 17.

Figure 17

Final response message after invoking the integration flow with a list of order numbers

The flow successfully retrieved the order numbers from the incoming message containing the order list and created an appropriate response message. That’s exactly the result you want to achieve.

Enriching Individual Messages with Additional Data

I start with this relatively simple example just to concentrate on the Splitter’s and Gather’s behavior. However, you can extend this simple example to something more useful by invoking an external OData data source and enriching the result message. Typically, you would want to enrich some basic data with additional data coming from external sources. In my case I want to retrieve order details for each of the order numbers I extracted.

If you have followed the SAP HCI series closely, you should be reminded of my article titled “Invoking an OData Service with SAP HANA Cloud Integration.” In that article I configured an OData connection for retrieving detailed data for order numbers. This is exactly what I do next.

I replace the second Content Modifier (which actually just sets the body of the single message artificially with the extracted order number) with a Request-Reply step invoking the OData service and providing useful data as a response for each single message the Splitter created. After I add the Request-Reply step into the integration flow, the scenario looks like the one in Figure 18.

Figure 18

Integration flow with Splitter, Gather, and the invocation of an external OData source

The configuration of the Request-Reply step is identical to the one described in my other article and can be found there. Note: The call of the Request-Reply step is executed for every order number of the original incoming message. I can’t stress enough the importance of this specific behavior of an integration flow making use of the Splitter step. All activities following the Splitter are invoked for each single message generated by the Splitter until either the end of the flow or the Gather step is reached. The invocation of the integration flow finally results in the response message shown in Figure 19.

Figure 19

Result message including order details retrieved from the external OData source

Integration solutions such as SAP HCI frequently have to handle messages containing lists of items. In this article the Splitter step was used to split up a message comprising several order numbers into individual messages containing one order number each. SAP HCI can treat the individual messages separately. Afterwards, SAP HCI can combine the results of each individual message processing chain back into one integrated response message using the Gather activity.

Dr. Volker Stiehl

Prof. Dr. Volker Stiehl studied computer science at the Friedrich-Alexander-University of Erlangen-Nuremberg. After 12 years as a developer and senior system architect at Siemens, he joined SAP in 2004. As chief product expert, Volker was responsible for the success of the products SAP Process Orchestration, SAP Process Integration, and SAP HANA Cloud Integration (now SAP HANA Cloud Platform, integration service). He left SAP in 2016 and accepted a position as professor at the Ingolstadt Technical University of Applied Sciences where he is currently teaching business information systems. In September 2011, Volker received his Ph.D. degree from the University of Technology Darmstadt. His thesis was on the systematic design and implementation of applications using BPMN. Volker is also the author of Process-Driven Applications with BPMN as well as the co-author of SAP HANA Cloud Integration and a regular speaker at various national and international conferences.

 

You may contact the author at editor@SAPpro.com.

If you have comments about this article or publication, or would like to submit an article idea, please contact the editor.