In fewer than 120 days, State Industrial Products moved part of its field sales force from an inefficient paper-and-pencil process (without a CRM system) to mySAP CRM 4.0 with Field Sales. With the old process, the field sales team would visit customer sites and take order requests using a paper order pad and pencil. They would write the order, look up pricing, and fax the order to company headquarters where an agent would enter the information into SAP R/3 Enterprise 4.7.

With this process we were unable to effectively manage the large amount of data generated (475,000 customers and 60,000 materials), leaving the sales force with limited access to key information. For example, customer contract pricing was very difficult to manage without an integrated and organized price reporting system. Also, our associates needed to view notes, activities, and historical sales information to best respond to customers’ service inquiries. Finally, our customers demanded more professional documentation — something that we were not providing with handwritten orders.

State Industrial Products has used R/3 with Microsoft Windows and SQL Server since 1998. We were confident in our ability to implement another SAP component because we had already upgraded twice. The latest was an upgrade to SAP R/3 Enterprise 4.7 in 2004. However, our group is small, comprised of one Basis team member, one ABAP developer, and one Sales and Distribution (SD) configuration team member. At first, we questioned whether we had the resources to support the existing R/3 operation while implementing mySAP CRM 4.0 and Field Sales. What follows are some of the most significant lessons learned from our project.

1. Implement one functionality at a time. A best practice is to implement CRM functionality one bite at a time. Our initial goal was to go live with SAP CRM via the big bang approach. That would have involved implementing Field Sales for select members of our field sales force, along with implementing People-Centric User Interface (People-Centric UI) for our outbound telesales centers (all locations) at the same time. In the blueprinting stage, we realized that the needs for each group were quite different. We determined that it would be difficult for the upgrade team to effectively focus on each of these needs at the same time because the project required deep involvement of the same functional resources in both implementations.

We decided to implement Field Sales first and deferred the People-Centric UI project. Figure 1 shows our Field Sales work plan summary. Our field associates had an immediate necessity to enter orders remotely via Field Sales and our outbound telesales group needed a highly customized user interface to effectively make sales calls (an average of over 50 per day) and record related activities and sales orders. The custom development efforts were somewhat more significant for People-Centric UI than for Field Sales. Lastly, the Field Sales implementation was more urgent — our telesales group uses Siebel CRM, but we had no relative application for our field representatives.

Figure 1

Work plan summary click here to view a larger version of this image

Keep in mind that an initial CRM implementation requires the most effort. For any CRM component to work, you must implement the basic mySAP CRM framework, which includes initial installation, configuration, and SAP Middleware if you plan to connect to a back-end system (such as R/3). Establishing SAP Middleware functionality takes significant effort, especially if you expect to extend any database tables. Plan on your ABAP developer spending at least half the time on SAP Middleware development and control, even through go-live.

2. Invest in high-performance hardware. Before you implement Field Sales, make sure to prepare your hardware so that your system operates at maximum efficiency. Items to consider are your disk, the mobile workstations, and a development application repository server (development ARS).

High Performance Disk. When you implement Field Sales, you post transactional data into two databases — the CRM database that contains all of the CRM information and the consolidated database (CDB), which contains all of the Field Sales-relevant data. Therefore, any R/3 information (customers, orders, pricing) you post to the CRM database you also post to the CDB.

We discovered that change transactions (e.g., an order status that the system changes when R/3 creates a subsequent delivery) can generate significant volumes of transactional data. The system passes the data in the form of a business document (BDoc) between the R/3 and CRM systems. The system then posts the BDocs automatically into the CRM database and the CDB. For example, a change in order status (creation of a delivery, post goods issue [PGI], or invoice) of a 10-line-item order creates BDocs that contain data for the order header, the 10 items, and all the related schedule lines for each change. The CRM database and CDB must process hundreds of records from dozens of BDoc structures, causing a significant amount of processing overhead and disk input/output (I/O).

As a result, a disk subsystem that supports the CRM database performs nearly twice as much work as an R/3 disk subsystem posting the same types of transactions. So, it’s best to review your disk and channel options and go with the options that provide optimum performance. Factors to consider:

• Disk Interface: Consider Fibre Channel (FC) over Serial-Attached SCSI (SAS). FC bandwidth can be up to four times greater than SAS.
  
  
• Disk Speed: Carefully review the manufacturer’s specifications for disk speed. Areas to review are disk rotation speed (15K RPM is typically found in a premium drive vs. 10K RPM in a standard drive). You should consider 15K RPM drives if you invest in an FC interface; otherwise, your disk’s processing bottleneck increases.
  
  
• Disk Size: Try to procure many, smaller drives instead of fewer, larger drives. Also plan for 50 percent more disk capacity, because SAP CRM reports much of the information twice.

Mobile Developer Workstations. Plan to obtain new workstations with maximum memory for your SAP Field Sales developers. We did, but not until after go-live. In addition to modest interactive response-time improvements and elimination of quirky program abends, you can save days of compile time compared with older workstations. We cut our full compile times in half (to about four hours) and our dynamic link library (DLL)-only compile time from an hour to 20 minutes by replacing our three-year-old processors and increasing the memory to 2 GB RAM.

Development ARS. Development servers are typically the smallest servers in an SAP environment, primarily because the database is smaller and the server load is much lighter than the training or production ARS servers. However, the development ARS server functionality is quite the opposite of a typical development server. You use it much more than the training or production counterparts because the system compiles SAP Field Sales on the development ARS server whenever any program changes are made, whereas the training and production servers only compile the program after completing all iterations of the development testing and transporting them to the target system. As with the developer workstations, a new, higher-end ARS can save days of development effort during a Field Sales implementation.

3. Review your findings to optimize performance. During and after our Field Sales implementation, we found that we had to adjust our processes to reflect issues we encountered. These issues included slow loading of customer data and incorrect feedback from a tool that determines and removes indexes not perceived as needed. By adding preventative measures, we were able to overcome the issues and keep our project on schedule.

Initial data loads from R/3 and work table reorganization. Our task was to load 60,000 materials, 475,000 customers, 2.4 million business partner records, and 1 million condition records into our CRM database. We had thought we were on schedule, allowing ourselves three weeks to load the data into our production CRM database. In the first 24 hours (on a Friday evening) we loaded all of the materials and nearly 200,000 customers, but we noticed that the rate of customers loading into the system was slowing down quickly. By Monday morning, we were loading customers at a rate that would cause the remainder of the loads to run another two months.

After reviewing suspicious tables and SAP notes related to database performance, our Basis administrator found two tables (SRRELROLES and SMW0REL), each of which contained several million rows and did not appear to contain any type of master data. These tables were “log” tables that needed to be reorganized periodically through transaction SM06_REORG2.

After configuring parameters within the job to retain the data in these tables, we purged the data in the two tables. Deleting the records was initially very slow, but throughput increased as the table became smaller. The initial run of SM06_REORG2 took approximately two days. Once the run was complete, we re-indexed the remaining tables and restarted loading the materials. The data loaded at a faster rate. We repeated the entire process (loading customer data, stopping to run SM06_ REORG2, and re-indexing) on a daily basis until we loaded all the information, finishing one day before go-live.

CRM Middleware Monitoring Cockpit. One of the CRM Middleware Monitoring Cockpit utilities reports all table indexes in use and maintains them (add, keep, or delete) based on their use. Use transaction SMWP, select menu option General Information, and click on Missing indexes. If you elect to use this utility, proceed with caution; while the utility almost always suggests valid actions, in this case it did not. The program removed indexes using fields SFABAK_V, SFABAK_N in table SMOVBFA and field SFABAP in table SMOVBKD, which negatively affected the performance of the sales order updates in CDB. After experiencing very poor response times for processing orders via the queued Remote Function Call (qRFC) inbound queue to the CDB, we researched the performance problem and added the indexes that the utility deleted. It’s best to record overall transaction response times before and after the change to ensure that this utility does not have any negative effect.

4. Proactively review SAP notes to improve performance. With any CRM implementation, you should first browse the list of relevant SAP notes. mySAP CRM, a less mature application than SAP R/3, has many SAP notes to review and implement before you populate your database with data. The following notes contain information that was of the greatest value to our implementation.

SAP note 740418: Alternative Globally Unique Identifier (GUID) generation for Windows. We initially planned to provide the sales representatives with one year of historical sales activity. Due to the delays we incurred with the initial load of business partners into CRM, we postponed the load of historical sales orders until after we went live. Working conservatively, we started to load sales orders from the previous month only. The load took much longer than expected — approximately three days, whereas we expected the load to take only a few hours. Again, with the large volume of data, we needed to constantly purge  the data in the SRRELROLES and SM0WREL tables and re-index the database. By this time we had a new wrinkle. In addition to the historical sales order loads running, we were live — which meant that the system was replicating current sales orders and customer data from R/3 in both CRM and CDB.

With historical sales order loads and daily activity burdening the system, response time suffered, even with a new, separate CRM database server. We loaded the remaining historical sales orders (at the rate of about 3,000 per day) for approximately two months. Having managed an R/3 database for nearly seven years, we were very surprised with the poor performance of the CRM database, and needed to answer three questions:

• Why was the disk activity so high? The disk lights were solid green, as if the system were hung up.
  
  
• Why was the processor activity so high?
  
  
• Why was the database fragmented by the end of each week? (Performance was much better on Monday than it was on Friday.)

Our Basis administrator found SAP note 740418, which relates to the GUID generation. A GUID is a 128-bit key that Microsoft Windows generates for data replication. Instead of sequentially assigning GUIDs, the system was randomly assigning them. After review, we implemented the SAP note, which caused the system to assign the GUID sequentially, and we experienced immediate, significant performance improvement.

SAP note 384971: Gateway parameters for a high interface load. Once we loaded the initial data (customers, pricing, and materials) into the CRM database, we extracted the information to import to Field Sales. We had a relatively small number of customers to extract (about 120), but had approximately 60,000 materials and 300,000 condition records to extract per associate. The initial subscription extracts to the CDB for Field Sales ran slowly; we figured that the extracts would take days at that pace. Our Basis administrator referred to SAP note 384971, which explains how to adjust the gateway parameters, reducing our overall extract time by about 80 percent.

For a list of other valuable SAP notes to consider with a Field Sales implementation, refer to Table 1.

5. Review laptop hardware and software specifications. You should consider the following hardware and software components for your field sales representatives’ laptops to ensure that Field Sales runs correctly.

• Processors. All Intel and AMD processors appear to work fine with Field Sales. The software installation and data imports took longer with Celeron processors.
  
  
• Memory. At the minimum, you should have 512 MB of memory, although it would be best to have 1 GB of memory in each laptop. We saw improvements in response time with laptops that had more memory, especially during initial installation and during order entry.
  
  
• Microsoft Office. You must have the full version of Microsoft Office installed because the system uses Microsoft Word as the reporting tool for the order document.
  
  
• ISP/Broadband Requirements. Broadband certainly is helpful, but not necessary if the mobile user doesn’t mind waiting periodically. We use ConnTrans (the standard SAP utility that facilitates communication between the mobile client and the CRM server) to send and receive data from the mobile client, which works very efficiently. Dial-up bandwidth was effective (typical ConnTrans operations on a daily basis take about one minute or so). However, program updates (typically about 10 MB each) take much less time with broadband access.
  
  
• Firewalls/Anti-Virus/Privacy Manager. Make sure you disable these security programs before you begin installing Field Sales. Also, ensure that mobile users do not block the ConnTrans operation via their Internet security software. In our case, some users who have McAfee Privacy Manager need to disable it so ConnTrans can work properly. We have not been able to correct the issue to use ConnTrans without disabling McAfee Privacy Manager.
  
  
• Pop-ups/Spyware. Clean up these programs before installing Field Sales. Using one of the Internet security programs or utilities such as SpyBot, AdAware, or Microsoft Windows anti-spyware (beta) is most effective.
  
  
• Java. Make sure the current version on the laptop is consistent with what Field Sales requires. Our version of SAP Field Sales uses Java version 1.3.10, although the current (non-SAP) version is 1.3.15. Make sure users do not install software that uses a more recent version — this causes the IPC to stop functioning!

Allow at least an hour to install SQL Server and Field Sales. Also allow enough time for the initial ConnTrans operation. In our case, importing the initial data via ConnTrans took about six hours per laptop.

6. Allow ample time to train users. Many sales representatives might tell you that they can learn programs on their own and don’t need training. This is not the case with Field Sales! While Field Sales is intuitive, we found that our sales representatives needed about four to six hours of training to learn how to use the product effectively.

Our training schedule looked like this:

Day 1 (full day): We brought our sales representatives in for training in our training environment for a day while we installed the software on their laptops. The training environment worked well, because all machines were set up consistently.

Day 2 (half day): We brought the sales representatives back for training, this time using their laptops with live data. We asked them to bring a live sales order with them to enter into Field Sales. We then reviewed the complete ConnTrans process with them, which included how to use the virtual private network (VPN) client. Don’t discount the ConnTrans training — this solicited many questions!

Bill Rady

Bill Rady is the Director of Information Services at State Industrial Products.

You may contact the author at brady@stateindustrial.com.

If you have comments about this article or publication, or would like to submit an article idea, please contact the editor.