After nine months of planning, preparing, customizing, and migrating data and applications for 4,500 SAP end users across 20 countries, MFC had only a narrow 72-hour window to launch the new system. The MFC project team was able to beat that mark, tallying a successful go-live in 71 hours.

Along the way, the Dallas-based MFC upgrade team learned several valuable lessons that apply to any company contemplating or in the midst of a large SAP upgrade.

1. Practice Makes Precision

Because MFC designs and manufactures products that are critical to national security, it has developed a culture of rigorous testing that extends to all departments. The IT department is no different, so the SAP upgrade team practiced six go-live events before finally initiating go-live in May 2008.

“It happens hundreds of times a year with missile tests and product tests — we go through hours and hours of simulation before a go-live or firing,” says Craig Vanbebber, senior media relations manager for MFC. “It’s really a culture within our business that says we’re going to go through and test this event to the nth degree until we get it right.”

To practice go-live, the team loaded a copy of the production environment into a sandbox environment. Using complete, up-to-date data, the team followed the upgrade procedures using SAP upgrade tools.

As it tested several iterations of the go-live event, the MFC project team composed a playbook that described each step along the upgrade path. The playbook, complete with screenprints of every step during the process and explanations for those steps, was given out to team members and used to gauge the exact timing of each step during the process.

“We didn’t want to have any surprises. We wanted to know the answer to every question before it came up,” says Chris Church, system integration analyst at MFC.

The team also recommends refreshing the QA system with a current copy of the production system just before the upgrade in order to get a dress rehearsal using real data. It also recommends testing at least one upgrade on a test cluster whenever applicable.

2. Audit Your Client GUI Versions

While the project team at MFC knew that many users were on GUI version 7.1 and others were on version 6.4, they were unaware that some users were on version 6.2. Users on versions 7.1 and 6.4 did not experience any errors after the upgrade, but the team was forced to upgrade all 6.2 users to 7.1 after the SAP upgrade.

As with many globally distributed companies, knowing which versions remote sites are running is a tricky endeavor. In this case, it was one that created extra work for the team as users on version 6.2 experienced transaction errors.

“It’s hard for us to know all the details for all the users at our off sites,” says Carl Neeley, system integration analyst on the team. “We found out on Monday after the go-live that we had some more upgrades to do.”

Luckily, SAP provides a direct upgrade path from GUI version 6.2 to 7.1, so the impact on the business was minimal.

3. Prepare for Password Problems

Strong passwords are a must in today’s defense industry, says Vanbebber, and the upgrade to SAP ERP 6.0 gave MFC an opportunity to upgrade the security of its SAP system by implementing case-sensitive, complicated passwords up to 40 characters. Previously, SAP logon screens had only required eight-character passwords.

Mandating password changes is tricky, however, because user lockouts can bog down the support staff and lead to widespread work slowdowns. Because another Lockheed Martin division had suffered password-related issues after a similar upgrade, MFC undertook a widespread communications campaign to prepare users for the change.

The first step was to set up an internal Web site to answer password-related questions and initiate an email campaign of instructions and reminders sent to all SAP users (Figure 1). The team also included password instructions and information on the new SAP logon screen at go-live.

Figure 1

The MFC project team prepared end users for changes to the password rules via email

The team was pleasantly surprised with fewer than expected user lockouts, well within a range the MFC service desk could handle. The MFC team also struggled with connectivity issues with an external system based on an old version of .NET, which forced the project team to temporarily require users of that system to implement all-uppercase eight-character passwords. Eventually the team was able to locate an SAP Note that fixed the issue.

4. Use the Downtime-Minimized Strategy

Because the upgrade window was only 72 hours long, the MFC project team had to carefully consider various testing strategies to streamline the process as much as possible. The team initially calculated that it would take 81 hours to perform and test the upgrade.

“There are two approaches that you can take — a downtime-minimized path or a resource-minimized approach,” says Church. “You have to choose between the two.”

The resource-minimized strategy is an option offered within SAP systems designed to reduce the amount of resources necessary to complete the upgrade by running the production system and upgraded system independently. The downtime-minimized strategy consumes more resources, but reduces downtime by allowing you to run parallel systems simultaneously during the upgrade.

By switching to the downtime-minimized strategy, the MFC team estimates it was able to shave 24 hours off of the projected upgrade time (additional software upgrades added some of that time back — see lesson 6) . Using this approach, users were able to access the old system with no performance problems — even as some upgrade processes were already in progress.

5. Avoid Project Bottleneck

One of the most challenging aspects of the MFC upgrade project was that it had to be completed concurrent with three other SAP projects.

“Normally, if you had a choice, you would not do this,” says Church. “But we had unavoidable business reasons for proceeding with the other projects at the same time.”

Because all four projects relied on several of the same key people, the MFC team struggled to find adequate time to devote to the upgrade project. Several key project team members relocated to other sites to pursue other major projects — further complicating post-go-live issues.

In fact, just ensuring that all necessary team members would be on site for the go-live weekend required a massive communications effort and the support of key executives who were aware of the issue, says Church.

6. Stick to the Baseline

Over the nine-month course of the upgrade project, SAP released several new software features that were later included in the scope of MFC’s upgrade project. These additions tacked on considerable time and expense to the project.

The team had planned and tested to upgrade to SAP ERP 6.0 SR2, which ships at support stack level 6. After months of planning and testing, MFC decided to upgrade to support stack 11 and add enhancement package 2.

That decision was made because newly-available features in both additions would support the company’s business intelligence (BI) initiatives. Church says switching the software baseline of the upgrade added hours to the go-live weekend and sent some aspects of the project back to square one.

“The change invalidated early testing, so we had to repeat steps — and repeating steps cost us at least two months in the project,” says Church.

7. Keep an Eye on Scheduled Jobs

Just before the upgrade began, the MFC project team saved all scheduled jobs. Once the system was upgraded, the team used transaction BTCTRNS2 to restart the jobs.

Some of the jobs that had been scheduled to run during the 72 hours of downtime during go-live weekend automatically started at that point, while others did not.

“We never completely understood why it occurred that way,” says Church.

Despite testing the upgrade multiple times, it took the project team considerable effort to discover why the glitch had not been discovered during validation testing.

“We had some key numbers we identified before the upgrade, and we wanted to verify that we got the same answers after the upgrade to ensure all the data was still there. Well, some of our key numbers had changed because some jobs had run unexpectedly,” says Church. “We had to react quickly to understand why our validation was not exactly correct, to understand job scheduling, and to determine that the data was valid so we could proceed.”

8. Rely on SAP Expertise

Because MFC was working on four SAP systems at the same time, the company had built a solid relationship with an SAP consultant who was familiar with the company’s landscape and needs. This relationship proved invaluable during the weeks immediately before and after the upgrade project. The SAP consultant was available to devote time to the project even when the rest of the team was too busy.

“By working with an expert, we were able to get through problems faster when our allowable time was constrained,” says Neeley.

9. Get Ready to Troubleshoot Performance Issues

Sluggish performance is a common aftereffect of many technical upgrades, and the MFC project was no different.

“Immediately after all of our users were back on board, we saw that we had sequential reads where we weren’t expecting them. Microsoft suggested that we update the SQL server statistics. This remedied part of that problem and immediately enhanced performance,” says Church.

The team realized further performance gains by updating the Database Shared Library (DBSL), which is the SAP kernel that communicates with the database. A few specific transactions continued to struggle after those issues had been resolved, but the project team relieved those issues by tweaking some selected ABAP code.

10. Craft Your Communications Strategy

It’s no secret that communication is key to a successful go-live event. MFC approached its communications strategy by carefully selecting employees outside the upgrade team — especially the service desk and desktop support groups — and by maintaining a robust communications channel with them.

The project team incorporated a variety of communications strategies to keep these employees aware of what was happening at each stage of the project, including emails, in-person meetings, and Web postings. The internal team communicated via phone messages and a special project-focused blog.

“With people working such odd hours, we needed a method for keeping everyone informed of our progress as we went through this long process,” says Church. “For example, when they wanted to know what time they had to come back in to work on Sunday, we were able to keep them posted.”

The Launch

In the end, MFC was able to upgrade an SAP system serving 4,500 end users dispersed around the globe with only three total days of downtime — two of them during the weekend. The team had to upgrade some GUI versions and troubleshoot performance issues, but thanks to meticulous planning and rigorous testing, MFC tallies its SAP ERP upgrade as a victory.

“We survived,” says Church. “And it was successful.”

Davin Wilfrid

Davin Wilfrid was a writer and editor for SAPinsider and SAP Experts. He contributed case studies and research projects aimed at helping the SAP ecosystem get the most out of their existing technology investments.

If you have comments about this article or publication, or would like to submit an article idea, please contact the editor.