Poor forecasting results are a risk that organizations face, but there are ways to mitigate that risk. You can address the poor results through multiple ways — through better implementation methods, by making sure end-user adoption of the system is better, or through an overall organizational model that supports continuous improvement. The root cause of many issues lies in the process design, technical configuration, and organizational accountabilities during the course of statistical forecasting technology implementations. While these causes are apparent, there are some less apparent causes of which many are not aware. Among the most common causes of low-quality statistical forecasts are:

• Lack of understanding about statistical forecasting principles
• Gaps in the composition of the demand planning/forecasting team
• Deficiencies in data quality and maintenance processes
• Ineffective technology configuration
• Expecting immediate results

In the sections that follow, we take a look at each one of these issues and present important considerations for your efforts to avoid them.

Lack of Understanding about Statistical Forecasting Principles

This problem begins on many levels. It usually starts with the assumption that any demand planning toolset can generate statistical forecasts on its own without thoughtful inputs and interpretation. Not only is this thinking wrong, but it often leads to the second problem, which is the assumption that because the toolset is fine, the people using it do not require much knowledge on the subject. As a result, core team members, leaders, and even project sponsors lack the depth of understanding needed for statistical forecasting development and use.

Teams are thus handicapped when faced with decisions that affect:

• Statistical forecasting methodologies
• Forecasting parameter implications
• Forecasting model applications
• Methods for effectively developing sound statistical forecasts to be used as a baseline in creating a demand plan

The result is usually major disappointment.

Yielding any toolset’s benefits requires a solid foundation combining academic knowledge, effective technology configuration, data management, and organizational design. Companies that are unable to distinguish good forecast numbers from bad ones often lack an understanding of the details of how statistical forecasts are developed in the demand planning toolset. Before blaming bad supply chain performance on poor statistical forecasts generated from the demand planning toolset, first consider whether the source of the problem stems from data cleansing, tactical refinement of forecast parameters, assignment of the right forecast models, or the underlying analysis required to identify the appropriate planning levels. Therefore, statistical forecast training is critical to organizational adoption and success.

The best time to start training is with the launch of an implementation, specifically as part of the project kick-off and initial blueprint design activities. Effective educational topics to consider include:

• Statistical forecasting overview
• Statistical forecasting process
• Inputs to statistical forecasting
• Statistical forecast parameters including definitions, how they are used in the process, and the effects of forecast parameters
• High-level overview of how the demand planning toolset generates statistical forecasts

During this primary training, be sure to focus on the toolset’s solution, terminology, capabilities, and constraints. Drawing a correlation to effective statistical forecasting practices and definitions can help differentiate expectations between business responsibilities and tool capabilities. There are certain things which a business or planner must do for forecasting to be effective — for example, setting the correct values for certain parameters based on their specific business knowledge and making sure that the results are acceptable. However, do not mistake statistical forecasting overview education for training in the use of the forecast toolset. Primary training should focus on the understanding and application of concepts and methodologies only. For example, SAP APO demand planning training must be kept separate and focused on the execution of hands-on processes and analytics within the SAP APO demand planning toolset.

If your company is past an implementation with no future ones on the horizon, this doesn’t mean that you have missed your window for educating the team. There’s always time for proper training. Training can include helping your team to identify specific forecast models that are applicable to their respective businesses, or to know how to selectively review forecast results to make sure the right models are being used, or if any refinements are required to specific parameters.

Of course, it always helps to train the team to be able to identify and establish a process to selectively review results or outputs of automated batch job forecasting results. For example, SAP APO demand planning periodic background jobs generate forecasts that demand planners must critically evaluate. However, in many cases, the planners never really look at the automated batch of generated forecasts because they either cannot make sense of it, or even if they look at it, they don’t know what is a good number vs. a bad number. The result is that nobody really knows if the forecast being generated is good, bad, or even relevant.

Learning how to refine algorithms can present a greater possibility for improving forecasts than technical fixes without effective architects. Instead of questioning the capabilities of the demand planning toolset, it can help instead to explore ways to educate users and help them improve analysis and refine forecasts.

Differentiated education and training can result in employees that know enough about statistical forecasting to make informed design decisions with appropriate expectations for the results to be produced. Also, communication of expectations can help set the stage for greater user acceptance in subsequent phases of an implementation.

The bottom line: Don’t expect to run the system on auto-pilot and get perfect results. A system is only as good as the intelligence you feed it. It is imperative that businesses use available functionality and configure the system to meet their needs — not just implement software and expect it to deliver results without human intervention or fine tuning.

Gaps in the Composition of the Demand Planning/Forecasting Team

Companies typically assume that a small number of demand planners is sufficient to develop and refine statistical forecasts for the entire global organization. We view this approach as taking the easy way out: Training fewer people means less of an investment. Then, perhaps a bit surprisingly, the company often asks people who generate demand plans in the IT department (or statisticians with demand planning toolset training) to make sure that the supply chain statistical forecasts are generated on time and without major errors or anomalies. This is another preventable trap. What looks like a lesser investment at first glance can actually turn out to be far more expensive in the long run.

Although IT (or even statisticians) can generate statistical forecasts and perform system maintenance and monitoring, they generally don’t have either the business knowledge or the awareness of market conditions to evaluate and refine statistical forecasts to reflect the organization’s strategy. These statistical forecasts rarely project strategies and market dynamics that influence how organizations intend to stay competitive in the marketplace.

As a result, when statistical forecasts are presented to the sales and marketing department without consideration of the marketplace effect, credibility is lost for the plans, the planners, and the whole department. It’s not uncommon for sales and marketing planners to feel that they can generate far more accurate demand pictures based on qualitative data (market intelligence) and personal experience because they are closer to the marketplace. When they harbor this feeling that their plans are better, it can lead to disconnected plans for the organization with no clear agreement on what to plan and who owns the plan. Furthermore, when statistical forecasts are based on pure statistics or demand history, they lag behind market trends.

Investments in the management of any change in the organization (such as roles, implementations of new tools, new processes, job expectations, or user adoption) can help mitigate the disconnection between supply chain, sales and marketing, and technology organizations. Change management efforts require early organizational investment, but they are key to creating a global demand planning/forecasting team.

Effective teams require effective involvement by senior leadership who must function as a liaison among the demand planning team, business, and IT management — and also act as the demand planning super user. Organizations with geographic or product complexity should consider business planners specific to their regions. For example, if a company has locations in Latin America with a certain portfolio of products, locations in the Middle East with another portfolio of products, and locations in Asia Pacific with a different portfolio of products, it makes sense to have planners specific to those regions who can incorporate their market knowledge. It is beneficial if these planners are part of the overall demand planning team.

For example, regional planners should be responsible for:

• Interacting on a daily basis with business sales and marketing users within that region to gain understanding of market situations
• Incorporating regional market awareness into the statistical forecasts generated by the demand planners for all products within that region
• Ensuring that all end users are trained and act as the liaison between the business and the regional users
• Recommending and implementing continuous improvements to the demand planning toolset and the statistical forecasting/demand planning process
• Working closely with IT to monitor performance and implement enhancements to the demand planning toolset

While change management involves some effort, the potential benefits of this type of demand planning and forecasting team setup are numerous. They include:

• Facilitating the development of credible, flexible statistical forecasts that other departments can use as a baseline
• Reducing the focus at a product-by-product level and encouraging managing only exceptions. This means there is no need to review forecasts for every product, but review forecasts for those products that exceed a certain agreed-upon threshold value that generated an alert.
• Involving all regions and markets in the development of a global demand plan to capture local market conditions
• Developing demand planning toolsets and statistical forecasting capabilities within an organization
• Laying the foundation for continuous improvement by improving end-user acceptance
• Mitigating the risk of losing knowledge due to attrition. This includes knowledge of the toolset itself, specific activities within the toolset, and a general knowledge of the conditions of a market/region. Other team members would be able to temporarily perform the daily activities of the person leaving the company because it would be an extension of their typical responsibilities, except for the fact that they may be dealing with a different set of products, customers, or region/market and the nuances associated with them.
• Reducing the time and effort spent on knowledge transfer for replacement employees by making training and onboarding (when someone new joins a team) more focused

The bottom line: Team structure and change management are critical factors that influence the success of any statistical forecasting implementation.

Deficiencies in Data Quality and Maintenance Processes

The most common cause of low-quality statistical forecasts is poor data quality. Two components that drive this problem are poor quality of historical data and ineffective ongoing data management and maintenance. Let us explain what we mean.

Poor Quality of Historical Data

The biggest problem here is that most companies assume that historical data doesn’t need to be cleansed because the forecasting toolset can be used to smooth outliers using pure statistics. It is true that many toolsets can be used in this way very effectively when outlier correction is executed on previously cleansed historical data — for example, data adjusted for promotions and business anomalies. However, if the data is not scrubbed to account for these promotions or business anomalies, statistical forecasts run the risk of having excessive quantities. This does not mean that outlier correction does not correct peaks and troughs in the data, but that the precision in how historical data is used to generate the forecast is lost. The result is that all too often data cleansing routinely starts too late, is under resourced, and lacks an owner. Because data cleansing intersects different teams, it is difficult for organizations to identify who should be in charge of it — should it be business, IT, or project resources?

When project management consciously defers data cleansing to save budget or effort, the consequences can last far beyond the implementation date. Trying to clean data in a production environment is more difficult. Companies often shy away from dedicating people to this task because it takes a lot of time and they feel that the time is better spent elsewhere. Furthermore, from a change management and business confidence perspective, poor and possibly erroneous results can affect the credibility of the toolset.

Ineffective Ongoing Data Management and Maintenance

Not all companies defer data cleansing. Many organizations see the value and necessity in cleansing data history. The problem here arises when the company doesn’t extend the data cleansing activities to all ongoing data maintenance and cleaning operations. An absence of data owners (that is, owners of specific data elements, such as a central master data management organization) and a maintenance infrastructure can produce forecast quality deterioration over time. Even when teams that own specific data elements (such as a centralized master data management team) are established, attrition plans are required to ensure that knowledge is not lost.

An effective way to help reduce this risk is to dedicate a team of business resources from the demand planning team who also focus on data cleansing pre- and post-go-live. By adding this responsibility to the job profile of the central demand planning group, companies can further help demand planners become more closely associated with business-driven cleansing events in the data. Intimate knowledge of data has the supplemental benefit of giving the planners deeper knowledge to help them fine-tune statistical models and create more informed forecasts analysis based on business conditions. Such ownership also allows the business demand planning teams to work more closely with IT and applications support, thus alleviating the risk of data cleansing being driven solely by the IT department.

The bottom line: Clean historical data is a prerequisite for developing good statistical forecasts.

Ineffective Technology Settings

The need to perform configuration housekeeping within the forecasting toolset is all too often overlooked. For example, the life cycle planning profile within SAP APO is a functionality that allows planners to plan forecasts for products that are at varying stages of the product life cycle (such as phase-in or phase-out). However, when life cycle planning profiles are not deleted or updated, statistical forecasts for the associated parts are sure to be erroneous and unacceptable to the business. Situations like this can lead organizations to conclude that statistical forecasting does not meet expectations and that alternative, non-statistical processes should be considered to support organizational planning objectives. Therefore, maintaining various applicable settings is critical to the integrity and quality of the statistical forecasts that the toolset is used to produce.

In SAP APO, the most common settings that need to be continually fine-tuned are:

• Statistical forecast models and parameters
• Promotions
• Life cycle planning (phase-in and phase-out profiles)

In the case of life cycle planning profiles, make sure they are maintained, updated, refined, or deleted as required. Ongoing support requires training, effective knowledge transfer, and institution of demand planning team roles, responsibilities, and routines.

Statistical forecast profiles, models, and parameters also require maintenance. Demand planners often assume that scheduled background jobs have calculated statistical forecast accurately and, therefore, they don’t need any further interference from the business. Due to this common behavior, the business demand planners might not validate the settings for statistical forecast generation or fine-tune the models and parameters. The tendency to use global settings and values to fine-tune forecasts (i.e., using one standard set of parameter values for all products, regardless of what type of products they are) also has risks. It is important to engage resources that have an understanding of statistical forecasting and fluency in the applicable forecasting toolset.

The bottom line: Part of the responsibility of a demand planner is to evaluate the appropriateness of statistical models and parameters to the products involved on an ongoing basis.

Expecting Immediate Results

A statistical forecasting implementation can take months or several cycles for the tool, processes, and users to mature. Unfortunately, most companies fall into the trap of looking for quick results and immediate improvements. Instead, they need to understand the reality of such an implementation.

Planners need proper time to learn how to use the toolset effectively, and organizations need to go through a few forecast cycles to truly understand what can be improved and how. A statistical forecast is nothing more than a directional estimate with degrees of confidence for other departments within an organization to leverage; it takes time and patience to learn and improve them. The goal is not to create the best, most accurate statistical forecast possible, but to provide sales and marketing departments with better baseline forecasts that they can leverage in developing the demand plan that drives the supply chain.

Better statistical forecasts can help organizations in their efforts to become more aware of the variables that have an impact on demand. If the correct metrics are monitored (e.g., statistical forecast accuracy versus sales forecast accuracy versus marketing forecast accuracy), the objective of the demand planning process shifts from developing accurate statistical forecasts to developing an accurate demand plan. Experience in helping organizations in their implementation of such processes has shown us that, initially, statistical forecasts provide a much better starting point for sales and marketing department planners. In turn, the sales and marketing department planners, over time, become more insightful and better with their forecasting. This results in sales and marketing forecasts that are more accurate overall than standalone statistical forecasts.

Unfortunately, in many companies, the fact that statistical forecasts are not as accurate as expected leads management to believe that the forecasting toolset is not as effective for developing statistical forecasts as they had believed. What they usually do not see is that statistical forecasts actually can help improve the overall forecast performance of the organization and that having statistical forecasts that are less accurate than expected is not necessarily a bad thing.

If the symptoms we describe are addressed, then the ongoing improvement of statistical forecast quality should help drive broader improvements in the overall forecast performance of an organization. Rather than make the mistake of not developing statistical forecasts altogether, organizations need to use their lessons learned and continue to optimize statistical forecasts based on collaborative, cross-organizational forecasts. The objective of such implementations should always be to improve demand planning by providing better baseline statistical forecasts. These can then be used to generate the demand plan that drives the supply chain.

The bottom line: The value of statistical forecasting should be measured over a period of time, not just a month or two months after go-live.

Key Takeaways

The quality of statistical forecasts is determined by a variety of causes, many of them not typically associated with the success or failure of a typical implementation. Often, factors other than the forecasting tool itself can determine the success or failure of an implementation. Informed organizations first look to transform their supply chain processes (and related organizational changes) and then use forecasting technology to enable these effective processes, rather than spend large amounts of effort (and money) to implement an IT-driven project for the purpose of automating existing processes. Implementing advanced, effective functionality, such as statistical forecasting in SAP APO, in a standalone, discrete manner cannot improve supply chain performance if the process and subsequent organizational effects are not aligned and integrated as part of the program itself.

Gautam Narayan

Gautam Narayan is a senior consultant with Deloitte Consulting LLP in the technology practice. His area of focus is supply chain planning, with specific experience, knowledge, and skills in demand planning and enterprise forecasting consulting services. He has consulting experience in demand management and process optimization, as well as broader supply chain strategy assessments and business transformation. Gautam is a specialist in advanced supply chain planning systems and has led the provision of consulting services in support of multiple, full life cycle implementation efforts for SAP Advanced Planning & Optimization (SAP APO) demand planning and Service Parts Planning (SPP) toolsets. He has experience working across a broad set of industries, including high technology manufacturing, healthcare/life sciences, consumer goods, and automotive.

You may contact the author at gnarayan@deloitte.com.

If you have comments about this article or publication, or would like to submit an article idea, please contact the editor.

Jerry Hoberman

Jerry Hoberman is a director with Deloitte Consulting LLP and leads their Northeast SAP practice with responsibilities for serving clients, mentoring practitioners, and managing operations. His 15 years of SAP implementation-related consulting experience has focused on helping organizations in their efforts to achieve transformational value through finance, supply chain, and customer process and organizational changes. Jerry has extensive experience with manufacturing, distribution, aerospace and defense, media and high-tech organizations.

You may contact the author at jhoberman@deloitte.com.

If you have comments about this article or publication, or would like to submit an article idea, please contact the editor.