Using Microsoft Tools for Business Process Management

May 3, 2005 - For the latest information, please see http://www.microsoft.com/biztalk

Introduction

Business processes are dependent and ordered activities that result in predictable and repeatable outcomes. Consisting of an organization’s operating procedures, institutional working knowledge, and information resources, business processes are designed to satisfy defined business objectives in an efficient and timely manner. In an efficient environment, the functional components of a process can be readily identified, adapted, and deployed to address ever-changing corporate requirements—a capability termed as business agility. By definition, business agility is an organization’s systemic ability to fluidly marshal and reconfigure resources in response to business requirements and opportunities. Business Process Management (BPM) tools are designed to provide for such agility by facilitating the creation and execution of highly transparent and modular process-oriented workflows that meet the operational performance standards IT organizations demand.

Automated business processes developed and executed within such an environment are characterized by the following attributes:

• Visibility of end-to-end process activities
• Process components and functionality that are exposed and self-describing
• Ability to integrate disparate information source and application functionality into a process
• Information flow and event notification that can be automated and monitored throughout a process
• Workflow participation that makes the most of desktop productivity and communication tools
• Service level agreements that can be specified, monitored, and enforced for activities in a process
• Ability to add, remove, or reconfigure any process activity or component, without disrupting the process
• Processes that can be monitored in real time or near real time
• Process designs that can accommodate any exception handling requirement
• Processes that can be easily replicated, extended, and scaled

With the support of XML and Web Services, BPM systems are transforming the way in which IT organizations are implementing and executing workflow components. XML applies structure to information, freeing it from any functional dependency on the software that operates on it. Web Services on the other hand provide the framework for application-to-application messaging and invocation over an unbounded network. BPM tools provide the additional support infrastructure to harness these capabilities to create, deploy, and execute the entire scope of workflow management, enterprise application integration (EAI), and trading partner integration (TPI).

This document examines how Microsoft’s tools for Business Process Management and supporting technologies facilitate the creation of processes that share the characteristics defined previously. The paper also describes how XML and Web Services are deployed within a BPM solution to achieve unprecedented modularity and extensibility. Finally, the document illuminates the gains in development and operational productivity that BPM technology engenders, which in turn enables real-time business agility.

A new paradigm

Just as standards-based Web servers and browsers facilitated the communication and distribution of information between people, BPM tools that employ standards-based XML and Web Services technologies will facilitate the wide-scale proliferation of automated and distributed business processes.

A defining characteristic of BPM technology is the elevation of design and development functions from the program layer to the information (document) and transport (messaging) layer. An application is no longer an opaque data-centric procedural construct; it is a messaging event or messaging agent capable of processing the exposed declarative properties of rich (XML) documents. Workflow processes, integration scenarios, or trading partner interactions consist of an orchestrated flow of messages that are routed, transformed, and processed according to message content, formatting requirements, and business rules.

Transparency and modularity are also defining characteristics of BPM technology. Not only are documents and messages exposed, self-describing, and extensible, but so are the communicating endpoint definitions, services, business rules, transformation maps, and process execution instructions that exchange and act upon the messages and documents. A process component can be “loosely coupled” with any other component so that a modification made to one activity or component in a process does not necessitate changes to other activities or components. Each component is functionally independent of any other component.

Microsoft Technology for BPM

Business Process Management technology represents a major conceptual reorientation of the methodologies of workflow development and deployment for business processes. As with any paradigm shift, it must result in significant benefits to be justified. Substantial evidence already indicates that users of BPM technology are achieving dramatic development efficiencies, accelerated returns on their investment, and most importantly, reduced resource requirements. Although XML, Web Services, and BPM platforms impose a new conceptual model on business process development and execution, the technologies required to do so are proven Microsoft® products that have been augmented to support this new paradigm. This section describes the core and supporting component technologies that constitute Microsoft’s process development and execution suite.

BizTalk® Server is Microsoft’s central platform for Enterprise Application Integration (EAI) and BPM and embodies the integration and automation capabilities of XML and Web Services technologies. BizTalk Server functions as a process execution engine and as a multitransport hub for messaging and document transformations.

Visual Studio® .NET is Microsoft’s flagship integrated development environment. The Orchestration Designer module found in previous versions of BizTalk Server is now an integral part of Visual Studio .NET with significantly more functionality. It is a visual development tool for building sophisticated workflows and processes that incorporate business rules, events, transactions, and exceptions and for linking these elements to implementation objects and messaging events. The assembled process generates an XML-based run-time script (BPEL) of the process that is executed in BizTalk Server.

SQL Server is tightly coupled with BizTalk Server and functions as its real-time data store for document tracking information and dehydrated instances of long-running processes.

Office 2003 redefines the functional concept and capabilities of Word and Excel by making XML the native file format. The applications can now behave like network clients, in the manner of a Web browser or e-mail client, but are capable of far more sophisticated and automated interactions with any source of XML information. Furthermore, in Office 2003, Microsoft also introduces InfoPath, an XML-based form application designed to address complex workflow documentation requirements. Because these applications can generate and decode XML documents with their respective schema definitions and processing instructions, they can directly engage in event-level interactions with BizTalk Server.

Active Directory® provides automated and federated authentication and authorization facilities as part of the process development and execution architecture. Its authentication and authorization capabilities facilitate sophisticated workflow processes that involve multiple participants, applications, documentation flows, and information sources. Active Directory also defines role-based participation attributes for workflow activities.

Host Integration Server and BizTalk Server Adapters facilitate integration with enterprise applications, legacy networking and transport protocols, and numerous data formats.

Microsoft Operations Manager and Application Center provide data center class system tools for building, monitoring, and scaling high-performance, mission-critical deployments of BizTalk Server and its supporting technologies.

Microsoft is at the forefront of XML, Web Services, and Business Process Management development and is committed to the implementation of these enabling technologies throughout its products. Nowhere are the potential capabilities of XML and Web Services more evident and maximized than within Microsoft’s integration, development, and productivity technologies. The core XML and Web Services capabilities found in the new releases of BizTalk Server, Visual Studio .NET, Visio, and Microsoft Office 2003 demonstrate a coherent vision for distributing EAI and BPA development and deployment activities, both along functional lines and among stakeholders. The details of this vision become apparent through an examination of the new features and functions of the component technologies described previously.

BizTalk Server 2004

BizTalk Server is the central product of the Microsoft Enterprise Application Integration (EAI) and Business Process Management (BPM) toolset and embodies the application integration and process automation capabilities of XML and Web Services technologies. BizTalk Server has two core functions:

• A process execution engine that manages the steps, applies the business logic, and invokes the supporting applications of a complex process and/or transaction set.
• A multitransport messaging hub for transforming and routing information to and from applications and process steps.

The release of BizTalk Server 2004 represents Microsoft’s third generation of BPM technology. The initial introduction of BizTalk Server 2000 demonstrated Microsoft’s early leadership in defining BPM functionality and supporting XML. BizTalk Server 2002 provided feature set refinements and performance enhancements. The new version of BizTalk Server is a major upgrade that incorporates the recommendations of thousands of users. BizTalk Server 2004 offers many new features and has been reengineered to provide substantial upgrades including improved performance and monitoring of process execution, robust Visual Studio .NET integration for programmatic control, and enhanced workflow modeling capabilities.

By leveraging the application integration and process automation capabilities of BizTalk Server and Microsoft Office, Microsoft has adopted the W3C XML Schema Recommendation. XML Schema is a specification that formally defines an extensive array of data type primitives and structural components for creating XML documents; it serves as a dictionary of abstract elements, attribute entities, and organizational rules. Creating XML documents that conform to the schema delivers a significant advantage: the meaning, function, and application of the document’s content is comprehensible to, and operable by, any XML-enabled application that can access the document’s underlying schema.

Every industry-specific initiative to develop a common vocabulary and set of procedures for the exchange and processing of information is based on XML Schema. By using XML Schema internally, BizTalk Server can import any XML schema definition natively without translation. This significantly reduces the time and effort required to facilitate sophisticated, interactive business scenarios—particularly those that involve the exchange of multiple document types. For example, consider the emergence of generic business-to-business (B2B) framework specifications for standardizing document content and data exchange procedures and behavior. There are typically hundreds of document types and exchange scenarios identified in each initiative specification, all of which are defined using XML Schema. By supporting XML Schema natively, the BizTalk Server messaging infrastructure and process execution engine can effectively support any B2B framework initiative.

Office 2003

XML Schema also figures prominently in the release of Microsoft Office 2003 as Word and Excel adopt XML as a native document format using an XML schema definition file. Also in Office 2003, Microsoft introduces InfoPath™, an XML-based form tool designed to propagate automated workflow capabilities throughout an organization. InfoPath allows workflow participants who create new information or perform analytical or content collection functions to generate, interact with, and exchange structured information. Typically, these activities are paper-bound or use a digital representation of paper. A form created in a word processing or spreadsheet program can be filled out easily enough, but the information that is entered in the form cannot be understood or processed without programmatic or manual intervention. Generating, conveying, extracting, manipulating, and reorganizing unstructured information in and from these formats is extremely labor intensive, inefficient, and costly.

InfoPath addresses this problem by creating smart forms that generate structured XML information, including processing instruction metadata. Underlying an InfoPath form is a template that incorporates one or more XML schemas, XSLT style sheets, embedded controls, and business logic instruction sets. The template controls the behavior of a form in the following ways:

• Generates multiple form views of the same information
• Constrains data types and values that can be entered in a form
• Defines and controls the dependencies for entering information
• Generates automatic, derived, and computed values
• Invokes events, prompts, and instructions based on dependencies
• Provides access to remote information sources
• Enables the incorporation of digital signatures

Form templates are created in a WYSIWYG design tool and require no procedural programming, predefined XML schemas, or XSLT style sheets. The schema and processing instructions are implicitly defined as the form template, which is represented in XML, is constructed from a palette of drag-and-drop controls, wizards, and dialog boxes. When an InfoPath form is populated, it generates an XML document that contains the entered and derived information, required metadata, processing instructions, and digital signatures of the participants who have accessed the form. The document also includes references to the template schemas and XSLT files required by other XML-enabled applications.

By enabling Microsoft Office applications to automatically recognize and process the information and metadata contained in a document, they become capable of engaging in event-level interactions. These “content-responsive” applications have the ability to facilitate automated workflow processes that take place between applications, and between applications and participants. This is the same fundamental premise of Web Services, and it redefines the functional concept and capabilities of these applications. They now behave as network clients, in the manner of a Web browser or e-mail client, but are capable of engaging in sophisticated and automated interactions with any source of XML information. Participants using these tools still engage in their workflow functions, but much more efficiently because of the elimination of manual processing tasks that are irrelevant to the effective execution of those functions.

Exchanging Information

BizTalk Server provides extensive transport and messaging services such as receive and send locations, adapters, pipelines and publish and subscribe distribution capabilities through a messagebox data store—all of which support content-based routing and processing. These facilities combined with the BizTalk Server document tracking and process monitoring capabilities, provide an overlay infrastructure for managing workflow processes. With BizTalk Server as the messaging and process hub and Office 2003 applications as XML-processing clients, participant involvement in workflows can be orchestrated, monitored, and qualified for reliability and performance metrics. This has the potential to radically alter the dynamics and efficiencies of workflow processing on an enterprise-wide basis.

The methodologies and technologies of process design, implementation, and execution are also undergoing a paradigm shift. As such, Microsoft is introducing key innovations in its BPM tools that will significantly improve process development and deployment. One of these innovations is the introduction of a set of high-level process-design and implementation tools that correspond to the roles of the participants involved in process development. These tools make it possible to graphically construct the business logic model of a process, link the steps in the model to actual implementation agents and components, and then generate an executable run-time instruction set of the finished process model in XML.
Roles of Business Analyst and Developer

Automating business processes is a collaborative activity that takes place between line-of-business professionals and programmers. Because each discipline has its own language and development issues, a communication and procedural divide separates their understanding of the development objectives. Consequently, software development is characterized by recursive revision cycles and ambiguity. Its methodology requires interpreting the apparent objective of a specification and translating it into a highly abstract form. While development notation systems such as UML provide business analysts with a structured approach for documenting use cases and specifications, programmers still have to interpret the documentation and translate its intent into a different language and format.

To take advantage of a business process paradigm that is exposed, loosely coupled, and document-driven, development tools and methodologies that incorporate these concepts are required. In creating these tools, Microsoft offers an alternative methodology for developing process-oriented applications. This methodology eliminates the inefficient interpretation and translation cycles that presently characterize application development.

Microsoft Visio remains the ideal tool for the business analyst. Designed for line-of-business professionals, Visio enables users to create diagrams of a business process by arranging, ordering, labeling, and linking various symbols that represent activities, events, decisions, flow, and transactions. When a process diagram is completed, Visio generates an XML representation of it in a Business Process Execution Language (BPEL) document. The BPEL representation of the designed process is then imported into Visual Studio .NET where each design object in the process will be implemented.

The corresponding product for developers remains Visual Studio .NET. Offering an extensive set of functionality, Visual Studio .NET contains a special project overlay template that provides programmers with a visual workspace. This workspace exposes the “binding” that exists between implementation objects, documents, and messaging infrastructure and the process steps created in the design stage. Though Visual Studio .NET is a programming environment, the method for implementing the process design does not resemble procedural programming application development. Instead, the visual workspace provides a collaborative environment where the programmer and process designer can work together on a diagrammatic object model of the process that both can understand.

Bringing the pieces together

In the workspace, Visual Studio .NET reconstitutes the diagrammatic representation of the business process. In addition to the primitive element design palette found in Visio, it also contains a palette of implementation objects: application integration components, COM objects, Web Services, XML documents, messaging and transport facilities, and transformations. These objects are visually “bound” to the design objects through messaging events or method calls using drop, drag, and connect activities. Furthermore, the implementation mechanisms for highly complex functions, such as transactions requiring two-phase commit, rollback, and so on, are built-in functions— thus eliminating the need to write complicated procedural code.

Because the primary dynamic of BPM exchanges is based on sending, receiving, inspecting, and transforming exposed XML documents, a BPM development tool should be capable of visually mapping the flow and exchange of information in a messaging event associated with a process step. This is facilitated through a special “Data” template that graphically represents the message flow in a process as well as the transformation of, and operations performed on, document elements through the process flow.

This assembly approach to building process applications incorporates the modular, loosely coupled, and exposed paradigm for creating any type of interaction or event. Each implementation binding or messaging event is functionally independent of any other binding or messaging event. A change on the design side does not affect the operation, function, structure, or integrity of the objects on the implementation side. In conventional process development, a complex integration or process scenario is embodied in opaque programming code. That code incorporates the structure of the endpoint objects, the process flow logic, the conversion of data formats, business rules, and the bindings to transport infrastructure. If a modification is required to any one facet of the integration module, the integrity of the entire exchange implementation is compromised. The risk of introducing unexpected behavior when modifying code has always been a pitfall of software development and it accounts for the hesitancy to make ongoing process changes in response to business requirements.

It is no accident that the tools and methodologies for developing business processes exemplify the modular, loosely coupled, and exposed paradigm that characterizes the processes themselves. This paradigm is fundamental to every aspect of the Microsoft BPM environment. The ability to visualize and comprehend complex business logic and its implementation mechanisms makes process development and maintenance infinitely more efficient and manageable—enabling organizations to flexibly adapt to new business requirements and opportunities.

Understanding BPEL

Real-world business processes are complex and incorporate numerous integrity controls: ACID transaction support, stateful persistence of long-running interactions, nested and parallel operations, compensation and exception mechanisms, acknowledgements, and correlation capabilities. As process complexity increases, the value of being able to design, implement, and document highly dependant and sophisticated behavior using a visual assembly methodology becomes more apparent. This is especially the case when processes require modification or one process design serves as the basis for another.

The Business Process Execution Language for Web Services (BPEL4WS or BPEL) is another key standard supported by Microsoft in BizTalk Server 2004. Developed collaboratively by Microsoft, IBM, and BEA Systems, BPEL was designed to orchestrate and coordinate Web Services so they can be engaged in collaborative and transactional behavior. The BPEL specification has been submitted to the OASIS standards body for review and eventual designation as a protocol standard.

While Web Services provide the methodology for application-to-application messaging and method invocation over an unbounded network, by themselves they cannot satisfy the operational requirements of a business process. A business processes is a set of dependant and ordered activities, the execution of which results in a predictable and repeatable outcome in a timely manner. BPEL will enable Web Services to ultimately meet these requirements. BPEL formally defines basic and structured activities that are used to compose sophisticated business processes. Because a BPEL instruction set is an XML representation of a process with a precise language and grammar structure, it provides a readable and understandable instruction set for documenting a process. In fact, the object primitives used in BizTalk Server Orchestration Designer are direct representations of basic and structured BPEL elements such as receive, invoke, sequence, flow, switch, partners, role, link, and source.

Business Rules in Business Processes

Business processes are driven by business rules, and the majority of modifications to a business process life cycle pertain to changes in business rules (as opposed to technology-related modifications). However, because business rules in conventional applications are embodied in opaque programming code, they cannot be accessed or modified easily and without potential disruption to running processes. There’s no argument that isolating business rules from procedural code or any process implementation mechanisms dramatically improves the efficiencies of managing and adapting business processes in response to new requirements or business conditions.

It is therefore of particular significance that, in BizTalk Server 2004, Microsoft introduces the Business Rules Composer. The Business Rules Composer consists of a business rule editor and engine for creating and processing sophisticated rule sets using a forward-chaining inference model. A rule set (or “Policy”) that drives a specific activity or function is created with the Business Rules Composer and becomes a resource object that is referenced in a BizTalk Server orchestration. Transparency and loose coupling governs the creation and implementation of business rules. A rule set incorporated within a BizTalk Server orchestration can be viewed, modified, or replaced both at design and run time, without affecting any other operational aspect of a process or interrupting running instances of the affected process.

Isolating, exposing, and publishing business rule sets as services that can be accessed by any application or process provides one of the most compelling value propositions for the Services Oriented Architecture paradigm. As such, the Business Rules Composer module is one of Microsoft’s most noteworthy products.

BPM and the Service Oriented Architecture

The next era of computing will be characterized by the detachment of information from applications, leading to a widely distributed Service Oriented Architecture.
• The meaning, function, relationships, and presentation of information will be self-describing and embedded in the information itself, using schema vocabularies and style sheet references.
• Information will be generated and published without knowledge of how it will be consumed or used.
• Applications will be capable of consuming information and the methods of other applications, while being consumed themselves.
• Processes will be self-configurable and self-modifying based on event-level interactions between rule sets and information.
• Entirely new applications and business models will evolve from this paradigm.

Microsoft’s BPM tools and the XML technologies on which they are built introduce the next era of computing based on the Services Oriented Architecture paradigm. In examining the innovations being introduced in Microsoft’s BPM toolset, it becomes apparent how cumulative XML functionality accrues: XML Schema enables Web Services and InfoPath. XML technologies embedded in BizTalk Server enable an entirely new model for application integration and process management. Each manifestation by itself has significant value, but combined, they offer the potential to facilitate wholesale efficiencies and innovative solutions. It is an object lesson in the whole being greater than the sum of the parts.

Operational Support for BPM Technology

It is one matter to have a vision of how things can work; it is another to actually make that vision work. The innovations engendered by XML require operational support context. On their own, they cannot be simply inserted into an organization’s existing infrastructure and expected to provide functional efficiencies, or to conform to the operational performance standards to which IT organizations are accustomed. Their value is actualized when implemented within a framework of complementary and supporting technologies that facilitate their use within an embedded infrastructure. Just as InfoPath and BizTalk Server complement each other, Microsoft’s array of proven enterprise applications (SQL Server, Active Directory, Host Integration Server, Microsoft Operations Manager, and Application Center) complement and support the implementation of these applications in a real-world, mission-critical context.

SQL server provides the transaction record data store for all BizTalk Server messaging events as well as dehydrated instances of long-running processes. Because SQL Server directly supports XML, it can maintain a record of the document instance associated with each message event as well. In this capacity SQL server provides the persistence capabilities of BizTalk Server.

The BizTalk Server application integration methodology is based on converting the native file formats of legacy and packaged applications into an internal XML representation, which is then transformed according to the requirements of the integration exchange. A prerequisite of this conversion function is knowledge of the native file formats. Microsoft’s Host Integration Server and other third-party adaptors provide this information to BizTalk Server in a comprehensible and usable way. Furthermore, these products provide transport gateways between supported transport facilities in BizTalk Server and the transport protocols used by legacy systems.

Lastly are the requirements for system performance, availability and scalability. Any enterprise-class application must be engineered to accommodate performance metrics monitoring, operational redundancy, and scalability at a granular level. This fine grain level of performance management, failure prevention, and scalability can be achieved with the support of Microsoft Operations Manager (MOM) and Applications Center. These tools are tightly integrated with all Microsoft server products and provide the operational assurances necessary for enterprise class information processing.

Conclusion

Over the last two years, based on thousands of successful deployments, BizTalk Server has clearly demonstrated that a platform based on XML technologies can dramatically improve the efficiencies and economics of application integration development. With the innovations introduced in BizTalk Server 2004, Office 2003, Visio, and Visual Studio .NET XML technologies will demonstrate an even greater potential to deliver operational efficiencies and benefits—and will, no doubt, assume a prominent role in enterprise computing.

Business Intelligence Meets Business Process Management

Powerful technologies can work in tandem to drive successful operations - Ventana Research 2006

The Corporate Challenge

Whether they involve distributing goods to customers, collaborating with suppliers or coordinating the efforts of employees, business processes are the foundation on which rest a business’s products, services and brands. They are the essential underpinning of an organization’s ability to function. In successful companies, business processes align resources to support the achievement of goals.

Business process management (BPM) is a methodology to ensure that those processes support a common set of goals and objectives. It involves automating and/or improving the effectiveness of process activities, tasks and outcomes to accomplish particular business purposes. Its goals include not only efficiency and productivity but, beyond them, control, responsiveness and improvement. Efficiency enables individuals to maximize the time they can devote to priority business tasks and to maximize throughput. Control assures that company resources are aligned optimally to execute strategies. Responsiveness and improvement support the competitive differentiation that enables a company to excel over others.

BPM, then, is about improving processes and implementing systems that enable the improvement. In pursuit of this mission, BPM starts with the following assumptions:

* As a business changes, so must its processes; as a result, they need to be revisited periodically.
* Processes are used by multiple organizations and stakeholders.
* Processes interact with systems and people. Those people can be employees, customers, suppliers or other stakeholders.

To attain the process goals of efficiency, productivity, control, response and improvement, companies first must understand their processes, the needs and skills of the people who use them, the changes that affect them and which areas need improvement. To understand these things, they need relevant information and the ability to analyze and apply it. Business intelligence (BI) tools provide this information, its contexts and appropriate analytics. BI delivers information that, when linked with BPM, gives people the input they require to improve business processes.

This last point is key: Both processes and tools are subordinate to the end of empowering people, at whatever level they occupy, to make informed business decisions that execute corporate strategy, improve performance and ultimately produce the best possible results.

Separation Inhibits Decision-Making

To benefit from improved process management and decision-making, however, companies first must bring together these two business enablers. Most organizations use BI and BPM technologies to serve separate purposes that seldom overlap. For the most part, BI deployments don’t focus on process, and BPM technology doesn’t provide metrics or an aggregate view of business. This situation reflects the predominant view that these are different technologies that each stands alone, delivering value to the business each in its own way.

This view is shortsighted, and soon it will be antiquated. Dynamic enterprises are beginning to forge links between processes and data, particularly in the areas of operational BI and decision-intensive processes. They are linking these two seemingly incompatible technologies in three distinct ways.

First, when initially implemented, most processes are sketchily defined. BI can generate analytics that help analysts tune the process to serve its purposes better. For example, process analytics can calculate measures such as the average process cycle time, durations of individual activities and time lapses between activities. Typically, BI used in this way will report on data about both transactions and process instances. This data becomes post-process feedback that can guide the process of tuning.

Second, most business processes manage the flow of data through an organization and involve decision points at which human interaction is required. For example, sales order processors need to obtain account approval before orders can be fulfilled. Business intelligence coupled with enterprise business process management can place the right data in the hands of the right users exactly when they must make a decision.

Third, many business processes are kicked off by information generated from BI. For example, trade promotion managers use BI to track the effectiveness of promotional activities. Their measurements help shape the next steps of the process. If the activities prove ineffective, managers can act to create more awareness or decide to bring the promotion to an end. In any case, decision-makers initiate the chosen business response process based on the BI information. The inputs gained through BI and BPM empower groups and individuals to execute strategy effectively and streamline tactics.

BI Sharpens Business Processes

The ultimate goal of business process management is performance management: managing the performance of the organization and its business network by using all assets in ways that achieve a common set of goals and objectives. Performance management enables all individuals to work across strategic, tactical and operational levels to align actions so that the organization produces rapid, effective responses to business challenges.

To connect processes with performance goals, companies need business intelligence capabilities, including metrics, key performance indicators (KPIs), executive dashboards and advanced reporting capabilities. They must go beyond just providing reports of basic operational metrics to facilitating access to aggregate data definitions and real-time information. Without BI, it is impossible to correlate process outcomes to corporate performance goals or to apply operational metrics to continuous process improvement.

BI is also a critical tool for analyzing process data. Without it, it’s virtually impossible to measure, evaluate and control business processes. Through process performance reports, BI gives business process managers the means to measure process execution as well as to gain insight into future workflow design improvements. BI tools map out process metrics such as throughput and flow rates in process modeling and then can be used to measure actual process execution. This data can be aggregated from the process implementation and displayed in real time in performance dashboards and reports.

Like corporate performance management systems, BI in a BPM environment supports high-level strategic metrics as well as drill-down analytics and sends alerts when process performance results begin to deviate from targets. In addition, by providing a platform for rule-triggered actions, BI can turn alerts into automated procedures for escalation and remediation in real time, allowing the organization to respond without delay to changes in the business environment. Ultimately, statistics distilled from actual operations can be fed back into the process model to begin the next cycle of performance improvement.

In many cases, business data is far more important than process data for analyzing and optimizing business processes. BI is the critical element of BPM here. To use the example of the mortgage loan approval process, the real-time value of credit scores is critical data for calculating the loan interest rate to be offered. In this situation, integrated business intelligence data is the major ingredient to support key decisions in the process.

BI is also critical to BPM because day-to-day business decisions must be made in the context of business process. BI not only harnesses the power of transactional data created by processes but also creates understanding that leads to action. Order fulfillment is an example of this intersection. In such a case, BI helps make these and other decisions:

* Should we process this order even though it exceeds the customer’s credit limit?
* Should we escalate this order to the regional sales manager because the credit score is low and this customer is a priority client?
* Should we extend more credit to this customer even though he or she has outstanding payments?

These key processing decisions cannot be made efficiently unless the data from each application and each process is incorporated in a BPM application. Integrating process and data on a readily available platform enables decision-making in context and in real time.

Integration of BI and BPM also addresses an issue in the relationship of enterprise software to business processes. An application represents a logical grouping of tasks that has been automated with the objective of reducing or supplementing human interaction. A business process is a set of activities performed by people and machines that are necessary to bring about a desired result. Thus, an application represents a subset of the tasks of a business process. The remaining steps of any business process continue to be human activities and interactions. The bottom line is that most IT applications automate only a subset of the tasks in a business process, and often they do so in an inflexible way.

Most applications in widespread use today do not help to streamline processes. In many companies, departments buy niche technology to solve their particular problems. Even though in recent years the application footprint of enterprise resource planning (ERP) has expanded, companies still find themselves with separate applications addressing customer relationship management (CRM), supply chain management (SCM) and supplier resource management (SRM). The problem with these applications is that they are focused on function, not business process. They provide no real linkage between transactions and processes when business processes cross individual systems and departments.

Another problem with functional applications is that they lack integration with each other. Even though enterprise application integration (EAI) technology can connect systems at critical data points, it really helps automate only transactions – not processes. Thus, integration, if used alone, actually can become a barrier to getting desired results.

Limits of Traditional BI

BI tools are widely used to complement enterprise applications. Application-specific BI tools do a great job of surfacing after-the-fact data captured in, say, an ERP or SCM application. Transactional data is stored and aggregated in a repository, such as an operational data store, for reporting, analysis and visualization. Application BI comes preconfigured to support the data schemas, application programming interfaces (APIs), security mechanisms and business logic of the particular transactional system. This approach speeds the extraction of information and helps maintain the integrity of source applications. Unfortunately, it also limits the scope of BI in decision-making.

BI reporting functionality often is included with an ERP or CRM solution, for example, in predefined or “canned” reports specific to that one application. The reports come from data without its process context, which typically spans multiple applications. Processes such as mortgage approval or call center routing, for example, require information from multiple applications and data sources. So the usefulness of the application-based reports is limited. Reporting embedded within each solution cannot satisfy the requirements of combining data and processes from disparate applications.

Another example of unnecessarily limited BI utilization occurs when data from multiple applications is homogenized in a data warehouse. In this situation, reporting and analytics provide enterprise-wide visibility, but the data loses business applicability – that is, the ability to facilitate collaboration and support decision-making in the context of a specific business process. An example of this is the different ways customer data appears within a data warehouse and outside of it. Data homogenized from sales force automation software (sales opportunity data) and a separate order management system (customer order data) provides good insight into customer buying trends and fits well in the structure of a data warehouse. However, it is missing the granular customer contact data required for the business process of escalation.

Traditional BI also has a latency problem: The information is not available quickly enough to support individuals making immediate decisions. Often the stored data is out of date for operational response; for example, a bank today needs to complete loan approvals in hours, not days. Hourly and near-real-time information is important for operational responsiveness to such competitive pressures.

Also, traditional BI information is not delivered in the context of a business process, but rather at the individual, entity or activity level. Information should be in the context of causal activities preceding a situation and consequential activities following it.

The latency and context issues compound another problem: BI information is not always actionable. Individuals receiving analytic information also need options for possible next steps. For example, call center managers must be given information that is timely (such as that call volumes are in excess now), and notification must be in a form that supports an active response (telling where to route those calls).

Even if the problems of integration, collaboration, latency, context and actionability in business intelligence usage were solved, corporate executives still would not have the complete foundation they need. The ultimate goal of BPM is performance management, and a key component of performance management is alignment: the process of linking strategy with corporate goals and objectives in a way that makes best use of the company’s resources by coordinating the efforts of every member of the organization. Alignment links strategies and goals across the organization from the present into the future. In other words, performance management is about managing the business moving forward, not just looking at past results.

Here traditional BI falls short in that it looks backward at results and analyzes what happened. While understanding the past is a necessary part of the performance management cycle, dynamic businesses need to look also at what is happening now – that is, to get real-time data about events and processes. And they need to project what will happen – to be able to anticipate the consequences of future choices, actions or plans. All these time-oriented steps are needed for good performance management, and so for proper business process management.

Plan for BI with BPM

Companies should do more with their BI deployments. Most businesses do not set performance targets and goals for processes and then monitor those. Nor do they map strategy to business processes.

Henry Ford did it. His strategy was to produce cars at a lower cost by increasing the throughput of production. He divided automobile production into specialized tasks and set time standards as target goals for each, to be able to monitor production rates. Concurrently, he employed accounting methods to measure the favorable and unfavorable cost impacts of faster or slower production speeds. By employing these methods he could optimize production to meet his cost goals.

Ford’s methods illustrate how setting corporate performance targets and goals trickle down to performance goals and targets for business processes. Today’s companies can use modern methods to align their own processes with goals.

To assure that process performance is connected to results, organizations need to include BI at each stage of the process performance life cycle. That life cycle includes process planning, execution and management.

At the first stage, “planning performance,” managers need to define, in order, the desired outcomes of a business process, the resources needed to support it and the strategy by which to achieve the outcomes. This should be a cross-functional exercise that involves all managers and line workers who participate in the process. They should set expectations via the creation of goals and planned outcomes and record these goals and expected outcomes for future plan-vs.-actual comparison. The next step is to determine resources and support requirements. Plans will become operational once the business rules and individual process steps that make up the overall process all are defined. In this first phase, managers make sure that the resources, structure and support are in place for proper process execution.

At stage two, “executing performance,” each step of the business process is carried out exactly as the plan and the rules specify. At each step, inputs, outputs, messages and process data are recorded to monitor the behaviors of users of the process.

At stage three, “managing performance,” managers analyze process outcomes and process behavior data for deviations from the performance plan and to determine what action needs to be taken. This action can be corrective, preventive or sustaining. Typically, business process managers give feedback to the process participants about the execution, and the process is changed as appropriate to pursue the targeted outcomes more effectively.

Most process managers stop here. But they ought to go further and fully close the performance management loop by providing feedback to stage one (planning performance). If processes are to be changed, new business rules, expectations, resources and support also must be determined and new process outcome expectations recorded. All these steps must be followed if optimal performance is to be attained.

To ensure that the goals are met, business unit and IT process managers should apply BI and BPM technology jointly at each stage of the process performance life cycle. In stage one, business and IT process planners need a flexible BPM modeling tool that meets both their needs. While the logic of an automated business process can be implemented directly in a programming language or script, successfully creating an automated business process usually requires collaboration between software developers and business users. For the most part, business-side people should be responsible for designing, maintaining and managing business processes. But since they aren’t programmers, they need to be able to do the design work in a simple, graphical environment in which they define process steps (workflow) and process parameters (basic business rules).

Integral to the effective design of these processes is the incorporation of parameters that will capture the performance data or events for later business intelligence reporting, analysis and business activity monitoring. Business and IT process planners must know this and work together to bake these key capture points into the process. They also must work together to create visual representations of how strategies map to business processes. Here, the same process designers must incorporate BI technology to map key performance indicators and corporate initiatives into the process flow diagram.

At stage two (executing performance) the automated business intelligence data capture defined in the process records key inputs, outputs and process data. It is important to remember that most business processes span multiple applications, so any process messages and transactional data have to be synchronized.

BI technology plays another key role is stage three (managing performance). Here, technology helps close the loop by summarizing performance outcomes. Scorecards, dashboards, alerts and portals play supporting roles with visual representations of the resulting data and events. Scorecards in particular provide an effective way to display process KPIs; they provide easy-to-understand icons such as traffic lights (in red, yellow and green), gauges and heat maps that show the performance of a KPI metric against a target process benchmark. Desktop office applications, particularly spreadsheets, also play a role since they extend performance analysis by serving up detail data, allowing drilling down to more detailed levels of information to spot exceptions and trends.

Convergence Improves Decision-Making

When BI and BPM are linked, they enable people to make better decisions – about business processes, within business processes and that initiate business processes. This linkage brings content and context to individuals and teams across the organization. Let’s look at some common situations in which the technologies can work together to improve business decision-making.

Companies can use BI and BPM to analyze and tune the efficiency of a business process. For example, businesses that process insurance claims must establish connections within a multisource data entry process that deals with thousands of claims per day. Claim processing moves a claim application from entry to validation to authorization to approval and ultimately to payment. The company needs to manage the time it takes to collate data from its many source points and to incorporate rigorous screening checks. Process designers should identify all points at which to customize claims, to perform additional checks and to establish what authorizations and business rules apply. To measure efficiency and effectiveness, process planners need to analyze the business model, identify bottlenecks and make improvements before executing the modified business process. BI provides these analytics for process efficiency and effectiveness by reporting on cycle times and process constraints.

A second example of BI and BPM convergence can be found in the order fulfillment process. Here, some process decisions are based on business data. When an order processor has determined that a priority customer has exceeded his credit limit, a trigger initiates an automated escalation process in which an alert is sent to the account manager and regional sales manager, notifying them of a delay in fulfillment. The alert notice can point to the open order (in the transaction system) and the sales history of the account (in the BI application) for them to review. The same alert also can be sent to the credit manager – the decision-maker – who sees not only the open order and the sales history but also the customer’s payment history. In this way business process and business intelligence work together in ways determined by the business role and data security level defined for each individual.

A third example of convergence occurs when BPM manages a process that is kicked off by information generated by BI. The trade promotion management process, mentioned earlier, includes creating a promotion, selling it and evaluating its effectiveness. When the promotion is launched, its process infrastructure should include the steps needed to measure its effectiveness. These measurements help shape reactions to the process as it continues. If the promotion proves to be effective, then a process must be started to deliver more product units to the retailers. If the promotion proves ineffective, another process has to begin to create more awareness of the promotion or, alternately, bring it to an end. In any case, decision-makers need to see all the appropriate business intelligence reports on the promotion, and from the same screen the marketing manager needs to be able to initiate the chosen business response process.

The Right Approach

To automate and improve business processes involves not only business process decisions but technology choices. Bringing together BI and BPM capabilities need not require a large investment in a single enterprise-wide technology. It is possible to procure a solution that can be implemented quickly without sacrificing the ability to scale and then built out incrementally. For many companies, this will represent the most manageable and cost-effective choice.

Business process management software should be able to integrate systems, trading partners and business users through flexible business processes. But as information technology continues to evolve, companies acquire an increasing number of systems and need to add more business users – and both these steps increase the challenges of implementing BPM.

The challenges begin with trying to get the several systems to work compatibly, a task that expands with each new business need and software upgrade. In addition to an increasing number of points of connectivity, business drivers such as just-in-time inventory and real-time reporting have introduced demands for accessibility and visibility into the processes that span these connected systems. BPM products have been in the market for more than a decade, but many are expensive and difficult to use and address only a single aspect of a company’s needs. What’s more, often they are hard to integrate with other BPM and BI technologies.

Typically, a company realizes the best return on investment when BI and BPM are brought together and integrated with a technology platform that enables joint development, management and interaction. Customers that desire to realize the benefits of BPM and BI technologies should consider the extensibility of the software platform as well as the degree to which the BI and BPM components are integrated. The platform also should extend to tools in everyday use, such as Microsoft Office. The best solution will make the most of both existing and new investments.

Business Process Management Overview

July 12, 2006 - Microsoft Corporation

What is Business Process Management?
A business process is a set of linked steps or activities that taken together result in a specific business outcome, either internal or external to the organization. Documenting a business processes involves describing what is done, why it is done, how it is done, who (or what system) does it, as well as how well it is done.
Business processes may be structured or unstructured, depending on the extent to which the underlying steps are fixed and therefore automated or changeable and generally executed by people or people interacting with systems.
Business processes underlie all organizational efforts, and the effectiveness with which they are carried out contributes directly to critical business goals such as customer retention, length of time it takes to fulfill a product order or service, or regulatory compliance.

BPM Defined
Business process management (BPM) is a management discipline that combines a process-centric and cross-functional approach to improving how organizations achieve their business goals. A BPM solution provides the tools that help make these processes explicit, as well as the functionality to help business managers control and change both manual and automated workflows.
Business process management has its origins in total quality management and business process reengineering. While it adds to these a technological framework, it is more than just the combination of these disciplines. BPM is an IT enabled management discipline that promotes organizational agility and supports the efforts of people to drive process change and rapid innovation. As such, BPM supports the alignment of IT and business activities both within the organization and with business partners and suppliers.
Although early implementations of BPM have unfolded in large enterprises, managing business processes is critical for any sized organization that would benefit from greater visibility into and control over the processes that support their business goals.

Who does BPM?
Business process management is inherently a cross-disciplinary exercise involving personnel from all areas of the organization—from the process owners who are responsible for getting the day to day operational work done, to the department heads who are responsible for managing divisional areas, to the CXOs of the organization providing oversight and direction. That said, most organizations appoint or hire a point person to oversee the BPM process. That person, often referred to as a business analyst or process architect, generally comes from the business side of the organization, but nevertheless has a strong understanding of IT, critical to effective liaison with the IT department.

What BPM Is Not
While business process management endeavors make use of IT technologies to automate and increase efficiencies, BPM is not an amalgam of existing technologies cobbled together and labeled BPM. Nor is it a form of application development which in which business requirements are thrown to IT to interpret and develop, a solution emerging only after extended time and often with poor success translating business requirements into functionality.
BPM is a fast, agile process in which business and IT work together using tools that enable them to arrive at solutions that not only support the organization’s efforts right now, but provide the framework enabling rapid adaptation to new challenges.

What is the BPM lifecycle?
Every organization’s day to day work practices—from shipping clerk to CEO—support the organization’s ability to remain competitive. Workflow practices that support organizational goals are valuable assets to the organization, a principle made more obvious when you consider the knowledge that needs to be exchanged for successful outsourcing or mergers. Not all workflow practices are valued for the same thing: in some cases the most effective practice is also the most streamlined and efficient; in others, the practice that best supports organizational goals is one that enables the greatest ingenuity and innovation.
Driving improvement in business processes has the advantage of carrying the same underlying logic of process improvement and total quality management initiatives. The difference, as we have already seen, is tying together processes that span unstructured to highly structured work—from people to systems—with agile software support.
We see the iterative stages of BPM lifecycle as divided into the stages shown in the diagram: planning, model and design, develop and deploy, manage and interact, analyze and optimize. Preceding the onset of the BPM lifecycle with a planning stage helps to ensure success.

Planning
Business process improvement doesn’t just happen, and as a strategist in the business world, it’s evident that attempting to implement organization-wide changes without first culturing buy-in and support is almost guaranteed to simply waste resources and sour employee good feeling.
Businesses need to focus their initial efforts on understanding their current business practices and planning on the best course of action to improve their current situation.
The initial planning stage consists of identifying and prioritizing a short list of candidate BPM projects, identifying key players whose input is critical to project success, and establishing the governance to ensure that the BPM project stays on track throughout all of the iterative stages of the cycle.

Choosing the Business Process to Improve
Which business processes are good candidates for an initial BPM project? Look for two things. Those processes that have considerable impact on an organization’s ability to achieve its goals either in the short or long term. And those processes, which if improved, will enable the organization to realize a high return on investment.
Business processes related to productivity can have considerable impact on organizational efficiencies in the near term. Look at areas where there are known complaints, whether from customers, trading partners or internal staff. At the root of these complaints are processes that are ineffective and require reworking, streamlining, or better management.
In those business areas where today’s actions have an impact 12 to 18 months down the road, again prioritize those processes that are in greatest need of rework to better support such goals as innovation, growth, and regulatory compliance.
Because organizational buy-in is so critical to the success of business process management, the project that has the greatest support for the BPM process may be the one your organization chooses to go with. A successful BPM experience in one area of the organization will help create enthusiasm for subsequent projects in other areas.
Having completed the preliminary planning, the iterative stage of the BPM lifecycle begins with formally modeling the business process and beginning design work to build an effective solution.

Model and Design
Modeling business processes that span people and systems within the organization, as well as those that reach across to the business partners in the supply chain, is the critical first step of the iterative cycle of BPM. Why? Because modeling directly supports a number of goals, first and foremost of which is to make explicit the opportunities for how the business process might be improved.
Modeling enables you to clearly and explicitly lay out each step in a business process, including the critical touch points across people and disparate systems. It is this mapping that enables informed process improvement; in addition, detailing processes through rigorous documentation enables organizations to meet other goals, including effective outsourcing and regulatory compliance.
A business analyst—usually someone with a foot both in business and IT—oversees the model and design process. During the modeling phase, the analyst works first with process owners and end users to formally define and document the target business process. This stage maps the process from end-to-end, not only across functional divisions (such as finance or marketing) within the organization, but also across to relevant business partners outside of the organization. More than simply a mapping process, modeling also makes explicit the business rules that underlie the various steps in the process—how each of the steps relate, constrain or derive from others. During this modeling stage, stakeholders are encouraged to share suggestions on how changes to the workflow process can better support business goals.
Once all the steps of the target business process has been defined and documented (and often graphically represented in a model), the analyst works with IT to determine how IT can design improved support. This means looking at the entire business process to assess where integration, automation or workflow redesign will improve efficiencies or increase agility. Obviously not all steps in a business processes (particularly those steps that represent highly unstructured human workflow processes) will benefit from being linked to an underlying technology; for the many that will, such linkage is the primary focus of the develop and deploy stage of the BPM lifecycle.

Develop and Deploy
Armed with the detailed model of the business process and the underlying business rules, the IT developer maps the business needs onto the underlying technologies that contribute to the complete solution. These technologies include line of business applications and messaging systems, the former supporting highly structured workflows and the latter unstructured workflows.
Using BPM tools to uncouple business rules from their underlying technologies, the IT developer abstracts these rules to a layer independent of the systems and applications, then joins the logical components into a “composite application” that combines the functionality of underlying systems. Not only can this composite application be rapidly assembled and reassembled, it provides outputs that individual systems alone cannot (such as real-time metrics across the entire business process).
Much of the IT developer’s job during this stage is to ensure that the solution incorporates the functionality, performance metrics, and user interface to meet business user needs. Once the solution is tested and deployed, business users and IT alike begin the manage and interact stage of the BPM lifecycle.

Manage and Interact
With the BPM solution for a specific business process in place, end users interact with the process as it runs though the various process stages, providing or requesting information (though form-based interfaces) as appropriate. At the same time, business users can monitor for potentially disruptive events along the various steps of the workflow process (such as an event indicating low stock of a production item), and take appropriate action as required.
In the background, IT staff manage the entire automated business process solution to ensure that the running process continues to meet capacity and availability standards.

Analyze and Optimize
The BPM solution provides a number of different means through which the business analyst or manager can continuously monitor and measure improvements both within single and across multiple business processes.
Benchmarks are established using service level agreements (SLAs) and key performance indicators (KPIs). Information derived from performance metrics is critical in driving the iterative process of optimizing the business practices and policies that support organizational goals.
The most effective BPM systems enables IT staff or non-technical business users to optimize of business rules in real-time, an iterative process that enables rule change, versioning and simple execution.

What are the benefits of BPM?
The reasons for embarking on a business process management effort are as varied as the organizations that undergo the endeavor, but most organizations are driven by the following benefits:
• Increased customer retention, gained through better, faster customer sales and services, as well as providing customers better access to resources and information through the various access channels.
• Reduced process time, gained through process optimization and efficiencies. Shorter process cycle times speeds time to market and time to service.
• Improved regulatory compliance, gained through improved process control, regulation and monitoring. One of the most effective ways to move toward compliance is to replace manual processes with automated ones, lowering the risk of errors and security breaches.
• Improved efficiencies across organizational boundaries such as departments, branches and trading partners (including both supply chain and outsourcing). Efficiencies are gained through improved visibility and control, as well as automation and integration.
• Reuse and create new IT assets, through integration with legacy applications and the creation of new composite applications that help to overcome their limitations.
• Greater personal productivity and satisfaction, resulting from greater insight into processes and improved workflow.
• Reduced risk, reduced waste and more profitable allocation of human resources.
• Increased agility through compression of BPM lifecycle, allowing for more rapid process innovation and response to changing business conditions.

What are the challenges associated with BPM?
Successful business process management initiatives require both near and long term planning and goal setting, and the goals and means by which to achieve them must be supported by executives across the organization. Without clear alignment on the goals and commitment to the BPM process, organizational resistance will defeat the initiative.
BPM requires strong communication both within and across departmental boundaries. Handoffs between departments are notorious for providing the greatest challenge to process improvement, and the BPM process is no exception in this regard.
A strong business analyst or process architect, key to ensuring that the BPM process remains on track, is critical for ensuring effective communication throughout the initiative. It is the individual in this role who ensures continuous alignment between business and IT; failure to keep business personnel engaged in the process can too readily result in IT solutions that fail to meet business goals.
Both the business analyst and the IT developer must ensure that the chosen BPM system works with existing systems and is capable of supporting current business practices. Since one of the primary goals is to integrate processes across both people and systems, the BPM solution must be flexible enough to ensure that integration efforts can meet across the divide that has traditionally separated business and IT.
Having outlined these challenges, it is obvious that the first BPM project will be the hardest: familiarity and expertise with the method is most likely limited, and organizational reluctance (to yet another initiative) will likely be at its greatest. But with the completion of the BPM lifecycle and the delivery of measurable improvements, subsequent iterations to fine tune the process will be smoother, as will the undertaking of new BPM projects.
As the iterative process of BPM unfolds, workers develop a more nuanced understanding of their workflow processes. At the same time, IT gains a more sophisticated understanding of the business processes they are supporting. For their part, business managers develop a better understanding of how technologies can support their needs, which increases their abilities to suggest targeted changes to the functionality of the composite application supporting their needs.

How can your organization get started with BPM?
It is critical that the organization appoint or hire a BPM process architect who understands both business requirements and technical solutions. In addition to garnering organizational buy-in and choosing an initial project that targets a weak process directly impacting the customer, as well as one that offers a high return on investment, it is imperative to choose the right business process management solution to support the BPM process.

Choosing a BPM Solution
The BPM solution market is highly fragmented. BPM enabling technologies span a broad spectrum of activities, but can be generalized as supporting either activities that are unstructured (such as ad-hoc or collaborative tasks) or activities that are highly structured and often transactional in nature. Unstructured or human-workflow activities are supported by tools that center on the information worker; structured (or straight through processing) activities are supported by traditional IT business applications and integration middleware.
Until recently, solutions have been focused on one area or the other. Because of that, it’s important to look for comprehensive BPM solutions that:
• support both human-centric and straight through processing activities
• offer a solution framework rather than simply a point solution (necessitating multiple point solutions)
• support process standards enabling integration across different platforms and with different line of business applications
• support integration across business partners
• support solutions for different industries
• provides business users with the ability to define the business rules that make up business processes, without IT programming
• provides visibility into business processes, enabling real-time monitoring and event management

BPM - A Cure for Institutional Memory Loss

By: Ken Mullins, MITRE
Wednesday December 12, 2007



Across the federal government, large numbers of baby boomers are reaching the age of retirement. In 2006, more than 60,000 people left the civil service. The federal Office of Personnel Management believes that 2009 will be the peak year for boomer retirements. What can federal agencies do now, to stop this massive loss of institutional knowledge?

Stepped-up recruiting and staffing efforts will help a little, as will additional emphasis on outsourcing. Still, those measures will not be enough to stem the flow of institutional know-how, which has been steadily accumulated for decades in the minds of the federal workers, now rushing for the exits. To prevent this loss of priceless, corporate memory, more business process modeling is needed – just as soon as possible.

Departing workers can be motivated to support such loss-prevention measures, if their managers challenge them to leave behind a legacy that will have the effect of preserving their hard won years of experience. Replacement workers, on the other hand, can also be motivated to proactively engage right away in an organization’s process improvement program. Incentives can motivate them to immediately share some of the best practices and lessons-learned, which are brought to their new job as part of their prior experiences.

Preserving Institutional Knowledge

Business processes reflect the means by which an organization provides value to the customers it serves. Business Process Modeling (BPM) is not just for organizations thought to be in financial trouble. For agencies faced with the threat of a brain drain, BPM can provide much needed relief. Because process models demonstrate how work is performed in an organization, they provide valuable insight into every aspect of business performance, from management, to operations, to customer service -- even to measuring and accounting for results. By capturing how work is performed before staff members retire, and having the results validated by the retirees themselves, the organization is left with much of the institutional knowledge accrued by those workers – in the form of documented, reusable process models.

The validated business process models can then be put to good use in a number of very constructive ways, including these:

* Development of training and orientation materials, for use by new staff members
* Development of standard operating procedures, prescribing the performance of specific tasks
* Establishment of business rules, used to guide decision making at all levels
* Publication of communications material, used to inform customers of the services available from the agency
* Preservation of a process baseline, from which gradual improvements can be planned
* Incorporation into the agency’s Enterprise Architecture (EA), since business processes, activities, and information flows constitute large segments of the EA
* Application to the development of business cases, used to substantiate the need for capital investments

Energizing Experienced Workers

Tapping into the work ethic of senior staff members, especially those nearing retirement, helps the organization in another way that can be easily overlooked. Many workforce studies have shown that older workers are energized by some fairly basic interests, like these:

* Motivating Work
* Sense of Belonging
* Pride of Mission
* Strategic Direction

If properly engaged, mature workers will earnestly strive to provide a view of how their work fits into the operation of the enterprise as a whole. As active participants in the success of the agency they work for, their institutional pride often motivates them to look beyond the obvious, while fleshing out an enterprise view of how the business really operates. Indeed, experienced workers will often willingly accept responsibility for process ownership and process management. With a little coaching on the mechanics, they will help to plan and execute a program of process design. Refuting the notion that their agency operates as a set of discrete units with hard-drawn boundaries, their years of experience seem to open their eyes to the fact that individual business units actually contribute to groups of interlocking workflows and information flows that often traverse organizational boundaries. Some processes - like the payment of a claim for government benefits - will be found to extend far across the enterprise before an output is produced that is valued by a customer. They are also more likely to recognize that the worth of such extended processes to the business must be measured more holistically – not just by adding the sum of the metrics for each segment of the process. As veteran performers of the work they prescribe in models they own and manage, they are also able to convey the credibility needed to secure essential buy-in from other members of the workforce.

Engaging Replacement Workers

What better way to start a new job than to be invited to weigh in – at periodic intervals during your first year (perhaps as part of doing rotations around the enterprise) – on the way the place works from a new employee’s perspective? Many people would react by passing on such an invitation, suggesting they don’t have enough information about the new work environment to comment intelligently on the way it works.

Suppose, however, that you were seriously encouraged to help your new organization by sharing your own experiences with similar work at former places of employment. Also imagine that you became convinced that constructive feedback on business processes and performance at the new place of business can be provided in a manner that adequately protects your identity. Wouldn’t that provide some incentive for you to immerse yourself sooner, in learning about your new organization, while striving to make a meaningful contribution to its process improvement program?

Many new employees, especially those well-credentialed, are eager to share their best ideas and practices with a new employer and their colleagues right away. To overcome any natural reticence to speaking out, employers should offer special awards and other forms of recognition to individuals who manage to get a process improvement suggestion adopted by the new organization and put into practice during the person’s first year on the job.

Final Thoughts

Faced with the departure of large numbers of employees, as baby boomers retire in droves from the federal workforce, government agencies can mitigate the risk of losing institutional knowledge by engaging in a comprehensive BPM exercise before all that knowledge is lost. To realize the best results however, the soon-to-be-retirees must feel compelled to leave for posterity, a record of hard-learned lessons and the practices they perfected over the years by relearning some of those lessons many times over. To obtain the greatest benefit from the legacy left behind by retirees, those hired to replace them should also be induced to engage in a thorough review of the BPM artifacts created by their predecessors. An agency can only benefit, it seems to me, from a situation that encourages new employees to share the beat practices they bring to their new jobs, while at the same time being rewarded for offering constructive suggestions to improve upon the fruits of their predecessors’ final labor.

Business Process Management 101: The Basics of BPM and How to Choose the Right Suite

Business Process Management is gaining adoption, but just what is BPM and how do BPM systems work? This article clears up some of the confusion and helps you choose the right product with a six-step guide to selecting a BPM suite.

By Tulu Tanrikorur
May 7, 2007

IT giants are either buying business process management (BPM) vendors or expanding on existing BPM capabilities. Business Intelligence vendors are partnering with BPM vendors, as are purveyors of business activity monitoring and business rule vendors. Plenty of businesses are adopting the technology, too, so just what is BPM, why is it gaining attention and how do you choose the right technology?

You would think that defining BPM would be straight-forward, but that's not the case, in part because business process management suite (BPMS) capabilities are now being found in many solutions solving different needs. Thus, the terminology is sometimes used rather loosely (not to mention the confusion with the other BPM, business performance management). Regardless, one thing is clear: BPM is getting a lot of attention. Forrester Research estimates that BPMS license, services and maintenance revenue will grow from $1.2 billion in 2005 to more than $2.7 billion by 2009 " an adoption trend vendors and practitioners alike can't ignore.

A common confusion about BPM surrounds the difference between the workflow systems of the 1990s and today's BPMS. Older, proprietary workflow systems managed document-based processes where people executed the workflow steps of the process. Today's BPM systems manage processes that include person-to-person work steps, system-to-system communications or combinations of both. In addition, BPM systems include integrated features such as enhanced (and portable) process modeling, simulation, code generation, process execution, process monitoring, customizable industry-specific templates and UI components, and out-of-box integration capabilities along with support for Web-services-based integration.

All of these ingredients translate to increased interest today in BPM suites because they bring businesses a higher level of flexibility for business processes while reducing risks and cost. Think of BPM suites as offering a way to build, execute and monitor automated processes that may go across organizational boundaries - a kind of next-generation workflow. Read on to learn what's inside the two styles of BPMS, what makes the technology tick and how to choose the right system for your organization and process needs.

PM suites incorporate multiple technologies, as shown in the diagram to the right.


Another area of confusion is about the terminology used to refer to the different types of BPMS. For example, you may encounter the term "pure-play BPM," describing vendors that focus solely on business process management, but given vendor consolidation and an increasing number of solutions with similar capabilities, this terminology should soon disappear.

Lightweight BPM functionality is sometimes found in packaged enterprise applications such as ERP, CRM and financial software. This is a type of "embedded BPM," since it is not intended to be used outside of the application domain.

BPM features have also emerged from the enterprise application integration (EAI) vendors. These are enhanced versions of Enterprise Service Bus (ESB) products. These systems support the design, development and execution of system-to-system processes while integrating them by means of Web services and asynchronous messaging.

In a nutshell, it is much easier to view BPM solutions in two basic categories: Front-Office BPM (FO-BPM) and Back-Office BPM (BO-BPM). While vendors are racing to provide both types of BPM solutions, the maturity of their capabilities in each area tends to depend on their history and depth of experience in each domain.

The FO-BPM solutions (with roots in human-centric workflow products) provide capabilities for person-to-person processes in which "work items" are created and routed along with any attached documents. A partial list of vendors that provide FO-BPM capabilities would include TIBCO, FileNet, IBM, PegaSystems, Global360, Oracle, DST Systems, BEA (with recently acquired Fuego) , Computer Associates, Ultimus, Savvion and MetaStorm, among others.


On the other hand, the back-office-oriented BPM solutions (such as ESB products) enable system-to-system integration in which a single process may rely on multiple external processes (and possibly heterogeneous platforms) to complete its work. These suites also manage aggregation and composition of all services for the enterprise. Vendors such as IBM, Oracle, TIBCO, Sun, CapeClear, WebMethods (soon to be acquired by Software AG), Fiorano Software, Sonic Software and BEA Systems, among others, have specialized products in this space.

Front-office- and back-office-oriented BPM systems are aimed at different problems. For example, front-office BPM applications typically involve transactions that are "short running." This may not be the case when an internal process is integrated with Web-service-based processes (possibly outside of an enterprise), which can result in "long-running" transactions. Managing the transaction context of a process involving multiple resource managers requires an alternative approach in BO-BPM. (We will cover this in more detail later on).

One major differentiation among BPM suites is in support business process lifecycles. Therefore, it's important to understand what's involved in each phase of a process. The diagram on the left depicts the basic BPM Lifecycle. The basic flow is similar for both types of BPMS, but we'll describe some specific differences later in the article.

Process Modeling

The modeling phase is typically supported by a visual process design tool. Some vendors have their own proprietary tools while others provide specialized plug-ins that extend existing IDEs or products such as Microsoft Visio or Eclipse-based modules.

A process consists of multiple activities (also know as "steps" or "tasks").These are created and linked to each other to form the flow of the process. Conditions that define how and when an activity must be called are also are defined during the modeling phase. Process activities can include attachments, such as documents or images, that are passed to the next activity along with associated data. In addition, process can be split into parallel execution paths and joined at a predetermined step. If needed, an instance of a process can be started by an external event, such as the arrival of an e-mail, message or document. These capabilities can be graphically depicted using the same visual design tool. The activities within a process either require human involvement or are processed without any human interaction. Therefore, the modeling tool needs to support person-to-person and system-to-system processes.

Since different types of activities usually have their own visual representations (such as activity icons and graphics), this area of design and presented an opportunity for standardization. One such standard that is widely accepted is Business Process Modeling Notation (BPMN).

The outcome of the process modeling phase is usually a pseudo-executable process flow model that can be exported in an executable language format supported by the BPM engine. Thus, there needs to be a mapping from a modeling language (such as BPMN) to a process execution language (such as XPDL or BPEL) understood by the BPM engine of your choice (more on this later).

To streamline the transition from business design to technical design, some process modeling tools are also capable of producing application design diagrams in Unified Modeling Language (UML). Advanced modeling tools also let you simulate processes and changing activity parameters such duration and resource utilization to test impacts on process efficiency.

Most BPMS vendors supply modeling tools that are optimized to work with their solution, but there are a number of vendors that specialize in process modeling (and analysis, to be covered later), including IDS Scheer, Proforma, Mega, iGrafx, Telelogic and Casewise.

Process Activity Implementation

To create the business logic of the modeled activities, one must provide the necessary instructions (beyond those required to manage the execution flow and pass necessary data). Further, the complex activities within a process often need to be programmed at a lower level to do such things as accessing databases or performing business calculations.

The development environment varies by vendor strategy, but this typically comes in the form of either an integrated development environment (IDE) that is integrated with the modeling tool or a separate IDE. Regardless, all vendors try to minimize the need for hard-coding by supporting a visual programming environment. The most common characteristic of BPM IDEs are meta-data-driven development and code-generation aimed at speeding implementation. The programming language used might be an inference-based business rule engine or a standard language such as Java or C++. To implement more complex business logic, it might be necessary to rely on different programming tools. To ensure flexibility, almost all vendors provide a set of APIs and Web service interfaces.

Front-office-oriented BPM systems typically let you design forms, define data fields, customize process templates, set up access control lists, configure integration capabilities and manage deadlines. Since back-office-oriented BPM solutions aren't intended to support human-centric workflow (and therefore don't require form capabilities), they focus instead on features such as back-end process composition, data/message transformation and automatic creation of service interfaces to ease implementation.

If your application requires integration with external systems, look for out-of-the-box support first. Usually, vendors offer prebuilt components to integrate with document/content management systems, security repositories, third-party packaged applications and middleware products.

Once process activities are implemented, the application is deployable and ready to execute on the vendor's chosen platform, such as J2EE or .Net.

Process Execution

The process engine that runs on the BPMS vendor's server is responsible for executing the process model exported for execution and activities programmed. A typical run-time environment also needs other low-level services, such as security, transaction and concurrency management. These services are either provided by the same vendor or the vendor relies on another technology, such as applications or servers. If the transactions involve multiple business partners, the challenge of managing those transactions gets more complex.

To control the execution of process activities, there are basically two approaches: orchestration and choreography. The distinction lies in the point of view on "who is the process owner" of an internal corporate process, an external public process (peer-to-peer/B2B) or combinations of both.

The orchestration approach mainly assumes that the process owner is always a single-entity that has total control of all message exchanges between activities. An example is a workflow application for an in-house loan-approval process.

The choreography approach, on the other hand, implies that the process is owned "jointly" by all participants, and the sequence of multiple message exchanges among participants are driven by mutually agreed "contracts." Examples of this include the fulfillment processes of a purchase order in a B2B environment.

Among process execution languages, BPEL more closely resembles the orchestration approach. Another standard, WS-CDL, mostly addresses the needs of the choreography approach. XPDL, on the other hand, is mostly seen in human-centric workflow tools/applications. In the future, expect BPEL to include similar capabilities from XPDL and WS-CDL.

Process Monitoring and Analysis

The ability to report business performance metrics is increasingly important and is a major differentiator among BPM suites. Given the large potential impact on process efficiency, BPM vendors are rushing to include this type of business activity monitoring (BAM) capability in their suite through acquisitions, partnerships or enhancements of existing tools.

Unfortunately, would-be customers are usually confused about how to best use BAM capabilities. Part of the reason for this confusion is that there are two types of BAM: real-time and historical. Yet the distinctions between the two are not clear to most.

Real-time BAM components are tightly integrated within the BPM solution to monitor metrics for work-in-progress (such as the number of process instances started, execution times of activities and so on). Advanced real-time systems can do such things as record an execution pattern signaling process failure so you can avoid them in the future. Real-time BAM also predicts work loads for better resource utilization. Filtering and reacting to business events is also a key feature. For example, based on a certain business condition (such as receiving an electronic order), the following actions can start: send e-mail to a user, automatically start another process instance and send alerts to dashboard/console.

Historical BAM components, in contrast, provide analysis reports to draw conclusions from purely historical execution of already completed processes. Process-specific key performance indicators (KPIs) and service level agreements (SLAs) can then be compared against this data. These tools can also integrate with existing data warehouses and may require a different server platform than the BPM server.

Collecting execution metrics also enables "closed loop" modeling in which process simulations can be adjusted to improve efficiency.

Are We There Yet?

The short answer to this question is "not completely," but we are getting rather close. Even though the industry has make much progress in foundational areas, it is still a highly dynamic market. Here are just a few of the areas that are in flux:

* Standards for managing person-to-person processes
* Ongoing revisions of message security and reliability for system-to-system processes
* Transaction propagation
* Business activity monitoring
* Integration of multiple process engines

There is currently a lack of strong support for a standard way of working with processes that involve human-interactions, even though there are recommendations to address them. BPEL currently does not handle person-to-person processes, so an extension has been proposed called "BPEL4People." Similar capabilities can also be found in XPDL.

The security issues are about message integrity, confidentiality and authentication. In SOAP-based communication, XML extensions are created to support the use of multiple security tokens (such as username, certificates, SAML) within the message. The example of such effort comes from the WS-Security specification.

Communication standards are a priority given the need to handle communication problems gracefully between any two systems (due to, say, a network or application outage). In SOAP-based Web-service integration, the two competing standards, backed by different vendors, are WS-RM and WS-R. They are both XML-based protocols that attempt to ensure that the sender of the message gets a reliable acknowledgement when the receiver has received the message (and preferably ignored any duplicate messages). Currently WS-RM seems to have broader vendor support, but reconciliation between the two standards is necessary.

One other area in flux involves how transactions are managed during system-to-system interactions with Web services. BPEL and WS-* extensions are becoming widely supported in this type of back-office BPMS. BPEL workflow transactions, for example, can be short or long running. WS-Transaction and WS-Coordination services are ways of supporting transactional Web services. While another standard, BTP (Business Transaction Protocol) supports similar needs, currently BPEL and WS-* extensions are getting more visibility.

On the BAM front, BPM vendors are determining how to best collect and present information from the customer's BAM tool of choice. While most BPM products provide operational metrics on dashboards, it's not easy to use BAM and BPM products from different vendors. Creating historical reports from BPM data and analyzing KPIs using existing reporting tools requires customization effort.

When disparate process engines (XPDL- or BPEL-compliant) from different vendors need to be integrated, it may be hard to reconcile different levels of standards, vendor-specific features and proprietary interfaces. For example, to start a process in one process engine and hand it off to another, you need proper communication. The WfXML initiative is an effort to address issues specific to this scenario.

Before You Buy

The selection of a BPMS demands unique analyses you won't find on a typical software evaluation checklist. Here are 6 things to look for in determining the BPMS that's right for your organization:

1. BPM systems aren't meant for all application needs. BPM systems are especially good for processes in which all activities are predetermined in order. Validate the business value of taking a BPM approach by developing a deep understanding of both the business problem and the capabilities of the BPMS. Document the detailed activities of your process and be sure the BPMS addresses core needs before committing to a product.

2. Next, consider the type of BPM solution needed: front-office-oriented BPM, for human-centric processes, or back-office-oriented BPM, for integration centric processes. It's possible that your environment may benefit from both types of BPM. If so, focus on products from either the same vendor or from vendors that have partnered to minimize integration efforts. If you use solutions from multiple vendors, keep in mind that you could end up with multiple process modeling and monitoring tools as well as server platforms.

3. Once you decide on the type of BPMS you're after, refer to the BPM lifecycle activities and architectural components covered earlier and analyze how your requirements are supported, in detail, by the prospective products. Make your product selection based on the total value of the BPMS rather than just standards support, particularly if you're considering front-office BPMS (an area in which standards aren't settled).

4. For front-office-oriented BPMS, ease of customization and out-of-box support for integration (such as with a content management or imaging systems) should be high on your list. If the same vendor provides content-management and collaboration modules as well, then confirm that their repositories, business modeling tools and administration consoles are integrated with all these components.

5. Among back-office-oriented BPM suites, you will notice that BPEL and WS-* support are widely implemented. Therefore, it's important to pay more close attention to the vendor's direction. Consider the vendor's support message transformation, routing and multiple-transport protocols. While most front-office-oriented BPM systems can be deployed on application servers, BO-BPM server requirements can vary. They may require installation of messaging-oriented middleware (MOM), application servers, subsets of application servers (such as a Web container) or some combination. Make sure you understand how the vendor solution is architected in order to wisely plan for its technology fit and impact on infrastructure.

6. Another issue you may want to consider is how rule engines are used in the BPMS. Rule engines are typically an optional component in most BPM suites, but you may find that certain vendors requires everything to run under a rule engine that needs a proprietary language. Determine whether the benefits of using that rule engine for the entire application are conclusive while also considering potential hiring and training needs tied to that environment and language.

BPM solutions bundle a lot of capabilities, so you should expect a learning curve to before you can take full advantage of a suite. Focus on proper training and establish best-practices and guidelines to create manageable deployments. Remember that matching the right type of BPMS to the processes considered is the most important decision you will need to make early on.