Design, development and implementation of a system-of-systems routing framework to support efficient planning, implementation and sustainment of data flows in support of organizational goals.
Key activities in this data-centric organization are data collection, processing, and distribution of large volumes of data to broadly distributed endpoints for data analysis and use. Data routing operations are key, with route planning and implementation being high value, high visibility activities. Due to the complexity of the data flows, the organization was experiencing increasing delays in flow implementation based on the long planning cycle and complex flows were often not fully documented as the flows were implemented, making post-implementation troubleshooting difficult. Due to the large number of systems involved, flow disruptions based on planned or unplanned system updates were increasing, and the work backlog was increasing beyond acceptable limits.
A framework was designed composed of three key components – system nodes, data sets and routing templates. The components were then implemented in software to facilitate the planning, processing and implementation of data flows. Individual systems were defined as functional nodes, with attributes specifying types of data that were accepted and actions that could be performed on the data at each node. Data types and routes were then analyzed and normalized to yield data flow templates that represented the patterns for approximately 80% of the data flows the organization uses. Interfaces between the systems were characterized and encoded within each template, streamlining the planning process and making data flow plans more uniform across the organization. Lastly, visual representations of the components were implemented in software, providing visualization of the flows and illustrating attributes of the data flowing between each pair of nodes.
The organization has realized significant improvements in the efficiency and consistency of producing route plans based on a template-based routing framework approach engineered by Business Transformation Institute, Inc. The templates provide easy-to-use detailed planning guidance for the community of route planners for the majority of the flows, and the framework supports customization for the 20% that require non-standard planning. The framework formalizes basic system functions and interface elements, reducing errors and allowing consideration of complex flows visually, which helps in determining resources needed to implement the data flows. Use of the templates also helps the teams identify potential problem areas and communicate efficiently both at planning and after implementation.
The Challenges:
Data collection and distribution are governed by complex policies and operational procedures, and implementation of data flows involves diverse service organizations in multiple divisions of the enterprise. Receiving, processing and routing data to the points where it is needed, as well as ensuring that only authorized personnel have access to data is key, but the systems and personnel involved in various steps of the processes have little insight into the end-to-end flows and the accompanying constraints, making it almost impossible to provide all of the information needed to optimize individual tasks for each flow at the time of planning.
Initially, route planning for data was performed manually, based on SME knowledge in a very small group of people. Expanding the numbers of planners was an obvious choice, but this led to significant inconsistencies in route implementation, and made capacity extrapolations more difficult. Frequently, the ‘as-built’ flows were not well documented, resulting in high diagnostic costs when a flow was not behaving normally.
Numerous services are involved in the end-to-end flow of the data; interfaces with many of the intermediary systems were not completely described, and unannounced changes often resulted in broad flow disruption.
Progression of the implementation tasks was tracked via a simple in-line comment service, making it necessary to read the entire list of comments in order to determine work status for an individual flow.
The Goals:
Goals of the project were to increase the accuracy, speed and completeness of dataflow planning, to improve the consistency of planning practices across planners, to characterize the interfaces between systems so that more automation could be used, and to increase the status visibility of the data flows to all stakeholders.
The Plan:
Approximately 80% of flows used ‘standard’ core paths, while 20% are completely custom. Based on information, a series of 28 templates was planned with the help of the SMEs. These templates are based on key characteristics of specific classes of data, and on the functional profiles of each system along the path. The templates contained information about types of data the system handles, the actions that system can take on data, and the outputs that could be expected from each system. Each template was constructed for use with one class of data, with add-on templates to handle variations. Use of the templates should provide an efficient and visual method for socializing information about each step in the data flow path, and would clearly identify interface points and dependencies.
For 80% of the flows, planning will involve instantiating the template for a particular data flow, and then completing additional steps to ensure the appropriateness and integrity of the flow; similarly, for the 20% of custom flows, system information is added as the route is planned. The resulting plans should not only result in more consistent data flow plans, but should also allow construction of a data flow inventory of the organization’s’ data assets.
The Results:
The process of templates construction yielded benefits beyond those that were planned; group discussions of the paths with the technical leadership resulted in greater consistency and agreement about the approach that should be used for each data group, as well as about the paths appropriate for the myriad of types of data that would use the templates. Subcomponents of the templates representing individual system functions were useful in talking with system owners, in uncovering undocumented features of some interfaces and in improving ongoing communications between different divisions.
As planning personnel have started to instantiate templates for route planning for each dataset, they are finding a large increase in efficiency – estimates are above 50%, although the planning users are still learning the new system, so additional efficiency improvements are expected. Use of visual data routes has improved the ability of the service providers to understand the specific needs for each dataflow, and to easily locate and review route planning information. Relationships with system owners and service provider groups has been strengthened by their participation in the planning processes, and change coordination has improved significantly. Sustainment personnel have noted that the ‘as-built’ flow plans dramatically reduce the time required to identify breakages in the path, primarily because they don’t have to spend time determining where they should look for the flow.
Lessons Learned and Next Steps:
Based on initial experiences, the organization is planning on expanding their use of the framework components for additional classes of data flows within the enterprise, with the hope that they can reduce the backlog of routing requests and improve their service delivery times. Automated integration with several reference systems is underway, further supporting efficiency in planning. The goal longer term is to provide proactive planning based on information delivered automatically from the system; for instance, understanding which flows need to be rerouted when an intermediary system is decommissioned, and performing the flow re-planning and implementation proactively. Increased flow resilience, based on an understanding of the entire flow inventory and an ability to move high-value flows off of struggling processors or to dynamically scale those processors, is also a goal longer term.
Note: Confidentiality is maintained for competitive reasons. Examples of environments with similar data flow challenges include organizations that provide networking services, health care provision and insurance services, communications providers, banking and insurance industries, and any organization that conducts large volume data analysis.
