From Traditional to Collaborative Forms of Modeling Data Architecture
The need to manage information assets in support of multiple use cases such as business process management (BPM), service-oriented architecture (SOA), data warehousing and business intelligence or master data management (MDM) initiatives are driving business and application architects to use different standards, methods and technologies than those responsible for data models. Without coexistence and synchronization of models between these approaches and tools, it will be impossible to understand the impact and scope of change to information assets.
- Historically, IT and business architects, analysts and designers had the ability to share a common set of structured analysis/structured design (SA/SD) set standards, methods and modeling tools. This allowed for a consistent way to define both data and processes.
- In support of BPM and SOA, business and IT architects, analysts and designers are moving to new standards, methods and tools, including the changed way data is defined at the conceptual and logical levels of detail by them.
- Therefore, in order to maintain a consistent understanding and view of the data it will be necessary for organizations to coordinate data models – and other related metadata – and allow them to coexist using multiple modeling standards, methods and tools.
- Ensure that the roles and responsibilities for creating and maintaining the data models and architecture are defined and understood, including those who are contributing to it from the enterprise, business and application architecture perspective.
- Ensure that there is synchronization between the different approaches and tools for data modeling and other forms of metadata related to information assets.
- For new efforts, strongly consider using BPM methods and tools for the conceptual level of data modeling, SOA-oriented methods and tools for the logical level, with SA/SD at the physical level for data integration and database design modeling as appropriate.
This research targets information leaders, methodologists and those individuals affected by discipline and technology changes needed to support the information modeling aspects of broader data architecture. In "Data Modeling and Data Architecture; A Required Strategy for Enterprise Information Architecture," we explained how information modeling describes the metadata necessary to understand the data, processes and rules that are relevant to the enterprise (see Figure 1). Here, we focus on a subset of that landscape – the data models – and, specifically, how organizations are moving from a single set of methods and tools to more collaborative conceptual and logical modeling across roles (see Figure 2) using disparate methods and tools.
Figure 1. Information Modeling Landscape
Figure 2. Sample Roles Involved in Data Architecture
Source: Gartner (September 2011)
Historically, defining and managing the data architecture (see Note 1) was primarily the responsibility of information management (IM) personnel, with data modeling being a key component. Where there was collaboration in defining the data architecture, it was generally through the use of traditional forms of modeling using SA/SD methods and tools across roles. However, we are now in a period of transition from the shared use of SA/SD methods to the disparate use of newer methodologies, standards and technologies. This will require a migration strategy for how the data architecture is defined, including a period of coexistence between these evolving disciplines and technologies that include data modeling. This transition plan may take up to 10 years or more to accomplish in many organizations. However, the ability of multiple approaches and tools to coexist based on different organizational roles and needs – especially in coordination with their implementation of BPM and SOA – is something that should be addressed by organizations to ensure a consistent and cohesive understanding and view of data. This research focuses on the issues related to collaborative data modeling across roles, which generally occurs most frequently at the conceptual and logical level of detail; however, the data architecture includes physical modeling of both structured and unstructured data as well (see Note 2).
As can be seen in Figure 3, there are three specific types of modeling tools used for strategic planning, analysis and design, which include support for the data architecture – SA/SD, business process analysis (BPA) and object-oriented analysis and design (OOA&D). Each support aspects of data modeling from the perspective of the different roles depicted in Figure 2. Though potentially using different notation standards and tools, these modelers are defining the same data and its use. Business architects and process analysts involved in BPM initiatives generally use BPA methods and modeling tools that include data concepts like "objects" or "classes." Application architects and designers involved in SOA initiatives tend to use OOA&D methods and modeling tools that include data concepts like "classes." Those responsible for data analysis and database design tend still to use forms of SA/SD methods and tools that include data concepts like "entities." In order to better integrate the data models across these approaches and tools there needs to be a coexistence strategy in the short term, and a longer-term migration strategy away from SA/SD methods to the newer ones. In fact, many of the leading data modeling/database tools now support OOA&D concepts within their tools, and most provide bridges from other modeling tools using BPA and OOA&D methods.
Figure 3. Examples of Metadata Tools Needed to Support Data Architecture
BPA = business process analysis; DBMS = database management system; OOA&D = object-oriented analysis and design; SA/SD = structured analysis/structured design
Source: Gartner (September 2011)
Definition: Most organizations continue primarily to use formalized SA and SD approaches to system analysis based on defining functional specification, formal data dictionaries and data flows within a set context. Functional decomposition is a basic tenet of SA/SD, moving from conceptual through logical and into physical schema. Historically, architects, business analysts and application designers used the same or similar SA/SD methods and modeling tools when defining processes and data and building applications that target implementation in a third-generation language, such as COBOL, FORTRAN and client/server architecture. SA/SD methods and modeling have also been extensively used in support of data warehousing projects to model requirements across the business and IT using the conceptual model as the basis for this work.
Numerous instances of SA/SD exist, such as the Structured System Analysis and Design Method (SSADM), information engineering, Ward/Mellor and Yourdon/DeMarco methodologies. Integration definition (IDEF) for function modeling is one of the most popular structured modeling techniques, and covers 14 viewpoints, including functional, information, organization and process. Historically, organizations have used SA/SD methods with a "waterfall" approach because of the sequential nature of model definition and decomposition; however, it is also possible to use "agile/iterative" and other approaches to SA/SD.
Trend Analysis: SA and SD represent a very mature set of principles and standards, with over 30 years in the field. While also used by business and application analysts, SA/SD is closely associated with traditional forms of data modeling using concepts like entities, attributes and entity-relationship diagrams. Most of the leading database design tools use these concepts to create physical database designs and generate database schemas. A number of commercially licensed SA/SD data warehouse models continue to be used and refined to capture industry best practices and requirements for data warehousing. Many of these models continue to deliver great business value to organizations, further supporting the value of classic data modeling tools and concepts. At the same time, data integration tool suites are adding – generally SA/SD – data modeling capabilities to support their diverse data delivery needs. The same concept may be used by different projects or change data delivery method.
From a BPM perspective, SA/SD has been superseded by BPM methods and BPA tools which follow a standard called business process modeling notation (BPMN). From an SOA perspective it has been superseded by object-oriented methods and modeling tools following standards like the Unified Modeling Language (UML) and domain-specific languages (DSLs). As a result, from the perspective of process and application design SA/SD is now limited to legacy development in many organizations – although some aspects of SA/SD persist in other methods.
Impact on Business Architects, Process Analysts and Workflow Designers
In short, there are compelling reasons for those focused on business architecture to use these methods for the modeling of process and data at multiple levels of detail as opposed to SA/SD methods and tools. However, since these tools use data concepts like "objects" or "classes," they are not identical to the way data is defined in SA/SD tools – creating a challenge for managing the broader data architecture.
Recommendation: Those supporting the business architecture should immediately convert from the use of SA/SD methods and tools to newer ones that support BPM.
While SA/SD approaches include a set of stable legacy methodologies, they are not suited for current programming paradigms based on object orientation, SOA and dynamic languages. Organizations will also find it increasingly difficult to find SA/SD-experienced staff as the baby boomer generation retires. Application analysis and design methods and tools based on OOA&D concepts are becoming increasingly integrated with Java and .NET development tools to increase developer productivity. They are also better than SA/SD at getting the right level of granularity and reuse of software services. Most come (or can be integrated) with a repository to manage the process and data-related metadata at multiple levels of detail. And, as an added benefit, many of the BPA tools use the same OOA&D concepts for data "classes and objects" at the conceptual level of detail.
In short, there are compelling reasons for those focused on the application/solution architecture to use OOA&D modeling of process and data at multiple levels of detail as opposed to SA/SD methods and tools. However, since these tools use data concepts like "classes," they are not identical to the way data is defined in SA/SD tools – creating a challenge for managing cohesive and consistent data models and architecture.
Recommendation: Use SA/SD approaches and tools for legacy applications that have existing SA/SD models and are undergoing incremental change. Use OOA&D methods and tools for new SOA applications and consider using them for legacy applications undergoing major change. Plan a broader migration strategy for moving legacy process and data models from SA/SD to OOA&D over the next five to 10 years.
Since data models are now appearing in other types of standards and tools, one way to address the problem is to "bridge and translate" data models, especially from the "class" format in OOA&D tools to the "entity" one in SA/SD tools. The alternative is to use the OOA&D classes to represent the design level of detail and generate the database schemas out of the OOA&D tools. Most organizations are opting to have their developers use the OOA&D class-to-database approach for projects using agile methods (with limited time for analysis). For projects where there is more time spent on data analysis and database design, they are bridging the OOA&D classes into their SA/SD tools and representing them as "entities" and "attributes." (Note: Most leading data modeling/database design tools have ways of importing classes and attributes from OOA&D tools.) Then, these entities and attributes are refined at the logical level of detail by data analysts prior to database administrators designing the physical file formats and database schemas in the SA/SD tool.
It should be noted that SA/SD approaches are still important from the perspective of data integration (see Note 3).
Recommendation: Decide which databases and files will be managed by developers using an OOA&D set of tools as opposed to those which will be managed by IM personnel using SD/SD tools or using data integration tools. Plan a migration strategy away from SA/SD to data integration tool suites or OOA&D methods and tools over the next 10 years.
Figure 4 shows different common points of interfacing models between SA/SD (including data integration), OOA&D and BPA methods and tools – including implemented services – at different levels of the data architecture. One of the most common points of interface for coexistence is from BPA to OOA&D IT analysis and then on to SA/SD physical database design and construction. A second common point of interface is from BPA workflow, service assembly workflow execution using OOA&D-built software and data services against SA/SD-built databases.
Figure 4. Paths for Coexisting SA/SD, OOA&D and BPA Methods and Tools
BPA = business process analysis; OOA&D = object-oriented analysis and design; SA/SD = structured analysis/structured design
Source: Gartner (September 2011)
This form of coexistence may include disparate, heterogeneous tools with some limited passing of shareable models and metadata between them, or they might be tools that are part of a "tool suite" from a single vendor sharing a single metadata repository, including, in some cases, metadata from other technologies and sources. For more information on the various approaches implementing model sharing as shown in Figure 4, and integrating the metadata across tools at different points of modeling, including through the use of metadata repositories.
Data architecture needs to include the data models and metadata that exist in multiple tools in different formats and in support of multiple use cases. Organizations must weigh the benefits of tighter integration and consistency of those models and metadata for use by traditional data modelers and data integrators using SA/SD approaches with those doing planning, analysis and design using the next generation of BPA and OOA&D modeling tools and data integration tool suites. Most will find that a coexistence strategy in the short term coupled with a long-term migration strategy away from SA/SD will be the most pragmatic decision.
This research is based on various Gartner research into the trends and revenues for the tool markets discussed. We have also based our findings on feedback from clients regarding their increased use of business process analysis and object-oriented analysis and design tools to model data (and process).
Source: Gartner RAS Core Research Note G00218922, Mike Blechar, Roxane Edjlali, 28 September 2011The meaning of data architecture is somewhat in the eye of the beholder. To some it may be documenting the physical structure of the databases and the use of that data. For others, like enterprise architects, it is frequently used as a broader term which includes conceptual, logical and physical definition of data and its integrity, security and use. Regardless, data modeling is a key subset component for documenting and understanding the data architecture.
Wikipedia defines data architecture in enterprise architecture as "the design of data for use in defining the target state and the subsequent planning needed to achieve the target state. It is usually one of several architecture domains that form the pillars of an enterprise architecture or solution architecture." (See http://en.wikipedia.org/wiki/Data_architecture.)
During the definition of the target state, the data architect breaks a subject down to the atomic level and then builds it back up to the desired form. The data architect breaks the subject down by going through three traditional architectural processes:
- Conceptual – represents all business entities.
- Logical – represents the logic of how entities are related.
- Physical – the realization of the data mechanisms for a specific type of functionality.
In the broader sense, data architecture includes a complete analysis of the relationships between an organization's functions, available technologies and data types.
This research focuses on the issues related to collaborative data modeling across roles, which generally occurs most frequently at the conceptual and logical level of detail; however, the data architecture includes physical modeling of data as well. In most organizations the physical design of structured data to be stored in the database is managed by IM personnel using SA/SD methods and tools. However, there is an increasing need to do logical-to-physical design of unstructured data and content as well – such as XML files. We tend to see developers defining and building these files using OOA&D tools. While this is not an ideal solution, this is the current practice in most organizations – and will remain so until some standard is created to automate BPA or OOA&D models into XML files and the leading tool vendors adopt that standard and implement it. That said, XML files are critical to information sharing and interoperability.
Shared information architectures require "interoperability by design" practices that balance the technical and information needs of external actors with other members in the information ecosystem. An example of a shared information architecture is the U.S. National Capital Region's (NCR's) Interoperability Program and Data Exchange Hub (see www.ncrnet.us). The NCR's information-sharing environment uses the National Information Exchange Model (NIEM) as the foundation to meet technology, business, application and information-sharing requirements for achieving regional interoperability by design.
Since most of the leading data warehousing and master data management tools use the physical schemas, either approach can work. But, many still expect there to be (logical) data models in a SA/SD format to improve end-user understanding and allow for multiple physical implementations of the same logical data attribute. Moreover, until the BPA and OOA&D standards provide better support for metadata semantics, the data modelers in most organizations will want to continue using entity-attribute modeling. Still, the data modeling/database design tool market is already in its later life cycle stages and new product licenses and maintenance revenues are already in decline – though we expect the leading data modeling/database design tools to continue to be supported for at least the next five to 10 years.
The need for data modeling at a conceptual and logical level remains and is being surfaced in data integration tools. Generating physical models and optimizing the schemas to support the various SLAs will likely go away in the coming five to 10 years with the development of in-memory DBMSs and the growing adoption of software-as-a-service and platform-as-a-service DBMSs.
So, a long-term transition plan from SA/SD to OOA&D or data integration tool suites is called for. Organizations should consider the evolution of data modeling capabilities offered in data integration tools suites and decide based upon the use case and maturity of the technology.
