Agile Data Logo

The Agile Data Architecture Process

Follow @scottwambler on Twitter!

This article explores what agile architects, or more pertinently, agile data architects, do. As you will soon see, this article overviews strategies and concepts of the Agile Modeling (AM) method, extending them with data-oriented techniques. Figure 1 depicts a flowchart overviewing how to choose between common activities required for successful data architects.

Figure 1. The agile architecture process (click to enlarge).

Agile Architecture process

There are effectively four key activities performed by agile data architects:

  1. Provide help to others
  2. Agile architectural modeling
  3. Explore complex architectural issues
  4. Invest in learning

1. Agile Data Architects Provide Help to Others

Agile data architects spend the majority of their time collaborating with their stakeholders, as you can see in Figure 2. Although Figure 2 was originally developed to overview enterprise architecture activities, it also applies to specialized architects (such as data architects or security architects) operating at different levels of architecture scope.

Figure 2. The enterprise architecture process (click to enlarge).

Enterprise AMDD

Data architects typically work with their stakeholders, overviewed in Table 1, to either apply, create, or evolve data architecture assets. Potential data assets include data guidance, data access control guidance, data models, data standards, reference architectures, executable test suites, and more. Data architects do this by helping stakeholders to:

  1. Understand and apply the architectural vision and existing data architecture assets. You want to make it as easy as possible for your stakeholders to leverage your data architecture work, and you do this by working closely with them to do so.
  2. Collaboratively develop new, or extend existing, data architecture assets that they require. By working closely with stakeholders to develop the data-oriented assets that they need you help to ensure that new assets are of actual value.
  3. Identify and obtain existing data architecture assets that they can apply without your assistance. Sometimes people simply need to be pointed to existing assets that apply to the situation that they currently face.

Table 1. Potential stakeholders of data architects.

Architect Type Potential Stakeholders

  • Enterprise architects
  • Standards bodies
  • Industry analysts

  • Industry architects
  • Product line architects
  • Solution architects
  • Business executives
  • Domain experts

Product Line
  • Domain experts
  • End users/customers
  • Solution teams
  • Enterprise architects
  • Component teams

  • Domain experts
  • End users/customers
  • Developers
  • Product line architects
  • Component teams

  • Domain experts
  • End users
  • Solution architects
  • Developers

2. Agile Data Architects Perform Agile Architectural Modeling

Let's start with a few definitions:

  1. Agile modeling. Agile Modeling (AM) is a practice-based methodology for effective modeling and documentation. Agile modeling is an evolutionary (iterative and incremental) approach to creating models and supporting documents that is highly collaborative in nature.
  2. Agile models. Agile models are sufficient for the context in which they are used, or as agilists like to say, they are just barely good enough (JBGE).
  3. Agile architecture modeling. This is the application of AM strategies in the creation and evolution of architecture models and related assets.
  4. Agile model driven development (AMDD). AMDD is the agile version of Model Driven Development (MDD). MDD is an approach to software development where extensive models are created before source code is written.

Figure 3 depicts a high-level overview of AMDD from the point of view of a component or solution team. Initial envisioning occurs during sprint/iteration 0 (also known as the initiation, ideation, or inception phase) at the beginning of an agile initiative. The goal of initial envisioning is to invest just enough effort to understand the problem domain and to identify what the team believes to be a viable architectural solution. Detailed modeling occurs during construction in a just in time (JIT) agile manner. This will include requirements exploration, architectural modeling, and design solutioning.

Figure 3. Agile model driven development (AMDD) process (click to enlarge).

Agile Model Driven Development

Figure 4 is a more detailed version of Figure 3, showing how initial envisioning is an evolutionary combination of requirements envisioning and architecture envisioning. Depending on the complexity of the problem faced by the team, and the skills and knowledge of the people involved, this envisioning effort may run from several hours to several weeks. Having said that, shorter is generally better than longer because agilists focus on producing models that are just barely good enough (JBGE).

Figure 4. A detailed look at the AMDD process (click to enlarge).

Agile Model Driven Development (detailed)

You can also see that Figure 4 expands upon detailed exploration, calling out detailed modeling and writing executable acceptance and developer tests. Detailed modeling is applied to work through high-level issues. We want to work as collaboratively as possible, exploring the complexities that we face using inclusive tools such as whiteboards and paper at first and then eventually shifting to more sophisticated tooling when and if appropriate once we've come to an agreement as to how to proceed.

Writing executable tests before writing the implementation logic is called test-driven development (TDD). TDD should occur at both at the requirements level with acceptance tests, sometimes called acceptance TDD (ATDD) or behavior driven development (BDD), and at the design level with developer tests. TDD requires practitioners with testing skills and discipline, two things that take time and effort to gain. With TDD you work through the detailed logic, and specify that logic, before you implement it. Being executable tests, rather than static documentation, the tests both specify as well as validate your implementation.

This begs the questions "Why do we need detailed modeling plus TDD? Why not just model like we did in the good old days?" There are three reasons to do so:

  1. Modeling addresses high-level complexities. We apply modeling for what it is good at, enabling people to collaboratively think through the complexities that they face and to provide a visual overview of those complexities.
  2. Executable tests addresses details. Executable tests are very good at specifying and validating details, they're not that good at working through high-level complexities.
  3. Overall maintainability. Decades of experience have shown us that people won't read detailed documentation, and even if they do they may not understand or follow it. They certainly are unlikely to update it. See the CRUFT formula for calculating the value of a document. But if the specifications add value, which they do in the form of executable tests because they become part of our overall regression test suite, then suddenly they add real value and as a result are likely to be maintained.

Figure 5 brings enterprise modeling into consideration, expanding upon Figure 3 to do so. The architecture owner (the "agile architect") on an agile team collaborates with enterprise architects to ensure they leverage organizational assets on the team. If a team includes specialized architects (not shown), such as agile data architects, then those people may work with similarly specialized enterprise data architects. More on this in a minute.

Figure 5. An overview of the agile architecture process (ideal) (click to enlarge).

Process of Architecture Modeling (ideal)
Figure 6 expands upon Figure 5 by showing how specialized architectural modeling fits in. Although the recommendation is for people to strive to become generalizing specialists, the reality is that the complexities faced within the IT world will always necessitate the need for specialists (or better yet, generalizing specialists who choose to focus on a single specialty). Such specialists may be focused on technical domains such as data, security, or networking; business domains such as finance or logistics; or platforms that you're working with. Figure 6 shows how you will iterate back and forth between focusing on general issues, the "bigger picture," and on detailed specialist issues. You will incrementally build your artifacts (models, code, documents, tests, "¦) over time.

Figure 6. An overview of the agile architecture process (realistic) (click to enlarge).

Process of Architecture Modeling (realistic)

3. Agile Data Architects Explore Complex Architectural Issues

Sometimes you just don't know how something will work in practice. Instead of just guessing, you can choose to reduce the risk of unknowns by making them known. Two common techniques for doing so are:

  1. Architecture spikes. Architecture spikes, or just spikes, were first introduced by Extreme Programming (XP). A spike is throw away, quick and dirty code, used to explore a narrow technical issue. For example, spikes may be created to explore how two technologies work together (e.g. can language X interact with database Y?), or how a technology works in general (e.g. what are the new features in the new release of API Z?) Spikes are created to explore unknowns, thereby reducing overall risk. Figure 7 overviews the process of creating spikes, showing how it's an evolutionary exploration. The spike process typically runs on the order of a few hours to a few days.
  2. Proof of concept (PoC) projects. PoC projects are typically run to explore high-risk issues, such as the purchase of an expensive technology (e.g. adopting a Financial package) or the transitioning from one platform/technology to another one (e.g. migrating from DB2 to Oracle). PoC projects tend to run for several weeks or months, and may require significant research, consideration of several options, and collaboration with vendors.

Figure 7. The process of creating architecture spikes.

Process of Architecture Spikes

4. Agile Data Architects Invest in Learning

An important responsibility for anyone in a senior role, including an architect role, is to share their skills and knowledge with others. There are various ways that you can help others to learn more about data architecture and data skills in general. In fact, PMI's Disciplined Agile (DA) tool kit includes the Grow Team Members process goal which provides a variety of strategies to support learning. This includes, but is not limited to strategies such as providing coaching office hours, making presentations, mentoring, coaching, running training sessions, and being involved in non-solo work such as pairing and mobbing.

And of course you should invest in your own learning. Agile data architects are generalizing specialists, overviewed in Figure 8. A generalizing specialist is someone with broad knowledge of the domain in which they work plus one or more specialties. In the case of a data architect, one of those specialties would be data and the broad knowledge that they would have includes other aspects of architecture and most likely agile database techniques.

Figure 8. The agile architecture skillset (click to enlarge).

Agile Architecture Skillset

5. Related Resources