The Agile Data Architect

An agile data architect is someone who guides the development and support of the data-oriented aspects of something in a collaborative and evolutionary (iterative and incremental) manner. “Something” may be a component, a solution, a product line, or even your entire enterprise.This article is organized into the following topics:

1. The Agile Data Architect Role

Agile data architects are considered specialists, or more accurately specialized architects. This can be a good stepping stone towards a more senior role, such as an architecture owner on an agile team or an enterprise architect. Or you may choose to remain specialized, it’s really up to you. Having said that, as I indicate below agile data architects still need to work towards becoming a generalizing specialist in that minimally they will need agile database development skills.

Depending on need, agile data architects may focus on one or more architectural levels:

Component. A component data architect will guide the development and support of the data aspects of a component, such as a microservice or API. If the component is data intense, such as a collection of data services, they may in effect be the architecture owner for that component.
Solution. A solution data architect will guide the development and support of the data aspects of a solution, such as an application or system. If the solution is data intense, such as a data warehouse, they may in effect be the architecture owner for the solution. In this case, they are likely to need broader architectural skills than just data, for example security and user experience (UX) skills for a data warehouse/business intelligence solution.
Product line. A product line data architect will guide the development and support of the data aspects of a product line, which is a collection of related solutions and services. Product line data architects will need to work closely with the architecture owner for the product line as well as with the data architects (if any) on teams working on the solutions and services
Enterprise. A enterprise data architect will guide the development and support of the data aspects of an entire organization, working very closely with your enterprise architects to ensure that their work fits into the overall direction of your enterprise.

For more information about these architecture levels, see Agile Data Architecture Context.

2. The Responsibilities of an Agile Data Architect

There are effectively four key activities performed by agile data architects:

Provide help to others. Agile data architects spend the majority of their time collaborating with their stakeholders to either apply, create, or evolve data architecture assets.
Agile architectural modeling. Agile data architects will apply Agile Modeling (AM) strategies to explore and capture data concerns. More on this below
Explore complex architectural issues. Sometimes you just don’t know how something will work in practice. Instead of just guessing, you can choose to reduce the risk of unknowns by making them known. Two common techniques for doing so are to build architecture spikes or run proof of concept (PoC) initiatives.
Invest in learning. An important responsibility for anyone in a senior role, including architects, is to share their skills and knowledge with others. And of course you should invest in your own learning.

I explore these activities in greater detail in the article The Agile Data Architecture Process.

3. Becoming An Agile Data Architect

Agile data architects need:

3.1 Agile Data Architects Need to be Generalizing Specialists

A generalizing specialist is someone who:

Has one or more specialized skills (e.g. User interface design, Project Management, Security Engineering, Marketing, …).
Has at least a general knowledge of software development.
Has at least a general knowledge of the business domain in which they work.
Actively seeks to gain new skills in both their existing specialties as well as in other areas.

People who are generalizing specialists are in the “sweet spot” between the two extremes of being either just a specialist or just a generalist, enjoying the benefits of both but not suffering from their drawbacks.

But what does this mean for an agile data architect? As I indicated early, data architects are certainly an example of a specialized architect, are are security architects, user experience (UX) architects, business architects, and others. Although specialized architects can prove to be valuable, particularly when you face a very difficult problem, they can also prove to be detrimental in practice due to their tendency to locally optimize around their specialty. As an agile data architect, you usually need to be more than just a specialized data architect.

Figure 1 depicts representative skill ranges for various architectural roles, including developers. These are examples, not prescriptions, as everyone is different. It is the general pattern that is important. You see that a solution architect has a wide range of skills. This person has deep skills in a narrow specialty, let’s assume data, from the implementation level all the way up to the enterprise level. They have broad skills applicable to solution architecture, hence their current role, but also deep skills in some aspects of product line, component architecture, and even implementation. They are well suited to be a solution architect given their broad skills in that, and their skills at other levels enables them to have more empathy for people working at those levels and to better interact with them as a result. Notice however that they don’t have solid skills in all aspects of solution architecture, they have more to learn (as does everyone else). They’re not just a specialized architect, however, someone with very deep skills in one area but weak in many others.

Figure 1. The agile architecture skillset (click to enlarge).

3.2 Agile Data Architects Need Agile Modeling Skills

Agile Modeling (AM) is a practice-based methodology for effective modeling and documentation. Agile modeling is an evolutionary (iterative and incremental) approach to creating models and supporting documents that is highly collaborative in nature. You can view Agile Modeling as a collection of core practices, as you see in Figure 2. These practices are applicable to all modeling domains, including both data and architecture.

Figure 2. The core practices of Agile Modeling (click on each practice for details).

Key AM concepts for data architects are:

Collaborate closely with stakeholders.
Take an evolutionary (iterative and incremental) approach.
Have the humility to realize that you don’t have all of the answers.
Tailor your way of working (WoW) to reflect your situation.
Embrace change.
Agile models are sufficient in that they are just barely good enough (JBGE) for the context in which they are used.
Detailed specifications should be captured as executable tests whenever possible.
High-level concepts should be captured visually and supported by concise descriptions whenever possible.
Take a multi-view approach, data being just one view.

3.3 Agile Data Architects Need Implementation Skills

Agile data architects are also familiar with, and ideally adept at, the agile database techniques stack overviewed in Figure 3 below. These are the critical skills required of anyone involved in the implementation aspects of an agile data initiative.

Figure 3. The agile database techniques stack.

Because agile data architects are expected to be generalizing specialists, and with the exception of industry data architects having at least rudimentary implementation skills, they should be knowledgeable about the following techniques:

1. Vertical slicing. Vertical slicing organizes functionality vertically into small, consumable pieces that may be potentially deployed into production quickly. These vertical slices are completely implemented – the analysis, design, programming, and testing are complete – and offer real business value to stakeholders.
2. Clean data architecture and clean data design. A clean data architecture strategy enables you to develop and evolve your data assets at a pace which safely and effectively supports your organization – in short, to be agile. Similarly, a clean data design

enables you to evolve specific data assets in an agile manner.

Agile data modeling. With an evolutionary approach to data modeling you model the data aspects of a system iteratively and incrementally. With an agile data modeling approach you do so in a highly collaborative and streamlined manner.
Database refactoring. A database refactoring is a small change to your database schema which improves its design without changing its semantics, enabling you to safely improve the quality of your data sources.
Automated database regression testing. You should ensure that your database schema actually meets the requirements for it, and the best way to do that is via automated regression tests for your database schema to ensure data quality.
Continuous database integration (CDI). Continuous integration (CI) is the automatic invocation of the build process of a system. As the name implies, continuous database integration (CDI) is the database version of CI.
Configuration management. Your data models, database tests, test data, and so on are important artifacts which should be put under configuration management just like any other artifact.