Data is clearly an important aspect of software-based systems, a fact that the information technology (IT)
industry has understood for decades, yet many organizations still struggle with their approach to data within
their software processes. As a consultant that has the privilege of working in a wide range of organizations, it
seems that about one in ten organizations are reasonable successful with their approach to data-oriented
activities, about six in ten think they're doing well when they really aren't, and the rest have a pretty good
idea that they have a problem but don't know what to do about it. It doesn't have to be this way.
So how do you know you've got a problem? Enterprise data professionals, including both data
and data administrators, will be
frustrated by the fact that developers on development teams ignore their advice, standards, guidelines, and
enterprise models. Worse yet the developers often don't even know about these people and things in the first
Developers will be frustrated by what they
perceive (often rightfully so) to be the glacial pace of enterprise data professionals to make or authorize
seemingly simple changes.
Agile data engineers often find themselves
stuck in the middle of these two warring factions, trying to get their work done while struggling to keep the
peace. If one or more of these problems is common within your organization you've got a problem.
To understand why your organization should adopt the AD method you need to consider the following topics:
- The Agile Data vision in a nutshell
- Warning signs that you've got a problem
- Towards a solution: The agile movement
- An agile movement for data professionals
- Improving data ways of working (WoW)
- The implications for IT professionals
1. The Agile Data Vision In a Nutshell
The goal of the Agile Data (AD) methodology is to define strategies that you can apply in a wide variety of
situations to work together effectively regarding data activities. This isn't to say that AD is a "one size fits
all" way of working (WoW). Instead, consider AD as a collection of ways of thinking (WoT), roles, and WoW that
people within your organization will apply to work together effectively when it comes to data.
2. Warning Signs That You've Got A Problem
It is very easy for organizations to deny that they have a problem. It can be very difficult for senior
management to detect problems until it's too late because the "bad news" that they need to hear is filtered out
long before it gets to them. It can be difficult for people elsewhere in the organization to detect problems,
perhaps everything is going quite well in their opinion - unfortunately the value system that they're using to
judge the situation isn't ideal, making them blind to the problems that they are causing. The following is a
list of potential symptoms, each of which may indicate that your organization has one or more challenges that
the Agile Data method may help you to address:
- People are significantly frustrated with the efforts, or lack thereof, of one or more groups.
- Software is not being developed, or if it is it is taking much too long.
- Finger pointing occurs, "the data administrators are holding up progress" or "the developers aren't
following corporate guidelines" are common complaints. Worse yet, the finger pointer doesn't perceive
that they are also part of the problem.
- Political issues are given higher priority than working together to development, maintain, and support
- Ongoing feuds exist between people/groups. Phrases like "you always" and "you never" are very good clues
that feuds exist.
- Well-known problems within your organization are not being addressed. Furthermore, suggestions for
improvements appear to be ignored, nothing happens and no reason for rejection is provided.
- People are working excessively long hours with little or no reward.
- Decisions affecting teams, in particular development teams, are made in an apparently arbitrary and
We need to find a way to work together effectively. There are clear differences between the data and
development communities, and between the development and enterprise communities. The fact that we're talking
about different communities is also part of the problem, arguably one of the roots causes. You have a
fundamental decision to make: Should you use these differences as an excuse to exacerbate existing problems
within your organization or should you revel in these differences and find a way to take advantage of them? I
prefer the latter. My experience is that the values and principles of the agile movement form the basis for an
effective approach to working together.
3. Towards a Solution: The Agile Movement
To address the challenges faced by software developers an initial group of 17 methodologists formed the Agile Alliance, in February of 2001. An interesting thing about this
group is that they all came from different backgrounds, yet were able to come to an agreement on issues that
methodologists typically don't agree upon. This group of people defined The
Manifesto for Agile Software Development, better known as the Agile Manifesto, for encouraging better ways
of developing software, and then based on that manifesto formulated a collection of principles which defines the
criteria for agile software development processes. The manifesto defines four values and
principles which form the foundation of the agile movement.
The manifesto is defined by four simple value
statements - the important thing to understand is that while you should value the concepts on the right hand
side you should value the things on the left hand side (presented in bold) even more. A good way to think about
the manifesto is that it defines preferences, not alternatives, encouraging a focus on certain areas but not
eliminating others. The Agile Alliance values
- Individuals and interactions over processes and tools.
- Working software over comprehensive documentation.
- Customer collaboration over contract negotiation.
- Responding to change over following a plan.
The interesting thing about these value statements is there are something that almost everyone will instantly
agree to, yet will rarely adhere to in practice. Senior management will always claim that its employees are the
most important aspect of your organization, yet insist they follow ISO-9000 compliant processes and treat their
staff as replaceable assets. Even worse, management often refuses to provide sufficient resources to comply to
the processes that they insist teams follow. Everyone will readily agree that the creation of software is the
fundamental goal of software development, yet insist on spending months producing documentation describing what
the software is and how it is going to be built instead of simply rolling up their sleeves and building it. You
get the idea - people say one thing and do another. This has to stop now. Agile developers do what they say and
say what they do.
To help people to gain a better understanding of what agile software development is all about, the members of
the Agile Alliance refined the values captured in their manifesto into a collection of
twelve principles. Agile software
development methodologies, such as Scrum and SAFe should conform to. These principles are:
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable
- Welcome changing requirements, even late in development. Agile processes harness change for the
customer's competitive advantage.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to
the shorter time scale.
- Business people and developers must work together daily throughout the initiative.
- Build teams around motivated individuals. Give them the environment and support they need, and trust
them to get the job done.
- The most efficient and effective method of conveying information to and within a development team is
- Working software is the primary measure of progress.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to
maintain a constant pace indefinitely.
- Continuous attention to technical excellence and good design enhances agility.
- Simplicity - the art of maximizing the amount of work not done - is essential.
- The best architectures, requirements, and designs emerge from self-organizing teams.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its
Stop for a moment and think about these principles. Is this the way that your software teams actually work?
Is this the way that you think teams should work? Re-read the principles once again. Are they radical and
impossible goals as some people would claim, are they meaningless motherhood and apple pie statements, or are
they simply common sense? My belief is that these principles form a foundation of common sense upon which you
can base successful software development efforts. A foundation that can be used to direct the data-oriented
efforts of IT professionals.
For a detailed explanation of the values and principles of the agile movement, you may find
Examining the Agile Manifesto of interest.
4. An Agile Movement for Data Professionals
One of the challenges that of the Agile Manifesto is that its focus is on software development and as a
result does not address some of the critical issues faced by data professionals. This observation helped to
motivate the development of the Agile Data method, and in particular the
Agile Data Ways of Thinking (WoT):
- Look beyond data
- Collaborate closely
- Be quality infected
- Embrace evolution
- Be enterprise aware
- Everyone agile
Given the agile mindset, we are now in a position to address the challenges faced by organizations that
result from traditional data ways of working (WoW).
5. Improving Data Ways of Working (WoW)
The relationship between data professionals and their stakeholders is often less than ideal within many
organizations. Yes, there are some organizations where these people often work together quite well but there are
always tensions - when a healthy tension exists between groups your organization can benefit, unfortunately
these differences often lead to conflicts that aren't healthy. Your organization may not experience every single
problem listed below although it likely suffers from a subset.
How can we address the problems that organizations face when it comes to the data aspects of your way of
Table 1 lists the potential challenges and the solution suggested by the Agile
Table 1. Improving our data WoW.
|Different visions and priorities. Developers are often focused on the specific needs of a single
initiative and often strive to work as much as possible in isolation from the rest of the organization. Data
engineers focus on the database(s) that they are responsible for, often "protecting"the databases by
minimizing changes to the them. Data administrators and data architects focus on the overall data needs of
the enterprise, sometimes to the virtual exclusion of the immediate needs of development teams. Clearly the
scope of each group is different, their priorities are different, and the issues that they deal with are
different. To make matters worse your stakeholders, including direct users all the way up to senior
management, have varying priorities and visions as well.
Agile Data implores IT professionals to work together, to understand and respect the viewpoints of your
|Over-specialization of roles. Specialists have a tendency to become too narrowly focused, to know
everything there is to know about a small slice of software development but can become oblivious of
everything else. It is quite common to find senior Java developers that have never heard about data normalization, or even understand why
you would want to do such a thing, and data architects that can't read a Unified Modeling Language (UML)
state chart diagram. Because they are overly specialized they often have difficulties relating to other IT
professionals. A common agile way of thinking (WoT) is that IT professionals should have a general
understanding of the overall software process and have one or more specialties. Because they are generalists
they understand the broad range of issues pertinent to the "software game"yet at the same time have specific
and valuable skills to offer to their team. People who are just specialists or who are just generalists are
at the extreme ends of the spectrum, as with most things in life it is far better to find a sweet spot in
between these two extremes. .
||What we really need are people who are
|Process impedance mismatch. One of the few things that processes such as
Disciplined Agile Delivery (DAD),
Scrum, DSDM, Crystal Clear, and
Agile Modeling have in common is that they all work in an
evolutionary (iterative and incremental) manner. Unfortunately many within the data community still view
software development as a serial or near-serial process. Clearly there is an impedance mismatch here,
indicating that the data community needs to rethink their approach. It is possible to take an evolutionary
approach to data, a change that will require
cultural and organizational changes
within your organization.
Agile Data implores enterprise and data professionals to be prepared to work following an incremental and
iterative approach, the norm for most modern development and the defacto standard for agile software
development. It also implores application developers to recognize that the existing environment, and future
vision for the organization, places constraints on their efforts.
|Technology impedance mismatch.
Developers work with objects and components whereas data professionals work with databases and files.
Software engineering principles form the underlying foundational paradigm for objects and components whereas
set theory forms the underlying foundational paradigm for relational databases (by far the most popular
database technology). Because the underlying paradigms are different the technologies don't work together
perfectly and an impedance mismatch exists. This mismatch can be overcome although doing so requires a
Agile Data requires that IT professionals work together closely, learning from each other as they do so.
Agile data engineers have the skills to map the application schema to the data schema, to write
data-oriented code, and to performance tune their work.
|Ossified management. The technology and techniques used by IT professionals changes rapidly, a
fact that we all know very well. As people progress up the corporate hierarchy they deal less with
technology and more with people issues, the end result being that managers have lost their technical edge.
The problem is that their previous development experiences, experiences on which they base technical
decisions, may no longer be applicable. We saw this as an industry when we moved from procedural
technologies to object-oriented technologies - what may have been a good decision on a COBOL team often
proves to be the kiss of death to a Java team. We're clearly seeing this problem once again as we move to
agile software processes. Management needs to change with the times.
Agile Data asks enterprise architects to work with senior management and educate them in the realities of
modern software development. Similarly application developers should work with and help to educate both line
and middle management.
|Organizational challenges. Common problems such as
poor communications or politics between
individuals and groups hurt the data aspects of software development just as badly as they hurt other
Agile Data implores IT professionals to work with one another and with your stakeholders, to respect them
and to actively strive to work together effectively.
|Poor documentation. Most documentation seems to be at one extreme or another: little or no
documentation or overly complex documentation that nobody reads. Mutually agreed to development standards
and guidelines, legacy system documentation, legacy database documentation, and enterprise models can be
valuable resources when written well. Agile
documentation is definitely critical to your success.
Agile Data directs IT professionals to follow the principles of Agile Documentation.
|Ineffective architectural efforts. Most organizations face significant challenges when it comes
to enterprise architecture, the most common challenge being that they don't even know where to start. Biased
enterprise architectures, those that overly focus on one view of the enterprise, lead to architectures that
do not adequately address the real needs of your organization. As the Zachman
Framework indicates, there are many potential views (which Zachman unfortunately refers to as
components, a loaded term) that you want to consider, a concept captured by
Multiple Models principle. These views
are data, function/process, network, people, time, motivation. Ivory tower architectures, those formulated
by teams that have removed themselves from the day-to-day realities of development teams, look good on paper
but unfortunately fail in practice. Developer's unwillingness to conform to the constraints imposed upon
enterprise architectural models, if
they even know that such models exist, is also a common and serious problem.
Agile Data advises enterprise architects to take a multi-view/model approach to architecture and to actively
work on a team to support and prove their architecture. The feedback from these efforts should then be
reflected in future iterations of the architecture.
|Ineffective development guidelines. Many organizations struggle to come to a collection of
development guidelines that all IT professionals will work to. There is a large number of causes for this,
including people not understanding the need to follow such guidelines, people unwilling to follow someone
else's guidelines, overly complex guidelines, overly simplistic guidelines, a "one size fits all" attitude
that leads to inappropriate guidelines for a specific platform, and an unwillingness to evolve guidelines
over time. When you have an effective collection of guidelines available to you, and everyone understands
and applies them appropriately, you can dramatically improve the productivity of your IT efforts. I maintain
coding guidelines and modeling style guidelines.
Agile Data implores enterprise administrators to write clear, effective, and applicable standards and
guidelines and to be prepared to act on feedback from the development teams.
|Ineffective modeling efforts. This is often the result of several of the previously identified
problems. People focused on a specific aspect of development will often produce models that wonderfully
reflect the priorities of that narrow view but that don't take into account the realities of other views. An
enterprise data model may present an excellent vision of the data required by an organization, but an
enterprise model that reflects the data, functional, usage, and technical requirements of an organization is
likely far more useful. A
UML class diagram may reflect the needs of a
single initiative, but if it
doesn't reflect the realities of the legacy data sources that it will access then it is of little value in
practice. Modelers, and IT professionals in general, need to work together and look at the full picture to
be truly effective.
Agile Data directs IT professionals to follow the principles and practices of the Agile Modeling methodology.
6. The Implications for IT Professionals
The Agile Data method defines several roles that IT professionals will fulfill. These roles, and how they work
together, are discussed in greater detail in The Roles of Agile
Data. Table 2 summarizes the implications of the Agile Data method for IT
professionals, implications that are also covered in The Roles
of Agile Data.
Table 2. Implications for IT professionals.
- Everyone must agree to a common vision as to what they are trying to accomplish and how they're
going to accomplish it.
- Agile software developers are willing to work with others and co-locate as needed.
- Documentation is a reality of software development. Choose to be agile in your approach.
- Agile software developers strive to be generalists with one or more specialties. This is called
being a "generalizing
- Agile software developers are flexible in their approach because one "process size" does not fit
- Software is your primary goal, enabling the next effort is your secondary goal.
- Application developers must work closely with stakeholders.
- Application developers must recognize that legacy systems and databases place constraints on them.
- Application developers should follow their organization's standards and guidelines and be willing to
provide feedback into their ongoing evolution.
- Application developers will work closely with enterprise architects, people who should become active
members of the team.
agile data engineers
- Agile data engineers work very closely with application developers to implement and support
data-oriented development efforts.
- Agile data engineers must be willing to work in an iterative and incremental manner, just as
application developers do.
- Agile data engineers will work with enterprise administrators to take advantage of and to help
evolve corporate meta data, standards, and guidelines.
- Enterprise architects focus on more than just data architecture.
- Enterprise architects work in an iterative and incremental manner.
- Enterprise architects actively work with development teams to communicate the architecture, to
mentor team members in architecture and modeling skills, and to gain real-world feedback that they
use to evolve the architecture.