Agile/Lean Data Governance: Proven Strategies

The goal of data governance is to ensure the quality, availability, integrity, security, and usability of information within an organization. The way that you go about this is up to you. Many traditional approaches to data governance seem to struggle in practice, I suspect in part because of the cultural impedance mismatch but also in part because traditional IT governance struggles in general. The bureaucratic, command and control approach typical of traditional governance strategies is a lot like herding cats, you do a lot of work but nothing much gets accomplished in the long run because it’s far too easy for the cats to simply go around whatever you want them to do. Agile/lean governance, on the other hand, is focused on motivating people to do the “right things,” enabling them to do so, and then monitoring and guiding their efforts where needed.

The goal of agile/lean data governance is to enable development teams to maintain and develop high-quality data assets within your overall IT ecology. A lean data governance approach promotes a healthy, collaborative relationship between data professionals and the teams that they’re supporting. In the Disciplined Agile (DA) toolkit we have explicitly adopted proven strategies for lean governance both of development teams and the organization in general. The approach is based on the observation is that the most effective way to govern the actions of intellectual workers is through motivating and enabling them, not by a command-and-control process. Unfortunately, traditional approaches to IT governance are often implemented in a command-and-control fashion.

This article is organized into the following sections:

1. Potential Challenges with Traditional Data Governance Strategies

Traditional data governance strategies often suffer from one or more common problems:

Data governance doesn’t fit into the overall governance effort. In a way it’s sort of questionable to even consider data governance, or development governance, or financial governance, or a myriad of other governance efforts, outside the scope of the overall organizational governance strategy. You need to optimize the whole governance strategy, not just individual parts.
Data governance efforts are ignored. Many development teams will choose to “work around” their organization’s data group, often because they find the data group too difficult or slow to work with. This is clearly problematic and an indication that your data governance efforts can’t possibly succeed if you can’t find ways to collaborate effectively with development teams.
Traditional data governance is too onerous. When it is difficult to work with your data governance team it motivates people to avoid working with them, thereby reducing the likelihood of them being compliant to your organizational conventions.
Traditional data governors are too slow to respond. This motivates developers to what they believe is best (even though they might not actually know what the best course of action is).
Data teams aren’t perceived as providing value.

2. Agile/Lean Data Governance Strategies

In addition to supporting proven agile database practices, the agile/lean development governance practices which you should consider adopting for your data governance efforts are:

Valued corporate assets. Guidance (such as database design conventions, modeling style guidelines, data naming conventions, and report design guidelines), metadata definitions, and reusable assets such as frameworks and components, will be adopted if they are perceived to add value to developers. You want to make it as easy as possible for developers to comply to, and more importantly take advantage of, your corporate IT infrastructure. When data standards are sensible, easy to understand, and easy to access then there is a significantly greater chance that people will actually follow the standards in practice. When you force people to conform to standards, when it makes it onerous for them to do so, then you reduce the chance that they will actually do so.
Usage-driven development. The whole cannot be defined without understanding the parts, and the parts cannot be defined in detail without understanding the whole. By taking a usage-driven approach you can understand how people will actually use your system, thereby enabling you to build something that meets their actual needs. A common mistake made by traditional data approaches is that they take data-driven approaches (understandable, given their biases) which gets them in trouble because data is too narrowly focused to drive things and it doesn’t reflect the needs of your overall governance effort.
Include data professionals as active participants on development teams. When your DM group is external to development teams it can foster a “them vs. us” mentality within your IT organization if you’re not very careful. You don’t need to have an external group to run your data governance activities, instead individual data professionals can do so as part of their responsibilities on development teams in a collaborative and timely manner. This is one of the fundamental concepts of the Agile Data method.
Educate developers. Developers need to understand why your MDM efforts are important, what the benefits are, and how to work together with your DM team. When they know why something needs to be done, and how to do it effectively, chances are much better that they’ll actually do it.
Allow teams to choose their way of working (WoW). Because teams vary in size, distribution, purpose, criticality, need for oversight, and member skillset, one process size does not fit all. The implication is that your approach to support data-oriented activities, including governance, will vary by team.
Align team structure with architecture. The organization of your data team should reflect the enterprise architecture structure. For example, a centralized data team will struggle to support an environment with a decentralized architecture.
Align HR policies with IT values. You need to establish specific incentives/rewards appropriate for the mindset of your technical staff to ensure that they follow your data governance strategy.
Align stakeholder policies and IT values. Your development efforts are driven and constrained by your stakeholders. Your stakeholders must be realistic in their demands on IT, and understand the implications of their decisions (you’ll need to educate them).
Business-driven pipeline. You should invest in the activities that are well-aligned to the business direction, return definable value, and match well with the priorities of the enterprise. This includes data-oriented activities as well. Unfortunately, many traditional data governance strategies often seem to reflect the priorities of the data bureaucrats among us, not the priorities of the business, resulting in data warehouses etc. being built which are underused
Embedded compliance. It is better to build compliance into your day-to-day processes instead of having a separate compliance process which often results in unnecessary overhead. Automation is critical. For example, Instead of holding reviews to ensure that development teams follow corporate data conventions, a time consuming and expensive endeavor, why not instead run an automated analysis tool against the database schemas on a regular basis to ensure that the data naming conventions are followed?
Flexible architectures. Architectures that are loosely coupled and highly cohesive that implement common architectural and design patterns lend themselves to greater levels of consistency, reuse, and adaptability.
Pragmatic governance body. Effective governance bodies focus on enabling teams in a cost-effective and timely manner. They typically have a small core staff with a majority of members being representatives from the governed groups.
Promote self-organizing teams. The best people for planning work are the ones who are going to do it. Knowledge workers should be respected as intelligent people who can determine their own strategies for working together. When given a bit of coaching and guidance, they can plan their work within established parameters, such as iteration boundaries. Self-organization doesn’t mean that the team is out of control, any given team must conform to your overall governance strategy, enterprise architecture, and so on.
Risk-based milestones. You want to mitigate the risks of your initiative, in particular business and technical risks, early in the lifecycle. You do this by having several milestones that teams work toward. The goal of each milestone is to address one or more risks. For example, Disciplined Agile Delivery (DAD) explicitly includes several risk-based milestones in its lifecycles (it supports several), one of which is “Proven architecture” that requires your architecture be proven through working code before Construction begins, thereby lowering overall technical risk.

3. Data Governance Success Factors

I’ve found that the following factors are critical to the success of your data governance efforts:

Recognize that IT governance, and better yet corporate governance, is the real goal. Data governance is just one part of your IT governance program, and it’s highly coupled to other aspects such as development governance, security governance, and so on. Focusing just on data governance puts you at risk of optimizing data governance in such a way that it doesn’t work well with the rest of your governance efforts, putting the entire governance program at risk. Remember the first of the Agile Data ways of thinking (WoT) to look beyond data, because data is only part of the overall picture.
The governance effort must be owned. If someone is not responsible for the governance effort it will very likely die a quick death in your organization. My experience is that the people most suited to be the owners of IT governance are the executive business stakeholders. When it comes to data governance, the people least suited to govern are data management professionals as they’re the ones being governed (amongst others). In short, don’t let your data governance efforts devolve into yet another political ploy of your data management group to try to grab additional power.
Have clear, quantifiable goals. What are you trying to achieve? Improved quality? Improved productivity? Improved time to value? Improved stakeholder satisfaction? Combinations thereof?
Measure and honestly report the results. It’s easy to talk about quantifiable goals, but it takes a fair bit of integrity to live up to your promises (and I’ll let the very serious lack of data around the effectiveness of data governance efforts speak for itself). It’s fairly easy to manage direct costs such as the salaries of the people involved with the governance effort, but a bit more difficult to measure indirect costs such as the opportunity cost of potentially lengthening decision and development life cycles due to increased governance (this issue is particularly acute for traditional governance strategies). Measuring the benefits can also be challenging, although as Doug Hubbard points out in How to Measure Anything it is possible if you think outside the box a bit. Automating metrics collection is an important aspect of lean governance.
Less is more. You need a lot less governance than what the pro-governance people believe, although probably a bit more governance than what the anti-governance people think. If you find that you need more governance it’s a lot easier to add it than it is to remove unnecessary governance activities once they’re in place.
Educate the people affected. If the people involved, including those being governed, don’t understand what you’re trying to achieve and also believe that any additional effort on their part is worthwhile then your governance effort will quickly fall apart. Furthermore, this sort of education is ongoing.

This book, Choose Your WoW! A Disciplined Agile Approach to Optimizing Your Way of Working (WoW) – Second Edition, is an indispensable guide for agile coaches and practitioners. It overviews key aspects of the Disciplined Agile® (DA™) tool kit. Hundreds of organizations around the world have already benefited from DA, which is the only comprehensive tool kit available for guidance on building high-performance agile teams and optimizing your WoW. As a hybrid of the leading agile, lean, and traditional approaches, DA provides hundreds of strategies to help you make better decisions within your agile teams, balancing self-organization with the realities and constraints of your unique enterprise context.

This book, Non-Invasive Data Governance, provides a detailed look at effective data governance. Data-governance programs focus on authority and accountability for the management of data as a valued organizational asset. Data Governance should not be about command-and-control, yet at times could become invasive or threatening to the work, people and culture of an organization. Non-Invasive Data Governance™ focuses on formalizing existing accountability for the management of data and improving formal communications, protection, and quality efforts through effective stewarding of data resources.

Non-Invasive Data Governance Strikes Again provides a blend of 50 applicable lessons learned and perspectives gained from years of assisting organizations worldwide to follow the popular non-invasive approach from the bestseller, Non-Invasive Data Governance.

I also maintain an agile database books page which overviews many books you will find interesting.