Agile Data

Agile/Lean Data Governance Best Practices

Follow @scottwambler on Twitter!

The goal of data governance is to ensure the quality, availability, integrity, security, and usability within an organization. The way that you go about this is up to you. Many traditional approaches to data governance seem to struggle in practice, I suspect in part because of the cultural impedance mismatch but also in part because traditional IT governance struggles in general. The command and control approach typical of traditional governance strategies is a lot like herding cats, you do a lot of work but nothing much gets accomplished in the long run.  Agile/lean governance, on the other hand, is focused on enabling people and motivating them to do the right things. 

The goal of agile/lean data governance is to enable development teams to maintain and develop high-quality data assets within your overall IT ecology. A lean data governance approach promotes a healthy, collaborative relationship between data professionals and the teams that they're supporting. Recently Per Kroll and I have been working on how to take a lean/agile approach to governance software development projects which has resulted in an IBM Whitepaper. The paper presents a collection of practices, many of which are applicable to IT governance in general (including data governance). The approach is based on the observation is that the most effective way to govern the actions of intellectual workers is through motivating and enabling them, not by a command-and-control process. This philosophy is backed up by findings from Dr. Dobb's Journal's Current State of Data Quality Techniques survey which found that a collaborative approach data management is more effective than a command-and-control approach, which in turn is better than no approach at all.  Unfortunately, traditional approaches to IT governance are often implemented in a command-and-control fashion. 

This article is organized into the following sections:

1. Potential Challenges with Traditional Data Governance Strategies

Traditional data governance strategies often suffer from one or more common problems:
  1. Data governance doesn't fit into the overall IT governance effort.  In a way it's sort of questionable to even consider data governance, or development governance, or SOA governance, or a myriad of other governance efforts, outside the scope of the overall IT governance strategy. You need to optimize the whole governance strategy, not just individual parts.
  2. Data governance efforts are ignored. The 2006 DDJ survey into the current state of data management practices showed that 66% of development teams will choose to "work around" their organization's data group, see Figure 1 below.  This is clearly problematic and an indication that your data governance efforts can't possibly succeed if you can't find ways to collaborate effectively with development teams.
  3. Data governance is too onerous. As you see in Figure 1 20% of development teams report that the data group within their organization is too difficult to work with. In some organizations this includes data governors.
  4. Data governors are too slow to respond. Figure 1 shows that 36% of development teams find that their data groups work too slowly, motivating developers to what they believe is best (even though they might not actually know what the best course of action is).
  5. Data governors aren't perceived as providing valueFigure 1 shows that 19% of development teams report that they believe that their data group doesn't add much value, often because of the additional bureaucracy involved with traditional approaches.
IT Governance

Figure 1. Reasons why development teams go around data groups.

2. Agile/Lean Data Governance Practices

In addition to supporting agile database best practices, the agile/lean development governance practices which you should consider adopting for your data governance efforts are:

  1. Valued Corporate Assets. Guidance (such as database design conventions, modeling style guidelines, data naming conventions, and report design guidelines), metadata definitions, and reusable assets such as frameworks and components, will be adopted if they are perceived to add value to developers. You want to make it as easy as possible for developers to comply to, and more importantly take advantage of, your corporate IT infrastructure. When data standards are sensible, easy to understand, and easy to access then there is a significantly greater chance that people will actually follow the standards in practice. When you force people to conform to standards, when it make it onerous for them to do so, then you reduce the chance that they will actually do so.
  2. Scenario-Driven Development. The whole cannot be defined without understanding the parts, and the parts cannot be defined in detail without understanding the whole. By taking a scenario-driven, also called a usage-driven approach, you can understand how people will actually use your system, thereby enabling you to build something that meets their actual needs. A common mistake made by traditional data approaches is that they take data-driven approaches (understandable, given their biases) which gets them in trouble because data is too narrowly focused to drive things and it doesn't reflect the needs of your overall governance effort.
  3. Include data professionals as active participants on development teams.  When your DM group is external to project teams it can foster a “them vs. us” mentality within your IT organization if you’re not very careful. You don’t need to have an external group to run your data governance activities, instead individual data professionals can do so as part of their responsibilities on development teams in a collaborative and timely manner.  This is one of the fundamental concepts of the Agile Data method.
  4. Educate developers. Developers need to understand why your MDM efforts are important, what the benefits are, and how to work together with your DM team. When they know why something needs to be done, and how to do it effectively, chances are much better that they’ll actually do it.
  5. Adapt the Process. Because teams vary in size, distribution, purpose, criticality, need for oversight, and member skillset, one process size does not fit all.  The implication is that your approach to support data-oriented activities, including governance, will vary by team.
  6. Align Team Structure With Architecture. The organization of your data team should reflect the enterprise architectural structure. For example, a centralized data team will struggle to support an environment with a decentralized architecture. 
  7. Align HR Policies With IT Values. You need to establish specific incentives/rewards appropriate for the mindset of your technical staff to ensure that they follow your data governance strategy.
  8. Align Stakeholder Policies and IT Values. Your development efforts are driven and constrained by your stakeholders. Your stakeholders must be realistic in their demands on IT, and understand the implications of their decisions (you’ll need to educate them).
  9. Business-Driven Project Pipeline. You should invest in the IT activities that are well-aligned to the business direction, return definable value, and match well with the priorities of the enterprise. This includes data-oriented activities as well. Unfortunately, many traditional data governance strategies often seem to reflect the priorities of the data bureaucrats among us, not the priorities of the business, resulting in data warehouses etc. being built which are underused
  10. Embedded Compliance.  It is better to build compliance into your day-to-day processes instead of having a separate compliance process which often results in unnecessary overhead. Automation is critical. For example, Instead of holding reviews to ensure that development teams follow corporate data conventions, a time consuming and expensive endeavor, why not instead run a static code analysis tool against the database schemas on a regular basis to ensure that the data naming conventions are followed?
  11. Flexible Architectures.  Architectures that are service-oriented, component-based, or object-oriented and implement common architectural and design patterns lend themselves to greater levels of consistency, reuse, and adaptability.
  12. Pragmatic Governance Body. Effective governance bodies focus on enabling development teams in a cost-effective and timely manner. They typically have a small core staff with a majority of members being representatives from the governed organizations.
  13. Promote Self-Organizing Teams. The best people for planning work are the ones who are going to do it. IT professionals should be respected as intelligent people who can determine their own strategies for working together. When given a bit of coaching and guidance, they can plan their work within established parameters, such as iteration boundaries. Self-organization doesn't mean that the team is out of control, any given team must conform to your overall governance strategy, enterprise architecture, and so on.
  14. Risk-Based Milestones.  You want to mitigate the risks of your project, in particular business and technical risks, early in the lifecycle. You do this by having throughout your project several milestones that teams work toward. The goal of each milestone is to address one or more risks. For example, Disciplined Agile Delivery (DAD) explicitly includes several light-weight milestones in its lifecycles (it supports several), one of which is "Proven architecture" that requires your architecture be proven through working code before Construction begins, thereby lowering overall technical risk.

3. Data Governance Success Factors

I've found that the following factors are critical to the success of your data governance efforts:

  1. Recognize that IT governance is the real goal. Data governance is just one part of your IT governance program, and it's highly coupled to other aspects such as development governance, security governance, and so on. Focusing just on data governance puts you at risk of optimizing data governance in such a way that it doesn't work well with the rest of your governance efforts, putting the entire governance program at risk. Remember the first agile data philosophy that data is only part of the overall picture.
  2. The governance effort must be owned. If someone is not responsible for the IT governance effort it will very likely die a quick death in your organization. My experience is that the people most suited to be the owners of IT governance are the executive business stakeholders. When it comes to data governance, the people least suited to govern are data management professionals as they're the ones being governed (amongst others). In short, don't let your data governance efforts devolve into yet another political ploy of your data management group to try to grab additional power.
  3. Have clear, quantifiable goals. What are you trying to achieve? Improved quality? Improved productivity? Improved time to value? Improved stakeholder satisfaction? Combinations thereof? 
  4. Measure and honestly report the results. It's easy to talk about quantifiable goals, but it takes a fair bit of integrity to live up to your promises (and I'll let the very serious lack of data around the value of data governance efforts speak for itself). It's fairly easy to manage direct costs such as the salaries of the people involved with the governance effort, but a bit more difficult to measure indirect costs such as the opportunity cost of potentially lengthening decision and development life cycles due to increased governance (this issue is particularly acute for traditional governance strategies). Measuring the benefits can also be challenging, although as Doug Hubbard points out in How to Measure Anything: Finding the Value of Intangibles in Business it is possible if you think outside the box a bit. Automating metrics collection is an important aspect of lean governance.
  5. Less is more. You need a lot less governance than what the pro-governance people believe, although probably a bit more governance than what the anti-governance people think. If you find that you need more governance it's a lot easier to add it than it is to remove unnecessary governance activities once they're in place.
  6. Educate the people affected. If the people involved, including those being governed, don't understand what you're trying to achieve and also believe that any additional effort on their part is worthwhile then your governance effort will quickly fall apart.  Furthermore, this sort of education is ongoing.