Agile Data

The Roles of the Agile Data Method

Follow @scottwambler on Twitter!

In this article I define the four high-level roles of the Agile Data method. Any single individual can hold more than one of these roles, and several people may be in the same role. An important concept to understand is that although the responsibilities of each role vary widely that any individual in a given role may not fulfill them all, or at least not focus on all of them.  For example an application developer may choose to focus on Java development on a specific project even though they also have significant modeling and project management skills. Agile IT professionals are generalizing specialists, and when they fulfill a role they will often focus on one or more responsibilities of that role, perhaps because they have been asked to “trouble shoot” those efforts or perhaps because they want to gain deeper experience in an existing or new specialty. This is an example of applying the sweet spot philosophy to your skill set, avoiding the extremes of being either a specialist or a generalist.

 This article is organized into the following sections:

  1. The mindset of agile software developers
  2. Working together effectively
  3. Short a few roles?


1. The Mindset of Agile Software Developers

An underlying assumption of the Agile Data method is that your organization want to take an agile approach to software development.  Agile software development reflects a shift of mindset by IT professionals, a new way of thinking. To succeed at the Agile Data method, people in these four roles must have this mindset, a mindset that is characterized by the following philosophies.

Agile software developers recognize the importance of working together effectively with others and will act accordingly.  They have the humility to respect and appreciate the views and abilities of others, without this humility they are unlikely to willingly choose to collaborate with other. A critical implication is that everyone is going to have to rethink the way that they work and be willing to change for the greater good.  The attitude that “my group is the center of the universe and everyone has to conform to our vision and follow our process” doesn't work well.

Agile software developers actively seek to define an overall approach that everyone agrees to. My experience is that processes imposed from the top are very likely to fail because all it takes is one group to reject the process and “go rogue”.  A better process improvement strategy is to organically grow a workable software process that reflects the needs of everyone involved.  Because IT professionals are intelligent people with valuable skills you are likely to find that a collection of principles that everyone agrees to is often the most important part of an effective process.  In the case of the Agile Data method this is the values and principles of agile software development as well as the Agile Data philosophies.

An interesting observation is that you likely need a lot less documentation describing how people should work together than you think.  For example, consider the processes by which you obtain food.  Clearly this is a very important part of your life because starving isn't much fun, yet I doubt very highly that you have a detailed process written down about how you do it. Furthermore, I doubt that your local grocery store has written procedures that you must follow in order to purchase food from them even though this is also a process that is critical to their success because it's hard to stay in business without any customers. Many aspects of software development can also be like this – once you have an agreed to process in place that everyone understands you can follow that process successfully. New people can get involved with and learn the process, I often see parents taking their children with them to the grocery store, just as I see new employees in IT departments.  The process can evolve over time, my local grocery store started accepting debit cards a few years ago and everyone involved can adjust accordingly. Yes, software development is a lot harder than shopping, but the basic ideas hold.  When there is a common vision in place, when people agree to and conform to that vision, and when people understand how to go about their part of the process then there is little need for detailed procedures (even if you had then people aren't going to read it anyway).    

Agile software developers are willing to “co-locate” with others as needed. You might need to give up your comfortable cubicle or office for a while to work in a shared team space, or even have someone share your office to work with you on a project.  This reflects the fact that communication and collaboration are critical to your success, you are much more effective working with others than you are working alone.  Because communication is a two way street you also need to listen as well as talk.

Agile software developers are generalizing specialists.  The implication is that everyone needs to have a wide range of skills and be willing to work with others to improve upon existing skills and to learn new ones. At the present moment it is unrealistic to expect many people to have this range of skills, many IT professionals are narrowly specialized in one aspect of the software process, but over time people can upgrade their skills if they choose to do so.  People new to the IT industry will always need to start somewhere and will clearly need time to come up to speed.

Agile software developers are also prepared to tailor their approach to meet the needs of the projects they are involved with.  For example a project team working on a reporting database may very well take a data-driven approach to development, using conceptual data models as the primary analysis artifact and physical data models as the primary design artifacts. A project team working on a business system is very likely to take a different approach, particularly where the target platform uses object technology such as Java, C#, or Visual Basic, using UML artifacts such as class diagrams, activity diagrams, and sequence diagrams as primary artifacts.  No one development process fits all applications

Agile software developers recognize that documentation is a necessity in their jobs, although something they can still be very effective at.  For example, enterprise architects recognize that the goal of enterprise modeling is to produce effective models that meet the needs of their audience, not to produce reams of documentation. They recognize that many traditional architecture efforts fail because developers are not willing to invest the time to wade through the documentation to learn the architecture. Application developers realize that system documentation is required to support future enhancement efforts and Agile DBAs realize that documentation is required that describes the data sources that they support.  Agile software developers will take an agile approach to documentation and produce well-written and concise documents that is just barely good enough for the situation at hand.


2. Working Together Effectively

Figure 1 presents a high-level overview of how people in the four roles interact with one another, indicating the types of information that they share with one another. The diagram distinguishes between the project-level efforts of Agile DBAs and application developers and the enterprise efforts of enterprise administrators and enterprise architects because we need to recognize that the potential division between these two types of efforts needs to be bridge.  At the project level the potential division between Agile DBAs and application developers also needs to be bridged, as does any division between people at the enterprise level. It is interesting to see that each group has something to offer to the other groups, the implication being that everyone needs to be prepared to work with everyone else. Agile Database Techniques


Figure 1. Interactions between Agile Data roles.

Let's examine each role in greater detail.


2.1 Agile DBAs

An Agile DBA (Schuh 2001) is anyone who is actively involved with the creation and evolution of the data aspects of one or more applications.  The responsibilities of this role potentially includes, but is not limited to, the responsibilities typically associated with the traditional roles of database programmers, database administrators (DBAs), data testers, data modelers, business analysts, project managers, and deployment engineers.  This is the type of role that a DBA within a small organization typically finds themselves in – a “data jack of all trades”.

Agile DBAs work closely with application developers, typically supporting a single larger team or several smaller teams as the case may be. Agile DBAs can often be responsible for several data sources (e.g. databases, files, XML structures, and so on) or at least be co-responsible for them.  For example, if two development team access the same database, and each of them have their own Agile DBA, then those two people will need to work together to evolve that database over time.  This is slightly different than Schuh's original vision of an Agile DBA – his focus was on how a DBA can be effective on a single team whereas Agile Data looks at the entire enterprise. 

The biggest potential change for Agile DBAs is that they will need to work in an iterative and incremental manner for with many project teams.  Modern development processes, such as Disciplined Agile Delivery (DAD), typically don't provide detailed requirements up front nor do they focus on detailed models (and certainly not detailed data models up front).  They evolve their models over time to reflect their changing understanding of the problem domain as well as the changing requirements of their stakeholders. Some project teams may choose to work in a more serial manner, they may even choose to produce a detailed conceptual data model early in their lifecycle, but those teams will be few and far between although you will be expected to support them too. Agile DBAs will need to communicate the constraints imposed by legacy data sources, working with application developers to understand those constraints and work appropriately. 

Agile DBAs will evolve their legacy data schemas over time, applying common data refactorings as appropriate and working with new tools to evolve and migrate their data schemas over time. This is a difficult but necessary task. Agile DBAs will also need to work with application developers to model their data needs, working with UML-based artifacts such class diagrams with some project teams and conceptual data models with other teams.  Agile DBAs will work with application developers to write and test database code such as stored procedures, data-oriented code within applications that interacts with their data sources, and even aid in mapping the application schema to the data schema. Performance tuning, both of the database and of data-oriented code and mappings within applications, is an important aspect of the job.  

Agile DBAs work with enterprise administrators who are responsible for maintaining and evolving the corporate meta-data describing your enterprise and your corporate development standards and guidelines.  Agile DBAs will use this information, and follow the standards and guidelines, as well as provide valuable feedback.  Agile DBAs will also interact with enterprise administrators and other Agile DBAs to evolve the various enterprise data sources over time.

Agile DBAs also work with enterprise architects to ensure that their work fits into the overall picture and to help evolve the enterprise architecture over time.


2.2 Application Developers

For the sake of the Agile Data method an application developer is anyone who is actively involved with the creation and evolution of the non-data aspects of a software application.  The responsibilities of this role can include the responsibilities traditionally associated to the “traditional roles” of programmers, modelers, testers, team leads, business analysts, project managers, and deployment engineers. Application developers work very closely with Agile DBAs who are responsible for working on the data aspects of one or more applications. 

Application developers must recognize that your although your primary focus is fulfilling the current needs of your direct project stakeholders that they also need to recognize that their project exists within the larger scope of your organization. This philosophy reflects Agile Modeling's principles Software Is Your Primary Goal and Enabling the Next Effort is Your Secondary Goal – in this case part of the next effort is ensuring that your project conforms to the overall enterprise vision.  Application developers are best served by recognizing that they are working on one project of many within their organization, that many projects came before theirs, that many projects will come after theirs, and therefore they need to work with people in the other roles to ensure that they do the right thing.

Application developers will adopt and follow agile software development processes such as Disciplined Agile Delivery (DAD). When it comes to modeling and documentation they are likely to enhance these processes with the principles and practices of Agile Modeling (AM).  All three of these processes, being agile, implores developers to work closely with their project stakeholders.  An implication is that developers are responsible for helping to educate their stakeholders, including both users and managers, in the basics of software development to help them make more informed decisions when it comes to technology. 

Application developers will often find that they are constrained by they organization's legacy systems, including legacy data schemas.  These systems will often be very difficult to evolve, and if they can evolve will often happen very slowly. Luckily Agile DBAs will be able to help application developers deal with the realities imposed upon them by legacy data sources, but they will need to work with enterprise administrators and more so with enterprise architects to ensure that their efforts reflect the long-term needs of your organization. 

Application developers will also need to recognize that they need to follow their organization's development practices, including the guidelines and standards supported by enterprise administrators.  Application developers are expected to provide feedback regarding the standards and guidelines, every in the organization should do so, and should be prepared two work with the enterprise administrators to develop guidelines for development environments that are new to your organization. 

Application developers also need to work closely with enterprise architects to ensure that their project takes advantage of existing enterprise resources as well as fits into the overall enterprise vision.  The enterprise architects should be able to provide this guidance and will work with your team to architect and even build your system.  Furthermore, application developers should expect to be mentored in “senior” skills such as architecture and modeling.  This approach makes it easy for your team to support enterprise efforts and helps to keep the enterprise architects grounded because they quickly discover if their architecture actually works in practice.


2.3 Enterprise Administrators

An enterprise administrator is anyone who is actively involved in identifying, documenting, evolving, protecting, and eventually sunsetting corporate IT assets. These assets include corporate data, corporate development standards/guidelines (e.g. modeling guidelines or coding guidelines), and reusable assets such as components, frameworks, and services. The responsibilities of this role potentially includes, but is not limited to, the responsibilities associated with traditional roles of data administrators, network administrators, reuse engineers, and software process specialists.  It is interesting to note that the role of data administrator, while clearly very important, isn't sufficient in an agile environment because of it's narrow focus on data.  Good enterprise administrators are generalizing specialists, one specialty could in fact be in data administration, who understand a wide range of issues pertinent to the enterprise.

Enterprise administrators recognize that there is more to this job than data administration, and like Agile DBAs they will work in an evolutionary manner when required to.  This is because enterprise administrators work closely with Agile DBAs, and to a lesser extent application developers, who will very often work in this manner. Enterprise administrators work with Agile DBAs to ensure that their databases reflect the overall needs and direction of the enterprise.  Enterprise administrators will find ways to communicate the importance of their role to Agile DBAs and application developers, and the best way to do this is to focus on things that will make them more effective in their jobs – few people refuse a helping hand.  Trying to “impose your will” through onerous processes or management edicts very likely won't work well, and in fact will likely result in you being blocked.

Enterprise administrators work with both Agile DBAs and application developers to ensure that these folks understand the corporate standards and guidelines that they should follow.  However, their role is to support the standards and guidelines, not enforce them. A good rule of thumb is that if you need to act as “standards police” then you have lost the battle and that it is very likely your fault because you didn't communicate the standards well, didn't gain support, or tried to enforce unrealistic guidelines. If the standards and guidelines make sense, if they're written well, if they're easy to conform to then data and application developers should be willing to follow them. However, when this is not the case, when the standards and guidelines aren't appropriate or place an inordinate burden on projects then enterprise administrators should expect pushback. Yes, some individuals may chaff at following standards and guidelines but that's something that project managers will need to deal with. 

When pushback occurs enterprise administrators work with the project team(s) explore and address the problem.  They are prepared to evolve the standards and guidelines over time to reflect lessons learned and the changing realities of your organization.  One size will not fit all – your relational database naming conventions may be very different from your Java naming conventions and that's ok because those are two different environments with two different sets of priorities.

Enterprise administrators work closely with enterprise architects to communicate the constraints imposed by the current environment to the architects. More importantly the enterprise administrators need to understand the future direction envisioned by the enterprise architects to ensure that their efforts support the long-term direction of your organization.


2.4 Enterprise Architects

An enterprise architect is anyone who is actively involved in the creation, evolution, and support/communication of enterprise models.  These models describe a wide variety of views, one of which may be data oriented, although network/hardware views, business process views, usage views, and organizational structure views (to name a few) are equally as valuable.  The responsibilities of this role potentially includes, but is not limited to, the responsibilities associated with the traditional roles of enterprise data architects, enterprise process architects, enterprise network architects, and so on. As with the role of enterprise administrator, the role of enterprise architect has a greater scope than just that of data – they instead look at the entire enterprise picture. Enterprise architects main job is to look into the future, to attempt to identify a direction in which the organization is going and hence determine how its IT infrastructure needs to evolve. Enterprise architects are naturally constrained by the current situation your organization finds itself in, its environment, and its capability to evolve.

Enterprise architects focus on a wide variety of architectural issues, data being one of many. Their main goal is to develop and then support enterprise architectural models.  It isn't sufficient for an enterprise architect to produce good models, they must evangelize those models, work with development teams, and educate senior management in the implications of the architecture of in system-related issues in general. In addition to the CIO and CTO of your organization your enterprise architects are likely to have the most visibility with senior management, therefore they need to be prepared to aid senior management to make strategic decisions.

Enterprise architects work with Agile DBAs and with application developers. The most important thing that enterprise architects can do is to “walk the talk” and roll up their sleeves and get actively involved with the project. This will earn the respect of the developers, dramatically increasing the chance that they'll actually understand and follow the vision of the enterprise architecture.  The advantage of this approach is that it provides immediate and concrete feedback as to whether the architecture actually works and provides valuable insights for how the architecture needs to evolve. 

Enterprise architects are prepared to work in an iterative and incremental manner. They are ill-advised to try to create an all-encompassing set of enterprise models at first.  Instead, envision an initial, high-level architecture and then work closely with one or more development teams to make sure it works.  Agile Modeling includes a practice called Model in Small Increments which is based on the idea that the longer you model without receiving concrete feedback, such at that provided by an actual project, the greater the chance that your model doesn't reflect the real-world needs of your organization.  Agile enterprise architects avoid ivory-tower architectures this way.

You may find The EUP Enterprise Architecture Discipline and The EUP Enterprise Business Modeling Discipline to be of interest.


3. Short a Few Roles?

Your organization may not have people in these roles right now.  Perhaps you're a very small organization with only a few developers or perhaps you're a mid-size organization that hasn't yet come to grips with the architectural and administration issues experienced by larger or longer-lived firms. Don't worry, you will. You're actually in an enviable position because you can do it right from the start.  You might not have anyone dedicated to the role of enterprise architecture, let a group of people, but the fact still remains that you will need to come to grips with the issues that enterprise architects deal with.  Perhaps enterprise architecture is something that your senior developers, and even your junior developers, discuss when they're building new systems but never take the time to put it on paper. That's all right, the important thing is that you think about it.  The same can be said of enterprise administration, perhaps it's something that your Agile DBAs do as part of their day-to-day jobs.