The Agile Data (AD) Method

Agile Data Logo

Implementing Question Stories: User Stories for Data Teams

A question story is a specialized user story specific to data-oriented requirements. Question stories represent thin slices of value that are deployable into production. This article addresses the following topics:

  1. What are question stories?
  2. Implementing a question story on an agile/Scrum team
  3. Implementing a question story on a continuous delivery team
  4. The implementation activities
  5. What technical work needs to occur?

1. What Are Question Stories?

The following is an example of a question story (click on it to enlarge):

Question story format

2. Implementing a Question Story on an Agile/Scrum Team

Let’s consider the case when a team is following a Scrum lifecycle. Figure 1 overviews an agile project lifecycle, though it should evolve into a continuous delivery lifecycle, where Initiation/Sprint 0 disappears because it occurred in the distant past, and the deploy phase disappears due to automation. Either way, the following advice still pertains.

Figure 1. The Agile Project Lifecycle (click to enlarge).

 

Figure 2 depicts the look-ahead data analysis work required for three question stories that are to be implemented during sprint #9 of an agile DW/BI initiative. Notice how each question story requires a different amount of data analysis effort because each question has unique data needs.

Figure 2. Look-ahead data analysis on an agile team. Click to enlarge.

Figure 3 depicts the overall development effort for a single question story, including both the look-ahead analysis required and the implementation portion called out in Figure 2.

Figure 4. Implementing a question story on a Scrum team (click to enlarge).

3. Implementing a Question Story on a Continuous Delivery Team

Now let’s consider the situation when a team is following a continuous delivery lifecycle.

Figure 4. A Lean Continuous Delivery Lifecycle (click to enlarge).

Figure 5 depicts the look-ahead data analysis work for the same three question stories from Figure 2, the difference being that the work is done on a just-in-time (JIT) basis rather than scheduled into fixed-length sprints. Note that the same amount of data analysis is still required for each user story as in Figure 3, but that the implementation time is no longer tied to a two-week sprint.

Figure 5. Look-ahead data analysis on a continuous delivery team. Click to enlarge.

Figure 6 depicts the overall development effort for a single question story, including both the look-ahead analysis as well as the implementation portion called out in Figure 5.

Figure 6. Implementing a question story on a continuous delivery team (click to enlarge).

4. The Implementation Activities

As you saw in Figure 3 and Figure 5 there are several activities that the two flows have in common:

  • Data Analysis – Explore Question. The data analyst(s), and perhaps the product owner/backlog owner, will collaborate with stakeholders to understand what their needs are.
  • Data Analysis – Explore Data Source(s). If the data required to answer is not yet in the data warehouse, then the analyst will need to determine which source(s) will provide that data. They will then need to perform sufficient data profiling to understand the data to determine if it is sufficient for their needs.  If it isn’t sufficient, they will need to look for other sources, renegotiate the requirement with the stakeholders, or even declare that it isn’t currently possible to provide an answer to the question story.
  • Data Analysis – Build Answer Prototype. For question stories with potentially complicated answers, you may find that you need to prototype your solution and ask for feedback so as to explore it more thoroughly.
  • Data Analysis – Validate Analysis. The work should be validated, ideally in a collaborative manner with one or more stakeholders, to ensure that it is correct. This could be something as simple as a discussion or a quick “show and tell.”
  • Specify Answer (Agile/Scrum only). An agile team will usually have a definition of ready (DoR) that defines the criteria that a work item must meet before they will work on it. This often requires additional specification work that continuous delivery teams don’t require. They still will work with stakeholders to explore their needs further if required, but they typically don’t need the overhead of capturing that information “on paper.” Instead, they just start implementing the solution.
  • Review Specification (Agile/Scrum only). The artifact(s) specifying a work item are reviewed by the team to ensure they meet, or exceed, their DoR.
  • Implement Answer. See the detailed description below.
  • Deploy Answer. On a Scrum team, you are likely to batch several answer implementations, whereas on a CD team, you ideally deploy once the answer is ready to be released. In both cases the work would need to meet the team’s definition of done (DoD).

 

5. What Technical Work Needs to Occur?

Figure 7 depicts how a question story can be implemented as a thin slice in a data warehouse following a Data Vault 2 (DV2) architecture. The green boxes represent major components of a DataVault 2 approach to architecting a DW/BI solution and the blue line indicates the vertical slice that implements the given question story.  In this case, new data elements from two sources, Data Source 1 and Data Source 5, are used. The new data elements from these two data sources are loaded into a raw vault, then into the data vault, into Data Mart 3, where they are then displayed on a dashboard. Other data that already existed in the raw vault and upwards was also used to implement the question story.

Figure 7. Implementing a question story as a thin slice (click to enlarge).

Vertical slice for a question story

The implied technical work to implement this question story includes:

  1. Data analysis of the two data sources. This requires the data analysis activities mentioned above.
  2. Extract and load data from the two sources. The data tables containing the requisite data elements are added to existing extraction process to bring them into the persistent staging area (PSA) and then into the raw vault. If you have not adopted a DV2-based strategy in your organization, this extraction would likely be into a data lake instead.
  3. Update of extract, transform, and load (ETL) from the raw vault to the sparse business vault. Load only the data elements required to answer the question to the sparse business vault, applying requisite business transformations as you do so to conform the data.
  4. Extract the requisite data into the data/info mart. The normalized data in the business vault is denormalized for reporting purposes into Data Mart 3.
  5. Update the dashboard to display the information. Dashboard widgets are added or modified to provide the answer to stakeholders. Other answers may be implemented as pre-defined reports or via managed self-service business intelligence (BI).

5. Related Resources

Recommended Reading

Choose Your WoW! 2nd Edition
This book, Choose Your WoW! A Disciplined Agile Approach to Optimizing Your Way of Working (WoW) – Second Edition, is an indispensable guide for agile coaches and practitioners. It overviews key aspects of the Disciplined Agile® (DA™) tool kit. Hundreds of organizations around the world have already benefited from DA, which is the only comprehensive tool kit available for guidance on building high-performance agile teams and optimizing your WoW. As a hybrid of the leading agile, lean, and traditional approaches, DA provides hundreds of strategies to help you make better decisions within your agile teams, balancing self-organization with the realities and constraints of your unique enterprise context.

 

I also maintain an agile database books page which overviews many books you will find interesting.