The Agile Data (AD) Method

Agile Data Logo

Choosing the Right Sprint Length for an Agile Data Team

How long should the sprint length be for an agile data team? It depends on several considerations that I explore in this article. Your goal should be to adopt what works best for your team in the context that it faces.

A sprint is a short time box, also referred to as an iteration, during which a team produces a new increment of whatever product or service they are working on. In the case of a data team, this would be a new (potential) release of a data product, data warehouse, business intelligence (BI) solution, and so on.

 

Common Sprint Lengths

The following table summarizes my thinking about some common sprint lengths.

Sprint Length Advantages Disadvantages
Four weeks or more. This is a spectacularly bad idea.
  • Comfortable for team members new to agile
  • Teams tend to fall into a mini-waterfall strategy, suffering from the inefficiencies and risks associated with working in a serial manner.
Three weeks
  • A comfortable sprint length for agile data teams that intend to remain working in an agile rather than a continuous delivery/DataOps manner.
  • Tends to be long enough to avoid most sprint-induced problems, such as the need to perform look-ahead data analysis across more than a single sprint. 
  • Experienced data professionals new to agile/lean ways of working (WoW) may struggle to make the leap to new WoW due to extra leeway of a longer sprint length.
Two weeks
  • Two weeks is common with other teams, particularly those in a Scrum or SAFe environment, enabling easier integration between them.
  • Potentially challenging at first, many data teams will find that they can learn to work this way with good coaching and improved tooling.
One week. This is my preferred sprint length for agile data teams, although it can require a fair bit of improvement work to get to this.
  • Short sprints motivates teams to adopt common DataOps practices and technologies to streamline their WoW.
  • It is straightforward to evolve into a continuous delivery WoW from here.
  • Can be stressful for people new to agile WoW.
  • Some agile ceremonies, such as demos and retrospectives, likely don’t need to be on a weekly basis.
One day or less. With a one day sprint you potentially release daily into production, often on a regular schedule (e.g. at 3 am every day). 
  • Release very regularly, shortening the feedback cycle.
  • Requires significant DataOps automation.
  • Very uncomfortable for experienced traditional data professionals.

I’d just like to say one thing about the one day or less sprint length. Are you still working in sprints at this point? In my mind you’re effectively working in a continuous delivery manner, albeit one in which you queue changes to release only once a day. This smells like a semantic issue to me, rather than a pragmatic one. 

 

Sprint Length Heuristics

I’d like to share the following words of wisdom about choosing the right sprint length for an agile data team:

  1. It depends. There isn’t an official, prescribed “best practice” for sprint length. Your real goal is to find a sprint length that works best for your time given your context, and that might even mean you abandon the idea of working in sprints all together.
  2. Shorter is better than longer. In general, shorter sprint lengths motivate you to find more effective WoW. They also shorten the feedback cycle, improving your ability to react to changes.
  3. Start with with a two-week sprint. When a team is relatively new to working in an agile manner my starting point for sprint length is two weeks. I’ll suggest this even when I suspect the team will struggle at first because I want to surface sprint-induced problems right away to see how people react to them. This can be frustrating for people, and risk low team morale at the start of an initiative, but it also forces people to focus on adopting new WoW and rethinking some of their previously held beliefs, and on finding ways to make it work out.

 

Factors that Affect Sprint Length

In order of impact, the following factors affect your team’s ability to shorten their sprint length:

  1. Mindset. Do you accept that working in short sprints is effective? Do you embrace change?  Do you strive to improve your WoW? Are you willing to work closely with others?
  2. Skillset. Do you have the agile data skills to work in short sprints?
  3. Domain complexity. The greater the complexity of the domain that you face, and in particular the greater the complexity and poor quality of the data that you’re working with, the more motivated you’ll be to work in longer sprints.
  4. Tooling infrastructure. Do you have the requisite tools and infrastructure to support DataOps and agile data WoW?

To summarize, the sprint length that is best for your agile data team depends on many considerations. Your goal should be to choose the right sprint length for your context, to strive to learn and improve over time, and to adjust accordingly as your situation evolves. There is no one right answer that works perfectly for every time, but you can discover what works best for your team right now.

 

Related Resources