Projects rely on knowledge. The more you know, the more successful your project is likely to be. Knowledge is a hard thing to define. What is useful knowledge? How do you get it? Why is it useful? Where does it live? How does my team access it? This post attempts to sketch out some answers to these questions.

A product with a purpose

Let’s start at the beginning: A project typically exists because of reasons given in a business case: unmet needs, an opportunity, whatever. This project must deliver certain outcomes, or benefits, in order for the return on investment to be realised.

So far so good, right?

To realise the benefits, however, there must be an output, or product, of the project. And this product needs to have something about it. A certain je ne sais quoi, if you will. It’s got to have some sort of clearly defined characteristics that will somehow enable the outcomes to be reached. It must, so to speak, have a purpose.

What does this purpose look like?

At Tigerspike, in the early stages of each project, we work together with our partners to define a ‘Change Statement’. This is a statement that summarises
1. 
what outcomes the product drives, 2. what it should do, 3. what makes it different and 4. how it should feel while achieving both those things.

For instance:

  1. The Sprocket App encourages sprocket users to buy more sprockets. 
  2. It does so by allowing sprocket users to search for the sprocket that’s right for them, save their sprockets for later and share their sprockets with their friends.
  3. Unlike other sprocket-search apps, it allows for instant access to any sprockets sprocket fans desire.
  4. It has a friendly, conversational, experience.

Put simply, the success of the project will be gauged by measuring the extent to which it delivers the expected benefits through the capabilities it possesses.

Dealing with the hypotheticals

It doesn’t end there, though.

In digital product development, we often think that we’ve got it all figured out from the start. After all, it’s obvious our users will want to use this feature, right? Wrong.

The features mentioned in the Change Statement are hypotheses. A hypothesis is a working assumption (or set of assumptions) that is being used for the purposes of making a claim—for instance, that the feature you think is so gosh darned cool is the right thing to do. But, unless you have empirical evidence to back up your claim, then it remains just that—a claim—no matter how high up the ladder you are.

What form can hypotheses take?

A hypothesis in a digital product can take many forms:

  • The ‘user need’ that you workshopped last week that everyone seemed so sure about. Did you involve any of your customers in the workshop? Did you do a survey? How big was your sample size?
  • That design you approved yesterday. Did you do any usability testing? Multivariate testing?
  • Those ideas in the backlog that marketing simply must have for the first release: why should the business spend serious money on an Agile team without some evidence?
  • And how about those outcomes? Where did they come from?

At Tigerspike, we utilise our data-driven approach called Catalyst which is designed to use hypotheses and turn them into valuable knowledge. We begin by clearly defining the hypothesis in terms of 1. our claim about some aspect of the real world, 2. our action to test the validity of that claim and 3. at data we expect to see.  We then use the learnings from the test to improve the chances of the next hypothesis we advance of moving us towards the needed outcomes.

For instance:

  1. We believe that prospective sprocket customers would more likely to convert into a sale with a custom recommendation.
  2. If we add an intelligent recommendations engine that uses machine learning to recommend sprockets to our customers.
  3. Then we will see our app sales figures increase by 15%.

What can I do with an hypothesis?

When we advance a hypothesis, we need to have a plan to validate it. Validating a hypothesis can be thought of as ‘performing an experiment’. After all, telling people you’re ‘running an experiment’ sounds better than validating an hypothesis, right?

What makes a good experiment, though? It’s surprisingly simple: your experiment needs to give you the right amount of confidence in your hypothesis to allow you to move forward.

It’s important to exercise care when designing your experiment. Different types can tell you different things. None of them will ever give you the full picture, but, used in the right way, an experiment can help you avoid wasting precious time and money.

In addition to choosing the right experiment for the task at hand, it’s just as important to think about when is the right time to do it. When uncertainty is high, at the start of a project, then cheap and quick experiments might make more sense as ideas and options are more plentiful. When things are more settled, more sophisticated experiments are probably justified.

On one hand, at the start of your project, when there is the most uncertainty, testing concept sketches for the important features in your app with a handful of users will give you some useful data and boost your confidence in what you’re doing is correct, but this approach is limited by the small sample size and solution fidelity.

On the other, implementing sophisticated analytics and pushing a feature to production will allow you to earn how your idea truly performs in the real world, but it would make little sense to do this right at the start of a project.

Back to the purpose

Remember that purpose we talked about?

The main outcome that your product is driving is in fact the all the changes described in each of the features in your backlog added together. This means that the outcome itself, to a greater or lesser degree, is hypothetical.

This figure describes the overall outcome of the project [Vector AZ] and how it is comprised of smaller vectors [Vector AB] or changes.

It’s therefore important to gain the right level of confidence in your objectives themselves so that your bullseye is in the right place. You’ll need to test them in much the same way as you would do with anything else in your product.

Moreover, each of the individual hypotheses that form your digital product is descended from the main purpose, but there is a bidirectional relationship between them: over time, each changes the other.

For instance, on one hand, a change in the world outside of the project will, undoubtedly require a new set of outcomes to be defined. In this case, you’ll need to check your backlog to see if all the little hypotheses need to be changed, moved or removed.

On the other hand, when your data indicates that lots of your small hypotheses should change, then perhaps all those changes taken together might make your outcomes different. Again, no harm in checking back on them!

The learning project

When we test an hypothesis by running an experiment, we generate data. Analytics, survey results, interview recordings…the types—and quantities!—of data can vary enormously.

On its own, data is not much use. It’s inert, inactive. But it’s when we apply it that data becomes something really valuable: knowledge.

The analysis of the data will tell you whether your hypothesis is valid or not. If it’s valid, then great. You have the knowledge that you can safely proceed to do the next piece of work that will move it along its journey to being a  real feature. If it’s invalid, then ask yourself why. Do you need to change the content of the hypothesis? Should you discard it and do something else instead?

But where does that knowledge live within a project team? Sure, you probably have a meticulously-kept wiki and highly-organised folder structure. However, the knowledge a team possesses is so much more than that.

Teams build collective knowledge as part of their day-to-day interactions. Vital portions of information about the project are stored in the brains of each of the teammates, ready to be accessed by others through conversation and dialogue. Teams incorporate newly-acquired knowledge—through experiments and other means—into the transactive memory of the team.

This process of creating, encoding, storing and retrieving collective memory, combined with the scientific method of developing and testing hypotheses is truly what makes a learning project.

For science!

You should treat developing your digital products like a scientist would: use the scientific method to reduce your uncertainty and develop your knowledge. But you also need to think like a psychologist: understand that your team is a ‘superorganism’ and possesses unique cognitive abilities, far greater than a single individual.

At Tigerspike, we combine evidence-based methods with a team-oriented product development approach to develop the best possible solutions. We’ve developed specialist tools and approaches that help us deal with uncertainty in a way that drives the outcomes vital to your business’ success.

So, what are you waiting for? Don your lab coat, get out there and get testing! For science!


Words: Richard Fort, Lead Business Analyst, Tigerspike London