• How to avoid opinion-based product prioritization
  • Original article by Tamzin Taylor
  • Translation from: The Gold Project
  • This article is permalink: github.com/xitu/gold-m…
  • Translator: Yuze
  • Proofreader: Yuhanlolo, GeniusQ1981

We can actually use the data to make better decisions

Product decisions are hard to make. In most companies, there are sometimes many other factors to consider when making a decision. Sometimes it comes from a competitor within the organization, a team leader or a boss, or even a person who wants to try something new. Sometimes the biggest obstacle to making a rational decision is your CEO or CFO, because their ideas are often hard to say no to.

Since this is a common question, how do successful app developers deal with the product decisions that come out of their head?

I interviewed about 20 top developers, including engineers, product managers, and growth directors from Google. From these brief interviews, they all showed three important traits:

  • They experiment to keep broadening their options.
  • Data is the most important part of the process.
  • They spread the value of data and the importance of sharing information in the company culture.

In this article, I’ll use two examples to show how top developers have solved these problems. In the first example, I was honored to have the opportunity to take the product-first approach that is often used internally at Google and apply it to the product 1Tap. In the second example, I was fortunate enough to be able to internally track the decision process used in the development of Memrise, the best app of 2017, to get their team’s perspective on the methodology.

Tap and Polaris methods

The Polaris method originated in Silicon Valley and has a relatively long history. Within Google, the Youtube and Gmail teams often use this approach to help them decide which features should be developed first. Based on my interviews with them, I found that this methodology was effective for the entire team. We have a dedicated team of professionals at Google, so I pushed them to work with the 1tap team to implement the methodology.

After all this talk, what is the methodology? Conceptually it is very simple, with only four parts: a pointing system, a diagram of user flow, a growth model, and a simple spreadsheet. After all, all data models need a spreadsheet as a carrier.

So let’s see how these 4 things work together to help 1tap solve the problem.

Set up a north pointing system

For products, this system is similar to the concept of KPIs. It measures the product’s ability to acquire users and keep them engaged, convert, and retain them. For example, if we are making a hotel booking app, this metric would be the number of user bookings. If we’re making an instant messaging app, that metric would be the number of messages sent.

1TAP’s vision is to make self-employment easier for a team. They have two apps on the Google Play Store: 1Tap Receipts, which automatically provides automatic data extraction and bookkeeping, and 1Tap Tax, which automatically calculates taxes through receipts and bills.

This time, we practice this methodology with 1tap Receipts. We quickly decided that the metric would be volume related. However, 1Tap’s chief growth officer tells us: “First, we need to figure out whether the goal of the product is to encourage individual users to upload multiple receipts, or to attract more users who upload only a few receipts.” If we take a closer look at the value 1tap provides to users, it’s clear that the more receipts an individual user uploads, the better 1Tap helps them manage their personal taxes. From this perspective, the core value measured by the Polaris system is about individual users, and the more receipts each user uploads, the better.

Define your user flow

There are two main steps to achieving this goal. First, make sure your app collects feedback from users. Second, review your metrics and determine how well your product is performing as your users go through each step of the process. The process is as simple as:

  • Define the critical events in your application.
  • Plot how the flow between events works.
  • Use statistics to see what percentage of users stay in each stream.

You can see that the graph has three main sections: your acquisition of new users, your return rate of old users, and the flow of critical events across the product.

New user acquisition is all about how users download your app. As long as you make sure you correctly set the UTM tag under all the links you place and associate it with your AdWords account, you’ll be able to easily see your new user acquisition path from the Google Play console.

But the user flow for each product is case-by-case. But it usually includes logging in, going live, and conversion rates (if it’s a paid product). Most of the time, there’s a magic moment for an app that you pay special attention to, which is the critical event stream. This stuff often represents the metric, the participation and conversion metrics, so it’s important to include it in the metric.

Finally, there’s the repeated flow, which is what leads the user back to the app.

When the flow chart is complete, the entire team will be able to talk about the product in the same way and understand how the interaction between the user and the product – from user acquisition, participation, transformation to return – works. This is the impact of the Polaris indicator.

In the case of 1tap, the graph looks like this:

You’ll notice that they don’t have any user flow analysis. The most important thing for 1Tap and the team is to recognize the moment when the user is amazed. This is the user activation phase. “The value of the product is that the user can see the automatically extracted information immediately, and a lot of users are amazed at how accurate our OCR technology is,” Jon says.

The final user flow diagram was produced after 10 iterations. Jon thinks the first version of this image is a complete mess. Too much detail makes the whole picture look confusing. Then the team works together to simplify the picture and focus on what’s important. Even better, organizing this graph also allows 1Tap’s app to remove a lot of redundant functionality. “If it wasn’t the point, we wouldn’t have put it in the app,” Jon says.

After collating the diagram, the 1tap team found that they had a breakpoint in the user flow. The problem is mainly when the user is exporting the data. Right now, they’re still in the experimental phase, but keeping these reports has helped them improve retention.

Build a growth model

The next step is to build the growth model. We will need the indicators we have previously found in the Polaris method to determine how we should play to our strengths and compensate for our deficiencies.

As in building user flow diagrams, the first version of the 1TAP growth model is very complex, about 16 pages long. However, by focusing on what is important, a directly feasible growth model emerges. The growth model is summarized as follows:

The key here is the receipt each user adds, which is the Polaris indicator. You can break it down into receipts submitted by monthly active users (MAUs). In turn, it can continue to be broken down into each user who adds receipts in different ways — scanning, emailing, manually typing — as well as new users and repeat customers.

When they first started to write 1Tap, they put a lot of thought into data analysis, analyzing almost everything users did on the app, and this is a point of pride for their team. However, Jon says his team often doesn’t know where to start to understand users because of this. Because there are just too many dimensions to look at. But this growth model helps them do that very well by finding the Polaris indicator.

But as Jon said, the point of this growth model is to help explain to the CEO where monthly users are coming from, and to help the CFO think about where the profits are coming from. In addition, products can use the system to make better decisions.

Create a spreadsheet

The final step in the process is to turn the model into a spreadsheet and help us evaluate our opportunities to see how they can help our product grow.

You may have already noticed that the growth model of 1tap turns into a giant spreadsheet. But in this case, the size of the table is important, and here is a part of what Jon and his team call a “calculator.”

Armed with the table, 1Tap began exploring the impact of various activities on the average number of receipts a user received over a dozen days. Jon found that the benefit of doing this was that they could see the benefit of small changes to the team. For example, when the conversion rate from download to registration increased by 2%, the total number of receipts doubled. In contrast, the increase from registration to activation to transformation has only 75% of the total impact.

The 1Tap team then decided to focus more on getting users to discover the “wow” moments earlier in the process (the sign-up phase), rather than simply paying to attract new users.

So what does this process and model bring to 1TAP?

First of all, it made their team understand what they were doing and what was important. CEO Nick no longer has to ask every day where the monthly paycheck comes from and how we can increase it. It’s all written out in the model. In addition, the product is more specific about what is needed later and how to prioritize. Now all decisions are made by this calculator. The CFO knows that by increasing revenue we can also increase retention. Basically everyone has a sense of what the team is going to do next.

Another important impact is understanding how to add new users. In the past they would not have had such an opportunity. Now they understand that by working with third parties to do some sort of sign-on, they can dramatically improve user acquisition and retention.

Finally, Jon says the model has helped his team know who to hire and when to hire. In the past, they often hired the wrong people at the wrong time and the team grew slowly.

Memrise

Memrise, a language-learning app, has 35 million users and teaches them in more than 200 languages. Kristina Narusk, their chief growth officer, thinks the company’s growth path has been very fortunate. They had Ed Cooke as CEO who had both the ability to innovate and the ability to run a good team. Ed often has a variety of ideas, but at the same time, it is very challenging to identify which of these ideas will actually lead to growth in the product.

For example, Kristina told me that one day three years ago, she received a text message from Ed. He bought a double-decker bus in the UK, decided to travel around Europe and record the way native speakers of different languages in different countries say certain things. This is a very rare thing to do, so our team has had to come up with a variety of solutions to deal with this bizarre idea and to combine it with the growth of the app.

Memrise came up with a six-dimensional validation to filter these fancy ideas:

  1. It must be iterable. We were able to start with a simple MVP and iterate until we perfected it.
  2. Have an immediate effect. New ideas need to make intuitive changes when they come to fruition.
  3. It has a long term effect. This idea needs to help our users in the long run, not just in the short run.
  4. It has to be quantifiable. We need to be able to quantify the impact of the product on the user.
  5. It can be localized to cover as many application markets as possible.
  6. Related to the product, and understandable.

When exploring new features for the future, the team uses this formula to determine what to develop and what not to.

As Kristina pointed out, we need to instill this creative and experimental way of thinking into the team and the company. Memrise divides the product development cycle into four phases: exploration, definition, development, and follow-up.

During the exploration phase, the product team draws the PRD, and users test the new idea. Experimentation involves innovations such as design, user experience, and content direction. For example, the team might have lots of passersby to test, or some online test or some long-term users to test new features. The entire team, including designers, linguists, and developers. However, Kristina found it difficult to keep developers calm when testing the product. They’re always upset that users don’t open new features properly.

In the case of Memrise Membus (the one driving around Europe above), during the exploratory phase, the bus drove first to Oxford to take some videos to test the idea. Instead of shooting all over Europe. Filming English videos in Oxford was the prototype for the idea. And it turns out that the benefit of that video idea to the whole product is very clear and very low cost. In the end, we settled on this plan.

When the idea is explored, we come to the stage of definition. At this stage, we will figure out what the new idea will do and how it will interact with users. At the same time, the details of the test will be clarified, such as:

  • Test platform
  • The target user
  • The target language
  • Test cycle length
  • Something to focus on
  • Analyze what kanban should include

The data team is also involved. They help ensure that the feature is built to capture the right data points so that the product team can assess the impact of the feature. These videos are used to create advanced learning modes and are available as part of the Professional subscription. Therefore, it is important to understand the impact of adding this pattern to the English course on the conversion rate of users in the professional version. At the end of the phase, the entire team acts as a user, providing feedback on the product idea and projected results.

When everything is ready, we will enter this stage of development. Because the development team is early and involved in the exploration and definition phase, the development is straightforward and everyone knows why the features are being developed.

When development is complete, it is up to the product manager to decide whether to officially enable the feature so that users can discover what has been updated and start testing the new features.

When a new version of the app is released, we get to the follow-through stage. At this stage, the team tends to answer questions like what about KPIs? What was the initial feedback from users? How will the new features affect your app’s rating? Ideally, as Kristina says, the moment is enjoyable because we can see how the results are produced.

YouTube video link: https://youtu.be/e2RPXKi4e90

At Memrise, they have a set of standards and charts for each experiment. “The day after launch, you’ll see that we’re refreshing the page all the time,” Kristina said. “See how the feature and new ideas perform. For our team, it’s exciting to see the results and let us know if what we’re doing is successful and people love it, or if we need to go back to the whiteboard and rethink.” Memrise eventually decided to add video and learning and native modes to all languages.

conclusion

As I said at the beginning of this article, making decisions is difficult. Making decisions away from the pat on the head may be even harder. However, by sharing these two different ways of making decisions, I hope I can help you get a new feel for making decisions about products. No matter what the process is, make sure the team is involved in the process, using the data to find the right changes, or the right activities for users to get involved.


What do you think?

If you have your own thoughts on decision-making and priorities, feel free to comment below, or tweet us, and follow us as we share the latest information.

If you find any errors in the translation or other areas that need improvement, you are welcome to revise and PR the translation in the Gold Translation program, and you can also get corresponding bonus points. The permanent link to this article at the beginning of this article is the MarkDown link to this article on GitHub.


Diggings translation project is a community for translating quality Internet technical articles from diggings English sharing articles. The content covers the fields of Android, iOS, front end, back end, blockchain, products, design, artificial intelligence and so on. For more high-quality translations, please keep paying attention to The Translation Project, official weibo and zhihu column.