The latest assignment I’ve got calls for a redesign of the Atlassian tutorial that introduces developers to our software developer kit (SDK) .  This is the first of a three-part series of blogs related to this project.  In this part, I discuss the project goals, the research behind the redesign, and some metrics for measuring the current and future success of the tutorial. Part 2 documents a process you can use for creating a URL funnel.  Part 3 provides some tips and tricks learned during the process.

Atlassian technical writers provide a series of blogs exploring the latest techniques in technical communication. These blogs are about our projects, experiments, and ideas. Through this series, we share the techniques we use to give our customers the sweetest documentation in the world.

The Tutorial and the Design Goal

Atlassian maintains the Atlassian Developers site (DAC for short) for software developers.  Developers may use the DAC documentation to learn how to create an add-on or customize Atlassian products for their employer. Or, DAC readers may be developing or customizing Atlassian products as their daily business. For example, Zen Foundation is a product you can buy developed by Stepstone Technologies for Confluence.

Regardless of which category a reader falls into, downloading the SDK and setting up a development environment is always the first step.  DAC teaches readers this process with a single tutorial made up of the following pages:

  1. DAC Welcome
  2. Installing the Atlassian Plugin SDK
  3. Create a Plugin Skeleton
  4. Start the Host Application with Your Plugin Installed
  5. Generate Project Files for Your IDE
  6. Write Some code
  7. Run your Changes

Of course, readers are free to skip around and leave the tutorial incomplete.   Users may enter the tutorial topics by searching as well.

Success with and perceived ease of use are important criteria for how likely a user is to adopt an information technology such as the Atlassian SDK. The goal is to redesign the tutorial so that the maximum number of readers complete the tutorial easily and successfully. If this goal is reached, then the documentation contributes to the SDK’s adoption rate.  A higher adoption rate could lead more third-party developers  to use the SDK to develop new features and add-ons for Atlassian products.

Usage and the User

The first phase in the redesign involved researching whether DAC readers were using the SDK tutorial and who those readers were. Also, I wanted to determine whether users were completing the tutorial successfully.

The good news is that visitors to the DAC site are interested in a tutorial and the topics it covers.  Three of the yearly top-ten pages on DAC are part of the tutorial and three others are highly correlated:

Top Ten Pages fo the Year

The top two organic search terms are for the SDK and two others in the top ten are highly correlated to learning about the development environment:

Top Search Terms for the Year

Reviewing the operating system of DAC readers, Windows is the clear leader:

DAC Reader Operating System

Given that DAC’s Eclipse IDE configuration page  is also in the top ten pages, it seems likely that the tutorial user is in a Windows environment and uses the Eclipse IDE.

Just how successful is the DAC tutorial design? To answer this question,  I searched Atlassian Answers to identify customers that were asking SDK questions, what kind of questions they were asking, and the level of response they got.  I also reviewed reader comments from the Feedback button and the Zendesk for support. There were about 10 tickets on both sites combined for readers reporting problems with Eclipse in combination with our SDK.

Anecdotally, the tutorial success didn’t appear to be high. However, the level of negative response was low in comparison to page views.  I needed a better, more conclusive means of measuring success before I redesigned the tutorial.  Doing some research in Google Analytics, it seemed that a funnel goal was perfect metric to determine a tutorial’s success.

The Goal Funnel Metric

A goal is a page on a site that serves as a conversion point.  The page may be a thank you page after a user submits a form or buys something.   In a tutorial’s case, the goal is for the reader to reach the last page in the tutorial — indicating the reader completed the tutorial.  The funnel is the path the reader takes on the way to the goal.  For the current tutorial, that path, as you may recall, is the following:

  1. DAC Welcome
  2. Installing the Atlassian Plugin SDK
  3. Create a Plugin Skeleton
  4. Start the Host Application with Your Plugin Installed
  5. Generate Project Files for Your IDE
  6. Write Some code
  7. Run your Changes

Readers can enter a funnel at any point in the path, and they can leave it.  A funnel’s conversion rate is:

Number of People Reach the Goal ÷ Number of People Entering Path

A Funnel Conversion Report illustrates how successful a site design is at “converting” readers.

Using the Funnel Conversion Report, I could measure how many users completed the tutorial. A high conversion rate would indicate a design was succeeding in guiding users through.  The conversion rate would not determine if users found the tutorial useful, for that I would continue to rely on user feedback.

Measuring the “Before” Tutorial

What constitutes a good conversion rate depends on the industry .   I’ve learned “good” could be 2%, 6%, and 20% from various sources — and none of the sources were looking at technical documentation.  Before I started, I needed some way of determining what was a good conversion rate for a successful tutorial.  Previously, I had written a tutorial for Atlassian bitbucket that received many positive comments from readers.  Though, just as with the negative feedback on the DAC tutorial, the number of positive comments was miniscule in comparison to the number of page views.

Still, many of the design techniques used in the bitbucket tutorial, I planned to use in the DAC tutorial’s redesign.   I decided to use the bitbucket tutorial as a control.  Further, I hypothesized that the limited feedback on both tutorials was predictive of the actual conversion rates.  If I was right, the conversion rate for bitbucket should be higher than the existing DAC tutorial.

To prove my lil’ hypothesis, I set up a goal funnel for each tutorial and measured for a week. After one week of measurement, the funnels reveal that my hypothesis is correct. The bitbucket tutorial has a significantly higher conversion rate:

Funnel Entered Made Goal Conv. Rate
bitbucket tutorial(PDF) 6891 1447 21.39%
DAC tutorial (PDF) 1791 144 8.04%

 

Based on these results, I decided that redesigning the DAC tutorial to use the bitbucket tutorial methods was a good idea. Further, I’ve hypothesized that after the redesign, the conversion rate on the DAC tutorial will improve.

To be continued…

In part 2 of this blog series, I’ll document the process I used for creating the tutorial URL Funnels.

What is Your Tutorial’s Funnel Conversion Rate? (Part 1 of 3)