Plain text beats graphic content: A/B testing

A/B testing can help to pinpoint the type of content that staff are most attracted to. I ran an experiment to find out which advert was most effective in an internal campaign.

The problem

The intranet staff directory gets a lot of complaints.”It’s never up to date,” being the main one. Last year, we ran a campaign to encourage staff to update their records. Incidentally, our intranet staff directory is not linked to any central human resources databases or Outlook contacts. We are stuck with a system designed outside of the central intranet team and the best we can do is to try to keep the information current.

On occasion, we have previously used a standard graphic advert (the yellow original) to encourage staff to check their directory details. It wasn’t a great advert and it wasn’t very effective. For this latest campaign, I decided to test the original advert against other variations to see if they sparked any interest from staff. I designed a series of adverts with different themes, some static, some animated and one plain text advert. I ran the test over a period of weeks to find out which was the most effective at encouraging staff to take actions to update their details.

Setting up the test

I used Google Website Optimiser to handle the A/B testing. I ran the adverts on the page where staff would normally go to search for phone numbers and configured Website Optimiser to serve the various adverts, using the original yellow graphic advert as the control. I defined a successful conversion as a member of staff going to the “How to update your record” page and clicking on the option to update their details.

Testing process and results

Website Optimiser works using cookies to serve one of the predefined adverts to a visitor. It will always display the same advert to the same visitor. It then measures how successful each advert is depending on which actions the visitor takes.

After a few weeks the numbers in Website Optimiser should start to solidify and you begin to see patterns in the results. The coloured lines show the conversion rates for each advert. This is a great visual tool for getting buy-in from stakeholders; the infographics clearly show the outperforming tests.

In this test, the plain text advert was getting the best conversions and showing up as the likely winner. The last stage in the testing process is to do a follow up experiment using just the original control advert and the proposed likely winner.

Peoplefinder AB test followup

It’s very satisfying to see the green light that signifies the end of test and a definite winner. Up against all the fancy graphics and animations, the plain text advert had a 98.9% chance of beating the original.

The experiment showed that the text advert was the best approach to take for this problem, for our staff on our intranet. With user testing, the results are often unexpected, reinforcing why we must always design for the user.

See also:

Release the test tubes; time for more user experiments

As part of our efforts to save money across all parts of our organisation, we are combining many of our human resources, finance and procurement services into one central place.

There are several intranets within our organisation. Currently, most staff come to the HQ intranet for their human resources content and services. But there are some parts of the organisation that do their own thing. The move towards a shared platform will simplify processes for staff as well as cut costs and improve efficiency in back-end processing and automation.

The move will be a staggered project, slowly introducing new ways of working with the ultimate goal of providing an online self-service platform for staff. We will kick off with human resources services such as updating bank details and sick absence records, and then bring in other services. We’ll move from paper-based and electronic, downloadable forms to online processing.

While the new applications are being designed and developed outside of the Digital Team (usually by technical developers with no user experience processes), and being housed outside of the core intranet as part of the greater digital workplace, I have to make sure that staff will be able to access the pages and find the services easily from the intranet. Much of the content and guidance already exists on the intranet but there will be some new guidance for specific front-line staff. So we’re giving it a little reshuffle and a bit of a rewrite.

Fortunately, I still have the test scripts from the information architecture phase of the intranet redesign project, last year. I just need to modify and add a selection of tasks and I’m ready to run the tests again.

Intranet redesign, Phase 3: wireframe designs and user testing

The previous phase of the project gave me 4 main intranet sections with specific content functions and a detailed map of all the content that was cherry-picked to make the migration to the new intranet. There were going to be a fair few chunks of content left ashore, and no doubt a few stowaways.

The top-level, homepage, main navigation options, as voted for by the willing and much appreciated members of staff who took part in the online experiments were:

News & features

Containing 2 types of news; corporate *must read* announcements and feature articles, promoted as the online staff magazine. In addition, this section would be home to the corporate calendar, press and media coverage, the monthly staff magazine, and our holistic section (sports, social, weather, travel, photo galleries).

Guidance & support

Look-up, reference and guidance material, policy and procedures, forms and templates. All the stuff you need in your day to day job.

Ministers & parliament

Content relating to official ministerial business. (Cardsorting put any guidance of this nature into this section, rather than the general guidance and support section.)

Organisation & vision

Content about our structure, goals, ways of working, strategies and initiatives. Promoting internal campaigns.

Contextual navigation

For this next phase, I ran some time trials, using 2 sets of wireframe prototypes (created in Axure) to test the existing navigation against the proposed contextual method. I created a set of tasks for face-to-face user testing, designed to bounce the participant around different sections of the site. Contextual navigation came out way ahead, with task completion times significantly faster than the existing method. Decision made.

I was then able to document a complete intranet navigation structure based on the 4 main sections, plus a supporting section for global pages (A-Z, search, help.)

Wireframe and prototype user testing

The next step in the project was to work out how best to present all this information on the intranet, how the navigation menus would work and to start to implement some of the design ideas from the project brief.

I spent a month making heavy use of a trial version of Axure (are you getting how many cost-effective tools were used in this project??) to create protoypes using the full intranet IA, implemented with the proposed contextual method of navigation.

Having spent some time setting this up in Axure, the product really came into its own when I started the iterative testing. I’d do short bursts of tests with just a handful of people, individually at their desks. Glaring problems immediately came to light and with a few adjustments in Axure, a press of a button gave me a new prototype in seconds. Then more testing and adjustments, round and round, until I was confident in the navigation and layout, and just before the trial license ran out 😉

Early wireframe prototype page (before final cardsorting results.) Note the drippy *Corporate services* label which crashed and burned in cardsorting as a possible name.
Early wireframe prototype page (before final cardsorting results.)
Note the drippy *Corporate services* label which
crashed and burned in cardsorting as a possible name.

Benefits of contextual navigation

One of the most striking effects of using contextual navigation was that it made the intranet seem smaller. Because left hand navigation menus were shorter on every page. Because they didn’t include all the unnecessary baggage of same-level branch options. And I think this made the intranet seem less daunting to people during testing. Wherever they were in the structure, they didn’t feel lost. Golden usability rule; take away choice and you make it easier (therefore faster) for the user.

Bye bye breadcrumb

I also got the idea that the left hand menu could double-up as a breadcrumb, always leading back to the homepage and always showing you the path you have taken. An added benefit (later implemented with colour coding) was that the menu acted as a depth gauge. The less darker menu colours, the nearer you were to the homepage, the more darker colours, the deeper you were in the structure. Removing the traditional breadcrumb that spans across the top of body content saved some screen space and would also speed up migration time since developers would not have to manually code breadcrumb navigation.

Flexible menus

I was very optimistic about all the benefits of switching to this type of navigation It is flexible. It is not rigid. But I found this the hardest part to explain to the developers who were stuck in a rigid mindset of using up to 3 levels of navigation levels and no more. The intranet is not an evenly distributed field of content that falls neatly into ordered layers. Some sections go deep. Others are shallow. But, most are a combination of both. Sections consist of sub-layers of varying depth. So it is illogical to try to force the structure. To try to make it fit. Or to limit the structure to a finite number of levels.

Examples from two pages showing flexible, contextual menus made from the following building blocks: I can go back to this page/section (dark shade, one or more of these) I am in this section (medium shade, just one of these) I can go forward to this page/section (light shade, one or more of these) A bullet point marker to show *I am here*
Examples from two pages showing flexible, contextual menus
made from the following building blocks:
I can go back to this page/section (dark shade, one or more of these)
I am in this section (medium shade, just one of these)
I can go forward to this page/section (light shade, one or more of these)
A bullet point marker to show *I am here*

Information and navigation design completed, I then moved on to visual design, which I’ll cover in my next post.

In this series

  1. Research, surveys and brief
  2. Information architecture and content audit
  3. Wireframe designs and user testing
  4. Visual design, HTML and CMS build
  5. Migration, content freeze/dual publishing
  6. Communications, launch and evaluation