Content Usability testing

Plain text beats graphic content: A/B testing

A/B testing can help to pinpoint the type of content that staff are most attracted to. I ran an experiment to find out which advert was most effective in an internal campaign.

The problem

The intranet staff directory gets a lot of complaints.”It’s never up to date,” being the main one. Last year, we ran a campaign to encourage staff to update their records. Incidentally, our intranet staff directory is not linked to any central human resources databases or Outlook contacts. We are stuck with a system designed outside of the central intranet team and the best we can do is to try to keep the information current.

On occasion, we have previously used a standard graphic advert (the yellow original) to encourage staff to check their directory details. It wasn’t a great advert and it wasn’t very effective. For this latest campaign, I decided to test the original advert against other variations to see if they sparked any interest from staff. I designed a series of adverts with different themes, some static, some animated and one plain text advert. I ran the test over a period of weeks to find out which was the most effective at encouraging staff to take actions to update their details.

Setting up the test

I used Google Website Optimiser to handle the A/B testing. I ran the adverts on the page where staff would normally go to search for phone numbers and configured Website Optimiser to serve the various adverts, using the original yellow graphic advert as the control. I defined a successful conversion as a member of staff going to the “How to update your record” page and clicking on the option to update their details.

Testing process and results

Website Optimiser works using cookies to serve one of the predefined adverts to a visitor. It will always display the same advert to the same visitor. It then measures how successful each advert is depending on which actions the visitor takes.

After a few weeks the numbers in Website Optimiser should start to solidify and you begin to see patterns in the results. The coloured lines show the conversion rates for each advert. This is a great visual tool for getting buy-in from stakeholders; the infographics clearly show the outperforming tests.

In this test, the plain text advert was getting the best conversions and showing up as the likely winner. The last stage in the testing process is to do a follow up experiment using just the original control advert and the proposed likely winner.

Peoplefinder AB test followup

It’s very satisfying to see the green light that signifies the end of test and a definite winner. Up against all the fancy graphics and animations, the plain text advert had a 98.9% chance of beating the original.

The experiment showed that the text advert was the best approach to take for this problem, for our staff on our intranet. With user testing, the results are often unexpected, reinforcing why we must always design for the user.

See also:


Shelf-stackers in the intranet supermarket

The job of the shelf-stacker is to fill the intranet shelves with junk. It’s a fairly easy job. They don’t have to think too much. It’s not difficult to shove something on a shelf. They might stack it next to something similar, but it’s not so important to them where it goes. Some of the shelf-stackers don’t stack items with the labels facing the right way. As long as it’s on a shelf.

They’re not bothered whether a customer takes an item off a shelf, has a little look, or not. They don’t mind if something has been sitting on a shelf for months and is out of date. That’s not their job. Sales figures don’t interest them. Top sellers? Supply and demand? Nah. In fact, most of the time, they don’t even bother looking at what it is that they put up on the shelves. Someone gives them something to stack, and off they go. Job done. Box ticked.

Shelf-stackers continue to fill the shelves. Some aisles become so cluttered with piles of junk that customers often complain that they can’t find what they want on the shelves. And when they give up looking and ask the shelf-stacker to help find something, it is a fruitless search. Some of the keener shelf-stackers may realise that customers aren’t visiting certain aisles. So they hang flashing signs above the aisle in the hope of enticing the customers. The customers learned long ago to avoid these aisles.

In the intranet supermarket, we need more than just shelf-stackers. We need intelligent people who care about their customers. Who realise that what they are stacking is ultimately for consumption by customers. Who understand that simple stacking alone does not tick the box. We need them to have an interest in the shelf life of their products, keen to find out if customers can find products easily. We need people with the good business sense to only stock what the customers want, eager and ready to tidy up messy aisles and to be more questioning about stacking whatever they are given, wherever they fancy.


Plain English for corporate intranet content

At work, I teach a course on writing for the intranet. It takes an afternoon and covers online writing and editing techniques, SEO, how to handle graphics and accessibility. I also touch on writing in plain English. For this part of the course I usually use the latest IT announcement as a demonstration. IT announcements highlight how not to write plain English. Right on the button. Every time. Guaranteed.

While useful for my course, it always annoys me that people from this department (who now call themselves ICT, the C for communication!) often fail to think about the people they are writing for. Their audience is the whole organisation, yet they write from their own point of view, using acronyms, technical jargon and internal project and process language.

So here’s our latest announcement. I’ve colour-coded it to help with my analysis:

Important Windows XP and Office Security Patches will be available to all workstations on your IT network from 4th October. This is part of an ongoing process to ensure all IT equipment is secure and to minimize the risk from computer viruses.

 What do you need to do?

You are advised to manually install these updates onto your desktop PC as soon as they are available. This is a simple process that will only take a few minutes of your time. For instructions on how to manually apply these security updates please click here.

If you are unable to apply these patches for whatever reason, they will be automatically applied to your workstation overnight 9th October, 20:00. However, we do advise you to manually apply these updates where possible.

Switching off your workstations

To ensure the security updates are correctly applied, you must restart your desktop PC after installation. This can be done when you shut down your PC at the end of the working day. Users are reminded that it is recommended that they ‘Shut Down’ their PC at the end of every working day – see below.

To shut down your PC, click on the ‘Start’ button and choose the ‘Shut Down’ option. Please remember to switch off your monitor as it is not switched off automatically by ‘Shut Down’. Some monitors have a very subtle ‘on/off’ switch that is flat and touch sensitive and located on the underside of the bottom right corner of the monitor; the blue light will be steady or flash until the monitor is switched-off.

Remote Workers

It is recommended that remote workers are onsite and logged in to their workstations when the Security Patchesare being applied. If this is not possible then the patches will be available remotely, however the update may take a little longer.


It is clear that the writer has no audience in mind. The piece switches from talking to me directly, to speaking about users, to some invisible person who isn’t specified because of using passive voice (“this can be done…”)

The message is inconsistent, taking ages to tell me how to shut down my PC but making me follow a link for the actual instructions to download the update, which is the point of the announcement. There’s a line that reads *see below*, below meaning the next sentence? And then there’s just plain daft stuff like recommending that I’m logged on to my workstation to download and install the udpate. No, really?

The message is mixed. Along the lines of “we have an update, you can download it yourself, or it will install automatically if you don’t, but, you know, you can do it yourself if you want.” Why not just install the update on my machine automatically and not bother me with having to read through this tripe? Is it that important that I know that this technical maintenance is going on?

The colour-coded headings below refer to the coloured sections in the original announcement.

Passive voice

I can spot passive voice a mile off. Years of editing online content tends to drum it into you. And there is a very good reason that IT people use it. To appear distant, to avoid the issue, to misdirect attention. Better for speech-writers and spin doctors. On the intranet, we want to get the point across quickly and clearly. Passive voice clouds the issue causing the brain to fire internal questions trying to fill in the gaps due to missing information in the text.


When We Capitalise The First Letter In Each Word It Makes It Really Hard To Read.

While not so bad in this IT announcement, you’ll often find capitalised project names, department names, process names and technical terms. In this announcement, what is so important about security patches or remote workers that they deserve capitalisation?

Bad names for links

Seeing a *click here* makes my blood boil. It is wrong for so many reasons. Staff using screenreaders will often request the software to group and read out all links on the page. Hearing click here, click here, more info, find out more, is not helpful. Similarly when the search engine reads the page and follows a *click here* link it does not help. It just registers a great page in the search index called “click here” that everyone is linking to. Even for staff who can see the page, it’s not clear from the link text what you will get if you click it. Link text should always accurately describe the target destination.


Important corporate announcements should just give me the facts. Not self-promotional fluff. I don’t need to know that what they are asking me to do is part of their ongoing process. And apologies to my non-English-writing readers, but there is a Z where there shouldn’t be.

Content rewrite

Instructions for how to download and install an update, including how to shut down your PC are already available elsewhere on the intranet. Assuming that the IT department can’t automate the updates, here’s how I would rewrite the announcement:

Important Windows XP and Office updates for your workstation will be available from 4th October
What do you need to do?
You can install the updates yourself. Read instructions on how to install the updates.

If you can’t do this yourself, we’ll update your workstation automatically at 20:00 on 9th October.

Remote workers
You’ll find it faster to install the update if you login while on site. You can still update remotely but it will take longer.


I know what I’m talking about here isn’t so important in the greater scheme of the workings of the organisation. But by taking just a few minutes to stop and think about what we are writing, we can help staff to save time in reading and understanding the finished piece. My rewritten example is shorter and clearer. The simple instructions makes it easier (and more likely) for staff to carry out the task. I have cut down the original announcement from 302 words to 78 words. With online corporate content, less is more.

CMS Content Redesign

Intranet redesign, Phase 5: migration, content freeze, dual publishing

The end is in sight

Remember that migration plan from phase 2? Having decided to publish back-issues of news stories to the start of 2009, I could quantify how much content we would need to migrate. I managed to cut the original content from 6416 entries down to 3000 entries. Some of this was by losing older news stories. The remainder I did by reorganising content into simplified chunks. Where a section had dozens of one-paragraph pages, I grouped and combined them into fewer pages. I removed out of date content. Cut out duplicated content. Barred the *what’s this doing on the intranet?* content.

So it was at this point in the project, around September 2009, that we agreed on an unofficial launch date of 1st January 2010. We reckoned we could populate the new intranet in that time. We would divert all our attention to the migration and bat off any new development work.

Migration checklist

We pulled a few extra resources from the web team to help out with migration. We had hands-on training for the CMS publishing and strict rules on page titles, metadata and file names.

I shared the master migration plan with the team through Google Docs which worked wonders for collaborating on our progress, showing up to the minute changes and no document check-out and check-in. Meh!

Content freeze

We announced a moratorium on intranet updates with the exception of corporate news and features and urgent ministerial and board messages.

Curious to note that having announced a content freeze, intranet content owners who had not done anything with their content for months and months suddenly wanted to update their pages. Don’t you love human nature?

Dual publishing

Because we had agreed to continue to publish news stories, I divided the migration work amongst the team with one person responsible for publishing ongoing news onto the existing intranet and also entering into the CMS for the new intranet. We had a few people working on news story back-issues. And a few working from the existing Dreamweaver content, manually building the green, blue and purple sections of the site.

Migration production line

Cut and paste. Cut and paste. Robot-like for weeks. The migration team laboured on. The traffic lights on our Google Docs spreadsheet slowly changed from red to green. We took this chance to do a little editorial work on pages as we moved them across, rewriting in plain English and reformatting pages to improve readability. I was also keeping an eye on our Google Search Appliance as it indexed the migrated content. Up and up went the page count. Perfect page titles all round!

A URL was leaked at some point and from studying Google Analytics I could see that there were about 60 people around the organisation looking at the new site before it was officially launched. I’m glad they were that interested!

Communications and launch

In my next post I’ll cover how we worked with our internal communications colleagues to prepare staff for the new intranet and how we have evaluated and evolved the intranet since launch.
In this series

Content Information architecture Redesign

Intranet redesign, Phase 2: information architecture and content audit

Back in the good old days when the IT department allowed me to use DOS (the operating system that I was brought up on,) I managed to do some jiggery-pokery and get a complete intranet file-listing. And a few deft keystrokes later, I was spreadsheet-a-go-go.


Content audit

My previous analysis already told me what was popular content on the intranet. The spreadsheet gave me the complete picture and what kind of state it was in, structurally.

Using our Google Search Appliance, I also generated a listing of all content that Google thought was on the intranet (having crawled from the homepage.) The resultant list contained 6416 entries (HTML, DOC, XLS, PDF) and was a fraction of the DOS listing, meaning that the webserver was filled with rubbish, albeit invisible to staff.

I then spent a long time sifting and sorting, grouping and batching, slicing and hacking. By the end, I had a good idea of what I was dealing with and what would make it into the final mix.

Top-level navigation

I decided that I could group the whole of the intranet into just 4 main sections. Big change. Big simplification. What I didn’t know for certain was what I was going to call these sections. So, with the core team, we came up with a good handful of ideas for the names of each of the 4 sections.

I started involving staff with an online, closed cardsort using all of our ideas for names. I invited staff from all over the organisation to take part. I tested a representative selection of all intranet content. The tests worked by showing the participant an intranet page or document and asking them which of 4 boxes they would put it in. Each question tested one section and the 4 boxes were labelled with a selection of names for that section.

I wanted to test a lot of content, against a lot of possible names with a lot of people, online. I had to find a way to distribute many smaller tests, testing different variations of sections with content while keeping the number of questions to a minimum, to encourage staff to complete the test. I was also working with a webserver with no coding capability (PHP or ASP). So I developed a simple set of HTML pages, which linked from question to question while passing the chosen answers to Google Analytics. I’d be able to filter the analytics for the test questions and answers.

For the intial test page which all participants landed on, I used Google Website Optimiser, setting up an A/B test which redirected participants to one of 12 different pages. Each of the 12 pages started an individual test containing 20 questions. The purpose of Website Optimiser is usually to find the best combination of elements or wording on a page to drive a particular outcome. In this instance, I just took advantage of Website Optimiser’s method of evenly distributing tests.

We advertised for participants across our family of intranets and got a really great response.

Cardsorting stats

Google Analytics report showing bursts of cardsorting activity during 2009
Google Analytics report showing bursts of cardsorting activity during 2009

I like looking back on these analytics. During the online cardsorting sessions I covered over 1000 content items and got nearly 36,000 individual responses. The first batch of tests was in February, designed to elicit our main navigation labels. Subsequent tests then checked that the chosen names would suit all the intranet content. It was a much larger set of tests, again using Website Optimiser for distribution and Google Analytics to capture results.

The Test Tube advert
The Test Tube advert

The “Test tube advert”

I attribute the success of the number of responses partially to the advert that I designed to entice staff to take the test. The advert lets people know what to expect: answer 20 questions. The text acknowledges that people would be supporting us in taking the test and also mentions that this is the second round of tests. There was no first round of tests; the line is there to give a sense of having missed out on something. I also chose *experiments* which I felt was slightly more sexy than *tests*. A bold, clear call to action encourages a click, and the test tubes are just dripping.

Cardsorting results

The stats from the cardsorting gave a clear visual indication of success or failure. Here is an example of results for a single card with the option of 4 placements (boxes). This card got 334 responses with nearly 80% concurrence of opinion. I tested iteratively until I was confident that everything fitted and would work well with staff.

Google Analytics results for a single card
Google Analytics results for a single card

Remaining information architecture

After the cardsorting exercises were finished I then got to work on creating the master IA for the intranet, creating the secondary and subsequent navigation levels. I could then map the initial intranet structure to the newly created structure, producing the content migration plan, ready for phase 5 of the project.

IA is such an important step in any web or intranet development project. And it is important that it is done early in the project, before any page layout or visual design.

In this series

  1. Research, surveys and brief
  2. Information architecture and content audit
  3. Wireframe designs and user testing
  4. Visual design, HTML and CMS build
  5. Migration, content freeze/dual publishing
  6. Communications, launch and evaluation