Plain text beats graphic content: A/B testing

A/B testing can help to pinpoint the type of content that staff are most attracted to. I ran an experiment to find out which advert was most effective in an internal campaign.

The problem

The intranet staff directory gets a lot of complaints.”It’s never up to date,” being the main one. Last year, we ran a campaign to encourage staff to update their records. Incidentally, our intranet staff directory is not linked to any central human resources databases or Outlook contacts. We are stuck with a system designed outside of the central intranet team and the best we can do is to try to keep the information current.

On occasion, we have previously used a standard graphic advert (the yellow original) to encourage staff to check their directory details. It wasn’t a great advert and it wasn’t very effective. For this latest campaign, I decided to test the original advert against other variations to see if they sparked any interest from staff. I designed a series of adverts with different themes, some static, some animated and one plain text advert. I ran the test over a period of weeks to find out which was the most effective at encouraging staff to take actions to update their details.

Setting up the test

I used Google Website Optimiser to handle the A/B testing. I ran the adverts on the page where staff would normally go to search for phone numbers and configured Website Optimiser to serve the various adverts, using the original yellow graphic advert as the control. I defined a successful conversion as a member of staff going to the “How to update your record” page and clicking on the option to update their details.

Testing process and results

Website Optimiser works using cookies to serve one of the predefined adverts to a visitor. It will always display the same advert to the same visitor. It then measures how successful each advert is depending on which actions the visitor takes.

After a few weeks the numbers in Website Optimiser should start to solidify and you begin to see patterns in the results. The coloured lines show the conversion rates for each advert. This is a great visual tool for getting buy-in from stakeholders; the infographics clearly show the outperforming tests.

In this test, the plain text advert was getting the best conversions and showing up as the likely winner. The last stage in the testing process is to do a follow up experiment using just the original control advert and the proposed likely winner.

Peoplefinder AB test followup

It’s very satisfying to see the green light that signifies the end of test and a definite winner. Up against all the fancy graphics and animations, the plain text advert had a 98.9% chance of beating the original.

The experiment showed that the text advert was the best approach to take for this problem, for our staff on our intranet. With user testing, the results are often unexpected, reinforcing why we must always design for the user.

See also:

Intranet roundup 2010

2010 has been a year of fast-paced progress for the world of intranets. You can almost feel the ground swell and ripple with growth and evolution. The main drivers have been awareness, interest and adoption of social tools within the workplace. Here are the main themes that I’ve spotted throughout the year.

Companies have implemented social tools both within and outside the firewall, to connect and engage with staff and customers. We’ve seen CEO blogging with both good and bad outcomes. We’ve seen whole intranets built on wikis. There has been some great buzz around commenting and moderation, with the general consensus that staff should be trusted to post and comment without the need for big brother monitoring. And the odd hint of using crowdsourcing tools to capture ideas from staff.

Some intranets have introduced individual staff profile pages (great for keeping the staff directory up to date) with activity streams, social networking, question and answer tools, rewards, badges and karma points.

Internal communications teams have had to learn that communication is a two-way street and that staff have a voice. With user-generated content, the intranet tables have turned. The intranet has shifted from being used centrally as a megaphone to a channel that can be used for listening. Staff can now create their own content through wikis, and whether content is sourced from central teams or not, staff can rate, comment on and share it with each other.

There has been a fair amount of buzz around what or whether to name your intranet. Great arguments being that it improves branding and adds personality. Arguments against, that the intranet is still known by staff as “the intranet”, even if it has a fancy name.

There have also been cries of “kill the intranet”, “the intranet is dead” and “long live the intranet!” Some people claim that with the fusion of intranet, extranet and website as the work space moves into the customer space, the intranet as we knew it really doesn’t exist any more. But this is certainly not the case across the board, especially for those who are slower to adopt social tools and use them to connect and engage with staff and customers.

Two camps are at odds over what to call this new, socially-integrated workspace; NetJMC rallying for the “workplace web” and IBF backing the “digital workplace“. There are those who insist we use the collaborate word in preference to the social word. And there are those who don’t see a problem with the word “intranet” and just see the role of the intranet changing and moving on with the times without having to label it.

The popularity of smart phones and tablet devices with improved user interfaces has given us the potential for mobile access. The ability to surf the intranet remotely via handheld devices will help staff who are not chained to a desk to view information and perform tasks from anywhere. A mobile intranet creates more potential for reach, in terms of internal communications to those members of staff who would otherwise find regular access to the intranet impractical.

There are more and more of us intranet professionals blogging and tweeting and connecting with each other; sharing ideas and advice around the globe, with a lot of buzz from the US, Canada, the UK, Italy, the Netherlands and Australia.

Social tools have also given us new problems to solve. Content has moved from being static pages to a fast-moving flow of consciousness, adding greater complexity to the age old problems of how to search this stuff and how to evaluate it.

On a personal note, I hope that 2011 will be the year when our intranet evolves socially. We don’t even get close to some of the themes that I’ve discussed here. To do so, we’ll need new platforms, new software and a new way of thinking. But I believe it is possible and dare I say, inevitable.

Top tip to improve intranet A-Z index pages

A-Z index pages are very popular on our intranet. More popular than search. Our staff like to find what they want by looking it up in a list. If they don’t find it in the list they may resort to the search box. By looking at what phrases staff use when they perform a search from an A-Z index page, we can get a good idea of what’s missing from the A-Z listing.

Google Analytics has an inbuilt function to segment visits where staff used the search function. It was this functionality that highlighted our A-Z index pages.

I like to visualise a visit to the intranet as a trip to the convenience store. A member of staff needs something. Some people enter the store and walk up and down the aisles trying to locate what they need (menu navigation). Some look up where to go in a catalogue (A-Z index). Some enter the store and immediately ask for help (search) and others ask for help as a last resort. It is this asking for help as a last resort (searching) that can benefit us.

When I looked at the segmented analytics showing only those visits in which a search occurred, it showed, unsurprisingly, that lots of people search from the homepage, equivalent to entering the store and immediately asking for help. Equally unsurprisingly, lots of our popular pages are showing up. But also high in the reports were our A-Z index pages, indicating that lots of staff are going to the A-Z listings and then having to search.

Having highlighted a problem, I can now take action. I can target the A-Z pages and produce analytics that will tell me which search terms are being used when staff resort to using search from these pages. I can then feed these terms back into the A-Z listings, if appropriate, and over time, improve our offering.

Staff who search from the homepage are just using their preferred method of getting to content. Staff who repeatedly search from an area of content deeper within the intranet structure may highlight a symptom that there is a problem with the content.

Intranet show and tell *Search* event: 2 Dec 2010

*Intranet search* was the subject of the Intranet show and tell event that took place at the Ministry of Justice in London on 2 December 2010. The bad weather meant that a lot of people could not make the event, but we managed to get 17 people around the table.

There was a diverse set of talks ranging from working with intranet content and metadata to back-end coding of the search interfaces and the nuts and bolts of the underlying systems.

I kicked off the talks with a quick history of how we have improved the search experience for staff by paying attention to metadata within our HQ intranet content, using Google Search Appliance. I also showed some of our search analytics and infographics that we are starting to use for data visualisation.

Simon Thompson (@thompsonsimon) and Angel Brown (@angelbrownuk) organised the event. Simon showed us some bespoke programming of Sharepoint to enable typeahead search results for both people and intranet content using jQuery.

Tyler Tate (@tylertate) managed to connect through Skype to do a remote session on *Dealing with diverse data* giving examples of search results page interfaces based on different data sources.

Tom Mortimer from Flax gave a very detailed presentation about restricting search results based on user credentials. And Rangi Robinson (@rangfu) from Framestore demo’d his staff intranet people search.

Great to see support from @Funnelback. And sorry we were unable to hear from Martin White from @IntranetFocus who was snowed in.

I noted a lack of talk about what I term *social search*, allowing staff to tag and rate search results. I would be interested to hear how this is being used by anyone.

Lastly, big thanks to @starhorseUK for adding the finishing touches to the room booking and for helping our guests into the building.