With the London 2012 Olympic Games only days away, I was interested in performing some remote user testing of the event’s official iPhone app to discover its pro’s and cons.
It has been a long time since I first published a blog post on remote user testing, 3 months to be exact. In that post, I compared two social traveling websites, AirBnB vs Wimdu.
This time around, along with testing the London 2012 official iPhone app, I was interested in testing UserTesting.com‘s Mobile Testing functionality. I have heard a lot about the feature and there is no better way to test it, then by diving into the tool yourself.
London 2012 iPhone App
For this test, we asked participants to install the London 2012: Join In app. According to the app’s creators:
In the summer of 2012 London and the UK will come alive with events, celebrations and activities during the Olympic and Paralympic Games.
The Official London 2012 Join In app is a mobile guide to help you plan, enjoy and share your Games experience.
This FREE app is an essential planning tool for everyone, whether you have tickets for a sporting event or not. From the start of the Olympic Torch Relay to the Olympics and Paralympics, the Opening and Closing Ceremonies, plus all the cultural, city and community celebrations happening across the UK, Join In is your essential companion.
So the app is multi purpose. It can be used before and during the Olympic Games to help those interested, or those in and around certain events , to stay up to date on what you can do and which events are taking place.
Setting Up The Test
In total, and in order to not to completely use up all of my spare time in the weekend, I limited the test to only 3 participants for the iPhone and 1 user for the iPad version of the app. Like I mentioned in my previous Remote User Testing tool post, I’ve got a full-time job, three kids and enough other things on my to-do list which prevent me from conducting full-scale research assignments. Enough moaning… back to the test.
This is a mobile device test that requires (1) a webcam provided by UserTesting.com and (2) a mobile device as requested by the client. Test must be done using an iPhone.
iPhone users, Any (using highest rated testers), 18-65 and older
You have just traveled to London for the Olympic Games of 2012. You have tickets to several events, but you are also interested in some events you don’t have tickets for. You have installed the official London 2012 app on your iPhone. You start the application to find out more about the Games, locations, event dates and more.
[NOTE] All participants were required to download, install and perform an initial startup of the app on a Wifi connection. After that, to properly simulate real world mobile usage, the participants were requested to perform the test using a 3G connection
- Use the app to find out on which date and in which venue the Men’s 100m Final is being held. When you reach the event’s page, proceed to the next task.
- You were able to get tickets in section 110 of the stadium. Locate the stadium’s map to find out where your seats are located. When you have found the map of the venue, proceed to the next task.
- You are curious to see what the inside of the stadium looks like. Try to discover the different media types (photo/video etc.) and share your thoughts on the quality with us.
- In the app, try to find a map view of the city of London and zoom to see where the Olympic Stadium is located.
- Try to find out where in the app you are able to purchase tickets to the different events.
- Last but not least, you would like to buy some Olympic goodies to take home with you. Locate the Online Shop and find the Men’s Team GB Clothing products.
- What was the most frustrating thing about your experience?
- What other ideas do you have about how it could be improved?
- What did you like about it?
- How likely are you to recommend this site to a friend or colleague (0=Not at all likely, and 10=Very Likely)?
The results were quite surprising. I only tested the app myself using a Wifi connection and found that the app was quite useful. Nevertheless, the remote user testing turned out some surprising results when using a 3G connection. Also, when searching through an event list the size of the Olympic Games, frustrations can arise when your confronted with poor navigation and function labeling.
Just for your information… the modern Olympic Games is home to 302 events in 28 different types of sport. Then take into account that there are over 40 venues for the London 2012 Olympic Games and that the city is expected to accommodate almost half a million visitors… well you do the math. After all, the visitors attending the London 2012 Olympic Games are often not the below average consumer and will most likely all, or at least very many, have a smartphone in their possession. Can I shout ‘Web Performance’?
- Filtering Option when Searching: Filtering when searching for an event is crucial in such an app. In total, the modern Olympic games counts 302 events for 28 sports. The way the app displays leaves much to be desired. In stead of first offering a break down per sport, the app displays each individual sport on a results page.
- Performance: The participants were asked to perform the test using 3G instead of wifi, this to simulate usage in a real mobile environment… on the move. It is clear that certain elements of the app, mainly the ‘Guide’, do not perform well under 3G.
- Media: The quality of media was experienced as being very good. It was unclear why certain venues did contain photo and video content and other venues did.
- Virtual Tours: The virtual experience video’s were perceived as good, however, some participants expressed the preference of better functions during the ‘fly by’ views. The video’s moved to fast.
- Map View: There were many ways to find venue locations. Via the event pages, via the maps and so on. However, with so many venues, not only for sports, the map was very cluttered. The filter, quick link and search functionalities didn’t stand out enough, or were poorly labeled. For instance, the ‘Zoom’ button was in fact a quick link function.
- Merchandising: With the Olympic Games being so commercially focused, I was surprised to see how difficult it was for participants to find a way to purchase tickets (which are still available) and official merchandise. The link to the ticket shop led you to an out-of-app, non-mobile-optimized website which was displayed in an in-app browser. As for merchandising, you were again led to an out-of-app website, this time fully optimized for mobile, but difficult to find.
- Information Architecture: In general, it can be said that the information architecture of the app is poor. With so much content, a better and more thoroughly tested structure and labeling of navigation, and presentation of content would be paramount to securing an excellent user experience. The app definitely loses points on this.
[icon_button type=arrowdown url=http://bqv.be/London2012Source]Download Video Source Files (Size: 125mb)[/icon_button]
I am always amazed to see how poor some mobile applications are developed. The London 2012 lies between good and poor. Good in quantity, poor in architecture.
Clearly, the London 2012 Join In app isn’t shy of content, but it is apparent that it has not been structured in the most optimized method. I have my worries about the app’s performance during the games. 3G did not seem to perform too well for some and the games haven’t even started yet.
On the positive side… the London 2012 is impressive. The amount of information is enormous and the media presented in the app is of great quality. Definitely an app worth having during the games.
[alert color=green title="Need help with remote user testing?" align=center]Let us help you. Check out our service’s page to learn more.[/alert]