Mobile Usability Testing Considerations [Video]

Mobile Usability Testing Considerations [Video]

Share these actual insights

If you are a regular visitor of my blog, you know that I have spent quite a bit of time on performing remote usability testing. In my latest test I performed a mobile usability test for the first time.

Performing mobile usability testing is quite exciting. Sure, testing websites and other desktop applications is still interesting, but dipping my toes in the new part of the testing pool was a real eye opener.

The Mobile Frontier & NUI

The Mobile FrontierAfter having read Rachel Hinman’s The Mobile Frontier I started to think more about the intricacies of designing for mobile. Interaction design for mobile goes far beyond the human computer interface techniques that we have grown accustom to. Gone are the HID’s like keyboards and mice, enter the World of NUI (Natural User Interface).

With new interface methods comes a whole new way of testing. Like Rachel mentioned in her book, there are no set of standards yet for designing for mobile, this is the new frontier. With no standards, mobile becomes a whole new ballgame all together, ditto when it comes to testing. No longer can we depend on a test person sitting at a desk somewhere using the desktop. The mobile user can be anywhere at anytime, using the mobile device for a plethora of purposes… so get ready!

How To Record Mobile Usability Testing Sessions

Back in the days of ‘follow-me-home’ research, researchers would observe participants in their home environment. With mobile, this turns into ‘follow-me-everywhere’ or as I found out recently, the term is ‘walk-along’ or ‘travel-along’.

There are many different ways to capture mobile usability testing sessions. Here are 3 of them briefly explained.

Mobile User Testing Rig

When searching around on the internet, you can find many MacGyver types out there blogging about their mobile usability testing rig. Rigs are basically tools to help you record a mobile user test, both video and audio for later analysis.

When conducting mobile user testing with rigs, there can be some limitations. Rigs are usually not meant to be used far from another computer, since they are connected to other computers recording the data. Considering this, it might skew the results somewhat, so be aware of the different types of rigs (desktop and mobile… oh the irony) and choose wisely.

To help you guys out, I can really recommend two links on how to create you own mobile user test setup:

  1. Make your own iPhone Usability testing sled for £5 by Harry Brignull – written in 2010, but still very useful and usable with other devices
  2. Cheap iOS Usability Testing – shameless self plug… I know it. even though it only focusses on iOS devices, I am sure that something could be done for other mobile OS’ and devices

UX researcher Nick Bowmast of MrTappy.com even offers a ready-made rig… but at with a price tag of $269 (excluding $37 shipping charges to pretty much everywhere in the world) it might hold people off. The rigs does require an additional device to record the session.

My personal opinion, is that it might well be worth it. Have a look at the following and tell me what you think… I think it is awesome!

Again, although rigs are important to capture the experience, they can impact the true-to-life use of the mobile device if they are not mobile themselves.

Remote Mobile User Testing (Rigless)

If you don’t want to purchase or create your own rig, or even want to create a test panel, then you can always consider remote mobile user testing. Like regular remote usability testing the method will allow you to conduct mobile user tests without recruiting your own panel or spending money on a rig.

There are some limits such as properly selecting your target audience, or even actual clients. Nevertheless, the method is useful and delivers great insights.

The four parties that I can think of off the top of my head, that offer mobile user testing are:

  1. Usertesting.com
  2. Userlytics.com
  3. Loop11.com
  4. Whatusersdo.com

Up to now, Usertesting.com delivers most value for money. Their remote mobile user testing features and turn-around time make it an ideal tool for anyone needing immediate, and high quality insights.

Now, without further delay, let me share with you a list of things you should consider prior to conducting your mobile user test. Again, this list is not set in stone, but it should help you create a bulletproof mobile testing hypothesis and environment in which to get started.

Eye Tracking Glasses

Just real quick… in a recent published study by Userzoom.com and KLI entitled “Travel Consumer Preferences for Mobile and Tablet”, I came across the use of a set of eye tracking glasses while conducting mobile field research. If you have got the money for this, this might well be your ticket to ultimate insights. I am pretty sure however, that the price tags are 4 figures or greater… so make sure you bring a fat wallet.

Mobile Usability Testing Considerations

To help to get a clearer picture of what you want to test, or what you want to try to discover, try to consider some of the following items in the list. I will try to elaborate on several of the items, especially those that do not seem self-explanatory.

Platform

What platform are you looking to gain actual insights on? There are many cross-platform apps out there, so have a look at your web analytics and find problems to tackle fo a specific platform.

  • Tablet
  • Smartphone
  • iPod type devices

Operating System

  • Latest OS
  • Most common OS
  • Most profitable OS

Just as important as Platform, and clearly related is the OS version you want to test. Even with Apple’s iOS which is easy to update, you will still find many users running older versions. This can be because they hardly ever connect their phone to the computer to let iTunes remind them that an update is available, or if you are like me… if the iOS version cannot be jailbroken, I won’t update. So make sure you find out for which OS version you are testing and consider any related functionalities that might or might not be present.

User Characteristics

Right off the bat, a simple, yet no so obvious. Is someone left- or right-handed?  Do they navigate/operate the phone primarily with the thumb (small interaction zone limits) or with their finger while the opposite hand holds the device. This is crucial to know when testing smartphones, since the size allows for both methods to be used.

  • Left-handed, right-handed
  • Thumb or Finger

Peripherals

Not everyone uses their mobile device the way you found it in the box. We all want to make our device more personal, or just protect it from damage, or just want to mount it in a location where we might not be able to hold the device. Cases, mounts, styli, keyboards, and headphones, we need to consider them. Personally I have a case with keyboard for my tablets, mainly to allow me to type more efficiently. I use the stylus for notes and headphones… well, when I don’t want to bother other people around me while I am on Skype, watching movies, or listening to music.

  • With or without headphones
  • With or without case
  • Bicycle/Car holders
  • Input type finger, keyboard, stylus

Environment

The environment is crucial. For instance, take one of the many fitness applications out there for your smartphone. Runkeeper is such an app. How do you test an application that is meant to work 95% of the time while you are running? How much will people physically interact with the application during a run? How important are features like the built-in camera compared to real-time pace/speed charts? The fact that they make the real-time pace/speed data available via audio cues, goes to show that it is not always about what you seen while using an app, so make sure to also think about peripherals. Clearly identify in which environment your application is used, and test accordingly.

  • At home or work
  • In car holder
  • Walking with phone in hand
  • Running/cycling with sport case (i.e. around arm)
  • Daytime/Nighttime

Connections

This is one of my favorites. We love performing mobile user testing. More often than not, we test using Wifi. We are just so used to working with Wifi that we tend to forget where mobile phones are also used… that’s right, while we’re on the move. Even when using a mobile broadband router as can be seen in the photo, we are still very dependent on our mobile network supplier. Let’s be real, even performing mobile user tests comparing one mobile network supplier with another is an interesting one!

  • 3G
  • Wifi
  • Wifi via Mobile Broadband

Not Definitive

Like I mentioned earlier in this post, this list is in no way definitive. If you have any points you would like me to consider, then please leave me a note in the comments.

Need help with mobile usability testing? Let us help you. Check out my service’s page to learn more.


Warning: Missing argument 3 for snp_create_popup(), called in /home/vitaluc0/public_html/actualinsights.com/wp-content/plugins/arscode-ninja-widget/arscode-ninja-widget.php on line 33 and defined in /home/vitaluc0/public_html/actualinsights.com/wp-content/plugins/arscode-ninja-popups/arscode-ninja-popups.php on line 1326
Receive FREE updates

Get FREE updates and UX product discounts directly in your inbox by simply signing up for my newsletter. You can unsubscribe at any time.

Your information will never be shared, period!
About Matthew Niederberger

Matthew works as Conversion Optimisation Manager at Ziggo BV. In his free time he enjoys family life as well as digging into online user research material whilst frequently generating some of his own, which he freely shares here on actualinsights.com.

Comments

  1. Great blog post Matthew, very useful tools to make people aware of.

    I did some mobile testing in a train station the other day on an iPad where we had limited connection for our online prototype. I found that the ‘GoodReader’ app for iPad was a really useful tool as it allows you to extract a zip file and run the prototype locally. It has limitations in that it’s very sensitive to touch and will display it’s own app menu quite a lot but otherwise a really useful tool when connection limited.

  2. HI David, I did not know about that Goodreader function. How cool! I tried something similar but with ‘offline’ apps that will allow you to download a website to the device and browse/test it from there. Did you guys put any local tracking code in to record clickstreams or heatmapping? Thanks for the compliment by the way :)

    By the way… Mr. Tappy’s Nick Bowmast is sending us a Mr. Tappy device to review for the website. Make sure you don’t miss it, I think that it will beat my expectations for mobile remote user testing.

Trackbacks

  1. […] Click here to view the embedded video. […]

  2. […] having mentioned MrTappy in a previous post, I dug a little deeper into the tool and decided to purchase one. I am glad I […]

  3. click here. says:

    click here.

    Mobile Usability Testing Considerations [Video] – actualinsights*

Leave a Reply

Share these actual insights

User Experience courses on sale at Udemy.com