Performing mobile user testing is quite exciting. Sure, testing websites and other desktop applications is still interesting, but dipping my toes in the new part of the testing pool was a real eye opener.
After having read Rachel Hinman’s The Mobile Frontier I started to think more about the intricacies of designing for mobile. Interaction design for mobile goes far beyond the human computer interface techniques that we have grown accustom to. Gone are the HID’s like keyboards and mice, enter the World of NUI (Natural User Interface).
With new interface methods comes a whole new way of testing. Like Rachel mentioned in her book, there are no set of standards yet for designing for mobile, this is the new frontier. With no standards, mobile becomes a whole new ballgame all together, ditto when it comes to testing. No longer can we depend on a test person sitting at a desk somewhere using the desktop. The mobile user can be anywhere at anytime, using the mobile device for a plethora of purposes… so get ready!
Back in the days of ‘follow-me-home’ research, researchers would observe participants in their home environment. With mobile, this turns into ‘follow-me-everywhere’ or as I found out recently, the term is ‘walk-along’ or ‘travel-along’.
There are many different ways to capture mobile user testing sessions. Here are 3 of them briefly explained.
When searching around on the internet, you can find many MacGyver types out there blogging about their mobile user testing rig. Rigs are basically tools to help you record a mobile user test, both video and audio for later analysis.
When conducting mobile user testing with rigs, there can be some limitations. Rigs are usually not meant to be used far from another computer, since they are connected to other computers recording the data. Considering this, it might skew the results somewhat, so be aware of the different types of rigs (desktop and mobile… oh the irony) and choose wisely.
To help you guys out, I can really recommend two links on how to create you own mobile user test setup:
UX researcher Nick Bowmast of MrTappy.com even offers a ready-made rig… but at with a price tag of $269 (excluding $37 shipping charges to pretty much everywhere in the world) it might hold people off. The rigs does require an additional device to record the session.
My personal opinion, is that it might well be worth it. Have a look at the following and tell me what you think… I think it is awesome!
Again, although rigs are important to capture the experience, they can impact the true-to-life use of the mobile device if they are not mobile themselves.
If you don’t want to purchase or create your own rig, or even want to create a test panel, then you can always consider remote mobile user testing. Like regular remote user testing the method will allow you to conduct mobile user tests without recruiting your own panel or spending money on a rig.
There are some limits such as properly selecting your target audience, or even actual clients. Nevertheless, the method is useful and delivers great insights.
The four parties that I can think of off the top of my head, that offer mobile user testing are:
Up to now, Usertesting.com delivers most value for money. Their remote mobile user testing features and turn-around time make it an ideal tool for anyone needing immediate, and high quality insights.
Now, without further delay, let me share with you a list of things you should consider prior to conducting your mobile user test. Again, this list is not set in stone, but it should help you create a bulletproof mobile testing hypothesis and environment in which to get started.
Just real quick… in a recent published study by Userzoom.com and KLI entitled ”Travel Consumer Preferences for Mobile and Tablet”, I came across the use of a set of eye tracking glasses while conducting mobile field research. If you have got the money for this, this might well be your ticket to ultimate insights. I am pretty sure however, that the price tags are 4 figures or greater… so make sure you bring a fat wallet.
To help to get a clearer picture of what you want to test, or what you want to try to discover, try to consider some of the following items in the list. I will try to elaborate on several of the items, especially those that do not seem self-explanatory.
What platform are you looking to gain actual insights on? There are many cross-platform apps out there, so have a look at your web analytics and find problems to tackle fo a specific platform.
Just as important as Platform, and clearly related is the OS version you want to test. Even with Apple’s iOS which is easy to update, you will still find many users running older versions. This can be because they hardly ever connect their phone to the computer to let iTunes remind them that an update is available, or if you are like me… if the iOS version cannot be jailbroken, I won’t update. So make sure you find out for which OS version you are testing and consider any related functionalities that might or might not be present.
Right off the bat, a simple, yet no so obvious. Is someone left- or right-handed? Do they navigate/operate the phone primarily with the thumb (small interaction zone limits) or with their finger while the opposite hand holds the device. This is crucial to know when testing smartphones, since the size allows for both methods to be used.
Not everyone uses their mobile device the way you found it in the box. We all want to make our device more personal, or just protect it from damage, or just want to mount it in a location where we might not be able to hold the device. Cases, mounts, styli, keyboards, and headphones, we need to consider them. Personally I have a case with keyboard for my tablets, mainly to allow me to type more efficiently. I use the stylus for notes and headphones… well, when I don’t want to bother other people around me while I am on Skype, watching movies, or listening to music.
The environment is crucial. For instance, take one of the many fitness applications out there for your smartphone. Runkeeper is such an app. How do you test an application that is meant to work 95% of the time while you are running? How much will people physically interact with the application during a run? How important are features like the built-in camera compared to real-time pace/speed charts? The fact that they make the real-time pace/speed data available via audio cues, goes to show that it is not always about what you seen while using an app, so make sure to also think about peripherals. Clearly identify in which environment your application is used, and test accordingly.
This is one of my favorites. We love performing mobile user testing. More often than not, we test using Wifi. We are just so used to working with Wifi that we tend to forget where mobile phones are also used… that’s right, while we’re on the move. Even when using a mobile broadband router as can be seen in the photo, we are still very dependent on our mobile network supplier. Let’s be real, even performing mobile user tests comparing one mobile network supplier with another is an interesting one!
Like I mentioned earlier in this post, this list is in no way definitive. If you have any points you would like me to consider, then please leave me a note in the comments.
Let us help you. Check out our service’s page to learn more.