Let's get connected:
In many of my other blog posts I have discussed user research through various methods. From surveys, to design feedback.
What I had not done, yet, was share my experience with performing remote user research. Technically I did, with my presentation ‘Prospective Optimization‘ at the Dutch Web Analytics Conference back in March, but not specifically on this website.
In this blog post I would like to share with you, a real life example of working with UserTesting.com. To be more precise, a comparison between Airbnb.com and Wimdu.com.
SKIP TO: User Testing Videos
I like to write my articles based on real life experiences in the world of conversion rate optimization. Although there are many worthy alternatives out there for remote user testing, such as Loop11, TryMyUI and YouEye, my practical experience with UserTesting.com is the most extensive.
In future articles I will definitely take the time to focus on the alternatives. Not only for remote user testing, but for all aspects of user research.
Airbnb has certainly established itself as a serious player in the field of travel. By creating a network of hosts, who make their homes, or rooms, available for (short) rental to travellers, it undoubtedly tapped into a vast resource of revenue still available in the online travel agency (OTA) world.
Is Airbnb unique? Some beg to differ, because they were not the first. The same goes for Wimdu. Is Airbnb an improved idea inspired by the likes of Couchsurfing.org? Is Wimdu another spin-off, of a spin-off? My opinion is… who cares. It should be clear that all of these companies are doing well, not only financially, but also in filling a requirement in today’s travel market. So, who are we to criticize?
Well, enough politics for now…
This research was basically performed with two goals in mind.
This form of user research, recording participants’ screens and audio while they perform a set of tasks is great because it allows you to do several things.
Basically, where analytics and feedback forms leave off, (remote) user testing picks up. This is true in more ways than one, because what (remote) user testing lacks is an easy way to quantify the findings.
Nothing is more laborious in the online world than performing and analyzing user research. It is good for the hours if you work freelance, but notoriously time consuming nevertheless. We have gotten lazy with all the great analytic tools on the market these days, that we forget that real insights, takes real-time, real effort… a lot of both to be honest.
The same is the case with remote user research. In several cases I have spent up to 1 hour setting up a test, 5 hours analyzing the results (average video length 20 minutes, analyzing takes me 1 hour per video) and around 8 to 16 hours compiling and documenting the results. Time and effort ultimately depends on the test in question.
Still, it is quicker than alternative methods of performing user research like in a lab.
So how did I go about setting up the test? In this specific test 5 participants were invited to perform a series of tasks defined around a single scenario. The tasks (some default ones from UserTesting.com) were meant to let the participants compare both websites and give their opinion on the experience.
FYI… The participants are pulled from a legion of testers who have signed up at UserTesting.com. In return for a small fee, they perform the test for you, which is included in the ‘per test fee’.
I have conducted of 100+ tests and I can happily report that the quality of participants is good. The only barriers now are geographical ones as the location of the participants is limited to USA, Canada and the UK. YouEye does support European testing at the moment, in case anyone was wondering.
I formulated the following scenario for the test:
Put yourself in the shoes of someone who wants to compare two sites before booking a Bed & Breakfast accommodation for your upcoming holiday to New York City. You will be visiting two sites to compare offerings.
Within the scenario, participants were asked to perform the following tasks:
When viewing the recorded test sessions, I take it upon myself to use a ‘Create Clip’ function whenever I find something that might be useful to discuss. In this case, I have done the same. Now, I did test 5 participants, but I will only show you the findings of 1 of them, just to prevent taking too much of your time and keep this article as enjoyable as possible.
Creating a clip helps you find a certain issue after reviewing.
Anyone interested in discussing the findings with me, you are welcome to do so by contacting me, or by sharing your thoughts in the comments below. I will consider making a compilation video of all the sessions and publishing them, but for now, I will limit the compilation video to just 1 participant, just to save some (personal) time.
DISCLAIMER: This list was created by myself and is in no way ‘complete’. Some shortcuts were taken for the purpose of publishing this blog post since I do all of this in my spare time, which, with 3 kids, is not much. I like to consider myself objective and skilled enough to detect possible issues on a website. I agree that because only 5 users were tested it is hard (near impossible) to claim significance, but I know that any finding could be a catalyst to improving the website. Since I work for neither Airbnb nor Wimdu, I cannot say what has already been tested and what has not. It could be that certain functionalities have been designed just as the user has experienced them (ie. Search Button on Homepages).
TIP: Click on the toggle below the video for a (short) summary of the findings in the Airbnb user research session.
TIP: Click on the toggle below the video for a (short) summary of the findings in the Wimdu user research session.
The preferences were evenly split. Wimdu did get some credits for being easier to book, but lacked in recognition. The latter is most likely (very non-scientific term) due to the fact that Wimdu is European based (Germany) and Airbnb US. Not an excuse, but surely worth considering. I think that both websites have enough work cut out for them in optimizing the user experience. My advice: Just keep testing.
Conducting user research this way is… well, call me a nerd, but it is fun. You are in charge of what you want to test, when, and to a large extent where. The insights gained through this research are powerful in a way that it is the users/visitors themselves that indicate what they like and don’t like. Gut feeling plays a small part, so the possible positive effect on conversion rate optimization based on user feedback is potentially huge.
No, there is no contact with the testers to ask additional questions or to steer a test while in progress. Some things you need to take for granted and take the feedback at such a value that it keeps your user experience and conversion rate optimization brain cells in motion.
When viewing the tests, I was glad that my trustlogo research importance was mentioned on several occasions, but I did miss a critical (or at least so for me) item. During my own pre-test, so to speak, I noticed that both websites don’t utilize some very key USP’s (unique selling points) at important stage in the booking process.
The one that caught my eye the most, and which was not visible while booking, was the payment process at Wimdu. So, in closing, please consider this…
At Airbnb, your are charged for you stay as soon as the host accepts your booking request.
Airbnb’s payment method differs from Wimdu. Critical, yes or no?
At Wimdu, you are not charged until 24 hours after arriving at your destination.
An important Wimdu USP hidden from view.
Exactly, (remote) user testing! Please share your experiences with remote user testing!
Let us help you. Check out our service’s page to learn more.
[ReviewAZON name="Airbnb Wimdu Research" id="6" display="multiinlineasins" asins="0470185481,0123849683,0321804813" trackingid="actualinsights-20" country="us" width="100%" float="left" displaytype="list" count="10"]
 |
 |
[...] Remote User Testing: Airbnb vs Wimdu http://t.co/qsKOkiDT via @actualinsights #ux #cro #usability Link – Trackbacks Posted in User experience (UX) | Permalink. ← How content density [...]
Nice post Matthew! Do you think showing reviews in the checkout process is worth testing for both Wimdu and Airbnb in a next user test?
I ask this question, because I found the reaction in the Wimdu video about recommendations interesting!
I would always test it. My personal opinion is that I do not think that it would help much. In the decision making process, the visitor has already taken the action to book a certain accommodation, validating that decision again in while booking seems a little too much repetition. I think, again personally, that other worries should be solved, such as payment security and for example the Wimdu payment system. Wimdu doesn’t pay the host until 24 hours after your arrival. This should make the booker feel safe to such an extent that if they arrive in some dump, they can still contact Wimdu and block payment. In the end, it all just depends on context.
I use Airbnb everyday and the reason why I always come back to find a place is because the website is very simple to use. The Q&A is quiet helpful and i can tell that I’m not very confortable with internet …
I was at first wondering if my review would appear on the website just the way I first wrote it. Airbnb does not change any word so it’s a trustful reviews system.
Anyway, this is my opinion as an Airbnb user and I also wanted to say that your test and comments are very interesting!
Hi Blancheneige. Thanks for the reply and compliments. I fully understand your point. I think that over the years we subconsciously build relationships with brands/websites. I can fully understand that Airbnb.com has that effect. With the community driven approach to accommodation rental the likelihood of switching to another brand such as Wimdu is decreased with every visit/booking. I am starting to compare it to the relationships we have to, lets say a computer’s operating system, or a mobile phone provider. At some point in time we have personally invested so much time and effort into building the relationship (learning all the ins and outs of a website/product, purchasing/reviewing products, building wishlists etc.) that we become reluctant to change. Change is just too damn difficult (pardon my French), even if it can sometimes benefit us in the end. Do you feel that this is how you experience, for example, your preference for Airbnb? Matthew
I use Airbnb all the time as well. I like how you pay Airbnb up front and if the reservation is accepted, you’re charged, but Airbnb does not release the money to hosts until 24 hours after the reservaton starts. If the host doesn’t want to accept the reservation, there’s not charge either.
Concerning payment, WIMDU now functions the same way Airbnb does, that is the total amount (rental + service charge) was taken off my account 2 days after I made the reservation. The owner of the chalet will not get the rental money until 24 hours after my arrival. That’s a very large sum of money that WIMDU has at their disposal for no cost for several months. They can do a lot of investing with that.
But my major complaint about WIMDU is that it turns out the owners of the chalet I want to rent for a week have not signed a contract with WIMDU, don’t know who they are and don’t want to work with them. In other words, WIMDU WAS ADVERTISING FOR THEM WITHOUT THEIR PERMISSION OR KNOWLEDGE. The owners asked WIMDU to cancel the reservation. No reply from WIMDU. Meanwhile, as indicated, the entire amount (rental + service charge, approximately 300 Euros) has been taken off my account. The owners refuse the WIMDU reservation and ask me to reserve directly with them or through another web site. I have written WIMDU 3 times and get NO REPLY. The owners work with several other sites and THE TOTAL COST IS LOWER ON ALL THESE OTHER SITES (as the owners pay the 10% service charge rather than the renter) AND ONLY THE SERVICE CHARGE IS TAKEN OFF YOUR ACCOUNT AT THE TIME OF RESERVATION, NOT THE ENTIRE COST.
So, for the time being, I have paid for the rental TWICE and remain (somewhat) hopeful that WIMDU will respond to my request for cancellation which was made through the correct channel. Next week I will be seeing a consumer organisation to ask them to contact WIMDU.
[...] It has been a long time since I first published a blog post on remote user testing, 3 months to be exact. In that post, I compared two social traveling websites, AirBnB vs Wimdu. [...]
[...] want to create a test panel, then you can always consider remote mobile user testing. Like regular remote user testing the method will allow you to conduct mobile user tests without recruiting your own panel or [...]
[...] Airbnb vs Wimdu [...]
I Like http://www.vaway.com also its a good site to list on because they partner with Airbnb, kayak and travelocity to list my property.
Ripped me off two months before I even got to San Francisco. I had (stupidly) entered my credit card number to (I thought) hold a vacation rental. They charged my card RIGHT AWAY, so, alarmed, I cancelled the reservation somewhere around 24 hours after making it. Now Wimdu claims they are keeping 12% of the total as a fee.
They claimed it was in the “terms and conditions”, which is about 12 pages long. NO where on the “complete booking” page does it say, if it turns out you don’t like the host, even if you cancel we’re keeping 12%. Now I’m out $134. I’ve begged and begged via email and they continously politely tell me they’re keeping my money.
Here in the US hotels do indeed take your credit card number but don’t charge you until you actually show up.
Some surveys will disclose a new product and have you not to
share this information. Enjoy searching for paid survey sites.
2) If you want to read, you will find companies you can find through bidding sites like o
- Desk, Elance, etc.
Wimdu is a clone of Airbnb except the service fee of the site are much higher in Wimdu! This is not from 6% to 12% + 3% like Airbnb. In Wimdu you pay 24% for the site service…
You can compare apartments which are on both sites.
Wimdu est un clone de Airbnb sauf que les frais de service du site sont bien plus élevé chez Wimdu! Ce n’est pas de 6% à 12% + 3% comme chez Airbnb. Sur Wimdu c’est 24% de frais de service pour le site…
Comparez les apartements qui sont sur les deux sites.
Correction of my previous post, it is 21% in Wimdu. It’s still more expensive than Airbnb.
Rectification de mon message précedent, c’est 21% chez Wimdu. Bon cela reste plus chere que Airbnb.
Started listing on Wimdu after AirBnB was subjected to new tax demands by our lovely government. If ever I’ve disliked an organization, this would be it.
1. CONTROLLING
If AirBnB is that cool older sibling who trusts you to know what you are doing, Wimdu is that nagging one who is consistently looking over your shoulder to make sure you’re responding to every single inquiry and punishing you with leeeeengthy politely-written essays – oops, emails – and the deactivation of your listings if you’re 1/ not, or 2/ declining, even if a male is requesting a stay in a space with the heading “all-female.”
I know when I receive a Wimdu message. My inbox has this cool way of notifying me with a message with the heading “New message on Wimdu.” I think it’s called an “e-mail.” It even contains a snippet of the incoming message if I click on it, which I do to see if it’s another male hoping to get lucky and score a stay in the female room before bothering with logging in to respond and whatnot. I don’t need big brother Wimdu to get all up in my face with a 3-page letter every time I don’t respond or decline someone. Either I’m full or am considering long-term residency applicants so don’t have the space for short-term stays at the moment. Stop needing to look over my shoulder at every turn.
2. INCOMPETENT
Twice now, I’ve had calendar blocking issues. On dates that had never been touched on Wimdu. Marked as unavailable. Thing with dates marked as such is that you can’t click on them on the otherwise interactive calendar. So you can’t change them to, well, available. Tried five times with refreshing plus five more on the drop-down calendar submission form below it to no avail – the screen didn’t even register that any information had been submitted for review or otherwise. So I relinquished to contacting support.
Two days. It took two days to clarify that the calendar was not interactive and, hence, could not be changed on the host computer end. Wasn’t the description above clear enough? You tell me, readers, you tell me, because Wimdu support obviously wasn’t getting it. And after the two days:
“Please give me the exact dates, price and minimum stay and I can [change availability] for you.” Dates not available had been specified in the email you just responded to. Dollar amount listed on all available dates on the calendar. Minimum stay, none, as it has always been in my default settings.
Did I mention that I had two separate guest reservations on standby waiting for this process to be completed, which I made clear several times in communication? Both ended up booking with us via a different service provider. Did I also mention that I’d had a similar issue on AirBnB last year but it was a glitch from my calendar updates rather than an “out of nowhere” unavailability, and AirBnB took care of it as soon as I contacted them?