Remote Usability Testing: Airbnb vs Wimdu [Video]

Remote Usability Testing: Airbnb vs Wimdu [Video]

Share these actual insights

In many of my other blog posts I have discussed user research through various methods. From surveys, to design feedback, but not remote usability testing.

So, let me share my experience with performing remote user testing with you.  In this blog post I would like to share with you, a real life example of working with UserTesting.com. To be more precise, a comparison between Airbnb.com and Wimdu.com.

SKIP TO: Usability Testing Videos

Why UserTesting.com?

I like to write my articles based on real life experiences in the world of conversion rate optimization. Although there are many worthy alternatives out there for remote user testing, such as Loop11, TryMyUI and YouEye, my practical experience with UserTesting.com is the most extensive.

In future articles I will definitely take the time to focus on the alternatives. Not only for remote usability testing, but for all aspects of user research.

Comparing Airbnb.com with Wimdu.com

Airbnb has certainly established itself as a serious player in the field of travel. By creating a network of hosts, who make their homes, or rooms, available for (short) rental to travellers, it undoubtedly tapped into a vast resource of revenue still available in the online travel agency (OTA) world.

Is Airbnb unique? Some beg to differ, because they were not the first. The same goes for Wimdu. Is Airbnb an improved idea inspired by the likes of Couchsurfing.org? Is Wimdu another spin-off, of a spin-off? My opinion is… who cares. It should be clear that all of these companies are doing well, not only financially, but also in filling a requirement in today’s travel market. So, who are we to criticize?

Well, enough politics for now…

Remote Usability Testing… let the users speak!

This research was basically performed with two goals in mind.

  • First is to show you how I use UserTesting.com to gain insights from participants.
  • Second is to learn more about this niche in the travel industry. Pure curiosity.

This form of user research, recording participants’ screens and audio while they perform a set of tasks is great because it allows you to do several things.

  • Listen to the participants speak their mind. If you get good participants, they will clearly speak their minds. They will share the positive and negative thoughts they have while participating in your test. In some situations, you can even hear a participants’ anxiety, or pleasure in performing the set tasks or using certain functionalities.
  • You get to see how participants use your website (or whatever it is you are testing). How they move their mouse, what links they click, and how the website performs for them.

Basically, where analytics and feedback forms leave off, (remote) usability testing picks up. This is true in more ways than one, because what (remote) usability testing lacks is an easy way to quantify the findings.

Qualitative Research = Manual Labor

Nothing is more laborious in the online world than performing and analyzing user research. It is good for the hours if you work freelance, but notoriously time consuming nevertheless. We have gotten lazy with all the great analytic tools on the market these days, that we forget that real insights, takes real-time, real effort… a lot of both to be honest.

The same is the case with remote usability research. In several cases I have spent up to 1 hour setting up a test, 5 hours analyzing the results (average video length 20 minutes, analyzing takes me 1 hour per video) and around 8 to 16 hours compiling and documenting the results. Time and effort ultimately depends on the test in question.

Still, it is quicker than alternative methods of performing user research like in a lab.

The Comparison Test: Airbnb vs Wimdu

So how did I go about setting up the test? In this specific test 5 participants were invited to perform a series of tasks defined around a single scenario. The tasks (some default ones from UserTesting.com) were meant to let the participants compare both websites and give their opinion on the experience.

FYI… The participants are pulled from a legion of testers who have signed up at UserTesting.com. In return for a small fee, they perform the test for you, which is included in the ‘per test fee’.

I have conducted of 100+ tests and I can happily report that the quality of participants is good. The only barriers now are geographical ones as the location of the participants is limited to USA, Canada and the UK. YouEye does support European testing at the moment, in case anyone was wondering.

I formulated the following scenario for the test:

Put yourself in the shoes of someone who wants to compare two sites before booking a Bed & Breakfast accommodation for your upcoming holiday to New York City. You will be visiting two sites to compare offerings.

Within the scenario, participants were asked to perform the following tasks:

  1. Go to www.airbnb.com. Look at the home page for five seconds. Then look away and answer this one question (without peeking!): What do you remember?
  2. You are looking for a place to stay in New York City. You and your partner will arrive on Friday, April 20th, and return home on Monday, April 23rd. Go ahead and look for a suitable place to stay.
  3. When you are happy with the place you would personally want to stay at, go through the motions of booking the accommodation. Make sure to stop just before ‘actually’ booking. Remember, please talk us through your journey. Tell us what you like and what you found distracting.
  4. Repeat steps 1,2, and 3 for www.wimdu.com
  5. Which site did you prefer? Why?
As you can see, the set of tasks basically repeat themselves for both websites being tested.

Findings

When viewing the recorded test sessions, I take it upon myself to use a ‘Create Clip’ function whenever I find something that might be useful to discuss. In this case, I have done the same. Now, I did test 5 participants, but I will only show you the findings of 1 of them, just to prevent taking too much of your time and keep this article as enjoyable as possible.

Creating a clip helps you find a certain issue after reviewing.

Anyone interested in discussing the findings with me, you are welcome to do so by contacting me, or by sharing your thoughts in the comments below. I will consider making a compilation video of all the sessions and publishing them, but for now, I will limit the compilation video to just 1 participant, just to save some (personal) time.

DISCLAIMER: This list was created by myself and is in no way ‘complete’. Some shortcuts were taken for the purpose of publishing this blog post since I do all of this in my spare time, which, with 3 kids, is not much. I like to consider myself objective and skilled enough to detect possible issues on a website. I agree that because only 5 users were tested it is hard (near impossible) to claim significance, but I know that any finding could be a catalyst to improving the website. Since I work for neither Airbnb nor Wimdu, I cannot say what has already been tested and what has not. It could be that certain functionalities have been designed just as the user has experienced them (ie. Search Button on Homepages).

Airbnb User Research Findings

  1. Star-icon by listing on Search Results Page: Unclear. In travel, this icon is too easily linked to rating/review score of an accommodation.
  2. Search flow: Location input field + search button, then followed below by Number of nights and persons. Wimdu’s example was appreciated a little more with search button under all input fields (see screenshot). This is a very subjective matter.
  3. Host video: Homepage displays video depicting a ‘host’. If you are a first time visitor and are new to the concept it might be a deterrent. Can I find a place to stay, or is this a site to list a place to stay…
  4. Filters: Extremely low contrast making the filters hard to find. Filters in bottom right of layout. When expanding filters, they become unusable (see screenshot).
  5. Extra costs: Cleaning costs of $50 steep for a $72/night apartment. Causes hesitation.
  6. Third Parties: When trying to book, signing in through Facebook took very long. This is a risky dependency at such a crucial stage in the process.
  7. Many Questions: Potential bookers must answer many questions before being able to book. Questions are supported by stating that ‘the host would like to know more’. Number of questions seem over the top and sometimes even irrelevant for purpose of travel. Hotel seems much more hassle-free at such a moment.
  8. Call-to-Action: Location of ‘continue’ call-to-action button is confusing since there is a require field just above it. Button seems to be linked to that input field, but is not causing an error message to pop up (see screenshot).

Wimdu User Research Findings

  1. Calendar: Start of the week different for US/EU users, Sunday vs Monday. Small difference, but big impact .
  2. Listing of Locations: New York, is both a state and a city. When searching, prioritize on city level instead of state or region (for EU). If people search, they will most likely already have a sense of the city they want to go to.
  3. Sorting by Recommendations: Be careful when sorting by recommendations when the number of recommendations are limited (ie. < 5 or 10). In past experience, the order in which visitors will seek out their desired product is price > reviews (recommendations). Default to price, low-to-high.
  4. Photo of Hosts: Placing a photo of the host on a search results page can be confusing, especially for first time visitors. Who are these people, reviewers, other travellers? The fact that the host is portrayed on the results page becomes more clear on the detail page. Progressive disclosure should be used here to prevent visitors from getting confused.
  5. Phone Support: Phone number below Book Now button very tempting to call. This would cause a channel shift and in the end give you a higher CPA.
  6. Booking Process: Booking seemed easier than with Airbnb. No list of questions. Trustlogo’s seemed a bit small and few. Location of the Complete Booking button too far to the right… move closer to the left, align with input fields.

Conclusion

The preferences were evenly split. Wimdu did get some credits for being easier to book, but lacked in recognition. The latter is most likely (very non-scientific term) due to the fact that Wimdu is European based (Germany) and Airbnb US. Not an excuse, but surely worth considering. I think that both websites have enough work cut out for them in optimizing the user experience. My advice: Just keep testing.

Conducting remote usability testing… well, call me a nerd, but it is fun. You are in charge of what you want to test, when, and to a large extent where. The insights gained through this research are powerful in a way that it is the users/visitors themselves that indicate what they like and don’t like. Gut feeling plays a small part, so the possible positive effect on conversion rate optimization based on user feedback is potentially huge.

No, there is no contact with the testers to ask additional questions or to steer a test while in progress. Some things you need to take for granted and take the feedback at such a value that it keeps your user experience and conversion rate optimization brain cells in motion.

When viewing the tests, I was glad that my trustlogo research importance was mentioned on several occasions, but I did miss a critical (or at least so for me) item. During my own pre-test, so to speak, I noticed that both websites don’t utilize some very key USP’s (unique selling points) at important stage in the booking process.

The one that caught my eye the most, and which was not visible while booking, was the payment process at Wimdu. So, in closing, please consider this…

At Airbnb, your are charged for you stay as soon as the host accepts your booking request.

Airbnb Payment Method

Airbnb’s payment method differs from Wimdu. Critical, yes or no?

At Wimdu, you are not charged until 24 hours after arriving at your destination.

Wimdu Payment USP

An important Wimdu USP hidden from view.

  1. How does that make you feel?
  2. Would seeing this USP make you feel safe booking?
  3. How would you test to see who else felt the same?

Exactly, (remote) usability testing! Please share your experiences with remote usability testing!

Need help with remote usability testing? Let me help you. Check out my service’s page to learn more.

Receive FREE updates

Get FREE updates and UX product discounts directly in your inbox by simply signing up for my newsletter. You can unsubscribe at any time.

Your information will never be shared, period!
About Matthew Niederberger

Matthew works as Conversion Optimisation Manager at Ziggo BV. In his free time he enjoys family life as well as digging into online user research material whilst frequently generating some of his own, which he freely shares here on actualinsights.com.

Comments

  1. Nice post Matthew! Do you think showing reviews in the checkout process is worth testing for both Wimdu and Airbnb in a next user test?

    I ask this question, because I found the reaction in the Wimdu video about recommendations interesting!

    • I would always test it. My personal opinion is that I do not think that it would help much. In the decision making process, the visitor has already taken the action to book a certain accommodation, validating that decision again in while booking seems a little too much repetition. I think, again personally, that other worries should be solved, such as payment security and for example the Wimdu payment system. Wimdu doesn’t pay the host until 24 hours after your arrival. This should make the booker feel safe to such an extent that if they arrive in some dump, they can still contact Wimdu and block payment. In the end, it all just depends on context.

  2. Blancheneige says:

    I use Airbnb everyday and the reason why I always come back to find a place is because the website is very simple to use. The Q&A is quiet helpful and i can tell that I’m not very confortable with internet …
    I was at first wondering if my review would appear on the website just the way I first wrote it. Airbnb does not change any word so it’s a trustful reviews system.
    Anyway, this is my opinion as an Airbnb user and I also wanted to say that your test and comments are very interesting!

    • Hi Blancheneige. Thanks for the reply and compliments. I fully understand your point. I think that over the years we subconsciously build relationships with brands/websites. I can fully understand that Airbnb.com has that effect. With the community driven approach to accommodation rental the likelihood of switching to another brand such as Wimdu is decreased with every visit/booking. I am starting to compare it to the relationships we have to, lets say a computer’s operating system, or a mobile phone provider. At some point in time we have personally invested so much time and effort into building the relationship (learning all the ins and outs of a website/product, purchasing/reviewing products, building wishlists etc.) that we become reluctant to change. Change is just too damn difficult (pardon my French), even if it can sometimes benefit us in the end. Do you feel that this is how you experience, for example, your preference for Airbnb? Matthew

  3. CookieMonster says:

    I use Airbnb all the time as well. I like how you pay Airbnb up front and if the reservation is accepted, you’re charged, but Airbnb does not release the money to hosts until 24 hours after the reservaton starts. If the host doesn’t want to accept the reservation, there’s not charge either.

  4. Concerning payment, WIMDU now functions the same way Airbnb does, that is the total amount (rental + service charge) was taken off my account 2 days after I made the reservation. The owner of the chalet will not get the rental money until 24 hours after my arrival. That’s a very large sum of money that WIMDU has at their disposal for no cost for several months. They can do a lot of investing with that.

    But my major complaint about WIMDU is that it turns out the owners of the chalet I want to rent for a week have not signed a contract with WIMDU, don’t know who they are and don’t want to work with them. In other words, WIMDU WAS ADVERTISING FOR THEM WITHOUT THEIR PERMISSION OR KNOWLEDGE. The owners asked WIMDU to cancel the reservation. No reply from WIMDU. Meanwhile, as indicated, the entire amount (rental + service charge, approximately 300 Euros) has been taken off my account. The owners refuse the WIMDU reservation and ask me to reserve directly with them or through another web site. I have written WIMDU 3 times and get NO REPLY. The owners work with several other sites and THE TOTAL COST IS LOWER ON ALL THESE OTHER SITES (as the owners pay the 10% service charge rather than the renter) AND ONLY THE SERVICE CHARGE IS TAKEN OFF YOUR ACCOUNT AT THE TIME OF RESERVATION, NOT THE ENTIRE COST.

    So, for the time being, I have paid for the rental TWICE and remain (somewhat) hopeful that WIMDU will respond to my request for cancellation which was made through the correct channel. Next week I will be seeing a consumer organisation to ask them to contact WIMDU.

  5. Shizzle says:

    I Like http://www.vaway.com also its a good site to list on because they partner with Airbnb, kayak and travelocity to list my property.

  6. bluesmama says:

    Ripped me off two months before I even got to San Francisco. I had (stupidly) entered my credit card number to (I thought) hold a vacation rental. They charged my card RIGHT AWAY, so, alarmed, I cancelled the reservation somewhere around 24 hours after making it. Now Wimdu claims they are keeping 12% of the total as a fee.
    They claimed it was in the “terms and conditions”, which is about 12 pages long. NO where on the “complete booking” page does it say, if it turns out you don’t like the host, even if you cancel we’re keeping 12%. Now I’m out $134. I’ve begged and begged via email and they continously politely tell me they’re keeping my money.
    Here in the US hotels do indeed take your credit card number but don’t charge you until you actually show up.

  7. Some surveys will disclose a new product and have you not to
    share this information. Enjoy searching for paid survey sites.
    2) If you want to read, you will find companies you can find through bidding sites like o
    - Desk, Elance, etc.

  8. Stef says:

    Wimdu is a clone of Airbnb except the service fee of the site are much higher in Wimdu! This is not from 6% to 12% + 3% like Airbnb. In Wimdu you pay 24% for the site service…
    You can compare apartments which are on both sites.

    Wimdu est un clone de Airbnb sauf que les frais de service du site sont bien plus élevé chez Wimdu! Ce n’est pas de 6% à 12% + 3% comme chez Airbnb. Sur Wimdu c’est 24% de frais de service pour le site…
    Comparez les apartements qui sont sur les deux sites.

  9. Stef says:

    Correction of my previous post, it is 21% in Wimdu. It’s still more expensive than Airbnb.

    Rectification de mon message précedent, c’est 21% chez Wimdu. Bon cela reste plus chere que Airbnb.

    • oui c’est exactement cela plus de 20% de commission et le service après-vente nul pour wimdu, pas de contrat entre les mains (à faire soi-même du côté propriétaire, pas de service de caution non plus, helpdesk qui répond 2 jours en retard pour juste émettre un ticket que notre problème va bientôt être traité… de qui se moque-t-on. Malheureusement le vacancier ne connaît pas tout cela en choisissant la plateforme wimdu plutôt que airbnb mais en tous les cas du côté propriétaire, il n’y pas de comparaison possible : airbnb est beaucoup moins cher et beaucoup efficace

  10. Started listing on Wimdu after AirBnB was subjected to new tax demands by our lovely government. If ever I’ve disliked an organization, this would be it.

    1. CONTROLLING

    If AirBnB is that cool older sibling who trusts you to know what you are doing, Wimdu is that nagging one who is consistently looking over your shoulder to make sure you’re responding to every single inquiry and punishing you with leeeeengthy politely-written essays – oops, emails – and the deactivation of your listings if you’re 1/ not, or 2/ declining, even if a male is requesting a stay in a space with the heading “all-female.”

    I know when I receive a Wimdu message. My inbox has this cool way of notifying me with a message with the heading “New message on Wimdu.” I think it’s called an “e-mail.” It even contains a snippet of the incoming message if I click on it, which I do to see if it’s another male hoping to get lucky and score a stay in the female room before bothering with logging in to respond and whatnot. I don’t need big brother Wimdu to get all up in my face with a 3-page letter every time I don’t respond or decline someone. Either I’m full or am considering long-term residency applicants so don’t have the space for short-term stays at the moment. Stop needing to look over my shoulder at every turn.

    2. INCOMPETENT

    Twice now, I’ve had calendar blocking issues. On dates that had never been touched on Wimdu. Marked as unavailable. Thing with dates marked as such is that you can’t click on them on the otherwise interactive calendar. So you can’t change them to, well, available. Tried five times with refreshing plus five more on the drop-down calendar submission form below it to no avail – the screen didn’t even register that any information had been submitted for review or otherwise. So I relinquished to contacting support.

    Two days. It took two days to clarify that the calendar was not interactive and, hence, could not be changed on the host computer end. Wasn’t the description above clear enough? You tell me, readers, you tell me, because Wimdu support obviously wasn’t getting it. And after the two days:

    “Please give me the exact dates, price and minimum stay and I can [change availability] for you.” Dates not available had been specified in the email you just responded to. Dollar amount listed on all available dates on the calendar. Minimum stay, none, as it has always been in my default settings.

    Did I mention that I had two separate guest reservations on standby waiting for this process to be completed, which I made clear several times in communication? Both ended up booking with us via a different service provider. Did I also mention that I’d had a similar issue on AirBnB last year but it was a glitch from my calendar updates rather than an “out of nowhere” unavailability, and AirBnB took care of it as soon as I contacted them?

Trackbacks

  1. [...] Remote User Testing: Airbnb vs Wimdu http://t.co/qsKOkiDT via @actualinsights #ux #cro #usability Link – Trackbacks Posted in User experience (UX) | Permalink. ← How content density [...]

  2. [...] It has been a long time since I first published a blog post on remote user testing, 3 months to be exact. In that post, I compared two social traveling websites, AirBnB vs Wimdu. [...]

  3. [...] want to create a test panel, then you can always consider remote mobile user testing. Like regular remote user testing the method will allow you to conduct mobile user tests without recruiting your own panel or [...]

  4. [...] exactly is Remote Usability Testing and how can your business benefit from it? How did Airbnb and Wimdu benefit from remote usability [...]

Leave a Reply

FREE Course!

Get FREE access to my User Feedback Fundamentals course at Udemy, a $29 value. Just sign-up to my newsletter.
Unsubscribe at anytime and keep access to the course.

Nope, we don't deal in spam, not even the canned sort.
x