User and Usability Testing Questions

What kind of User Testing does Userlytics do?

Both unmoderated as well as moderated (“Live Conversations”). Userlytics enables you to usability test the user experience of your websites and competitor websites, released and unreleased Mobile Apps, advertisements, and social media content, at all stages of the design and development process.
We capture the interaction, the verbal commentary of the participants and of the moderator if any, AND the participants themselves and their context through their webcam, in a picture-in-pìcture format, so you can be sure they are your target Persona.

How can I User Test on Mobile Devices?

We have native applications for Android & iOS for mobile testing of apps, prototypes, and mobile web browsing. You can also leverage our online/offline capability, allowing you to send participants to a physical store and compare the e-commerce experience with the in-person equivalent! We also offer an alternative methodology that allows the user to natively test the UI on their mobile device while viewing instructions and answering questions on their desktop/laptop computer.

Who benefits from User Testing?

Brand managers, Product Managers, Advertising and Marketing Agencies, User Experience professionals, Marketers, Mobile App and Website Designers and Developers all benefit from the capability to conduct remote usability testing on their prototypes, websites, mobile apps and digital assets within the context of participants’ homes and offices.

Do you have any recommendations or suggestions about how to create a good test script?

Yes, one of our senior UX Consultants hosted an in-depth webinar about this topic. You can see it here

What changes do I have to make on my website or mobile App to conduct user testing?

None. You do NOT need to add snippets of code anywhere on your website. You do NOT need to insert an SDK into your Mobile App. You can conduct remote usability testing at any stage of the design and development and launch of your website, mobile app or other digital asset, with no type of intrusive modifications to your proprietary code. Full compatibility with all user experience design, prototyping and prototype delivery platforms

How does remote unmoderated user testing work?

There are a number of different ways you can leverage Userlytics for remote usability testing:

  • You can use your own participants, or have them provided by a 3rd party user experience respondent panel, or ask Userlytics to recruit and incentivize the participants, or leverage our over 600,000  member proprietary panel. You can configure participant redirects to specific landing pages for completes, abandons, etc. Respondents download a small app that checks if they have the necessary equipment (e.g.: webcams, when applicable) and that takes them through the tasks and questions you have defined
  • You can test a website, a wireframe, a set of static images, a video, a PowerPoint or Google Slides presentation, or any other type of digital asset accessible from a computer, including desktop applications.
  • You can test a mobile App that is available on the App Store or Google Play, or an unreleased App using TestFlight or similar, on both smartphones and tablets.
  • You can simultaneously view the screen interactions, listen to what the participant hears, and see the participants’ face and surroundings in a “Picture-in-Picture” (PiP) format video.
  • You can use our customizable templates for all types of use cases, work flows and scenarios for easy task and exit survey question formulation.
  • You can leverage skipping logic in the design of your unmoderated usability testing script, as well as in the design of a custom screener survey
  • You can create an invitation link for participants that forces them to register their name and email, or one that incorporates a unique identifier in the link (e.g.: for use with 3rd party panel providers) which you can send or post anywhere.
  • You can use a customizable respondent feedback invitation widget that you can add on any website page.
  • You can access the full video and survey question results, share access to specific projects with clients and colleagues, and download videos, quantitative metrics and survey results for your own reporting format.
  • You can access a hyperlinked “Video Highlights” section with key comments and events linked to specific videos and timeline moments.
  • You can download specific video clips with or without captions of annotation titles.
  • You can automatically create “Highlight Reels” of clips from different participants and/or studies linked by a common theme through the use of tags.
  • You can request an AI generated audio transcript of the videos with time stamped text for easy searching of videos and links to specific timeline moments.
  • You can request an analysis and executive summary, crafted by one of our Professional UX Consultants, of results with detailed recommendations for each issue uncovered.

How does Live Conversations (Moderated Usability Testing) work?

Very easy and very similar to an unmoderated user experience test, the main difference is you have a scheduling tool where you define the availability of your moderator (unless you wish to have Userlytics moderate). Invited participants then confirm the specific date/time they will do the test and both the moderator and the participant receive reminders prior to the selected date/time.
One very cool and unique aspect of the Userlytics moderated testing platform (“Live Conversations”) is that participants do NOT need to download ANY application, extension or plug-in; the system is completely “in-browser” and 100% HTML based, so participant onboarding is extremely seamless and can take place within corporate firewalls!

Where are your Servers located, and where is Data Stored & Processed?

Our cloud hosting providers are based in the US, Amazon Web Services (Virginia) and Linode (Texas).

Are you GDPR Compliant? CCPA? ICO?

Yes, we are fully GDPR compliant, as well as registered in the ICO of the UK and fully CCPA compliant.

Are you certified under the EU-US Privacy Shield?

Yes. You can see our certification here

What languages do you support?

You can write your test instructions and questions in any language or alphabet.
In addition, our system supports the automatic localization of our recorder UI and participant landing pages into the following languages:

English
Spanish (Spain)
Spanish (Latam)
French
German
Italian
Dutch
Norwegian
Chinese (Traditional)
Chinese (Simplified)
Japanese
Korean

We can add additional languages upon request with a minimum 2 week lead time.

In addition, our AI based automatic transcription of video sessions is available in the following languages:
English (US)
English (UK)
English (Australia)
French
German
Spanish (Spain)
Spanish (Latam)
Dutch
Portuguese (Brazil)
Italian

Can I have Observers in my Moderated Usability Testing (“Live Conversations”)

Yes, you can have up to 5 observers in every session, who can watch and listen to the moderator and the participant as they go through the session

How does Mobile App remote user testing work?

We leverage various methodologies. You can use our native iOS and Android mobile apps, for recording the screen of the mobile device, the audio and the webcam view (“picture-in-picture”). You can even leverage this methodology to record offline user experiences and compare your offline with your online channels. An alternative method that is sometimes useful is through projecting the interacting screen of the smartphone or tablet (Android or iOS) onto the participant computer screen, and recording the screen and the participant through their webcam.
Either way, since you are able to see participants, not just their screen and audio, you can ensure respondents match your target Persona.
See the following blog post on methodologies for remote usability testing on mobile devices.

What kind of instructions should I use for remote Usability Testing?

You can direct participants to perform any goal or task or question, in any order, including leveraging branching logic, to help you optimize the user experience of your websites, mobile apps, online advertising and content, and discover the reasons behind your analytics and funnel conversion results. You can also leverage pre-formatted system usability scale questions, and net promoter score questions, and have results automatically tabulated (as well as time on task, success/failure, SEQ, surveys, etc.)
We provide a set of customizable templates.
You might ask about:

  • Site navigation
  • Mobile App downloads and interactions
  • The understanding of an advertising message
  • How do customers find your website or App or those of your competition
  • The process for satisfying a need or objective
  • Ease of use and intuitiveness
  • Design, video, colors, layout
  • Specific processes such as: registration, login, check out, utilities, etc.
  • If they would use your site or recommend it to a friend
  • Other user experience & usability questions specific to your user testing objectives
  • Comparisons with competitors, production and prototype assets, offline and online media

What is prototype usability testing?

Prototype user testing allows you to easily and quickly identify issues with the user interface and user experience of your prototype, mobile app, ad concept, storyboard, or wireframe during the early stages of design or development, and prior to spending significant sums of money on multivariate testing. Early detection of issues provides the opportunity to make necessary course corrections early in the design and development process, when change is less costly.
Early user experience testing of websites and mobile apps gives a better understanding of what features your users will appreciate, how and why they react to your message, and limits problems identified after your next product launch, campaign or upgrade. They also answer the question of “Why” participants engage in the behaviour you detect through analytics.

How does user testing of an unreleased Mobile App prototype work?

We can work with any platform you use, whether low fidelity prototypes hosted on Invision, Adobe XD, Figma, Proto, or similar, or unreleased high fidelity prototypes on the App Store or Google Play leveraging TestFlight or similar. We project manage the whole process of setting up the targeted participant profiles and Personas with the software required for conducting the mobile app usability test.

What kind of files can be uploaded for remote user testing?

You can user test any type of file (JPG, .GIF, .PNG, PDF, HTML, JS, CSS, etc), or Mobile App (Android, iOS, Free, Paid, released, unreleased…)
If you have a proprietary file format that you are unsure of, please contact us and we will be happy to review your needs and work with you to make your remote usability test project a success.

Is remote user testing secure?

We have implemented commercially reasonable technical and organizational measures designed to secure uploaded files from accidental loss and from unauthorized access, use, alteration or disclosure. Our participants have signed up to non disclosure clauses, and we can project manage a customized NDA from usability test respondents as part of the test workflow. Another alternative or complement is to set up a test on a password protected client site, with a temporary password provided to user test participants.

How can I conduct remote usability testing of purchases?

For privacy reasons, user experience test panel participants are normally prohibited from completing purchase processes with their own personal credit card information. As an alternative, we can facilitate the use of dummy credit card information for such purposes, including the use of an application that delivers unique codes to each participant.
However, if a real purchase is necessary, we can project manage participant reimbursement.

What if I make a mistake setting up a remote usability test?

You can run a “preview” of the test flow/journey/script, and then edit your directions, goals, tasks and questions or demographic selections until you are satisfied. You even change the test script or maximum session time or number of respondents in real time after your test has been launched and is live.

How quickly are results received with unmoderated remote user testing?

If you are using your own usability test participants, it depends on how quickly they respond to your invitation. Our platform processing time takes between 1 to 5 minutes, so you can receive results extremely quickly.
When using the Userlytics Panel, with broad based demographics, after a thorough QA review of every user test video, results are available in as little as 24 hours, sometimes within hours.
For narrowly based demographic targets, it depends on the filters and behavioural screener used to set up the project, and thus how narrowly defined the target Persona is.

For how long can I access usability test results?

You may access your remote user testing results at any time within 365 days after they have been delivered. You can also download the user experience test video results anytime within the 365 day period and keep and edit and share as needed.

How do I conduct custom usability tests?

Please contact us to learn more.

How much does a Usability Test cost?

Pricing for remote user testing depends on the following factors:

    • Usability test participant demographic profile – Target Persona
    • Maximum user experience test session time
    • Number of user experience test participant completes desired
    • Whether usability test design – scripting is required
    • Whether user test review, analysis and recommendations are required

Pricing can be monthly or annual-licensing based, or Ad-Hoc, on a per usability test project basis.
Contact us for information on how we can help you achieve your unmoderated user experience testing goals.

When do I pay for the test credits?

If you have a DIY account, then you would pay for the credits when you go to launch the test, after you have fully setup the test.
If you have a Custom Project or Enterprise Plan, then you would purchase the credits upfront and they would be available in your account for when you are ready to run your upcoming test(s).

I reside outside of the US. In which currency do I pay?

We can manage payments in either US dollars or Euros, depending on your location, using a Credit Card. For Custom Projects or Annual Enterprise plans, wire transfers or PayPal may also be used.

How long is a typical unmoderated usability test session?

We can conduct user testing with the maximum session time set to 5, 10, 15, 20, 30, 60 minutes or any maximum duration you choose.

However, we recommend designing unmoderated usability test sessions for approximately 20 min session times, with the maximum session time set to 30 minutes to catch outliers. For moderated usability test sessions (“Live Conversations”) a default session time of 60 minutes is used.

What are the browser requirements to use and access all functions of Userlytics?

We recommend using Microsoft Internet Explorer 8 or higher, Mozilla Firefox 4.x or higher, Apple Safari 4.x or higher, and Google Chrome 4.x or higher.

Can Userlytics manage user experience studies with persons with disabilities?

We are working on making the platform 100% accessible for testing of participants with disabilities.

If you are thinking of launching this type of study, please consult us first at operations@userlytics.com so we can assist you.

Our Global Participant Panel

Can I use participants based outside of the United States?

Yes, you may invite participants from our Global Participant Panel, or use a 3rd party panel provider, or invite participants yourself by sending them invitation links or posting them on your website or social media sites. Contact us for a customized recruitment quotation from anywhere in the world, at any incidence level.

How many participants should I use for a Usability Test?

  • Experts generally agree that using 5-15 participants per qualitative user test iteration will uncover 80% of usability issues, and is the most efficient manner to conduct agile user experience testing. We have found that it is more efficient and cost-effective to conduct numerous small sample studies iteratively, and then use multivariate testing at the end of the process with larger samples to conduct final optimization. Please refer to the following reference materials:

What kind of user test participant profiles can I select?

You can filter our remote user testing panel for education, age, gender, employment status, household income, education, civil status and similar. In addition, you may define a behavioural screener to select usability test participants based on highly specific demographic profile criteria, both B2B and B2C, from around the world to target your specific target Person for your remote usability test.

Where do usability test panel participants come from?

The Userlytics user experience test panel comprises crowd-sourced respondents from all backgrounds. However, most of our clients ask us to custom recruit highly specific demographic participants from around the world

Who are Userlytics’ highest rated user testing participants?

Our remote user testing participants are rated by our clients and we use those ratings as a selection basis for remote user test projects. In some cases our clients ask us to recruit first time usability test participants, in other cases we are asked to provide the same participants for a set of repeat sessions.

Can I get a result I am unhappy with replaced?

  • Yes, you can! If you rate a result as a “1 Star”, within 7 days of receiving the result (or simply by letting operations@userlytics.com know), we will conduct a second in-depth review. We will replace the result if:
    • Participant was not following a “Think Aloud” protocol
    • Participant was not making a good faith effort to follow instructions
    • There were technical issues with the recording

If however, the instructions were confusing (and you had not taken advantage of our optional Professional Services and had us design the test script), or, there was a bug in the test asset you directed participants to, then participants will be paid their incentives, and the credit(s) used will not be replaced (We can however still delete the result from your account if you wish)

Can I use the same participant invitation link for multiple sessions?

Yes. Participants can reuse the invitation link multiple times. It will generate a different code each time, which can be used each time with the Userlytics Recorder

Our Global Participant Panel

Our Features

Our Features

What Unmoderated Usability Testing Features are available?

    • Capture of computer or mobile device screen interactions;
    • Capture of participant audio commentary;
    • Capture of participant and context;
    • Integrated “Picture-in-Picture” (PiP) video of participant, verbal “think aloud” commentary, and screen device interactions;
    • Skipping logic;
    • Configurable testing methodology (webcam, screen, audio recording + metrics, or screen, audio recording + metrics or audio recording + metrics)
    • Personal Identifiable Information (PII) protection;
    • User testing of production sites, mobile apps and desktop applications;
    • UX testing of prototypes, wireframes, PNG sketches, unreleased mobile Apps;
    • Usability testing of Android or iOS, free or paid, production or prototype mobile apps;
    • In-linetime stamped hyperlinked annotations;
    • Downloadable videos (MP4), as well as downloadable CSV files  (time on task, success/failure rates, System Usability Scale, Net promoter Score, demographic  specs and other quantitative data);
    • Downloadable or shareable video clips with annotation subtitling;
    • Downloadable or shareable highlight reels with annotation subtitling;
    • AI generated testing session audio transcriptions;
    • Usability Test Design, Scenario – Script preparation;
    • User Experience Test review and analysis and executive Summary report with recommendations;
    • User test respondent recruitment and incentives with a global panel of over 600,000;
    • 3rd party user testing respondent panel integration;
    • White labelling – private labelling of dashboard results;
    • Project management.

What Operating systems and Prototyping platforms is Userlytics compatible with?

Mac, Windows, iOS, Android… We cover all the main operating systems. And you can use any prototyping platform such as Invision, Adobe XD, Proto, Axure, Figma, Justinmind…

Can I annotate my remote user testing videos?

You can annotate each user experience test video with key events and comments, and then share an aggregate list of annotations with each one hyperlinked to the relevant video and timeline moment.

What does picture-in-picture (PiP) recording include?

It refers to a recording of the webcam view of the usability test participant, together with the recording of the screen the participant is interacting with, as well as the audio “think aloud” commentary of the user testing participant and the UX moderator, if any.

What does PII Protection mean?

This unique feature enables the blocking of screen recording during sensitive moments in the test journey, to protect personal identifiable information, while continuing to record the webcam view of the participant and their “think aloud” commentary.

Can I create a Highlight Reel?

Yes. You can create and download or share individual video clips, even captioning them with annotation subtitles, as well as automatically create and download or share a highlight reel of clips from different participants and/or studies grouped by a common theme through the use of tags.

Can I have my picture-in-picture video recording sessions transcribed?

Yes. You can have them transcribed through the use of an AI system, search for key terms, and click on the results to go directly to the relevant video and timeline moment

Is it possible to do an unmoderated usability test with children?

Yes it is possible, however, a test with children requires certain protocols to be met in order to ensure all guidelines and laws are being followed. We have created a guide for this purpose here.