User and Usability Testing Questions
Both unmoderated as well as moderated (“Live Conversations”). Userlytics enables you to usability test the user experience of your websites and competitor websites, released and unreleased Mobile Apps, advertisements, and social media content, at all stages of the design and development process.
We capture the interaction, the verbal commentary of the participants and of the moderator if any, AND the participants themselves and their context through their webcam, in a picture-in-pìcture format, so you can be sure they are your target Persona.
We have native applications for Android & iOS for mobile testing of apps, prototypes, and mobile web browsing. You can also leverage our online/offline capability, allowing you to send participants to a physical store and compare the e-commerce experience with the in-person equivalent! We also offer an alternative methodology that allows the user to natively test the UI on their mobile device while viewing instructions and answering questions on their desktop/laptop computer.
Brand managers, Product Managers, Advertising and Marketing Agencies, User Experience professionals, Marketers, Mobile App and Website Designers and Developers all benefit from the capability to conduct remote usability testing on their prototypes, websites, mobile apps and digital assets within the context of participants’ homes and offices.
Yes, one of our senior UX Consultants hosted an in-depth webinar about this topic. You can see it here
None. You do NOT need to add snippets of code anywhere on your website. You do NOT need to insert an SDK into your Mobile App. You can conduct remote usability testing at any stage of the design and development and launch of your website, mobile app or other digital asset, with no type of intrusive modifications to your proprietary code. Full compatibility with all user experience design, prototyping and prototype delivery platforms
There are a number of different ways you can leverage Userlytics for remote usability testing:
- You can use your own participants, or have them provided by a 3rd party user experience respondent panel, or ask Userlytics to recruit and incentivize the participants, or leverage our over 600,000 member proprietary panel. You can configure participant redirects to specific landing pages for completes, abandons, etc. Respondents download a small app that checks if they have the necessary equipment (e.g.: webcams, when applicable) and that takes them through the tasks and questions you have defined
- You can test a website, a wireframe, a set of static images, a video, a PowerPoint or Google Slides presentation, or any other type of digital asset accessible from a computer, including desktop applications.
- You can test a mobile App that is available on the App Store or Google Play, or an unreleased App using TestFlight or similar, on both smartphones and tablets.
- You can simultaneously view the screen interactions, listen to what the participant hears, and see the participants’ face and surroundings in a “Picture-in-Picture” (PiP) format video.
- You can use our customizable templates for all types of use cases, work flows and scenarios for easy task and exit survey question formulation.
- You can leverage skipping logic in the design of your unmoderated usability testing script, as well as in the design of a custom screener survey
- You can create an invitation link for participants that forces them to register their name and email, or one that incorporates a unique identifier in the link (e.g.: for use with 3rd party panel providers) which you can send or post anywhere.
- You can use a customizable respondent feedback invitation widget that you can add on any website page.
- You can access the full video and survey question results, share access to specific projects with clients and colleagues, and download videos, quantitative metrics and survey results for your own reporting format.
- You can access a hyperlinked “Video Highlights” section with key comments and events linked to specific videos and timeline moments.
- You can download specific video clips with or without captions of annotation titles.
- You can automatically create “Highlight Reels” of clips from different participants and/or studies linked by a common theme through the use of tags.
- You can request an AI generated audio transcript of the videos with time stamped text for easy searching of videos and links to specific timeline moments.
- You can request an analysis and executive summary, crafted by one of our Professional UX Consultants, of results with detailed recommendations for each issue uncovered.
Very easy and very similar to an unmoderated user experience test, the main difference is you have a scheduling tool where you define the availability of your moderator (unless you wish to have Userlytics moderate). Invited participants then confirm the specific date/time they will do the test and both the moderator and the participant receive reminders prior to the selected date/time.
One very cool and unique aspect of the Userlytics moderated testing platform (“Live Conversations”) is that participants do NOT need to download ANY application, extension or plug-in; the system is completely “in-browser” and 100% HTML based, so participant onboarding is extremely seamless and can take place within corporate firewalls!
Our cloud hosting providers are based in the US, Amazon Web Services (Virginia) and Linode (Texas).
Yes, we are fully GDPR compliant, as well as registered in the ICO of the UK and fully CCPA compliant.
Yes. You can see our certification here
You can write your test instructions and questions in any language or alphabet. In addition, for formative studies, you can set the language of the recorder UI and participant landing pages into any of the following languages:
We can add additional languages upon request with a minimum 2 week lead time.
In addition, our AI based automatic transcription of video sessions, provided by Amazon Alexa, is available in the following languages:
Modern Standard Arabic
Yes, you can have up to 5 observers in every session, who can watch and listen to the moderator and the participant as they go through the session
You can direct participants to perform any goal or task or question, in any order, including leveraging branching logic, to help you optimize the user experience of your websites, mobile apps, online advertising and content, and discover the reasons behind your analytics and funnel conversion results. You can also leverage pre-formatted system usability scale questions, and net promoter score questions, and have results automatically tabulated (as well as time on task, success/failure, SEQ, surveys, etc.)
We provide a set of customizable templates.
You might ask about:
- Site navigation
- Mobile App downloads and interactions
- The understanding of an advertising message
- How do customers find your website or App or those of your competition
- The process for satisfying a need or objective
- Ease of use and intuitiveness
- Design, video, colors, layout
- Specific processes such as: registration, login, check out, utilities, etc.
- If they would use your site or recommend it to a friend
- Other user experience & usability questions specific to your user testing objectives
- Comparisons with competitors, production and prototype assets, offline and online media
Prototype user testing allows you to easily and quickly identify issues with the user interface and user experience of your prototype, mobile app, ad concept, storyboard, or wireframe during the early stages of design or development, and prior to spending significant sums of money on multivariate testing. Early detection of issues provides the opportunity to make necessary course corrections early in the design and development process, when change is less costly.
Early user experience testing of websites and mobile apps gives a better understanding of what features your users will appreciate, how and why they react to your message, and limits problems identified after your next product launch, campaign or upgrade. They also answer the question of “Why” participants engage in the behaviour you detect through analytics.
We can work with any platform you use, whether low fidelity prototypes hosted on Invision, Adobe XD, Figma, Proto, or similar, or unreleased high fidelity prototypes on the App Store or Google Play leveraging TestFlight or similar. Userlytics is compatible with any prototype platforms that can be tested through a link.
We project manage the entire process of setting up targeted participant profiles and Personas with the software required for conducting the mobile app usability test.
You can user test any type of file (JPG, .GIF, .PNG, PDF, HTML, JS, CSS, etc), or Mobile App (Android, iOS, Free, Paid, released, unreleased…)
If you have a proprietary file format that you are unsure of, please contact us and we will be happy to review your needs and work with you to make your remote usability test project a success.
We have implemented commercially reasonable technical and organizational measures designed to secure uploaded files from accidental loss and from unauthorized access, use, alteration or disclosure. Our participants have signed up to non disclosure clauses, and we can project manage a customized NDA from usability test respondents as part of the test workflow. Another alternative or complement is to set up a test on a password protected client site, with a temporary password provided to user test participants.
For privacy reasons, user experience test panel participants are normally prohibited from completing purchase processes with their own personal credit card information. As an alternative, we can facilitate the use of dummy credit card information for such purposes, including the use of an application that delivers unique codes to each participant.
However, if a real purchase is necessary, we can project manage participant reimbursement.
You can run a “preview” of the test flow/journey/script, and then edit your directions, goals, tasks and questions or demographic selections until you are satisfied. You even change the test script or maximum session time or number of respondents in real time after your test has been launched and is live.
If you are using your own usability test participants, it depends on how quickly they respond to your invitation. Our platform processing time takes between 1 to 5 minutes, so you can receive results extremely quickly.
When using the Userlytics Panel, with broad based demographics, after a thorough QA review of every user test video, results are available in as little as 24 hours, sometimes within hours.
For narrowly based demographic targets, it depends on the filters and behavioural screener used to set up the project, and thus how narrowly defined the target Persona is.
You may access your remote user testing results at any time within 365 days after they have been delivered. You can also download the user experience test video results anytime within the 365 day period and keep and edit and share as needed.
Please contact us to learn more.
Pricing for remote user testing depends on the following factors:
- Usability test participant demographic profile – Target Persona
- Maximum user experience test session time
- Number of user experience test participant completes desired
- Whether usability test design – scripting is required
- Whether user test review, analysis and recommendations are required
Pricing can be monthly or annual-licensing based, or Ad-Hoc, on a per usability test project basis.
Contact us for information on how we can help you achieve your unmoderated user experience testing goals.
If you have a DIY account, then you would pay for the credits when you go to launch the test, after you have fully setup the test.
If you have a Custom Project or Enterprise Plan, then you would purchase the credits upfront and they would be available in your account for when you are ready to run your upcoming test(s).
We can manage payments in either US dollars or Euros, depending on your location, using a Credit Card. For Custom Projects or Annual Enterprise plans, wire transfers or PayPal may also be used.
We can conduct user testing with the maximum session time set to 5, 10, 15, 20, 30, 60 minutes or any maximum duration you choose.
However, we recommend designing unmoderated usability test sessions for approximately 20 min session times, with the maximum session time set to 30 minutes to catch outliers. For moderated usability test sessions (“Live Conversations”) a default session time of 60 minutes is used.
We recommend using Microsoft Internet Explorer 8 or higher, Mozilla Firefox 4.x or higher, Apple Safari 4.x or higher, and Google Chrome 4.x or higher.
It means that the test has been completed (for now). We will keep our testers updated if there is an opening of new slots. When you receive an invitation to take one of our tests, please make sure to respond as soon as possible.
We are working on making the platform 100% accessible for testing of participants with disabilities.
If you are thinking of launching this type of study, please consult us first at email@example.com so we can assist you.
The “Starting Point URL” is the website, app or prototype your participant will immediately be pointed to when they begin a user experience study. The decision to include a Starting Point URL depends on whether you are conducting an unmoderated test or a moderated test, and what your test goals are.
If you are conducting an unmoderated study and would like to immediately point your participant to a URL at the beginning of the test, you will need to fill the starting point URL. When you place your URL in this field, the user will be sent to that location at the beginning of the test. However, if you would prefer to ask a few questions before directing the user to a site or app, toggle “Define Destination URL(s) inside the TestScript.” This will allow you to place your URL at a later task.
If you are conducting a moderated study, you do not need Starting Point URL. By default,“Define Destination URL(s) inside the TestScript” option will be toggled if you select “Live Conversation” as your test type. This is because our updated Live Conversations recorder allows you to direct your participant to your website, app, or prototype at any point during the session by clicking the “Share URL With Participant” button.
You can use any combination of video, audio and screen recording for tests taken on mobile devices with one exception: If you are testing a website on iOS devices, you cannot record participants via video due to technical limitations; only audio and screen recording is permitted for website tests taken on an iOS device.
If video recording on iOS devices is vital to the success of your study, please contact our operations team at firstname.lastname@example.org, and we will assist you.
Our Global Participant Panel
Yes, you may invite participants from our Global Participant Panel, or use a 3rd party panel provider, or invite participants yourself by sending them invitation links or posting them on your website or social media sites. Contact us for a customized recruitment quotation from anywhere in the world, at any incidence level.
- Experts generally agree that using 5-15 participants per qualitative user test iteration will uncover 80% of usability issues, and is the most efficient manner to conduct agile user experience testing. We have found that it is more efficient and cost-effective to conduct numerous small sample studies iteratively, and then use multivariate testing at the end of the process with larger samples to conduct final optimization. Please refer to the following reference materials:
You can filter our remote user testing panel for education, age, gender, employment status, household income, education, civil status and similar. In addition, you may define a behavioural screener to select usability test participants based on highly specific demographic profile criteria, both B2B and B2C, from around the world to target your specific target Person for your remote usability test.
The Userlytics user experience test panel comprises crowd-sourced respondents from all backgrounds. However, most of our clients ask us to custom recruit highly specific demographic participants from around the world
Our remote user testing participants are rated by our clients and we use those ratings as a selection basis for remote user test projects. In some cases our clients ask us to recruit first time usability test participants, in other cases we are asked to provide the same participants for a set of repeat sessions.
- Yes, you can! If you rate a result as a “1 Star”, within 7 days of receiving the result (or simply by letting email@example.com know), we will conduct a second in-depth review. We will replace the result if:
- Participant was not following a “Think Aloud” protocol
- Participant was not making a good faith effort to follow instructions
- There were technical issues with the recording
If however, the instructions were confusing (and you had not taken advantage of our optional Professional Services and had us design the test script), or, there was a bug in the test asset you directed participants to, then participants will be paid their incentives, and the credit(s) used will not be replaced (We can however still delete the result from your account if you wish)
Yes. Participants can reuse the invitation link multiple times. It will generate a different code each time, which can be used each time with the Userlytics Recorder
- Capture of computer or mobile device screen interactions;
- Capture of participant audio commentary;
- Capture of participant and context;
- Integrated “Picture-in-Picture” (PiP) video of participant, verbal “think aloud” commentary, and screen device interactions;
- Skipping logic;
- Configurable testing methodology (webcam, screen, audio recording + metrics, or screen, audio recording + metrics or audio recording + metrics)
- Personal Identifiable Information (PII) protection;
- User testing of production sites, mobile apps and desktop applications;
- UX testing of prototypes, wireframes, PNG sketches, unreleased mobile Apps;
- Usability testing of Android or iOS, free or paid, production or prototype mobile apps;
- In-linetime stamped hyperlinked annotations;
- Downloadable videos (MP4), as well as downloadable CSV files (time on task, success/failure rates, System Usability Scale, Net promoter Score, demographic specs and other quantitative data);
- Downloadable or shareable video clips with annotation subtitling;
- Downloadable or shareable highlight reels with annotation subtitling;
- AI generated testing session audio transcriptions;
- Usability Test Design, Scenario – Script preparation;
- User Experience Test review and analysis and executive Summary report with recommendations;
- User test respondent recruitment and incentives with a global panel of over 600,000;
- 3rd party user testing respondent panel integration;
- White labelling – private labelling of dashboard results;
- Project management.
We are compatible will all major operating systems, including Mac, Windows, iOS, and Android.
Userlytics is also compatible with most major prototyping platforms, including Figma, Invision, Proto.io, Axure, Marvel App, Justinmind, and Adobe Xd. In addition to these platforms, we are compatible with any prototyping platforms that can be tested through a link.
You can annotate each user experience test video with key events and comments, and then share an aggregate list of annotations with each one hyperlinked to the relevant video and timeline moment. Learn more about how to create annotations here.
It refers to a recording of the webcam view of the usability test participant, together with the recording of the screen the participant is interacting with, as well as the audio “think aloud” commentary of the user testing participant and the UX moderator, if any.
This unique feature enables the blocking of screen recording during sensitive moments in the test journey, to protect personal identifiable information, while continuing to record the webcam view of the participant and their “think aloud” commentary.
Yes. You can create and download or share individual video clips, even captioning them with annotation subtitles, as well as automatically create and download or share a highlight reel of clips from different participants and/or studies grouped by a common theme through the use of tags. Learn how to create highlight reels here.
Yes. You can have them transcribed through the use of our automated video transcription capability, provided by Amazon Alexa. Our advanced transcriptions allow you to search for key terms spoken aloud by participants, and go directly to the relevant video and timeline moments. Learn how to request session transcriptions here.
Yes it is possible, however, a test with children requires certain protocols to be met in order to ensure all guidelines and laws are being followed. We have created a guide for this purpose here.
Yes, you can conduct quantitative tests with participants across multiple types of devices, not just one. In the Userlytics study builder, when asked to designate your participant’s device, simply select the option that says “Multi-Device.” Read more about how to do this here.
Note – if you do not see this option and would like to conduct quantitative tests with your participants, please contact your Account Manager so they can adjust your account settings.