When deploying COTS (commercial-off-the-shelf) software such as Magento, “How much should I test in UAT?” is always an interesting question. After all, you’re buying 3rd party software so that you don’t have to pay for all of the development, testing, and maintenance. You won’t need to test the software’s base functionality. Yet don’t you still need to test something? Different organizations and projects are going to have different UAT testing approaches and needs. We’ve seen the gamut: either you’re testing every single aspect of the software, only testing major job functions, or simply attending the sprint demos instead of hands-on testing.
What Do You Need to UAT?
First, keep in mind what you are trying to accomplish with UAT: confirm that the software allows your users to complete their tasks. Magento, along with its library of extensions, provides industry-standard eCommerce. As such, you’ll have confidence that it is suitable for the job, but no COTS product is built to your specific business processes, ecosystem, and data. Whether you’ve configured or customized, the more your practices vary from or enhance industry standards, the more UAT you will need to perform to make sure your users’ needs are met. Your UAT plan should include:
- The most used functionality – You don’t need to test every combination and scenario, but you should confirm the basic functionality meets your users’ needs. Hopefully, you have requirements and have identified and prioritized your use cases/processes. When testing processes, be sure to cover your major user groups. For example, if you support both B2B and B2C, test for each user group.
- Workflow changes – Your business might require workflows that are different than standard. For example, if you choose to add or remove types of order status to create a unique order workflow, you’re going to want to make sure that is tested.
- Customizations – This is where you don’t get the benefit of others’ testing, so customizations should be tested to make sure they function as expected and meet the business need.
- Integrations – Look at data moving between Magento and other systems to make sure it appears in all systems as desired. Think about your entire data life cycle and include integrated systems such as ERP, CRM, tax, payment, and email. This can actually be the biggest part of your testing effort if you are in a highly integrated environment.
- Device types – If your system will run on multiple device types (phones, tablets, computers), you’ll want to make sure tasks can be completed on all device types. If this isn’t your very first online store, have your IT team analyze historic device type usage. You may learn that the majority of your users use one type of device and can concentrate testing there. (We analyzed usage patterns for a B2B customer and were surprised to find the majority of users came through iPhone using Safari, not computers as expected.)
Usability testing isn’t on the list above because usability testing should be part of your design and development process. If you haven’t done that before you get to UAT, you’re in trouble! However, any defects discovered during any phase of testing, including UAT, may have a usability component, so don’t neglect to consider that as you work through resolving issues.
Build a UAT Team
Second, consider who needs to be involved with your UAT and ensure you can schedule time with the people you need. Functionality you want to test will include internal and customer-facing processes, and these present different challenges to manage.
- Internal – You’ll need representatives from the user groups who will perform the key processes you’re testing. It’s always a challenge to pull people away from their “day jobs” to test, but it’s not ideal to have a manager (who isn’t currently hands-on with the process) or a proxy do the testing. You’ll need to be pretty clear about the time and duration of testing activities, ensure that you are focused on high-priority activities, and work with the affected teams to reach a satisfactory plan that won’t interrupt the business more than absolutely necessary.
- External – Ideally, you will have volunteer customer participants test your eCommerce site. What better way to ensure it really works for them? This can be hard to do, especially in the B2C space. But if you did design right, you talked with your customers to get feedback and improve your design concept. Usually these are customers who are engaged with your company and your site, so maintaining that relationship and extending it to UAT is ideal. Some companies will use proxies (internal team members or testers-for-hire) for customers in UAT, but this isn’t ideal because it won’t truly represent the customer’s interests and knowledge.
Make It Easy
Third, organize, organize, and organize some more! We’ve seen businesses stumble through UAT and leave everyone unhappy because they didn’t have a clear plan and good communications tools in place. Remember, your testers aren’t members of IT. Here’s some things that will make it easier for them and get better results for you.
- Make a schedule – Nothing makes users feel less cooperative than not being clearly told when and for how long you’ll need their help. They’ll need to reschedule meetings and plan their work around the testing schedule, so get that information to them as early as possible.
- Make a plan – Identify the specific use cases and scenarios to be tested, make sure these are documented so that your testers can follow them, and ensure that the underlying data and integrations to perform these tests is in place. If you need to test order submission and fulfillment, you’ll need accurate customer, product, and payment data in place. Nothing puts a spoke in the UAT wheel like telling all your testers that you need to replicate the product catalog, please come back tomorrow!
- Establish simple communication – When your testers have an issue or encounter a defect, you want it to be really easy for them to get you all the information you need to take action. Let them know how to submit and how much detail you need. There are UAT tools that you can utilize to simplify this, but however you choose to do it, make sure the process is documented and as simple as possible. It’s a good idea to walk testers through any processes before testing begins.
- Triage/follow-up – Nothing frustrates users like submitting defects and not hearing back! Ensure that you have a robust process to quickly triage defects, gather more information if needed, resolve, communicate, and retest. If you determine that a reported issue isn’t really a defect or isn’t priority to fix now, ensure that is communicated as well. If you don’t close this communication loop, users will assume you haven’t addressed the problem they reported and their confidence levels will drop, which can have a negative impact on user adoption.
In short, all software functionality is not created equal, and the time and resources you have for UAT will be limited, so focus on prioritization and excellent execution of your UAT. This will reduce project risk as well as reduce user burn-out. A win-win.
