Performance Testing: Real User Monitoring (RUM) vs Synthetic Monitoring (STM)
November 9, 2018
by Ohad Hagai
User experience (UX) is the name of the game today. Every millisecond counts on publishing and eCommerce sites. This is where the “Real User Monitoring (RUM) vs Synthetic Monitoring (STM)” dilemma begins.
All respectable online publishers and eCommerce vendors today are catering to a global audience. The digital revolution and evolving localization tech has erased all geographical boundaries. But these developments have also introduced many technical challenges, especially when it comes to performance.
The parameters that need to be introduced into performance testing are constantly rising as internet and mobile technologies evolve. For example, mobile usage has introduced a wide range of variables – operating systems (Android and iOS), processing power (processor, RAM, etc), and more.
Looking to test your performance? Chances are you have reached the point where you have to pick between RUM and STM. Which one is better for you?
What is Real User Monitoring (RUM)?
RUM measures the response times and performance KPIs on the actual devices/browsers running your website. These results are collected and aggregated at the user end, and reported back to the analytics backend to show you page load speeds, bounce rates, and other vital performance metrics.
Since the issues and errors you see are exactly the same as your users are facing, RUM gives you the “real deal” when it comes to performance testing.
What is Synthetic Monitoring (STM)?
Also known as proactive monitoring, STM simulates real user interactions with your website and tries to mimic real user behaviour to create an educated performance assessment. STM solutions can be configured to run periodically from desired geographical locations through available networks.
Real User Monitoring (RUM) vs Synthetic Monitoring (STM) – The Comparison
Real User Monitoring is gaining momentum due to the real-time results that it actually fetches from the end user’s browser. On the other hand, Synthetic Monitoring is more of a “hands on” methodology that combines manual and automated approaches based on a predetermined series of scenarios.
We have decided to compare these two performance testing techniques based on 5 parameters that are most crucial to the modern online business.
1 – Performance Monitoring – RUM
Real User Monitoring tracks, monitors, and registers every interaction the visitor makes. The data gathered and displayed is based on real pages being loaded by real users in real locations. On the other hand, STM involves the deployment of scripts to simulate the path and behaviour of your users.
Real User Monitoring simply has the edge when it comes to performance.
RUM solutions can often detect and diagnose the root cause of problems. For example, you can pinpoint problems with specific browsers or locations that would not be revealed using synthetic testing, which limits its ability to serve as a dedicated performance monitoring tool.
2 – 3rd Party Vendor Accountability – RUM
Most RUM solutions today give you the ability to generate performance reports, with in-depth analysis and breakdowns. This can help you make sure that service level agreements with 3rd party vendors are being met. Also, all performance issues can be eliminated faster since RUM pinpoints the root of the issue.
STM has no such functionality since it doesn’t have any attribution capabilities, nor can it be used to locate the root of the problem in any way.
3 – False Negatives – RUM
Synthetic Monitoring works on a series of scripted scenarios, which run through predefined networks and create virtual user environments that online publishers and eCommerce vendors can test out. However, the fact remains that these are a finite number of scenarios that cannot possibly cover all existing use cases.
STM simply can’t gauge the unpredictable variables that can potentially impact the user experience through different stages of the digital journey.
4 – On Demand – STM
STM allows you to measure performance even when the site is not being accessed by live customers. This can be a useful prior to releasing new features (implementing new 3rd party tags) or before a huge maintenance release, which can potentially have huge user experience implications.
However, it must be taken into consideration that none of the above are ongoing procedures, which still gives Real User Monitoring the edge in the long term.
5 – Setup and Maintenance – RUM
Cybercrime has risen exponentially in recent years. With more and more breaches being reported, today’s online publishers need a solution that can be automated and integrated seamlessly into their ecosystem. Real User Monitoring has the upper hand here due to its ability to operate automatically.
STM requires human intervention to pre-define the simulation parameters and additional staff to break down and digest the results, before starting to work on locating the exact issue. These additional setup and maintenance stages eat up additional time and resources. Another win for RUM.
The Bottom Line
RUM’s biggest advantage is that is doesn’t have to simulate situations, which makes it more reliable for constantly monitoring and fixing performance issues as they happen. As “smart” as STM has become today, there is only a finite number of situations and issues that can be simulated in a limited period of time.
Synthetic Monitoring does have its pros, as it can be deployed fast and has good on demand capabilities, but its not ideal for long-term performance monitoring and it also suffers from inherited limitations – inability to simulate (and detect) all issues, lack of automation capabilities and higher maintenance requirements.
While STM can be used as a supplementary solution, it’s becoming more and more evident that Real User Monitoring is the better performance testing option.