It is no secret that a key area of web scraping in 2023 will be centred around real-time data.
When asked about their goals for collecting data in the coming year, 62% of eCommerce organisations said that real-time data would be at the forefront. This is one of the key findings from the new Oxylabs white paper, Alternative Data Defines Competition in the US & UK Ecommerce Sectors, which demonstrates the importance of real-time data collection eCommerce businesses.
In conjunction with Censuswide, Oxylabs surveyed 500 UK-based and 501 US-based senior data decision-makers from UK and US ecommerce companies, gathering insights on attitudes towards public web data collection in the sector.
As most web pages today are subject to frequent content changes, real-time web scraping is an essential function for any digital business. In addition, real-time web scraping will be increasingly important to eCommerce businesses in the future, allowing them to track information on their competitors’ websites and other online stores.
As companies begin to rely more on efficient and low-cost data collection methods such as external data gathering, used by 40.54% of respondents, there is a clear shift towards gaining more actionable insights.
Tomas Montvilas, chief commercial officer at Oxylabs, said: “These goals might indicate that businesses have yet to harness the true potential of real-time data. Greater data delivery speeds and instant insights can prove itself to be the next edge of competition that eCommerce companies are faced with. As competitors adopt real-time data for purposes such as monitoring, others will want to follow suit, causing a feedback loop. I expect to see real-time data keep climbing in importance in the coming years.”
These trends show that eCommerce companies must consider new challenges in their business environment. For applications with low analytical requirements, such as dynamic pricing, real-time data is most valuable. Meanwhile the most widely used use case for external data was cited as market research—which necessitates high levels of involvement with the data.
According to the report, US-based companies will focus more on collecting and utilising external data (43.91% vs 37.20%), whereas their non-US counterparts will pay closer attention to managing existing infrastructure—56.09% vs 44.49%. On the other hand, UK-based companies are planning to focus more on real-time acquisition (65.94% vs 58.28%) and data quality assurance (28.35% vs 18.96%). This contrast shows that the two regions have unique web scraping environments, as each one faces specific issues in this regard.
Montvilas concluded: “As eCommerce has grown exponentially, businesses are increasingly dependent on data. Today’s organisations feel compelled to measure everything from the number of transactions per hour to the lifetime value of their customers. Web scraping has become an essential part of all decision-making processes for businesses large and small, which means that eCommerce companies can now use real time data to gain a competitive edge in difficult economic times”.
