01 April 2025

A recent whitepaper by Oxylabs reveals that 2024 saw a significant surge in demand for web intelligence, with 74% of businesses in the UK and the US expressing an increased preference for information-based decision-making across various industries. This finding emerges from a survey of over 500 web scraping and data collection professionals conducted by Oxylabs in collaboration with the research partner Censuswide.

The report identifies infrastructure development and maintenance as a critical challenge faced by 61% of respondents involved in web scraping, while a staggering 86% noted that data parsing — transforming raw data into usable formats — remains time-consuming and resource-intensive. These factors have prompted many organizations to explore AI-driven solutions to address the complexities of web scraping.

Interestingly, despite the challenges associated with in-house data collection, 89% of respondents prefer to retain control over web data collection processes, with only 11% opting to outsource these tasks completely. This trend signifies a considerable opportunity for external web intelligence service providers to deliver scalable and reliable data collection services tailored to the needs of contemporary businesses.

The potential of AI to assist web intelligence professionals with data parsing can significantly reduce delays in information acquisition and improve web unblocking efficiency. The report underscores how delayed data collection adversely affects business operations, with 98% of professionals reporting negative impacts from data delays — 21% characterized this impact as severe. When parsing efforts are disrupted, a notable 95% of respondents reported experiencing adverse effects within 24 hours.

Resource intensity is a prevailing concern, with 75% of developers spending 10 to 40 hours each week on parsing tasks. The challenges of identifying complex parsing patterns across multiple URLs were acknowledged by 58% of respondents, while 56% cited difficulties associated with managing dynamic website layouts. Frequent layout changes have compelled 57% of developers to modify parsers several times a week, and 31% reported making daily adjustments. For more than half of the developers surveyed, time emerged as the primary cost associated with parsing, further fuelling the need for improved web intelligence collection tools.
“Web intelligence has transitioned from a strategic advantage to an essential business requirement,” said Žydrūnas Tamašauskas, CTO at Oxylabs. “It is crucial for external service providers to empower organizations to thrive amid the challenges of data-driven decision-making.”