Developments in data collection technologies led to a novel type of information, alternative data, that is used to enrich traditional finance. In this article, I explore an Oxylabs and Censuswide survey that displays the utility and the level of adoption of alternative data across the UK financial sector.
Investment managers once relied exclusively on traditional data such as earnings and market reports to make decisions. However, as the quest for alpha became increasingly challenging, new sources of data from social media, public websites, and applications – collectively referred to as alternative data – emerged to create new growth opportunities for firms.
The use of alternative data has grown exponentially – to the point where its use is leading to significant irreversible changes in how markets behave. This has led to concern about its risks, particularly with flawed data modeling and incorrect market signals.
To address this issue, Oxylabs – in cooperation with Censuswide – surveyed 252 senior data decision-makers from leading UK-based financial services companies and published a whitepaper. This article will explore those findings and give you an overview of the risks, benefits, and overall role of alternative data in the finance sector.
Alternative data, defined
Alternative data is literally an “alternative” to traditional data such as regularly published economic reports, investment news, press releases, and corporate annual reports. Sources include mobile application downloads and usage info, social media sentiment analysis, website traffic information, and search data.
Initially used by hedge fund investors, the movement caught on quickly to include private equity investors, followed by growth and value-oriented fund managers. The use of alternative data has been particularly critical to collective intelligence investing, where traders predict the actions of the crowd and base their trading activity accordingly.
At present, alternative data is used by investment firms of all sizes and is even accessible to day traders and casual investors. The key is to know where to find it, how to extract it, and how to analyze it to find critical market insights.
What insights does alternative data provide?
Alternative data provides specialized insights that aren’t necessarily found in traditional sources of information like government market reports, company financial statements, or the media. These can include:
- Government contract activity
- Product recalls
- Social media sentiment from networks like Twitter and Reddit
- CEO compensation
- Work visas to specific countries and states
- Corporate flights to specific locations
- Politician trading trends
- Page views on information-rich websites like Wikipedia
- Credit agency data
Not all alternative data is equal. In some cases – such as politician trading activity or government contracts – alternative data is uncoupled from economic activity and is used for speculative purposes. In other cases – such as product recalls or social media sentiment about a company – alternative data can go beyond traditional economic reports to provide key insights into the health of a business.
How Alternative Data Is Used
As the amount and types of data available from applications and public websites continue to grow, so do use cases for alternative data. Sources of this information continuously emerge as more people and companies come online and share their data, leading to critical insights that can include:
Data extraction techniques like web scraping (more on that below) can be used to track the prices of millions of products from online stores to measure the effects of price shocks on inflation.
Data from credit reporting agencies in tandem with application performance statistics can be used to analyze the performance of a specific company or sector.
Stock price predictions
Data can be scraped from public discussion websites (such as the infamous WallStreetBets group on Reddit) to predict stock price movements. Other use cases in this category include data extracted regarding earnings announcements and additional company information that may cause shifts in prices.
Real estate values
Location data extracted via downloaded applications can be used to estimate foot traffic in specific areas to determine possible changes in real estate values.
Satellites have been used to zoom in on oil tankers to estimate their tank levels. These estimations have then been used to predict upcoming impacts on oil producers and related commodity prices.
Overall company health
Supplier payment history produced from inquiries to credit agencies can reveal insights into the overall health of a company.
The Exponential Growth of Alternative Data
Alternative data has experienced explosive growth in recent years. According to Grandview Research, the alternative data market was valued at $1.06 billion in 2019, and is expected to grow at a compound annual growth rate (CAGR) of 40.1 percent from 2020 to 2027.
Our survey results confirm alternative data’s massive growth, specifically in the UK. We found that data inquiries from the financial sector increased nearly three times in the past year when compared to previous years, and that total spending on data has increased over six times in the past five years. Other key insights included:
- 60 percent of respondents use alternative data sources as a way to improve decision making
- 26 percent of respondents noted their data needs have increased by a significant margin
- 38 percent of respondents increased data department budgets
- 36 percent of respondents list high risks of legal complications as a primary concern with regards to data extraction
Our research suggests that there is a rush towards the adoption of alternative data across the sector. We also learned that financial firms are meeting their data needs in a variety of ways depending on their resources and requirements – and that the process is continuously evolving as the use of alternative data increases.
How Businesses Obtain Alternative Data
Depending on a firm’s size, scope and budget, data is collected in several ways that include:
Public Website Data Extraction (Web Scraping)
Web scraping – the process of extracting data from public websites – uses scripts (also called “bots”) to visit web pages and collect publicly available data. Companies typically conduct web scraping in-house or through a third party. Our research determined that 40 percent of respondents use web scraping, with 36 percent outsourcing their requirements vs. 36 percent scraping data via an in-house team. Some firms mix up their strategy, with 27 percent choosing to combine both practices.
In-house scraping is typically favored by larger companies that employ their own developers, system administrators, data analysts, and data specialists. While the use of in-house scraping requires a significant capital outlay and maintenance budget, it offers considerable benefits such as increased flexibility and customization.
Outsourced web scraping is a cost-effective option that allows a firm to collect data through the use of customizable web scraping tools. Many small to medium size tools choose to outsource web scraping because it costs less, reduces management burden, and allows firms to focus resources on deriving insights instead of the extraction process itself.
Integrations with third-party databases
Roughly 60 percent of financial firms integrate their operations with data providers to purchase and consume data in a single step rather than manually collecting, parsing, and cleaning the data. While this option costs relatively more when compared to web scraping, the primary benefit of this method is that it allows companies to focus solely on obtaining insights.
Manual data collection & cleaning
Approximately 51 percent of firms manually collect and process data via a spreadsheet or other data processing application. The use of manual processing on its own for all data requirements is relatively inefficient, and likely supplements other extraction methods.
The Dark Side of Alternative Data
Estimating a risk-and-reward equation for the use of alternative data can be more of a challenge when compared to established data sources. Traditional sources of information often point to a clear relationship between the economy and output. For example, if sales data rises in a given month, this may indicate consumer confidence and economic growth. In that case, the relationship is clear, whereas many types of alternative data lead to insights based on speculation.
Other risks associated with alternative data span privacy concerns to issues with data modeling, and can include:
- Loss of competitive advantage from not using alternative data
- Provenance risk – a type of risk that is concerned with terms of service violations on websites targeted for data extraction
- Privacy violations due to the inclusion of personally identifiable information in a data set
- Accuracy risk that can manifest in inaccurate trading signals
- Risk of missed opportunities due to delays in extraction and analysis
- Data that is not incorporated correctly into a model
- Model output improperly linked to the trading process
- Loss of intellectual capital due to high employee turnover
While the use of alternative data may bring significant risks, the industry is innovating to mitigate those risks in various ways. Firstly, it should be widely understood that extracting data can be complicated. Collecting, implementing, and deriving insights is a highly complex operation that requires a combination of technology and human resources. Therefore, a standard mix of talent usually requires IT professionals, data scientists, security analysts, and portfolio managers. Also essential is the use of an integrated analytics platform to derive insights that can be combined with traditional financial data that lead to differentiated market insights.
Since alternative data is highly specialized, firms must be able to identify the correct data type and implement a rigorous testing procedure to ensure efficacy. Fluid data architecture is required to manage varied alternative data types, and the system should be able to handle multiple data feeds via an application programming interface (API) along with scalable processing power to accommodate increased demand over time. In addition, data modernization, curation, and industrialization are required to get the data into a useful state for algorithms – and this requires highly specialized knowledge.
Ready to learn more about alternative data?
It appears as if the use of alternative data is here to stay – and investment markets may be irrevocably changed as a result. While there are many benefits and challenges of using alternative data, its use is no longer optional as it now appears that firms must use it to stay competitive.
Our research confirms that roughly 35 percent of firms outsourcing data acquisition have issues finding reliable partners and tools. To learn more about how to efficiently collect data and use it to augment your strategy download our whitepaper to discover who is using alternative data, how it’s being extracted, and the best ways to obtain actionable insights for your business.
About the Author
Julius Černiauskas is the CEO of Oxylabs, a global provider of premium proxies and data scraping solutions that helps businesses to realise their full potential by harnessing the power of data. Cerniauskas’ experience and understanding of the data collection industry have allowed him to implement a new company structure, taking product and service technology to the next level, as well as securing long-term partnerships with dozens of Fortune 500 companies. He regularly speaks on the topics of web scraping, big data, machine learning, technology trends, and business leadership.