Blog Post

Search engines, big data and network effects

Search engines are intermediaries in a two-way market between users and advertisers. Their huge stocks of data about users and their preferences can help search engines offer better services to all parties. But does this make market entry difficult for new players? And can we see network effects emerging in the search engine market?

By: Date: November 22, 2016 Topic: Innovation & Competition Policy

Search engines respond to queries by providing relevant and valuable information about the topics that their users are looking for. They also act as intermediaries which match consumers with providers of services or sellers of products. To monetise their work, search engines collect and process data from users and sell advertising slots to companies. By analysing the users’ data, they can improve the quality of the search engine algorithm and provide more relevant organic search results, but they can also design personalised advertising strategies for companies’ products and services. Companies are thus more successful in placing their products, and consumers receive recommendations tailored to their interests.

Advertising slots are allocated on a competitive basis to the companies that are willing to pay the highest amount to get some prominence in the platform when the user is searching for relevant terms. Slots are auctioned and the winners have to pay a fee to the search engine whenever a potential consumer clicks on the displayed ad (pay-per-click strategy). Search engines base their revenues almost exclusively on selling advertising slots (see for example some relevant statistics in Liem and Petropoulos, 2016).

So, in this two-sided market environment, advertisers assign a higher valuation to participating in the search engine platform when they expect to have a higher probability of matching with as many consumers as possible. More consumers can increase the value for companies to be advertised in the search engine in two main ways. First, companies obviously see that they can be connected with more potential buyers as the pool of relevant consumers increases. Second, search engines with more users can analyse the additional data and improve their matching algorithms. They can then provide more personalised services to advertisers which can in turn have higher chances of success in attracting consumers. Given the competitive allocation of advertising slots, more consumers, therefore, implies higher revenues for the search engine as advertisers are willing to bid more for prominence.

Consumers, on the other side of the market, value information that leads them directly and quickly to what they are looking for, to minimise their search costs (for example, consumption of time). Hence, they mostly prefer to only visit the search engine of the highest quality (according to their standards and criteria) even if they have the option to use different search engines simultaneously.

In betwen these two sides, search engines compete in quality and efficiency of search (for example, accuracy of results, page load speed, real time relevance) in order to attract more consumers who will in turn bring more advertising revenues which can be used for further improving their quality to attract even more consumers, and so on. So, it is of interest for the search engines to build up a large stock of users’ information that can be analysed in order to provide good quality services which attract more consumers and advertisers.

How important is the scale of data collection for the improvement of services?

In general, the answer depends on the complexity of the algorithm in place, the number and the noise in observations. But in the context of data related to human behaviour and preferences, the amount of data and the quality of predictive power seem to be positively related.

When data are drawn from human actions, noise rates are often high. Social scientists have long argued that one way to circumvent the poor predictive validity of attitudes and traits is to aggregate data across occasions, situations, and forms of actions. This provides an early suggestion that more (and more varied) data might indeed be useful when modelling human behaviour data. The implication for predictive analytics based on data drawn from human behaviours is that by gathering more data from individuals (aggregated by the modelling), one could indeed hope for better predictions.

Fortuny, Martens and Provost (2013) demonstrate that when predictive models are built from sparse, fine-grained data—such as data on low-level human behaviour—economies of scale are important and we continue to see marginal increases in predictive performance even to very large scale. However, the curve between predictive power and data size do seem to show some diminishing returns to scale, albeit without any ceiling that limits the increasing pattern.

Their empirical results are based on data drawn from nine different predictive modelling applications, from book reviews to banking transactions. The study provides a clear illustration that larger data indeed can be more valuable assets for predictive analytics similar to those used in search engines.

Big data and quality driven competition

Since the quality of search engines increases with data gathering and analysis, search platforms with a larger stock of information than their competitors can have a competitive advantage. They can perform better and attract more users and consequently more advertisers. This suggests that the amount of data processed is linked with market power in the search engine market. Incumbents with a large stock of information and experience in data analytics can protect their market position against new entrants and firms that are far behind in data accumulation. Moving to the frontier of the search market and establishing a large base of users requires new entrants to offer very attractive prices for the advertising slots and at the same time to bear the costs of developing a good quality dynamic algorithm that can attract consumers. This is an investment that can only pay off in the long run. Bing, for example, in its effort to increase its installed base of users incurred losses for many years before starting to be profitable.

Note that in many markets product differentiation among competitors is multi-dimensional. Consumers’ preferences are affected differently by each of the multiple dimensions. If I need to subscribe to a mobile operator my choice depends not only on the quality of the network but also on whether I want to combine mobile with TV and home internet, whether I wish to call often in another country or how much mobile internet I intent to use. The plurality of options and preferences lead to multi-dimensional market competition where there is room for entry and growth of firms that specialise in specific dimensions.

The search engine market is by construction centralised and product differentiation has only on dimension, the quality of service. (Some could argue that privacy is another dimension, but this is a secondary choice parameter, especially since it is difficult for the users to rank competitors with respect to the privacy protection they provide). That means that the search engine of higher quality than its rivals can increase its market power relatively fast or even dominate the market, especially under the implementation of efficient methods of machine learning and data analytics.

Moreover, in many cases, companies that own a popular search engine can use the collected data for their operation in other data-driven markets to improve the quality of their services and consequently their market position.  For example, since 30% of mobile searches are about location, queries in the search engine about finding a particular location can improve the quality of navigation services through the additional data collected by the search engine. So, the search engine can help the “mother” company to enter to new markets and improve its market position through data accumulation and analysis.

Are there network effects in the search engine market?

Direct network effects in an online platform exist when an increase in the number of users or in usage of the platform directly increases the users’ value from accessing the platform. Facebook is a typical example of a platform where the value of using the network highly depends on the number of friends that also use the network. In search engine platforms, on the other hand, such direct effects are absent as each user derives individual benefits from its use without any direct effects to other individual users.

Indirect network effects exist when the increase in the number of users or usage of the network spawn increases in the value of a complementary product or network, which can in turn increase the value of the original. The relevant question is if there are indirect network effects in the search engine market which increase the value of participation for users.

Hal Varian, the chief economist at Google, argued at  a recent Bruegel event on big data and market competition that it is the accumulated experience in the market that helps search engines to learn how to improve the quality of their services (learning by doing effect) just as it happens in normal businesses. He claims there are no indirect network effects involved on the users’ side:  “The higher the number of customers a business has, the higher the revenue of the business, revenue which can be reinvested in the maintenance and improvement of the business so as to attract more users.”

While experience clearly matters, the impact of learning by doing effect is greater than in normal businesses due to the importance of data in improving the search engine algorithm. The addition of new users makes the search engine more competent to provide high quality services through the processing of the additional data collected by the newcomers. That generates extra value for the advertisers and further increases search engine’s revenue as well as its incentives to improve quality and attract even more consumers.

However, I would argue that indirect network effects are also present. We should not ignore that search engines are two-sided markets in which both sides are inter-dependent. What happens on one side has an impact on the other side and affects the value of accessing using the search engine for both sides of the market. The addition of new users increases the value of using the search engine for the existing ones not only because of the increase in quality (due to the learning by doing effect) but also because more users attract more advertisers. That leads to more matching opportunities for the existing users through the sponsored links of the search engine and therefore their value from accessing increases.

Network effects may also extend to organic search results given that the ranking of firms depends to some extent to firms’ choices such as their search engine optimisation strategies. While the algorithm of the search engine is secret, published guidelines provide an overview of how firms can increase their chances for high rankings in the organic search results. The arrival of new users increases the incentives for firms to get a better ranking position in order to attract more users by investing in search engine optimisation and the quality of their website. So, they provide better web services to their visitors (existing and new) and help them to reach what they are looking for in a more timely and efficient fashion.

When we assess market competition in the search engine market we need to take into account the importance of big data in defining market power. Especially when users do not pay any price to access search platforms and the latter compete in other key dimensions such as the (data-driven) quality of service.


Republishing and referencing

Bruegel considers itself a public good and takes no institutional standpoint. Anyone is free to republish and/or quote this post without prior consent. Please provide a full reference, clearly stating Bruegel and the relevant author as the source, and include a prominent hyperlink to the original post.

View comments
Read article

External Publication

Economic Implications of Further Harmonisation of Electronic Communications Regulation in the EU

One of the ways in which the European Commission has sought over the years to strengthen the European single market is by means of increased harmonisation of the regulation of electronic communications. To the extent that the European Union functions as a confederation of somewhat autonomous member states, however, there are both practical and political limits to the degree of harmonisation that is realistically desirable or achievable.

By: J. Scott Marcus and Christian Wernick Topic: Innovation & Competition Policy Date: August 11, 2017
Read about event

Past Event

Past Event

Geo-blocking in the digital single market

Geo-blocking is a discriminatory practice that is wide-spread in EU. It prevents online customers from accessing and purchasing products or services from a website based in another member state

Speakers: Marine Elgrichi, J. Scott Marcus, Fabian Paagman, Bertin Martens, Georgios Petropoulos, Agustin Reyna, Gareth Shier, Werner Stengg and Roza von Thun Topic: European Macroeconomics & Governance, Innovation & Competition Policy Location: Bruegel, Rue de la Charité 33, 1210 Brussels Date: May 30, 2017
Read article Download PDF More on this topic

Working Paper

From start-up to scale-up: examining public policies for the financing of high-growth ventures

What are the challenges of financing scale-ups, and how can long-term public policies support the creation of a better scale-up environment?

By: Gilles Duruflé, Thomas Hellmann and Karen E. Wilson Topic: Innovation & Competition Policy Date: April 10, 2017
Read article More on this topic More by this author

Blog Post

High expectations for 5G confront practical realities

The next wave of mobile network innovation is provoking great excitement in the industry. And indeed, there is substantial potential for improvement. However, the exact form of the technology and the appropriate policy support are still far from clear. And we should beware of over-ambitious promises about the impact and uptake of new network technologies.

By: J. Scott Marcus Topic: Innovation & Competition Policy Date: March 14, 2017
Read article Download PDF More on this topic

External Publication

Extending the scope of the geo-blocking prohibition: an economic assessment

This paper was prepared for the European Parliament at the request of the Committee on Internal Market and Consumer Protection.

By: J. Scott Marcus and Georgios Petropoulos Topic: Innovation & Competition Policy Date: February 27, 2017
Read article More on this topic More by this author

Blog Post

Big data and first-degree price discrimination

What’s at stake: first-degree price discrimination - or person-specific pricing, had until recently been considered a theoretical case with unlikely real-world application. Yet the increasing availability of big data could make this possible. We review recent contributions on this issue.

By: Silvia Merler Topic: Innovation & Competition Policy Date: February 20, 2017
Read article More on this topic More by this author

Blog Post

How good a shield is Privacy Shield?

Privacy Shield was put in place in 2016 to ensure that transfers of personal data from the EU to the US would be in compliance with European Union privacy law, and thus permissible. The institutional framework of Privacy Shield was weak, and depended on the good will of the US administration. Recent actions by the new administration, including the famous executive order forbidding residents from 7 predominantly Muslim countries to enter the US, may have (presumably unintended) effects on Privacy Shield. To preserve the validity of Privacy Shield in European Courts, strong EU-US cooperation and potentially additional agreements may become necessary.

By: J. Scott Marcus Topic: Innovation & Competition Policy Date: February 7, 2017
Read article More on this topic More by this author

External Publication

Policy and Politics in the Era of the Industrial Internet: How the Digital Transformation Will Change the Political Arena

The digital transformation has already had an impact on policymaking, and this trend will continue in the years to come. How will the political process change and how can influencers guide this change?

By: Giuseppe Porcaro Topic: Innovation & Competition Policy Date: December 7, 2016
Read article Download PDF More on this topic

Policy Contribution

Going local: empowering cities to lead EU decarbonisation

Decarbonisation and digitalisation are reshaping the European energy system, which will become more decentralised and interconnected with other sectors. Cities have the opportunity to be the key drivers of decarbonisation, but this will require the implementation of a new bottom-up governance system. This paper outlines a four-step mechanism in order to achieve decarbonisation at city level.

By: Simone Tagliapietra and Georg Zachmann Topic: Energy & Climate Date: November 30, 2016
Read about event More on this topic

Past Event

Past Event

Digital platforms: A policy and research agenda

The number of digital platforms is currently rising in many countries and sectors. What are the opportunities of platforms and which kind of regulation and policy framework is necessary to promote healthy competition?

Speakers: Bruno Basalisco, Diane Coyle, Jacques Crémer, Werner Stengg, Nicolas Petit, Georgios Petropoulos and Simon Wilkie Topic: Innovation & Competition Policy Location: Bruegel, Rue de la Charité 33, 1210 Brussels Date: October 20, 2016
Read article More by this author

Blog Post

The industrial internet will transform policymaking

The ‘internet of things’ will bring major changes in many areas of life, including the political arena. What will be the new communication tools, strategies and narratives for policymakers?

By: Giuseppe Porcaro Topic: Innovation & Competition Policy Date: September 28, 2016
Read article Download PDF More on this topic More by this author

External Publication

New network neutrality rules in Europe: comparisons to those in the U.S.

This paper explains the similarities and differences between European and U.S. net neutrality rules.

By: J. Scott Marcus Topic: Innovation & Competition Policy Date: September 15, 2016
Load more posts