SEO Effect

The effect of search engine optimization on the search results of web search engines (SEO Effect)

The overall goal of the project is to describe and explain the role of search engine optimization from the perspective of the participating stakeholder groups by analysing search results/search result pages for optimized content as well as quantitative and qualitative surveys of search engine users, search engine optimizers and content providers. Thus the external influence on the results of commercial search engines can be described and quantified for the first time. The project contributes to theory building in information science by extending existing information-seeking models by a component of external influence on the search results.

Insights

The project focuses on informative content; it examines how documents that play a role in answering information-oriented search queries can be influenced externally.

Methods

We combine methods from computer science (screen scraping, data analysis, machine learning) and the social sciences (experiments, surveys, qualitative interviews).

Classification

In the documents, we identified more than 40 indicators for search engine optimization (SEO) that we use in our analyses, resulting in a reliable classification.

Software and datasets

We develop open source software for collecting and analysing results from commercial search engines. We make different datasets with SEO results available.

The project focuses on informative content; it examines how documents that play a role in answering information-oriented search queries can be influenced externally. This sets the project apart from pure questions of online marketing, where the focus is on the optimization itself and not on the consequences for the compilation of result sets. Methodologically, the project is characterized by a triangulation of methods of data analysis from computer science and social science methods. The interdisciplinary basis of the analysis is unique and will significantly advance the understanding of search engines in general and the influence search engine optimization has in particular.

In the first project phase, we developed software that can automatically query search engines and analyze the returned results to measure the effect of search engine optimization. The results from these analyses were combined with findings from a representative online survey and further investigations of search engine users, search engine optimizers, and content producers.

In order to obtain a comprehensive picture of the influence of search engine optimization, we will include the perspectives of non-commercial content producers and search engine operators in the second project phase. We will also conduct large empirical studies to measure the impact of SEO on how users form their opinions on topics of societal relevance, e.g., politics and health.

Results

Expert interviews

As a basis for the online survey and the software development we conducted 15 interviews with stakeholders involved in search engine rankings: Search engine optimizers, content managers, and online journalists (Schultheiß & Lewandowski, 2020).

  • The interviewees assumed that SEO is hardly known to the users and that the user opinions of SEO strongly depend on their SEO knowledge. We used these assumptions to form hypotheses for the representative online survey.
  • The interviews also provided us with valuable clues on how SEO can be identified on websites. We have transferred these SEO indicators into software development.

Representative online survey

We conducted a representative online survey with 2,012 German Internet users to find out how familiar they are with SEO and what attitudes they have towards it (Schultheiß & Lewandowski, 2021a, 2021b). The results widely confirm the assumptions from the interviews:

  • Less than half (43%) of Internet users know that ranking improvement is possible outside of ads.
  • Only 8% of users are familiar with the term “SEO,” while 13% can correctly name one or more SEO techniques.
  • Organic results are associated with SEO by only a small fraction of users.
  • With increasing knowledge of SEO, a more positive opinion could be observed.

Laboratory study

  • In a laboratory study with 61 subjects (Schultheiß, S., Häußler, H., Lewandowski, D., 2022), we investigated how the quality of optimized as well as non-optimized health-related websites is evaluated. We have also classified the website types, looking in particular at commercial or non-commercial motivations. The key results are as follows:

    • SEO is more often performed by commercially motivated websites, for example pharmaceutical companies, than by non-commercial providers, such as ministries.

    • Non-optimized websites are attributed the higher expertise by the subjects.

    • The subjects justify these ratings in particular with the assumed competence or seriousness of the website operators.

Software implementation

  • We developed a multidimensional method for measuring SEO effectiveness and a software tool that detects whether SEO measures have been taken on the URL (Lewandowski, D., Sünkler, S., Yagci, N., 2021 & Sünkler S., Lewandowski D., 2021a, 2021b).  
    • For our method, we use a model with 48 indicators based on an extensive literature review and the aforementioned interviews with SEO experts (Schultheiß & Lewandowski, 2020).
    • An initial analysis of Google search results on three datasets (a total of 1,914 queries and 256,853 results) shows that most of the pages found in Google are likely to be at least optimized, which is consistent with the statements of SEO experts, i.e., it is difficult to become visible in search engines that do not use SEO technology (Lewandowski, D., Sünkler, S., Yagci, N., 2021).

Publications

Schultheiß, S.; Lewandowski, D.; Mach, S.; Yagci, N. (2023). Query sampler: generating query sets for analyzing search engines using keyword research tools. PeerJ Computer Science. https://doi.org/10.7717/peerj-cs.1421
Schultheiß, S. (2023). How search engine marketing influences user knowledge gain. Development and empirical testing of an information search behavior model. CHIIR '23: ACM SIGIR Conference on Human Information Interaction and Retrieval. https://doi.org/10.1145/3576840.3578297
Lewandowski, D., & Schultheiß, S. (2022). Public awareness and attitudes towards search engine optimization. Behaviour & Information Technology, 1–20. https://doi.org/10.1080/0144929X.2022.2056507 Preprint
Schultheiß, S., & Lewandowski, D. (2022). Data set of a representative online survey on search engines with a focus on search engine optimization (SEO): a cross-sectional study. F1000Research, 11(376). https://doi.org/10.12688/f1000research.109662.2
Schultheiß, S.; Häußler, H.; Lewandowski, D. (2022). Does Search Engine Optimization come along with high-quality content?: A comparison between optimized and non-optimized health-related web pages. CHIIR '22: ACM SIGIR Conference on Human Information Interaction and Retrieval, 123–134. https://doi.org/10.1145/3498366.3505811 YouTube
Lewandowski, D.; Sünkler, S.; Yagci, N. (2021). The Influence of search engine optimization on Google’s results: A multidimensional approach for detecting SEO. 13th ACM Web Science Conference, 2021. https://doi.org/10.1145/3447535.3462479
Schultheiß, S.; Lewandowski, D. (2021a): Misplaced trust? The relationship between trust, ability to identify commercially influenced results, and search engine preference. Journal of Information Science. https://doi.org/10.1177/01655515211014157
Schultheiß, S., & Lewandowski, D. (2021b). (Un)bekannte Akteure auf der Suchergebnisseite? Ein Vergleich zwischen selbst eingeschätzter und tatsächlich vorhandener Suchmaschinenkompetenz deutscher InternetnutzerInnen. In T. Schmidt & C. Wolff (Eds.), Information between Data and Knowledge. Information Science and Its Neighbors from Data Science to Digital Humanities. Proceedings of the 16th International Symposium of Information Science (ISI 2021), Regensburg, Germany, 8th—10th March 2021, 218–246. https://doi.org/10.5283/epub.44946
Sünkler, S., & Lewandowski, D. (2021a). Den Einfluss der Suchmaschinenoptimierung messbar machen: Ein halb-automatisierter Ansatz zur Bestimmung von optimierten Ergebnissen auf Googles Suchergebnisseiten. In T. Schmidt & C. Wolff (Eds.), Information between Data and Knowledge. Information Science and Its Neighbors from Data Science to Digital Humanities. Proceedings of the 16th International Symposium of Information Science (ISI 2021), Regensburg, Germany, 8th—10th March 2021, 273–298. https://doi.org/10.5283/epub.44946
Sünkler, S., & Lewandowski, D. (2021b). Ist die Webseite suchmaschinenoptimiert? Vorstellung eines Online-Tools zur Analyse der Wahrscheinlichkeit der Suchmaschinenoptimierung auf einer Webseite. In T. Schmidt & C. Wolff (Eds.), Information between Data and Knowledge. Information Science and Its Neighbors from Data Science to Digital Humanities. Proceedings of the 16th International Symposium of Information Science (ISI 2021), Regensburg, Germany, 8th—10th March 2021, 299–306. https://doi.org/10.5283/epub.44949
Schultheiß, S., & Lewandowski, D. (2020). “Outside the industry, nobody knows what we do” SEO as seen by search engine optimizers and content providers. Journal of Documentation 77(2), 542-557. https://dx.doi.org/10.1108/JD-07-2020-0127
Lewandowski, D. (2022). Suchmaschinenoptimierung. In: Diskursmonitor. https://diskursmonitor.de/glossar/suchmaschinenoptimierung/
Hinz, K.; Sünkler, S.; Lewandowski, D. (2023).  SEO im Wahlkampf. In: K. R. Korte, M. Schiffers, A. von Schuckmann & S. Plümer (Eds.), Die Bundestagswahl 2021. Wiesbaden: Springer VS, pp. 1–28. https://doi.org/10.1007/978-3-658-35758-0_19-1

Working Papers

Schultheiß, S., Häußler, H., & Lewandowski, D. (2021). User evaluation and perception of optimized websites for health-related queries: a user study within the SEO Effect project. Working Paper. https://osf.io/qxzeg/
Sünkler, S., Yagci, N., & Lewandowski, D. (2021). Development and software implementation of a preliminary model to identify the probability of search engine optimization on webpages. Working Paper. https://osf.io/u8d62/
Schultheiß, S., & Lewandowski, D. (2020a). A representative online survey among German search engine users with a focus on questions regarding search engine optimization (SEO): a study within the SEO Effect project. Working Paper. https://osf.io/3ukcf/
Schultheiß, S., & Lewandowski, D. (2020b). Expert interviews with stakeholder groups in the context of commercial search engines within the SEO Effect project. Working Paper. https://osf.io/5aufr/

Research data and software

All research data, software and the working papers detailing methods and implementation can be found in the OSF repository, https://doi.org/10.17605/OSF.IO/JYV9Ra.

A demo tool is available at https://searchstudies.org/research/seo-effekt/demo.

Project team

Dirk Lewandowski
Nurce Yagci
Sebastian Sünkler
Sebastian Schultheiß

Funded by the German Research Foundation (DFG – Deutsche Forschungsgemeinschaft):
Funding period: 05/2019 - 07/2021, grant number 417552432.
Funding period: 04/2022 - 03/2025, grant number 467027676.