Posted on November 15, 2022 by Michael Cortez What is an SEO Competitor Analysis An SEO competitor analysis can help you gain valuable insights into how you are performing in search vs. your competitors. The process will also reveal where your competitors are getting their traffic from. And it should help inform your SEO priorities and strategy. In this article, we will see how you can use the Python library Advertools to bring your competitive analysis visualization skills to a whole new level. Here is a video that walks you through the process we cover in this blog post. Want us to help your business grow visibility, authority, and rank at the top of search results? Check out Webfor’s award-winning SEO services and connect with us to see how we can help. Now let’s get back to the Python Competitor Analysis. Why use Python for an SEO Competitor Analysis When SEOs think of traditional tools they use to perform competitor analyses, many will overlook an extremely powerful tool: Python. The possibilities to analyze and visualize data with Python are endless. This is an area of SEO that is growing rapidly as the search landscape continues to get more complex through AI/ML algorithm updates, increased competition, and more. Learning how to use Python for SEO will not only help you automate processes and save large amounts of time. Python can also help you make better and more informed decisions based on deeper insights into data. Let’s dive into how to use Python to: analyze your most important keywords analyze your most important topics identify how you compare versus the top-ranking websites for your target keywords There are many ways the Python script can be used. One of the best is to analyze by topic. This data can help you understand who is the authority for any topic in Search. You can then use this to: reverse engineer their SEO and content strategy identify gaps in topics that your competitors are covering (that you do not) investigate why they are outperforming you organically. (Do they have more unique content? Are they using certain types of schema to gain rich results? etc.) This process will analyze the Search Engine Results Page (SERP) and output a SERP heat map that shows you the top-ranking websites for a topic or target set of keywords. Within a few minutes of setting up the script on Google Colaboratory (Colab for short), you will be able to automate your competitor analysis process and gain tremendous insight into your competition and search engine results. This will save you hours of cleaning and processing data. In addition, it outputs a visualization from which you can quickly extrapolate data and truly understand the competitive landscape. The SERP heatmap (screenshot above) visualizes the top competitors based on the number of times they are ranking at each position of Page 1 for one of your topics or target keywords. Furthermore, it calculates the average position, percentage of coverage, and the total number of appearances in SERPs for each of the Top 10 sites. This data is extremely useful in helping identify your true competitors for key topics and keywords. How to perform an SEO competitor analysis using Python We will be using Python to perform competitor analysis to analyze the most important keywords for your business. This will help you understand who is the top performer in organic search, how many keywords you and your competitors are ranking for, average position for you vs. competitors, and more. Our script will output both a visualization of your top keywords and a CSV file containing all the relevant ranking data. This competitor keyword analysis is much different than the traditional keyword gap analysis that you might get from your favorite SEO tools. What tools do you need to get started? Let’s chat briefly about Advertools. Then, we’ll dig into the code itself and set the script up. Finally, we’ll review how the Google Colab notebook (make a copy!) works. What is Advertools? Advertools is a powerful Python library built by Elias Dabbas. This library was designed with marketers in mind. As Dabbas writes: “A digital marketer is a data scientist. Your job is to manage, manipulate, visualize, communicate, understand, and make decisions based on data.” Advertools has a large number of built-in functions to help you automate your analysis and save time on common SEO and SEM processes. What can Advertools do? Here is a list of some of the tools built into the Advertools Python library: SEM Generate SEM keywords Create text ads on a large scale Create text ads from description text SEO Robots.txt tool XML sitemap tool SEO spider and crawler Crawl strategies Crawl headers Log file analysis Parse and analyze crawl logs Reverse DNS lookup Analyze Search Engine Results (SERPs) Google’s Knowledge Graph Text and Content Analysis URL structure analysis Emoji tools Extraction of structured entities Text analysis Word tokenization Social Media Twitter API YouTube API Index & Change Log We will be using the Analyze Search Engine Results (SERPs) function to replicate Google searches for your primary keywords and analyze/visualize the data. How to do a competitor SERP heatmap analysis using Python and Advertools Now, let’s run your competitor analysis using Python and Advertools. At the end of the post, I will include links to the Google Colab notebook, as well as an example CSV output file that you will receive after running the analysis. Set up a custom search engine First, you will need to set up a custom search engine to extract the keyword ranking data straight from Google. From the programmable search engine create page Enter a Name for your search engine Click Create Here is a link to the Programmable search engine homepage and within a few clicks, you should be able to set up your own programmable search engine. Here is a resource from Google on how to set it up step-by-step. The piece of data you will need from here is your Search Engine ID (screenshot above). Google allows you to run up to 100 searches per day for free. If you go beyond 100 searches per day, it is $5 for every 1,000 search queries. I use this to analyze topics/smaller sets of targeted keywords and have never needed to go beyond 100 daily searches. Set Up Custom Search API Key in Google Developer Console Google Developer Console will provide you with your Custom Search API key. This is the final piece of information you will need to get your custom search engine up and running so you can replicate search results and run a wide range of analyses. Go to Google Developer Console and click the Get a Key button. Create a new project and add a name or select an existing project in your Google Developer Console. Once you have both your Custom Search API key and your Search Engine ID, you have all the items you need to run the script and perform your competitor analysis. How to run the SERP heatmap Python Script Here are the steps we will cover to run the SEO SERP Heatmap Python Script: Install the Python Libraries Add Keywords and your Search Engine ID/API Key Run the serp_goog Function View results of the serp_ranking output Export data to CSV file Read data in from the CSV file Run the serp_heatmap function Invoke the serp_heatmap function View your SERP Heatmap Data Visualization Now you have everything you need to run the Google Colab notebook. You’re ready to perform your SERP heatmap SEO competitor analysis using Python. Let’s review how the notebook works and cover where you will need to add your API key and your Search Engine ID into the script. 1. Install the Python Libraries The first step is to install the Advertools library and import it. This will make the Advertools functions available in your Google Colab notebook that allows us to run some of the advanced analyses below. 2. Add Keywords and your Search Engine ID/API Key Following the screenshot above, add your keywords and the information for your custom search engine in the relevant field. You can follow the attached video if you want to see an example of the code being ran. 3. Run the serp_goog function Now that you have added all of that information, you can go ahead and run the code. This creates a variable called serp_rankings that invokes the Advertools serp_goog function and your custom search engine to pull the keyword rankings from the topic or keyword group that you are analyzing. 4. View the results of the serp_rankings output Now you will run the serp_rankings.head(25) function which outputs the first 25 results in our dataframe. You can see the output of the first six results. A dataframe is very similar to an Excel worksheet or a Google Sheet. 5. Export Data to CSV Now we are ready to export the dataframe to a CSV file. This will prepare the file so you can download it from Google Colab to your computer. 6. Read your CSV file The next cell creates a variable called df and Pandas read_csv function processes or reads the CSV file we just created. 7. Run the SERP Heatmap Function This is where the magic happens. The serp_heatmap function processes the dataframe, counts the domains ranking in each position, groups them by domain, counts the total coverage, and then plots out the data. Once you run this cell you are ready to invoke the serp_heatmap function and pass in your dataframe. 8. Call or Invoke the serp_heatmap function The final cell will call the serp_heatmap function and pass in the dataframe of the rankings you pulled from your custom search engine. This will then output the SERP heatmap visualization that we looked at earlier. 9. View your SERP Heatmap Data Visualization Based on our set of keywords, here is the output. For the “running shoes” topic keywords, we analyzed Runner’s World currently has the best average position, highest coverage, most appearances, and the most spots in Positions 1 and 2. We could further dig into the data to extrapolate additional insight. But I will leave that to you during your testing and analysis. Conclusion There are many ways a marketer could use Python for competitor analysis. The SERP heatmap method is one option to gather a great number of insights into organic rankings, top organic competitors, identify top-performing content, and more. Other popular ways to use Python for your SEO competitor analysis include performing topic modeling to determine how you stack up to your competitors for topics and identify gaps in topic coverage. This can be a helpful way to get a high-level overview that is extremely insightful and informative for your overarching marketing strategy. Here is a word cloud example output you can get from running topic modeling on a data set using Latent Dirichlet Allocation (LDA) and Python. Here is an example of topic modeling by analyzing the distance of topics and the most relevant terms/entities for each topic. This can be extremely useful to understand your competitor’s topical authority. In a future Python SEO blog post, I will share the topic modeling script above. There are many other ways to use Python to analyze competitors. Comment and let me know what type of Python scripts you or your agency use to analyze SERPs or competitors. Want more content about Python and SEO? Check out my recent blog post about how to use Python for Named Entity Recognition and Analysis. If you are into technical SEO make sure to read my blog on how to use Python to Automate Screaming Frog SEO Analysis. Is your business ready for the transition to Google Analytics 4? Read our Google Analytics 4 (GA4) Guide to Almost Everything! If you are interested in SEO, connect with me on LinkedIn. I like to share SEO tips and talk about the state of SEO.