How do you pull data from Google Trends using Python?

By Mr. Data Science

A brief overview:

Throughout this article, I will:

• Describe the Pytrends python library and the syntax you need to get Google Trends data
• Demonstrate how to get google trends data using several examples

These are my assumptions:

Since I cant cover everything in this article, I will assume the following:

• You have Python>3.3 installed on your machine and know how to use it
• You are able to install new python libraries on your own
• You want to programmatically get google trends data

Google Trends and the Pytrends library:

Google Trendsis a Google owned website that analyzes search term popularity across various regions and languages. On theGoogle Trends site, you can see trending keywords, assess the popularity of specific search terms over different time periods, explore related keywords, and much more. I encourage you to check out their site if youve never been there before. Beware, youll likely go down a rabbit hole ;).

As a curious and innovative human being, you might be interested in tapping into this wonderful resource to make predictions or understand human behavior. The way you do that is using thePytrends python library. Pytrends, according to thePytrends PyPI page, is an unofficial Google Trends API that can be used to extract useful search information using a simple interface. Beware that, due to the unofficial API status, any changes to Google Trends backend could impact a users ability to use the library. Make sure Pytrends is still active and up to date when you read this article.

Setting up your Python environment:

To use the Pytrends API, you will need to do the following:

• Make sure you have an internet connection
• Open your command prompt and make sure you can run python and pip from your command prompt. Seethis linkif you need help
• Runpip install pytrendsto install the Pytrends library.
  • Note that during installation, pip may need to install Requests, lxml, and Pandas. This is normal behavior as these libraries are required for Pytrends to work properly.
• Run pip freeze to display the installed python libraries. verify that pytrends appears on the list.

Getting Started

The first thing you want to do is establish a connection with google. There are several options for adjusting connection parameters (such as using proxies, timeouts and retries), but that is beyond the scope of this discussion. For now, lets just connect using the following commands.

In[1]:

After establishing a connection with Google, you must build a payload. For an API, the payload contains relevant request information that helps the service understand what data you want to obtain. A simple payload building example is shown below.

In[2]:

In the example, several parameters were used to tell google the type of information we would like to request.

• kw_list: a list of keywords to get data for (limit of 5 keywords at a time)
• geo: the geographic location.
• timeframe: the time frame of interest.

Lets look at a few examples:

After youve built the payload, you can start to extract useful information using several Pytrends methods. Several examples are shown below:

Example 1: Interest over time

First, lets take a look at how these keywords changed in popularity over time. lets get the interest over time and print the top 20 rows to see what the data looks like first.

In[3]:

Now that we know what the data looks like, lets plot the popularity of each keyword over the time period we requested:

In[4]:

Out[4]:

In the plot above, we can see that:

• Chicken is more popular than ai and space (makes sense since people are generally more interested in food)
• The popularity of chicken is growing over time whereas ai and space popularity are rather stagnant
• There is a spike in chickens popularity at the beginning of each year (All those new years resolutions)

While silly to compare these to each other, you can start to see the power of using google trends for inference. If I was a betting man, I would say chickens popularity will continue to grow Lets see how its popularity compares to other meats

In[5]:

Out[5]:

The results indicate that chicken is much more popular than other meats. Dont race to invet in chicken however, the percentage change in popularity appears to be fairly consistent among the different meets

In[6]:

Out[6]:

Example 2: Related topics, queries, and suggestions

While interest over time is interesting, you may want to use google trends to find related topics, queries, and/or suggestions. To do so, you can use the related_topics(), related_queries(), and suggestions() methods.

First, lets look at related topics:

In[7]:

df_rt is a dictionary of related_topics information. Specifically, the dictionary contains a keyword key and two related topics subkeys (rising and top). If you are looking for rising popularity topics, use the rising results. If you just want the top search results, look in top Lets look at the top 5 rising results:

In[8]:

In the rising set of results, we can see that instant pot and air fryer searches are among the top rising topics related to chicken. Next, lets see the top topics:

In[9]:

These results are less exciting because the word chicken is in 3 of the topic_title rows. However, you can see that the topic_type or category changes. To google, these types of searches provide somewhat different results.

Next, lets look at related queries:

In[10]:

Like related_topics, related_queries returns a dictionary containing top and rising query results.

Finally, we can get suggested search keywords as shown below:

In[11]:

Unlike related_topics and related_queries, suggestions does not require that a payload be built first. Instead, the keyword is passed to the method. The suggestions are just like the autocomplete information you get when typing in your search.

Example 3: Trending searches and top search charts

In the final set of examples, lets look at trending and top search results. This use case is more exploratory than the others and helps answer the question, what are other people searching for. First, lets look at trending searches.

In[12]:

Out[12]:

On 1/23/2021, these were the trending search results in the United States. Simply change pn to the location you desire in order to get trending information for your area of interest.

to get top results, you must include the date. host language(hl), timezone(tz) and geographic location(geo) are optional inputs.

In[13]:

Out[13]:

An IMPORTANT note on multiple api requests:

Before concluding our discussion on using Pytrends, I wanted to leave you with a brief warning. Like any site, Google has specific terms of service that you should review prior to using Pytrends. If you overuse the Pytrends API, Google WILL throttle your access to their site. According to the Pytrends PyPI page, the Google Trends request rate limit is not publicly known. Apparently, one user reported that 1,400 sequential requests of a 4 hour timeframe got them to the limit, but this likely varies with query size and request frequency. Upon reaching the rate limit, Google will throttle future requests. At least 60 seconds of sleep will be necessary once you hit the limit. Please note that this may change over time. I will do my best to keep this post current.

A quick review of what youve learned:

If youve made it this far, you are more than ready to use Pytrends in your application. Specifically, youve learned:

• How to install the Pytrends library
• How to establish a connection with Google and Build an API payload using Pytrends
• How to get interest over time for multiple keywords by geographic location, timezones, and date ranges
• How to get keyword related topics, queries, and suggestions
• How to identify trending and top search results

If you have any feedback or suggestions for improving this article, we would love to hear from you.

References:

• Pytrends 4.7.3.https://pypi.org/project/pytrends/
• Google Trends.https://trends.google.com/trends/?geo=US
• Installing Packages.https://packaging.python.org/tutorials/installing-packages/

Connect With Mr Data Science:

MrDataScience.com,GitHub,Medium,