A Very Precise & Fast Way to Pull Google Trends Data AutomaticallyA one-stop-shop script to automatically pull google trends by exact keywords using Pytrends COVID-19 has provided a boon for Google Trends usage in the U.S. (see chart below). After all, its a free publicly available tool that provides access to actual search requests across google search engine. More companies have leveraged Google Trends to gain insights on category trend, consumer top searched queries, competitors performance amid the black swan pandemic event. Each Google Trends request consists of 5 unique parameters:
CHALLENGES: There are two main challenges to pull Google trends data in scale. 1. Individual keyword by keyword manually pulling is time-consuming. Although Google Trends provides the Compare function to compare keywords, the downside is that it scales the results from 0 to 100 based on the most popular term entered. The less popular keyword will lose sensitivity quickly if you compare it with a popular one. For example, if you compare Nike brand with Supreme brand, you will basically get a flat line for Supreme. In this situation, youll get big errors when reporting Supreme search growth trend. Thus, it is suggested to pull Nike trend and Supreme trend individually and separately. You can still afford the time until you have too many keywords to pull. E.g. when to compare 50 apparel and footwear brands, you need to download 50 excels and combine them together. 2. Current available automation python methods dont query exact keywords, meaning theyre not accurate. There are many available methods to pull google trend data using python. However, none of them builds automatic codes that could pull EXACT KEYWORDS. Its not unusual that words often have multiple meanings. For example, Patagonia could either be a clothing company or a region in South American. Simply search Patagonia will give us ambiguous results as it contains both search terms. We need to make sure we search the right keywords. GOAL: The goal is to provide a solution to pull google trends data for multiple keywords exactly, individually, and automatically. Specifically, well pull google trends data for six apparel/footwear brands (Nike, Adidas, Under Armour, Zara, H&M, Louis Vuitton) in three countries (US, UK, Germany). Using Python, there are four steps to achieve this:
Step 1: Install pytrends APIFirst of all, we need to install that package called pytrends, which is designed to pull google trends using python. Simply execute the following code from your Terminal. You could find the comprehensive documentation of the pytrends API here. pip install pytrendsWe will then import the necessary packages. import pandas as pdimport pytrends from pytrends.request import TrendReq pytrend = TrendReq() Step 2: Get exact keywordsAs discussed earlier, we need to be precise on the keywords to avoid ambiguity. Pytrends provides a function called pytrend.suggestions that could return several suggestions for a keyword. Usually, the first suggestion is the most popular one. mid column contains those exact keywords wed like to search. KEYWORDS=['Nike','Adidas','Under Armour','Zara','H&M','Louis Vuitton']KEYWORDS_CODES=[pytrend.suggestions(keyword=i)[0] for i in KEYWORDS] df_CODES= pd.DataFrame(KEYWORDS_CODES) df_CODES Step 3: Get Google trends data by exact keywordsNext, well set those 5 parameters. EXACT_KEYWORDS=df_CODES['mid'].to_list()DATE_INTERVAL='2020-01-01 2020-05-01' COUNTRY=["US","GB","DE"] #Use this link for iso country code CATEGORY=0 # Use this link to select categories SEARCH_TYPE='' #default is 'web searches',others include 'images','news','youtube','froogle' (google shopping) Then, well write codes to pull google trends data by exact keywords by country using the above parameters. Individual_EXACT_KEYWORD = [list(x) for x in Individual_EXACT_KEYWORD] dicti = {} i = 1 for Country in COUNTRY: for keyword in Individual_EXACT_KEYWORD: pytrend.build_payload(kw_list=keyword, timeframe = DATE_INTERVAL, geo = Country, cat=CATEGORY, gprop=SEARCH_TYPE) dicti[i] = pytrend.interest_over_time() i+=1 df_trends = pd.concat(dicti, axis=1) And we will do some cleaning work and change those exact keywords back to readable brand names. df_trends.columns = df_trends.columns.droplevel(0) #drop outside headerdf_trends = df_trends.drop('isPartial', axis = 1) #drop "isPartial" df_trends.reset_index(level=0,inplace=True) #reset_index df_trends.columns=['date','Nike-US','Adidas-US','Under Armour-US','Zara-US','H&M-US','Louis Vuitton-US','Nike-UK','Adidas-UK','Under Armour-UK','Zara-UK','H&M-UK','Louis Vuitton-UK', 'Nike-Germany','Adidas-Germany','Under Armour-Germany','Zara-Germany','H&M-Germany','Louis Vuitton-Germany'] #change column names Step 4: Visualize Google trendsIn the blink of an eye, we get our google trends data. Finally, lets visualize Louis Vuitton's google trends across countries. As we could see, Louis Vuitton clearly gets hard hit by COVID-19, just like many other brands and industries. import seaborn as snssns.set(color_codes=True) dx = df_trends.plot(figsize = (12,8),x="date", y=['Louis Vuitton-US','Louis Vuitton-UK','Louis Vuitton-Germany'], kind="line", title = "Louis Vuitton Google Trends") dx.set_xlabel('Date') dx.set_ylabel('Trends Index') dx.tick_params(axis='both', which='both', labelsize=10) SUMMARYGoogle Trends tool usages see surging growth as companies closely monitor shifting consumer behavior during COVID -19. To help with this, we build a light application to pull Google Trends data precisely and automatically. Finally, stay positive, thanks for reading! |