Cara menggunakan tabulate nested dictionary python

I suppose the same could be said of extracting values from nested JSON structures. Even the most skilled programmer can be brought to tears when working with a JSON object that consists of a mix of deeply nested data structures. The process of extracting the values can feel messy and disorganized at best. The more data there is, the bigger the mess.

Course created by Author

In this tutorial, I’ll walk you through a step-by-step method to extract the values you need from any JSON. A word of warning: this tutorial is not meant for newbies to JSON, lists or dictionaries. If you’ve never heard of a list index or a dictionary key-value pair, I would suggest reviewing one of the many great tutorials available on the web or YouTube. Once you feel more comfortable with the subject, come back to continue learning and growing.

Housekeeping

JSON vs. Lists vs. Dictionaries

First things first, when it comes to the terms “JSON”, “list” and “dictionary”, we have to do some important housekeeping. JSON, or JavaScript Object Notation, is a broader format used to encompass dictionary and list structures as shown in the image below.

JSON: List and Dictionary Structure, Image by Author.

The technical documentation says a JSON object is built on two structures: a list of key-value pairs and an ordered list of values. In Python Programming, key-value pairs are dictionary objects and ordered list are list objects. In practice, the starting point for the extraction of nested data starts with either a dictionary or list data structure. When extracting Nested Data the question(s) should be: Is the data nested in a Dictionary or List data structure? What is the combination of data structures used? Is the first data structure used a dictionary or a list?

“It has long been an axiom of mine that the little things are infinitely the most important.” — Sir Arthur Conan Doyle,

If it seems like I’m making a big deal about the terminology, it is because I am. When it comes to extracting nested data the details matter. Data structures change the deeper the data is nested in the JSON structure and knowing those distinctions are important. The initial data structure may be a list but then change to a dictionary as the data is extracted. The key to extracting data from a JSON object is recognizing the mix of data structures used to store the data. If you struggle to recognize the data structure in a JSON object, it’s likely that you’ll struggle to extract the values you want. In most cases, this results in applying the wrong extraction technique.

The table below is a brief refresher on the techniques used to extract data from a JSON structure.

Data Types and Extraction Methods, Image by Author

One final note before starting our example. In Python Programming, the term “data structure” is rarely used when describing lists and dictionary. The commonly used term is “data type”. I use the terms data type and data structure interchangeably throughout this tutorial. I use the term data structure because it conveys the idea that the data structures are the fundamental building blocks of the JSON object. The usage of the term data type in Python is not of less importance however it does not convey the same meaning as a key to understanding nested data extraction.

Real World Data

Let's Get Started

One of the best ways to learn is by working through real data with a mix of list and dictionary data structures. In this tutorial, we’ll use real data from the . This API returns about 250 records with a mix of dictionaries, lists and other data types. Our objective is to extract the 'AFN’ value from the dictionary key-value pair 'code':'AFN' as shown in the image below. The 'AFN' is nested in two list structures and one dictionary structure.

REST Countries API Data, Image by Author

The Sample Code

Clicking this link will allow you to access the sample code in the following examples. The link will take you to a course I developed on learning to extract nested JSON data. The course has helped hundreds of students learn to extract nested data. You don’t have to purchase the course to obtain the files. The filenames are single_json.py and multiple_json.py.

Extracting Single Items

In this example, we’ll start by extracting data using a combination of list and dictionary extraction techniques as shown in the preceding table. In the Python code below, I start by providing the logic to import the data, the solution and then the workflow to derive the solution. I recommend following all the steps as shown below. The workflow steps are explained below the Python code.

Python Code:

Workflow Steps:

  • Step 1: import requests: this line imports the Requests HTTP library for Python. It is the library we use to connect to a Restful API. If you haven’t already installed it, you can install it from the command prompt or virtual environment using the pip install requests command.
  • Step 2: url = 'https://restcountries.eu/rest/v2/all' this line stores the web address for the REST API. The address is stored in the url variable.
  • Step 3: response = requests.get(url): this method is used to connect to the Restful API, https://restcountries.eu/rest/v2/all, to extract the data. The data returned is stored in the response variable. In technical terms, this is referred to as the response object.
  • Step 4: storage = response.json()returns a JSON object of the result (if the result was written in JSON format, if not it raises an error). Think of the .json()as a storage format used to exchange the data. In this instance, we store the content in the 'code':'AFN'0 variable.
  • 'code':'AFN'1: this returns the Python data type used to store the data. In this instance, the data type returned will be a list ( 'code':'AFN'2). Looking back the table I provided earlier, the data can be extracted using the list index [0,…]. You should always use the 'code':'AFN'3function to determine the data type. If you know the data type, you know the correct extraction technique to use.
  • 'code':'AFN'4: this provides the number of items in the list. Each number represents the index of an item in the list and the index can be used to extract the value.
  • 'code':'AFN'5: the 0 represents the first item in the list AND is used to extract the item from the list. Once the item is extracted, a new data type is now exposed. The 'code':'AFN'3function in step 8 is used to show the data type.
  • 'code':'AFN'7: the new data type will be 'code':'AFN'8. The dictionary means that the data can be extracted using a key. In our data, the key used to extract the value is 'code':'AFN'9. The 'code':'AFN'9 key is used in step 9. Take note, the data type changed from 'code':'AFN'2 in step 5 to 'AFN'2 in step 8.
  • 'AFN'3: the 'code':'AFN'9 key in the dictionary is used to output 'AFN'5. Once the item is extracted, a new data type is now exposed. The 'code':'AFN'3function in step 10 is used to show the data type.
  • 'AFN'7: the new data type will be 'AFN'8. The list data type means we use the index operator 'AFN'9 to extract the next set of values. In this instance the index value will be 0. So we use it in step 11. Take note, the data type changed from 'AFN'2 in step 8 to 'code':'AFN'2 in step 10.
  • Step 1: import requests2: the indexStep 1: import requests3 is used to output Step 1: import requests4. Once the item is extracted, a new data type is now exposed. The 'code':'AFN'3function in step 12 is used to show the data type.
  • Step 1: import requests6: the new data type is'AFN'2. The dictionary means the data can be extracted using a key. In our data, the key used to extract the value is Step 1: import requests8 . The Step 1: import requests8 key is used in step 13
  • Step 2: url = 'https://restcountries.eu/rest/v2/all'0: The Step 1: import requests8 key is used to output the Step 2: url = 'https://restcountries.eu/rest/v2/all'2 which is the value we're looking to output.

So, let’s summarize the two steps that are repeated until we arrive at the value we want to extract. The first step is to determine the data type and the second step is to apply the extraction method. If the datatype is list, then use the index operator with square brackets. However, if the datatype is a dictionary, use the dictionary key with curly brackets.

Extracting Multiple Items

While extracting a single list item from a JSON structure is an important first step, it is not common to extract only a single value. In real-world data, values in JSON objects are stored as collections. In the image below, the 'code':'AFN'9 and Step 1: import requests8 dictionary keys and values have multiple entries in the list data structure. Our real world example, has 250 of these entries, so our goal in this section of the tutorial will be to extract these and the remaining values.

REST Countries API Data, Image by Author

Fortunately, we can extract these values by building on the workflow steps we used to extract a single value from the JSON structure. I won’t list those steps again. Since the list and dictionary data structures are iterable we can use a Step 2: url = 'https://restcountries.eu/rest/v2/all'5 structure to traverse through our values. So, lets’ add that to our code in step 14. We’ll also need to use the Step 2: url = 'https://restcountries.eu/rest/v2/all'6 and Step 2: url = 'https://restcountries.eu/rest/v2/all'7 functions to enumerate the number of items in the list. The workflow steps are explained below the Python code.

Python Code:

Workflow Steps:

Step 2: url = 'https://restcountries.eu/rest/v2/all'8 The Step 2: url = 'https://restcountries.eu/rest/v2/all'9 variable contains the list 'code':'AFN'2returned in Step 4. The Step 2: url = 'https://restcountries.eu/rest/v2/all'7 function is used to count the number of items in the list. The Step 2: url = 'https://restcountries.eu/rest/v2/all'6 function generates a sequence of the numbers based on the number of items in the list. The number is passed to the url3 variable each time the Step 2: url = 'https://restcountries.eu/rest/v2/all'5 iterates through the Step 2: url = 'https://restcountries.eu/rest/v2/all'9 list. In line 48, the url6 shown in step 14 comes from step 13. We’ve encased it in the Step 2: url = 'https://restcountries.eu/rest/v2/all'5 structure and modified the code from url8 → url9 . The url8 references a single item in the list. However, the url9 captures multiple items in the Step 2: url = 'https://restcountries.eu/rest/v2/all'9 list. Each time the Step 2: url = 'https://restcountries.eu/rest/v2/all'5 iterates,the url9 is incremented to capture the next set of values.

Conclusion

The process of unearthing nested data can at times feel daunting, circuitous and exasperating. It does not lend itself easily to introductory techniques such as iterating thorough Step 3: response = requests.get(url)5. Inevitably, the complexity of the JSON structure requires alternating between multiple dictionaries and list extraction techniques to extract the data. Getting the hang of it requires a repeatable process but more importantly practice. I’ve included several additional practice problems with solutions (click here).