Recipe Objective
In python, while operating on list, we might need to store each loop output in a dataframe with each iteration.
So this recipe is a short example on how to append output of for loop in a pandas dataframe. Let's get started.
Get Closer To Your Dream of Becoming a Data Scientist with 70+ Solved End-to-End ML Projects
Table of Contents
- Recipe Objective
- Step 1 - Import the library
- Step 2 - Setup the Data
- Step 3 - Appending dataframe in a for loop
- Step 4 - Printing results
- Step 5 - Let's look at our dataset now
Step 1 - Import the library
import pandas as pd
Let's pause and look at these imports. Pandas is generally used for data manipulation and analysis.
Step 2 - Setup the Data
df= pd.DataFrame({'Table of 9': [9,18,27], 'Table of 10': [10,20,30]})
Let us create a dataframe containing some tables of 9 and 10.
Step 3 - Appending dataframe in a for loop
for i in range(4,11): df=df.append({'Table of 9':i*9,'Table of 10':i*10},ignore_index=True)
Comparing to append function in list, it applies a bit different for dataframe. As soon as any dataframe gets appnended using append function, it is note reflected in original dataframe. To store the appended information in a dataframe, we again assign it back to original dataframe.
Step 4 - Printing results
print('df\n',df)
Simply use print function to print new appended dataframe.
Step 5 - Let's look at our dataset now
Once we run the above code snippet, we will see:
Scroll down to the ipython notebook below to see the output.In this post, you’ll learn how to create an empty pandas dataframe and how to add data to them. Specifically, you’ll learn how to create the dataframe, create one with columns, add rows one-by-one and add rows via a loop.
- Create an Empty Pandas Dataframe
- Create an Empty Pandas Dataframe with Columns
- Create an Empty Pandas Dataframe with Columns and Indices
- Add Data to an Empty Dataframe
- Append Data to an Empty Pandas Dataframe
- Append to Empty Pandas Dataframe with a Loop
- Conclusion
To start things off, let’s begin by import the Pandas library as pd:
import pandas as pdCreating a completely empty Pandas Dataframe is very easy. We simply create a dataframe object without actually passing in any data:
df = pd.DataFrame() print(df)This returns the following:
Empty DataFrame Columns: [] Index: []We can see from the output that the dataframe is empty.
However, we can also check if it’s empty by using the Pandas .empty attribute, which returns a boolean value indicating if the dataframe is empty:
>> print(df.empty) TrueCreate an Empty Pandas Dataframe with Columns
There may be time when you know the columns you’ll want in a dataframe, but just don’t have the data for it yet (more on that in appending data to an empty dataframe below).
In order to do this, we can use the columns= parameter when creating the dataframe object to pass in a list of columns. Let’s create a dataframe with the following columns: Name, Age, Birth City, and Gender.
df = pd.DataFrame(columns=['Name', 'Age', 'Birth City', 'Gender']) print(df)This prints out the following, indicating that we now have an empty dataframe but with columns attached to it:
Empty DataFrame Columns: [Name, Age, Birth City, Gender] Index: []Create an Empty Pandas Dataframe with Columns and Indices
Similar to the situation above, there may be times when you know both column names and the different indices of a dataframe, but not the data.
We can accomplish creating such a dataframe by including both the columns= and index= parameters. Let’s create the same dataframe as above, but use the Name column as the index and fill in some sample indices:
df = pd.DataFrame( columns=['Age', 'Birth City', 'Gender'], index=['Jane', 'Melissa', 'John', 'Matt']) print(df)This returns the following:
Age Birth City Gender Jane NaN NaN NaN Melissa NaN NaN NaN John NaN NaN NaN Matt NaN NaN NaNNow, technically, this isn’t an empty dataframe anymore. It’s simply a dataframe without data. We can verify this by using the .empty attribute:
print(df.empty)This returns False.
Add Data to an Empty Dataframe
Now that we have our dataframe with both columns and indices, we can use .loc to add data to it. If you want to learn more about .loc, check out my tutorial here.
Let’s add some data to the record with index Jane:
df.loc['Jane',:] = [23, 'London', 'F'] print(df)This now returns the following dataframe:
Age Birth City Gender Jane 23 London F Melissa NaN NaN NaN John NaN NaN NaN Matt NaN NaN NaNAppend Data to an Empty Pandas Dataframe
Similar to adding rows one-by-one using the Pandas .loc, we can also use the .append() method to add rows.
The .append() method works by, well, appending a dataframe to another dataframe.
Let’s add the same row above using the append method:
df2 = pd.DataFrame( [['Jane', 23, 'London', 'F']], columns=['Name', 'Age', 'Birth City', 'Gender'] ) df = df.append(df2) print(df)This returns the following dataframe:
Name Age Birth City Gender 0 Jane 23 London FTo speed things up, we can also use a for loop to add data, as explore below.
Append to Empty Pandas Dataframe with a Loop
There may be times when you need to add multiple pieces of data to a dataframe. This can be simplified using a for loop, to, say, read multiple files and append them. To learn more about Python’s for loops, check out my post here.
In the example below, we’ll just work with different lists, but the method works the same if you read data from multiple iterative files.
We use the ignore_index = True argument to ensure that we create new indices. Otherwise, each index could be duplicated when reading in multiple dataframes.
df = pd.DataFrame( columns=['Name', 'Age', 'Birth City', 'Gender']) people = [ ['Jane', 23, 'London', 'F'], ['Melissa', 45, 'Paris', 'F'], ['John', 35, 'Toronto', 'M'] ] for person in people: temporary_df = pd.DataFrame([person], columns=['Name', 'Age', 'Birth City', 'Gender']) df = df.append(temporary_df, ignore_index=True) print(df)This returns the following dataframe:
Name Age Birth City Gender 0 Jane 23 London F 1 Melissa 45 Paris F 2 John 35 Toronto MConclusion
In this post, you learned how to create an empty dataframe, both with and without columns. Following that, you learned how to append data to an empty dataframe, both a single time as well as how to do it with a for loop. To learn more about the Pandas .DataFrame() class, check out the official documentation here. To learn more about the .append() method, check out the official documentation here.