Create dataframe in for loop python

Recipe Objective

In python, while operating on list, we might need to store each loop output in a dataframe with each iteration.

So this recipe is a short example on how to append output of for loop in a pandas dataframe. Let's get started.

Get Closer To Your Dream of Becoming a Data Scientist with 70+ Solved End-to-End ML Projects

Table of Contents

• Recipe Objective
  • Step 1 - Import the library
  • Step 2 - Setup the Data
  • Step 3 - Appending dataframe in a for loop
  • Step 4 - Printing results
  • Step 5 - Let's look at our dataset now

Step 1 - Import the library

import pandas as pd

Let's pause and look at these imports. Pandas is generally used for data manipulation and analysis.

Step 2 - Setup the Data

df= pd.DataFrame({'Table of 9': [9,18,27], 'Table of 10': [10,20,30]})

Let us create a dataframe containing some tables of 9 and 10.

Step 3 - Appending dataframe in a for loop

for i in range(4,11): df=df.append({'Table of 9':i*9,'Table of 10':i*10},ignore_index=True)

Comparing to append function in list, it applies a bit different for dataframe. As soon as any dataframe gets appnended using append function, it is note reflected in original dataframe. To store the appended information in a dataframe, we again assign it back to original dataframe.

Step 4 - Printing results

print('df\n',df)

Simply use print function to print new appended dataframe.

Step 5 - Let's look at our dataset now

Once we run the above code snippet, we will see:

Scroll down to the ipython notebook below to see the output.

In this post, you’ll learn how to create an empty pandas dataframe and how to add data to them. Specifically, you’ll learn how to create the dataframe, create one with columns, add rows one-by-one and add rows via a loop.

• Create an Empty Pandas Dataframe
• Create an Empty Pandas Dataframe with Columns
• Create an Empty Pandas Dataframe with Columns and Indices
• Add Data to an Empty Dataframe
• Append Data to an Empty Pandas Dataframe
• Append to Empty Pandas Dataframe with a Loop
• Conclusion

To start things off, let’s begin by import the Pandas library as pd:

import pandas as pd

Creating a completely empty Pandas Dataframe is very easy. We simply create a dataframe object without actually passing in any data:

df = pd.DataFrame()

print(df)

This returns the following:

Empty DataFrame
Columns: []
Index: []

We can see from the output that the dataframe is empty.

However, we can also check if it’s empty by using the Pandas .empty attribute, which returns a boolean value indicating if the dataframe is empty:

>> print(df.empty)
True

Create an Empty Pandas Dataframe with Columns

There may be time when you know the columns you’ll want in a dataframe, but just don’t have the data for it yet (more on that in appending data to an empty dataframe below).

In order to do this, we can use the columns= parameter when creating the dataframe object to pass in a list of columns. Let’s create a dataframe with the following columns: Name, Age, Birth City, and Gender.

df = pd.DataFrame(columns=['Name', 'Age', 'Birth City', 'Gender'])

print(df)

This prints out the following, indicating that we now have an empty dataframe but with columns attached to it:

Empty DataFrame
Columns: [Name, Age, Birth City, Gender]
Index: []

Create an Empty Pandas Dataframe with Columns and Indices

Similar to the situation above, there may be times when you know both column names and the different indices of a dataframe, but not the data.

We can accomplish creating such a dataframe by including both the columns= and index= parameters. Let’s create the same dataframe as above, but use the Name column as the index and fill in some sample indices:

df = pd.DataFrame(
    columns=['Age', 'Birth City', 'Gender'],
    index=['Jane', 'Melissa', 'John', 'Matt'])

print(df)

This returns the following:

         Age Birth City Gender
Jane     NaN        NaN    NaN
Melissa  NaN        NaN    NaN
John     NaN        NaN    NaN
Matt     NaN        NaN    NaN

Now, technically, this isn’t an empty dataframe anymore. It’s simply a dataframe without data. We can verify this by using the .empty attribute:

print(df.empty)

This returns False.

Add Data to an Empty Dataframe

Now that we have our dataframe with both columns and indices, we can use .loc to add data to it. If you want to learn more about .loc, check out my tutorial here.

Let’s add some data to the record with index Jane:

df.loc['Jane',:] = [23, 'London', 'F']

print(df)

This now returns the following dataframe:

         Age Birth City Gender
Jane      23     London      F
Melissa  NaN        NaN    NaN
John     NaN        NaN    NaN
Matt     NaN        NaN    NaN

Append Data to an Empty Pandas Dataframe

Similar to adding rows one-by-one using the Pandas .loc, we can also use the .append() method to add rows.

The .append() method works by, well, appending a dataframe to another dataframe.

Let’s add the same row above using the append method:

df2 = pd.DataFrame(
    [['Jane', 23, 'London', 'F']], 
    columns=['Name', 'Age', 'Birth City', 'Gender']
    )

df = df.append(df2)

print(df)

This returns the following dataframe:

   Name Age Birth City Gender
0  Jane  23     London      F

To speed things up, we can also use a for loop to add data, as explore below.

Append to Empty Pandas Dataframe with a Loop

There may be times when you need to add multiple pieces of data to a dataframe. This can be simplified using a for loop, to, say, read multiple files and append them. To learn more about Python’s for loops, check out my post here.

In the example below, we’ll just work with different lists, but the method works the same if you read data from multiple iterative files.

We use the ignore_index = True argument to ensure that we create new indices. Otherwise, each index could be duplicated when reading in multiple dataframes.

df = pd.DataFrame(
    columns=['Name', 'Age', 'Birth City', 'Gender'])

people = [
    ['Jane', 23, 'London', 'F'],
    ['Melissa', 45, 'Paris', 'F'],
    ['John', 35, 'Toronto', 'M']
]

for person in people:
    temporary_df = pd.DataFrame([person], columns=['Name', 'Age', 'Birth City', 'Gender'])
    df = df.append(temporary_df, ignore_index=True)

print(df)

This returns the following dataframe:

      Name Age Birth City Gender
0     Jane  23     London      F
1  Melissa  45      Paris      F
2     John  35    Toronto      M

Conclusion

In this post, you learned how to create an empty dataframe, both with and without columns. Following that, you learned how to append data to an empty dataframe, both a single time as well as how to do it with a for loop. To learn more about the Pandas .DataFrame() class, check out the official documentation here. To learn more about the .append() method, check out the official documentation here.

How do you create a for loop in a data frame?

How do I put the results of a loop into a DataFrame in python?

Can you loop a DataFrame in python?

How do you add a column to a DataFrame in for loop in python?