How do you remove duplicates from two lists in Python?

Python – Ways to remove duplicates from list

This article focuses on one of the operations of getting the unique list from a list that contains a possible duplicated. Remove duplicates from list operation has large number of applications and hence, it’s knowledge is good to have.

Table of Contents Show

Python – Ways to remove duplicates from list
Python | Difference of two lists including duplicates
Remove duplicates from list using Set
Remove Duplicates from a list using the Temporary List
Python merge list without duplicates example
Combining two lists and without duplicates and don’t remove duplicates in the original list
Share this:
Python: 5 Ways to Remove Duplicates from List

Method 1 : Naive method
In naive method, we simply traverse the list and append the first occurrence of the element in new list and ignore all the other occurrences of that particular element.

# Python 3 code to demonstrate 
# removing duplicated from list 
# using naive methods 

# initializing list 

test_list = [1, 3, 5, 6, 3, 5, 6, 1]

print ("The original list is : " + str(test_list))

# using naive method 
# to remove duplicated 
# from list 

res = []

for i in test_list:

if i not in res:

res.append(i)

# printing list after removal 

print ("The list after removing duplicates : " + str(res))

Output :

The original list is : [1, 3, 5, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]

Method 2 : Using list comprehension
This method has working similar to the above method, but this is just a one-liner shorthand of longer method done with the help of list comprehension.

# Python 3 code to demonstrate 
# removing duplicated from list 
# using list comprehension 

# initializing list 

test_list = [1, 3, 5, 6, 3, 5, 6, 1]

print ("The original list is : " + str(test_list))

# using list comprehension 
# to remove duplicated 
# from list 

res = []

[res.append(x) for x in test_list if x not in res]

# printing list after removal 

print ("The list after removing duplicates : " + str(res))

Output :

The original list is : [1, 3, 5, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]

Method 3 : Using set()
This is the most popular way by which the duplicated are removed from the list. But the main and notable drawback of this approach is that the ordering of the element is lost in this particular method.

# Python 3 code to demonstrate 
# removing duplicated from list 
# using set() 

# initializing list 

test_list = [1, 5, 3, 6, 3, 5, 6, 1]

print ("The original list is : " + str(test_list))

# using set() 
# to remove duplicated 
# from list 

test_list = list(set(test_list))

# printing list after removal 
# distorted ordering 

print ("The list after removing duplicates : " + str(test_list))

Output :

The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 3, 5, 6]

Method 4 : Using list comprehension + enumerate()
list comprehension coupled with enumerate function can also achieve this task. It basically looks for already occurred elements and skips adding them. It preserves the list ordering.

# Python 3 code to demonstrate 
# removing duplicated from list 
# using list comprehension + enumerate() 

# initializing list 

test_list = [1, 5, 3, 6, 3, 5, 6, 1]

print ("The original list is : " + str(test_list))

# using list comprehension + enumerate() 
# to remove duplicated 
# from list 

res = [i for n, i in enumerate(test_list) if i not in test_list[:n]]

# printing list after removal 

print ("The list after removing duplicates : " + str(res))

Output :

The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]

Method 5 : Using collections.OrderedDict.fromkeys()
This is fastest method to achieve the particular task. It first removes the duplicates and returns a dictionary which has to be converted to list. This works well in case of strings also.

# Python 3 code to demonstrate 
# removing duplicated from list 
# using collections.OrderedDict.fromkeys() 

from collections import OrderedDict

# initializing list 

test_list = [1, 5, 3, 6, 3, 5, 6, 1]

print ("The original list is : " + str(test_list))

# using collections.OrderedDict.fromkeys() 
# to remove duplicated 
# from list 

res = list(OrderedDict.fromkeys(test_list))

# printing list after removal 

print ("The list after removing duplicates : " + str(res))

Output :

The original list is : [1, 5, 3, 6, 3, 5, 6, 1] The list after removing duplicates : [1, 5, 3, 6]

Article Tags :

Python

Python list-programs

python-list

Practice Tags :

python-list

Read Full Article

Python | Difference of two lists including duplicates

The ways to find difference of two lists has been discussed earlier, but sometimes, we require to remove only the specific occurrences of the elements as much they occur in other list. Let’s discuss certain ways in which this can be performed.

Method #1 : Using collections.Counter()
The Counter method can be used to get the exact occurrence of the elements in the list and hence can subtract selectively rather than using the set and ignoring the count of elements altogether. Then the subtraction can be performed to get the actual occurrence.

# Python3 code to demonstrate 
# Difference of list including duplicates 
# Using collections.Counter() 

from collections import Counter

# initializing lists 

test_list1 = [1, 3, 4, 5, 1, 3, 3]

test_list2 = [1, 3, 5]

# printing original lists 

print("The original list 1 : " + str(test_list1))

print("The original list 2 : " + str(test_list2))

# Using collections.Counter() 
# Difference of list including duplicates 

res = list((Counter(test_list1) - Counter(test_list2)).elements())

# print result 

print("The list after performing the subtraction : " + str(res))

Output : The original list 1 : [1, 3, 4, 5, 1, 3, 3] The original list 2 : [1, 3, 5] The list after performing the subtraction : [1, 3, 3, 4]

Method #2 : Using map() + lambda + remove()
The combination of above functions can be used to perform this particular task. The map function can be used to link the function to all elements and remove removes the first occurrence of it. Hence doesn’t remove repeatedly. Works with Python2 only.

# Python code to demonstrate 
# Difference of list including duplicates 
# Using map() + lambda + remove() 

# initializing lists 

test_list1 = [1, 3, 4, 5, 1, 3, 3]

test_list2 = [1, 3, 5]

# printing original lists 

print("The original list 1 : " + str(test_list1))

print("The original list 2 : " + str(test_list2))

# Using map() + lambda + remove() 
# Difference of list including duplicates 

res = map(lambda x: test_list1.remove(x)

if x in test_list1 else None, test_list2)

# print result 

print("The list after performing the subtraction : " + str(test_list1))

Output : The original list 1 : [1, 3, 4, 5, 1, 3, 3] The original list 2 : [1, 3, 5] The list after performing the subtraction : [1, 3, 3, 4]

Article Tags :

Python

Python Programs

Python list-programs

Read Full Article

Remove duplicates from list using Set

To remove the duplicates from a list, you can make use of the built-in function set(). The specialty of set() method is that it returns distinct elements.

We have a list : [1,1,2,3,2,2,4,5,6,2,1]. The list has many duplicates which we need to remove and get back only the distinct elements. The list is given to the set() built-in function. Later the final list is displayed using the list() built-in function, as shown in the example below.

The output that we get is distinct elements where all the duplicates elements are eliminated.my_list = [1,1,2,3,2,2,4,5,6,2,1] my_final_list = set(my_list) print(list(my_final_list))

Output:

[1, 2, 3, 4, 5, 6]

Remove Duplicates from a list using the Temporary List

To remove duplicates from a given list, you can make use of an empty temporary list. For that first, you will have to loop through the list having duplicates and add the unique items to the temporary list. Later the temporary list is assigned to the main list.

Here is a working example using temporary list.

my_list = [1, 2, 3, 1, 2, 4, 5, 4 ,6, 2] print("List Before ", my_list) temp_list = [] for i in my_list: if i not in temp_list: temp_list.append(i) my_list = temp_list print("List After removing duplicates ", my_list)

Output:

List Before [1, 2, 3, 1, 2, 4, 5, 4, 6, 2] List After removing duplicates [1, 2, 3, 4, 5, 6]

Python merge list without duplicates example

Simple example code using set with + operator.

a = ['hello', 'world'] b = ['hello', 'universe'] unique = list(set(a + b)) print(unique)

Output:

Another method uses NumPy

You have to import a NumPy module for it.

import numpy as np list1 = [1, 2, 2, 5] list2 = [2, 5, 7, 9] res = np.unique(list1 + list2) print(res)

Output: [1 2 5 7 9]

Combining two lists and without duplicates and don’t remove duplicates in the original list

Use set to remove duplicate without removing element from the original list.

list1 = [1, 2, 2, 5] list2 = [2, 5, 7, 9] res = list(set(list1 + list2)) print(res) print(list1) print(list2)

Output:

[1, 2, 5, 7, 9]
[1, 2, 2, 5]
[2, 5, 7, 9]

You can do also convert the list to a new set.

set_1 = set(list_1) set_2 = set(list_2)

Do comment if you have any questions or suggestions on this Python list topic.

Note: IDE:PyCharm2021.3 (Community Edition)
Windows 10
Python 3.10.1
AllPython Examplesare inPython3, so Maybe its different from python 2 or upgraded versions.

Degree in Computer Science and Engineer: App Developer and has multiple Programming languages experience. Enthusiasm for technology & like learning technical.

Facebook
WhatsApp
LinkedIn
More

Twitter
Print
Reddit
Tumblr
Pinterest
Pocket
Telegram
Skype
Email

Python: 5 Ways to Remove Duplicates from List

In this article, we will learn what is a list in python. As a python list is a collection of multiple elements even containing duplicates, sometimes it is necessary to make the list unique. Here, we are going to study the multiple ways to remove duplicates from the list in python. So, let's get started!