Pre-processing refers to the transformations applied to our data before feeding it to the algorithm. Data Preprocessing is a technique that is used to convert the raw data into a clean data set. In other words, whenever the data is gathered from different sources it is collected in raw format which is not feasible for the analysis. Show Need of Data Preprocessing
This article contains 3 different data preprocessing techniques for machine learning.
1. Rescale Data
Code: Python code to Rescale data (between 0 and 1) Python
[[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]0 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]1 import [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]3 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]4 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]5 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]6 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]7 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]8 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]9 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]0 [[ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 1. 0. 1. 1. 1.] [ 1. 1. 1. 0. 0. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1. 1. 1.] [ 0. 1. 1. 1. 1. 1. 1. 1.]]7 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]2 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]3 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]5 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]7 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]9 [[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 # importing libraries 1[[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 # importing libraries 3[[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 # importing libraries 5[[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 # importing libraries 7[[ 0.64 0.848 0.15 0.907 -0.693 0.204 0.468 1.426] [-0.845 -1.123 -0.161 0.531 -0.693 -0.684 -0.365 -0.191] [ 1.234 1.944 -0.264 -1.288 -0.693 -1.103 0.604 -0.106] [-0.845 -0.998 -0.161 0.155 0.123 -0.494 -0.921 -1.042] [-1.142 0.504 -1.505 0.907 0.766 1.41 5.485 -0.02 ]]4 # importing libraries 9import 0
How to do data preprocessing of a CSV file?Steps involved in data preprocessing :. Importing the required Libraries.. Importing the data set.. Handling the Missing Data.. Encoding Categorical Data.. Splitting the data set into test set and training set.. Feature Scaling.. What are the steps in data preprocessing in Python?There are 4 main important steps for the preprocessing of data.. Splitting of the data set in Training and Validation sets.. Taking care of Missing values.. Taking care of Categorical Features.. Normalization of data set.. How do I clean a CSV file in Python?Cleaning Missing Values from Data. dropna() – Drop all rows that have any NaN values.. dropna(how='all') – Drop only if ALL columns are NaN.. dropna(thresh=2) – Drop row if it does not have at least two values that are not NaN.. dropna(subset=[1]) – Drop only if NaN in specific column.. What are the 5 major steps of data pre processing?Data Preprocessing Steps. Data quality assessment.. Data cleaning.. Data transformation.. Data reduction.. |