Data preparation: Dealing with duplicates in data(practical example)

In previous post i have said that duplicate values create biased while training model and have explained how to deal with duplicate values step by step. if you have not read that post make sure you read that (link--https://www.blogger.com/blog/post/edit/950003373928384423/4720144211787135428) as this post going to be practical implementation of theory explained in that post . or just continue reading as i shall explain while showing how it actually work in practical.

Steps---

1. finding duplicate values through data exploration ---

before we start finding duplicates we need to first import dataset and set columns

so ones data is imported we can start exploring it use dtypes, describe and other methods to explore data

see the image below and observe carefully you will find that we have printed two shapes first one being shape of entire data frame and the other one being shape of unique customer_id . and what we found is that there are 12 observations more in data frame as compared to unique values in customer id column.

2. Now as we have find out there are 12 duplicates we just need to remove them pandas makes it too easy to remove duplicates. we just have to apply dropduplicates method and we are done that easy. remember . strategy we are using is keeping the first value and dropping all other values.

Comments

Data preparation : Dealing with missing values

Missing values are probably the most common headache you are going to have as machine learning engineer. missing values are the ones who screw the whole algorithm and make model give wired results(predictions). Treating missing values--- 1. Use exploration to detect the missing values -- detecting missing values is crucial because lot of machine learning models fail because of missing values. 2.Find how are missing values are coded-- missing values could be codded in the data in one or more of following formats. -NULL -a string or number--eg.-9999,0,"NA","?"etc. 3. Treatment strategy-- - if some column has lot of missing values then its better to get rid of that column. - remove row-- suppose very few rows have missing values then remove those rows. -Forward or backward fill-- sometimes its just better to use fill which work by filling value of nearest neighbours in null cell. it is useful when data is in some order say in order of

Feature Engineering is easy ..!!

Features are input to machine learning algorithm. and the row features that are provided to us are not the best, they are just whatever just has to be in data. Feature engineering have a goal to convert given features into much more predictive ones so that the can predict label more precisely . feature engineering can make simple algorithm to give good results. at a same time if you apply the best algorithm and do not perform feature engineering well you are going to get poor results. feature engineering is a broad subject people dedicate their entire careers to feature engineering. there are some steps in feature engg that we need to follow and repeat most of the times to get job done. steps--- 1. Explore and understand data relationships 2. Transform feature 3.Compute new features from other by applying some maths on it 4. Visualization to check results 5. Test with ML model 6. Repeat above steps as needed Transforming feature--- Why transform feature ?-- 1. To improve distribu

AI hub

Search This Blog