Learn Data preparation steps for machine learning just in 5 min..!!!

Machine learning is hot topic now day and will be in future. data preparation is key when it comes to machine learning. with good data preparation a simple ml model can give very good and satisfying results but if data is not well prepared and you use very good quality/sophisticated ml algorithm(with good prediction precision) it is going to fail ,your model will just take garbage input and give out garbage output. keep in mind data preparation often makes more difference then the algorithm it self.

Ultimate goal of data preparation is to insure that machine learning model works optimal way.

Steps of data preparation ---

Explore to understand problems in data---there are many methods are used to explore data hrad(), tail(),shape(),describe() are just example of such methods ,and also we can use plots for this purpose.
Remove duplicates
Treat missing values
Treat errors and outliers
Treat null values
scale the features
split dataset

now understand that we don't just do these steps in this order . and often we just need to preform each of the step more then ones and maybe not even ones all depends on data.

Comments

Data preparation : Removing duplicates

Duplicate values not only increase size of dataset but also create bias while training model. Duplicate cases are over weighted thus create bias in training machine learning model. suppose there is customer data and one customer showed up 100 times and others just showed up only ones this defiantly confuse the model resulting in yielding wrong predictions Steps to remove duplicate values --- Explore data and identify duplicate values (use exploratory analysis)[to learn about exploring data check out other posts on the blog]. identify duplicates cases using --- -Unique id:-- if we are lucky enough we have given each entity with unique id. then it it bit easy to remove duplicates . ...

Feature Engineering is easy ..!!

Features are input to machine learning algorithm. and the row features that are provided to us are not the best, they are just whatever just has to be in data. Feature engineering have a goal to convert given features into much more predictive ones so that the can predict label more precisely . feature engineering can make simple algorithm to give good results. at a same time if you apply the best algorithm and do not perform feature engineering well you are going to get poor results. feature engineering is a broad subject people dedicate their entire careers to feature engineering. there are some steps in feature engg that we need to follow and repeat most of the times to get job done. steps--- 1. Explore and understand data relationships 2. Transform feature 3.Compute new features from other by applying some maths on it 4. Visualization to check results 5. Test with ML model 6. Repeat above steps as needed Transforming feature--- Why transform featu...

Data preparation: Dealing with duplicates in data(practical example)

In previous post i have said that duplicate values create biased while training model and have explained how to deal with duplicate values step by step. if you have not read that post make sure you read that (link-- https://www.blogger.com/blog/post/edit/950003373928384423/4720144211787135428 ) as this post going to be practical implementation of theory explained in that post . or just continue reading as i shall explain while showing how it actually work in practical. Steps-- - 1. finding duplicate values through data exploration --- before we start finding duplicates we need to first import dataset and set columns so ones data is imported we can start exploring it use dtypes, describe and other methods to explore data see the image below and observe carefully you will find that we have printed two shapes first one being shape of entire data frame and the other one being shape of unique customer_id . and what ...

AI hub

Search This Blog