Data preparation : Scaling

Scaling is very important operation for machine learning . improper scaling creates bias in model. we don't want to have is data column that have very lager value range for example we have people age and salary we see that salary of a person is going to be much grater then the age hence it will create bias in model.

There are several methods used for scaling---

most common ones are--

1. Z score--

taking z score usually means normalizing the values with mean=0, standard deviation=1

2. min max -if distribution is far from normal then min max method is used

remember deal with missing values and errors and outliers before scaling because they create bias.

Comments

Feature Engineering is easy ..!!

Features are input to machine learning algorithm. and the row features that are provided to us are not the best, they are just whatever just has to be in data. Feature engineering have a goal to convert given features into much more predictive ones so that the can predict label more precisely . feature engineering can make simple algorithm to give good results. at a same time if you apply the best algorithm and do not perform feature engineering well you are going to get poor results. feature engineering is a broad subject people dedicate their entire careers to feature engineering. there are some steps in feature engg that we need to follow and repeat most of the times to get job done. steps--- 1. Explore and understand data relationships 2. Transform feature 3.Compute new features from other by applying some maths on it 4. Visualization to check results 5. Test with ML model 6. Repeat above steps as needed Transforming feature--- Why transform featu...

Data preparation: Treating errors and outliers

Errors are outliers are the those values in dataset which are far away from mean value of that column in which they exist or we can say they are very small or large as compared to other values in that column. suppose we are observing column of price and we see value that is ten times bigger then other values then it could be error or outlier. to identify either it is error or outlier we need to apply some domain knowledge or we can say field knowledge. outliers are mainly caused by variability in measurement or experimental errors. but sometimes they could be useful and help us understand data better if we apply domain knowledge. for example we find a outlier in car price of cars data set then we check its other features and we found that that car is luxury car then we understand it was outlier but a useful one and it was not just a error. Dealing with errors/outliers--- 1. Detect errors/outliers -- use data exploration to identify errors /outliers 2. methods to identify errors a...

Data preparation : Removing duplicates

Duplicate values not only increase size of dataset but also create bias while training model. Duplicate cases are over weighted thus create bias in training machine learning model. suppose there is customer data and one customer showed up 100 times and others just showed up only ones this defiantly confuse the model resulting in yielding wrong predictions Steps to remove duplicate values --- Explore data and identify duplicate values (use exploratory analysis)[to learn about exploring data check out other posts on the blog]. identify duplicates cases using --- -Unique id:-- if we are lucky enough we have given each entity with unique id. then it it bit easy to remove duplicates . ...

AI hub

Search This Blog