You will often need to perform some form of preprocessing on your dataset before running a data clustering algorithm. In this post I will introduce one important preprocessing method that is often overlooked when clustering data. That method, my friends, is called data scaling.
What is data scaling?
As you may already know, clustering algorithms work by computing distances (i.e. dissimilarities) between data points in the dataset and grouping together points that are close in proximity. The method used for calculating the distance will be different depending on the algorithm used. One trait most of these methods share is high sensitivity to attributes (also called features) with numerically larger values compared to other attributes in a dataset. These features will bias the distance calculations and can cause the clustering algorithms to produce subpar clusterings.
To illustrate this, let’s say you have a dataset containing SAT and GPA scores for 5000 college students that you’d like to segment into 4 groups.
| Student | SAT | GPA |
| 1 | 1419 | 1.35 |
| 2 | 1384 | 1.16 |
| 3 | 784 | 3.58 |
| 4 | 748 | 3.31 |
| 5 | 1124 | 0.98 |
| … | … | … |
| 4996 | 683 | 2.02 |
| 4997 | 739 | 3.58 |
| 4998 | 741 | 3.21 |
| 4999 | 1088 | 1.01 |
| 5000 | 595 | 1.60 |
After running K-means on the dataset, you create a scatter plot showing the student scores and the clusters each student is assigned to.
It’s immediately apparent looking at the plot that a large number of students were mislabeled by the algorithm. SAT scores have a significantly larger numerical scale (400 to 1600) than that of grade point averages (0 to 4). SAT scores therefore have a much larger influence on the distance measurement K-means uses to group the students.
As can be seen from the plot, K-means leaned heavily on the SAT scores to cluster the data. To ensure that both GPA and SAT has an equal weight in the clustering both features need to be transformed so they are on the same scale. This is exactly what data scaling is for.
Here’s the scatter plot we get after scaling the data set and performing k-means:
Data Scaling Techniques
There are two main methods for scaling data. The first method is called normalization and the second method is called standardization.
Normalization
Normalization uses the minimum and maximum to transform features onto the same scale. Below is the formula used to normalize each feature:
Here’s what the student scores dataset would look like when each feature is normalized:
| Student | SAT | GPA |
| 1 | 0.849167 | 0.3375 |
| 2 | 0.820000 | 0.2900 |
| 3 | 0.320000 | 0.8950 |
| 4 | 0.290000 | 0.8275 |
| 5 | 0.603333 | 0.2450 |
| … | … | … |
| 4996 | 0.235833 | 0.5050 |
| 4997 | 0.282500 | 0.8950 |
| 4998 | 0.284167 | 0.8025 |
| 4999 | 0.573333 | 0.2525 |
| 5000 | 0.162500 | 0.4000 |
Normalization rescales the dataset features between 0 and 1 or -1 and 1. Normalization is typically used when the numerical distribution of the features is not known. Normalization fails however on datasets with outliers. Allow me to explain why.
Suppose you have a small dataset containing the following points.
| X | Y |
| 0 | 1 |
| 2 | 3 |
| 5 | 3 |
| 10 | 13 |
| 15 | 17 |
| 20 | 30 |
| 22 | 23 |
| 24 | 19 |
| 990 | 25 |
| 1000 | 27 |
As you can see from the table, the last two data points are outliers. Using normalization, the data points will transform into the following
| X | Y |
| 0.000 | 0.000 |
| 0.002 | 0.083 |
| 0.005 | 0.083 |
| 0.010 | 0.500 |
| 0.015 | 0.666 |
| 0.020 | 0.458 |
| 0.022 | 0.583 |
| 0.024 | 0.708 |
| 0.990 | 0.833 |
| 1.000 | 1.000 |
Even though normalization shifts the values between 0 and 1, the outliers still remain as outliers. If we were to compare these points using a distance function, the [] feature would have more influence on the measurement.
Standardization
Standardization uses the mean and standard deviation to transform a dataset. The formula for transforming each feature is shown below:
Here, µ is the mean of the feature and σ is the standard deviation of the feature. This is what the same student score dataset from earlier would look like when standardized.
| Student | SAT | GPA |
| 1 | 1.393781 | -0.447397 |
| 2 | 1.283359 | -0.634040 |
| 3 | -0.609587 | 1.743211 |
| 4 | -0.723163 | 1.477980 |
| 5 | 0.463083 | -0.810861 |
| … | … | … |
| 4996 | -0.928232 | 0.210768 |
| 4997 | -0.751557 | 1.743211 |
| 4998 | -0.745248 | 1.379747 |
| 4999 | 0.349506 | -0.781391 |
| 5000 | -1.205864 | -0.201813 |
Standardization is very helpful in cases where the features in the dataset are normally distributed. But your data doesn’t necessarily have to be normally distributed in order to use standardization. Unlike normalization, standardization does not squeeze your data into a bounding range, meaning that it will still work if there’s outliers present.
That’s all, folks!
There are many other data scaling techniques that can be used, but normalization and standardization are the two methods that are mainly used in practice. In my next post I will discuss fuzzy clustering.