A single data point can make or break a business forecast. Finding such values can help separate normal noise from data trends. It is a good practice to identify outliers to discover the actual behavior of the data.

Did you know Gartner estimated that by 2027, organizations that prioritize semantics (understanding meaning and context of data to identify unusual values, such as outliers) in AI-ready data will increase agentic AI accuracy by up to 80% and reduce costs by up to 60%?

Definition of an Outlier in Data

An observation that differs significantly from the overall pattern of a dataset is known as an outlier. It can be either too high, too low, or incongruent with respect to other observations in the dataset.

An example of a dataset and its observations is shown below:























420



450



460



480



490



5000


The observation of 5000 does not align with the general rule of the dataset and is considered an outlier. Identifying an observation as an outlier does not necessarily mean it is an error. There are several reasons that can lead to an observation being an outlier. It can be a case of data entry error, measurement error, or a genuine variation or an unusual event.

Types of Outliers

Outliers can be classified based on the nature of differences among observations in a dataset.

1.     Global Outliers: These observations differ from almost all other observations in a dataset. An example is a financial transaction worth $50,000, which would be an outlier in a dataset containing financial transactions worth less than $500.

2.     Contextual Outliers: An observation may be contextually unusual but may not differ from the other observations in the dataset. For example, the consumption of electricity in an office building may not be unusual during business hours. However, consumption of electricity at midnight may be unusual and may be an outlier.

3.     Collective Outliers: A group of observations, though not unusual individually, may be unusual when examined together. For example, it may not be unusual if an application on the computer requests data from a server a few times. However, it may be unusual if a group of applications makes a large number of requests to the server.

Considering these types of outliers helps analysts understand the ways and means of their detection.

Why Do Outliers Matter in Machine Learning?

Extreme values may be present in a given dataset. The ways and the extent to which outliers may influence statistical analysis and learning may vary with the kind of machine learning algorithm used, the nature of the data, and the goal of the analysis.

·       Outliers may alter the mean and the standard deviation.

·       Outliers may increase the prediction error in a machine learning model.

·       In a clustering analysis, outliers may be misclassified with other observations.

·       Not detecting and recognizing outliers may lead to business risks.

·       An extreme value may be misclassified as a data error.

Outlier detection may help identify the aforementioned situations.

However, the aim is not to eliminate all unusual values. The aim is to assess whether all observations are valid and relevant to the analysis.

Techniques for Spotting Outliers

There are a variety of techniques for spotting outliers, and the most appropriate technique is determined by the business context, the number of variables in the dataset, and the distribution of the dataset. The following techniques are a few examples.

1.    The Interquartile Range (IQR) Method

This method is used to identify outliers by looking at the spread of the middle 50% of the data. This method does not depend on the normality of the data being analyzed.

In this method, the IQR is calculated using the following equation:

IQR = Q3 − Q1

Where Q3 and Q1 represent the 75th percentile and the 25th percentile of the data, respectively.

In this method, values outside the lower and upper bounds are outliers. The bounds are calculated using the following equations:

Lower bound = Q1 − 1.5 × IQR

Upper bound = Q3 + 1.5 × IQR

Using this method, if Q1 = 10 and Q3 = 20, the IQR would be 10. The lower and upper bounds would be −5 and 35, respectively. Therefore, any value in the data set is considered an outlier if it is less than −5 or greater than 35.

This method is best utilized to identify outliers for exploratory data analysis and data sets where box plots are used to present the data.

2.    The Z-Score Method

According to this method, values are considered outliers if they are far from the mean in relation to the standard deviation. Relative to other data points, an absolute value of Z greater than 3 would indicate an outlier.

The equation to calculate Z is:

Z = (x − μ) / σ

Where x represents a data point, μ represents the mean, and σ is the standard deviation. An observation with a score of −3 is 3 standard deviations below the mean.

This approach works well with data distributions that approximately follow a normal distribution. With heavily skewed data, this method may not work well.

Suitable for: Data distributions that approximately follow a normal distribution.

3.    Isolation Forest

Isolation Forest is a machine learning method where the goal is to isolate data by making a series of random splits, where observations that are different from the rest are isolated in fewer splits.

In comparison to other methods, Isolation Forest can be effective when data contains an unusual combination of values.

Suitable: Effective with high-dimensional data.

4.    DBSCAN

DBSCAN determines data points that are part of a dense cluster, and if there are data points that are not part of any cluster, then DBSCAN identifies those points as noise.

DBSCAN can be used to identify customers who show purchasing behavior that is significantly different from a group of customers that shows dense and similar purchasing behavior.

Not all data points that are noise are really errors.

Suitable: Finding unusual data in a dataset that contains meaningful and dense clusters.

5.     Dealing with Outliers

After outliers are identified, there are various methods to treat outliers. However, before taking any action, the reason why the observation is an outlier should be investigated.

·       Remove outliers: If there is evidence to suggest that the value was entered in error, then the value should be removed.

·       Clip outliers: If an observation is an outlier due to chance, then the value can be set to a limit.

·       Perform transformations: Right-skewed data can often be approximated as normal by taking the natural log of the data. Log-transformations should be performed on a data feature only after determining the feature exhibits right-skewness and after determining that log-transformation on the data feature in question will assist in modeling the feature.

·       Use more robust methods: Median-based statistics and methodology can also assist in minimizing the impact of outliers.

·       Be discerning: If outlying data does represent aberrant activity (fraud, equipment failure, spikes in consumer demand, etc.), it should be represented.

To gain more clarity on outliers in data, download this A Practical Guide to Detect and Handle Outliers for Data Analysts.

Best Practices to Identify Outlying Data

A statistical hypothesis testing approach should not be the only approach to outlying data identification. Good business practice dictates:

·       Familiarize yourself with the data: Becoming intimately acquainted with the data in analysis is important.

·       Identify distribution: Data that does not conform to expectations should be flagged for further analysis.

·       Compare methods: Outlying data may not be flagged by all methods of analysis.

·       Investigate flagged values: Review your observations against the source records and your domain knowledge to determine if the cases warranted flagging.

·       Avoid data leakage: During training, establish detection thresholds for each rule. Apply the rules to validation and test datasets.

·       Document decisions: Explain the rationale for modifying, leaving, or deleting values.

To gain a better understanding of data science concepts like outlier. Explore the best data science certifications, such as Certified Data Science Professional (CDSP™), which is good for freshers and data science professionals.

Final Thought!

Outliers can be detected using numerous methods and approaches. It is useful to integrate different methods in order to adapt to the idiosyncratic nature of data.

It is best practice to evaluate a potential outlier to determine if it is a data error prior to removal.


Google AdSense Ad (Box)

Comments