Anomaly Detection for Categorical Data

Outlier detection system discovers the novel or rare events, anomalies, vicious actions, exceptional phenomena. It is mandatory to find these anomalies in data mining because the presence of these objects usually makes the database inefficient. An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism. Finding objects that do not conform to well-defined notions of expected behaviour in a dataset is called outlier detection. Outlier detection is a pre-processing step for locating these non-conforming objects in data sets. This outlier detection is a challenging process in large scale database since it has high dimensional data with low anomalous rate. Here outliers are defined formally and the optimized ways to detect outliers is also proposed here. Optimization in outlier detection is achieved by a new concept of holoentropy which combines entropy and total correlation. It is a more effective and efficient practical phenomenon in outlier detection methods. It can be used effectively to deal with both large and high-dimensional datasets.