•Two new feature selection methods are proposed based on joint mutual information.•The methods use joint mutual information with maximum of the minimum criterion.•The methods address the problem of ...selection of redundant and irrelevant features.•The methods are evaluated using eleven public data sets and five competing methods.•The proposed JMIM method outperforms five competing methods in terms of accuracy.
Feature selection is used in many application areas relevant to expert and intelligent systems, such as data mining and machine learning, image processing, anomaly detection, bioinformatics and natural language processing. Feature selection based on information theory is a popular approach due its computational efficiency, scalability in terms of the dataset dimensionality, and independence from the classifier. Common drawbacks of this approach are the lack of information about the interaction between the features and the classifier, and the selection of redundant and irrelevant features. The latter is due to the limitations of the employed goal functions leading to overestimation of the feature significance.
To address this problem, this article introduces two new nonlinear feature selection methods, namely Joint Mutual Information Maximisation (JMIM) and Normalised Joint Mutual Information Maximisation (NJMIM); both these methods use mutual information and the ‘maximum of the minimum’ criterion, which alleviates the problem of overestimation of the feature significance as demonstrated both theoretically and experimentally. The proposed methods are compared using eleven publically available datasets with five competing methods. The results demonstrate that the JMIM method outperforms the other methods on most tested public datasets, reducing the relative average classification error by almost 6% in comparison to the next best performing method. The statistical significance of the results is confirmed by the ANOVA test. Moreover, this method produces the best trade-off between accuracy and stability.
IntroductionAlterations in motor activity are an extremely important characteristic and one of the leading symptoms of major functional psychiatric disorders. These pattern disturbances can be ...observed in schizophrenia. Actigraphy is a non-invasive method that can be used to monitor these changes, and recent studies emphasize its significance in the early identification of disorders like schizophrenia.ObjectivesThis study uniquely focuses on distinguishing latent liabilities for schizotypy from manifested schizophrenia using specific actigraphy features.MethodsActigraphy data were collected using specialized devices from the University of Szeged and Haukeland University Hospital datasets (Berle et al., 2010). At Haukeland University Hospital patients with chronic schizophrenia (N=23) (so-called: manifested group) were collected, separately, at the University of Szeged, healthy university students were recruited and screened for latent tendencies towards shizotypic pathological development. In the latter study, two main groups were formed based on their scores: a positive schizotypy factor group (so-called: latent group) (N=22) and a control group (N=25), with actigraphy data.Utilizing the pyActigraphy library (Hammad et al., 2021) and wavelet analysis, features such as activity mean, interdaily stability and sleep movement characteristics were derived. Feature selection employed machine learning algorithms, notably Logistic Regression, Random Forest, ANN, and AHFS aided by Shapley values and Click Forming Feature Selection for insight into the most influential features.ResultsThe three models exhibited similar performance with a 60% accuracy threshold. In the latent group, sleep-related movements have a substantial impact, while in the manifested group, in addition to sleep characteristics, features like RA, IV, ADAT, M10, the mean activity level (all of which decreased), and the ratio of zero values also play a significant role. In the latent group, features related to the length of small amplitude movements were dominant, particularly the increased values, along with a decrease in the density of large movements.ConclusionsOur study indicates that in the latent phase of schizophrenia, actigraphy features related to sleep are most significant, but as the disease progresses, both sleep and daytime activity patterns are crucial. Sleep disturbances may signal early susceptibility, with nighttime movements offering clearer insights. These variations might be influenced by medication effects in the manifested group, reflecting the broader challenges in schizophrenia research where the drug-free study of patients remains elusive. Further studies should explore these features in the Clinical High Risk and prodromal groups to refine our understanding of the development of the disorder.Disclosure of InterestNone Declared
Streaming feature selection (SFS), is the task of selecting the most informative features in dealing with high-dimensional or incrementally growing problems. Several SFS algorithms have been proposed ...in the literature. However, they do not consider all feature subsets at the redundancy analysis step due to computational concerns. Moreover, they do not reconsider previously removed features which leads to losing most of the useful information. In this paper, the redundancy analysis step is defined as a binary optimization problem. Then, a binary bat algorithm (BBA) is adopted to find the minimal informative subsets. In this way, a large number of feature subsets can be considered effectively at the redundancy analysis step. In addition, an effective priority list is used to maintain previously removed redundant features. Such a list allows the re-examination of informative features. As a result, it is possible to consider the mutual information between features that are not streamed in an small time interval. Experimental studies on fifteen different types of datasets show that our approach is superior to state-of-the-art online and offline streaming feature selection methods in terms of classification accuracy.
A review of unsupervised feature selection methods Solorio-Fernández Saúl; Ariel, Carrasco-Ochoa J; Martínez-Trinidad, José Fco
The Artificial intelligence review,
02/2020, Letnik:
53, Številka:
2
Journal Article
Recenzirano
In recent years, unsupervised feature selection methods have raised considerable interest in many research areas; this is mainly due to their ability to identify and select relevant features without ...needing class label information. In this paper, we provide a comprehensive and structured review of the most relevant and recent unsupervised feature selection methods reported in the literature. We present a taxonomy of these methods and describe the main characteristics and the fundamental ideas they are based on. Additionally, we summarized the advantages and disadvantages of the general lines in which we have categorized the methods analyzed in this review. Moreover, an experimental comparison among the most representative methods of each approach is also presented. Finally, we discuss some important open challenges in this research area.
Cardiovascular diseases (CVD) are among the most common serious illnesses affecting human health. CVDs may be prevented or mitigated by early diagnosis, and this may reduce mortality rates. ...Identifying risk factors using machine learning models is a promising approach. We would like to propose a model that incorporates different methods to achieve effective prediction of heart disease. For our proposed model to be successful, we have used efficient Data Collection, Data Pre-processing and Data Transformation methods to create accurate information for the training model. We have used a combined dataset (Cleveland, Long Beach VA, Switzerland, Hungarian and Stat log). Suitable features are selected by using the Relief, and Least Absolute Shrinkage and Selection Operator (LASSO) techniques. New hybrid classifiers like Decision Tree Bagging Method (DTBM), Random Forest Bagging Method (RFBM), K-Nearest Neighbors Bagging Method (KNNBM), AdaBoost Boosting Method (ABBM), and Gradient Boosting Boosting Method (GBBM) are developed by integrating the traditional classifiers with bagging and boosting methods, which are used in the training process. We have also instrumented some machine learning algorithms to calculate the Accuracy (ACC), Sensitivity (SEN), Error Rate, Precision (PRE) and F1 Score (F1) of our model, along with the Negative Predictive Value (NPR), False Positive Rate (FPR), and False Negative Rate (FNR). The results are shown separately to provide comparisons. Based on the result analysis, we can conclude that our proposed model produced the highest accuracy while using RFBM and Relief feature selection methods (99.05%).
Feature selection on incomplete datasets is a challenging task. To address this challenge, existing methods first employ imputation methods to complete the dataset and then perform feature selection ...based on the imputed dataset. Since missing value imputation and feature selection are entirely independent, the importance of features cannot be considered during imputation. However, in real-world scenarios or datasets, different features have varying degrees of importance. To this end, we proposed a novel incomplete data feature selection framework that considers feature importance. The framework mainly consists of two alternating iterative stages: M-stage and W-stage. In the M-stage, missing values are imputed based on a given feature importance vector and multiple initial imputation results. In the W-stage, an improved reliefF algorithm is employed to learn the feature importance vector based on the imputed data. In particular, the feature importance output by the W-stage in the current iteration will be used as the input of the M-stage in the next iteration. Experimental results on artificial and real missing datasets demonstrate that the proposed method outperforms other approaches significantly.
•A new feature selection framework involving alternating iterations is proposed for incomplete data.•The proposed framework can select the most relevant features under the influence of missing values.•The framework imputes incomplete data based on current feature importance then updates it by a relief-based method.•Comparison results on 10 benchmark datasets show that the proposed method achieves better performance.
With the rapid increase of the data size, it has increasing demands for selecting features by exploiting both labeled and unlabeled data. In this paper, we propose a novel semi-supervised embedded ...feature selection method. The new method extends the least square regression model by rescaling the regression coefficients in the least square regression with a set of scale factors, which is used for evaluating the importance of features. An iterative algorithm is proposed to optimize the new model. It has been proved that solving the new model is equivalent to solving a sparse model with a flexible and adaptable ℓ 2;p norm regularization. Moreover, the optimal solution of scale factors provides a theoretical explanation for why we can use {||w 1 || 2 , . . .,||w d || 2 } to evaluate the importance of features. Experimental results on eight benchmark data sets show the superior performance of the proposed method.
Because of the overgrowth of data, especially in text format, the value and importance of multi‐label text classification have increased. Aside from this, preprocessing and particularly intelligent ...feature selection (FS) are the most important step in classification. Each FS finds the best features based on its approach, but we try to use a multi‐strategy approach to find more useful features. Evaluating and comparing features’ importance and relevance makes using multiple strategy and methods more suitable than conventional approaches because each feature is measured based on several perspectives. Nevertheless, the ensemble FS merges the final performance results of various methods to take advantage of different methods’ strengths and better classify. In this article, we have proposed an ensemble FS method for multi‐label text data (MLTD) for the first time using the order statistics (EMFS) approach. We have utilized four multi‐label FS (MLFS) algorithms with various particular performances to achieve a good result. In this method, as one of the most important statistics methods, Order Statistics was used to aggregate the ranks of different algorithms, which is robust against noise, redundant and inessential features. In the end, the performance of EMFS, executing six MLTDs, was evaluated according to six performance criteria (ranking‐based and classification‐based). Surprisingly, the proposed method was more accurate than others among all used MLTDs. The proposed method has improved by 1.5% compared to other methods. This value is based on the results obtained based on six evaluation criteria and all tested data sets.
Machine Learning and Data Mining algorithms are used extensively to enhance the performance of Intrusion Detection Systems. The number of training instances and the dimensionality of data are crucial ...factors affecting the performance of the model built during the training of any supervised learning algorithms. A sufficient proportion of instances having relevant features from all classes of attacks and normal traffic are considered most desirable while building the classification model that classifies the network traffic into attack and normal. This paper proposes a methodology to improve the accuracy of the model by giving importance to the relevant features that can contribute to model building. The feature selection using correlation‐based and information gain‐based techniques during training and testing contributes much to the detection of stealthier attacks and minority attacks. Then the features of the less detected attacks are identified as the second phase of the filter that is used to improve the performance. The relevant features of stealthy attacks are identified based on the correlation of corresponding features of the attack and normal data as the attacks are made stealthy mostly by making it resemble the normal traffic. Finally, the attacks that are rarely found in the training data are oversampled to improve their detection. CICIDS 2017 data set is employed as it comprises stealthier attacks generated using modern tools. NSL KDD data set is also used for evaluation to compare the proposed work with existing literature as it is used in most of the available literature. The results show superior performance with an accuracy of 99.8%, false positive rate of 0.2%, and a detection rate and 99.8%.
Feature selection aims to eliminate unimportant and redundant features or to select effective and interacting features. It is a challenging task to accurately measure the relationships of candidate ...features, the selected features and categories in the selection process, especially for high-dimensional and small-sample-size data. To this end, a new measure named Dynamic Feature Importance (DFI) is proposed, as well as its corresponding feature selection algorithm named Dynamic Feature Importance based Feature Selection (DFIFS). In order to obtain higher classification accuracy with smaller number of features, a newly Modified-Dynamic Feature Importance based Feature Selection (M-DFIFS) algorithm is developed by combining DFIFS with classical filters. Based on experiments with 14 public high-dimensional datasets, the lately M-DFIFS algorithm shows significantly better performance than five typical filter algorithms in terms of their average accuracy with acceptable computing time. When using random forest as the classifier, M-DFIFS brings a great advantage in the number decrease of selected features. Hence the new feature selection framework “Filter + DFIFS” is verified very effective to solve problems of obtaining high accuracy with a few features.
•A novel measure of the dynamic importance of features in the selection process.•A new feature selection algorithm based on Dynamic Feature Importance.•An effective “filter+proposed algorithm” based feature selection framework.•Smaller feature subset selected by proposed algorithm with high accuracy.•Good applicability to high-dimensional and small-sample-size data.