The outcome of collective decision-making often relies on the procedure through which the perspectives of its members are aggregated. Popular aggregation methods, such as the majority rule, often ...fail to produce the optimal result, especially in high-complexity tasks. Methods that rely on meta-cognitive information, such as confidence-based methods and the Surprisingly Popular answer, have succeeded in various tasks. However, there are still scenarios that result in choosing the incorrect answer. We aim to exploit meta-cognitive information and learn from it, to enhance the group’s ability to produce a correct answer. Specifically, we propose two different feature-representation approaches: Response-Centered feature Representation (RCR), which focuses on the characteristics of the individual response, and Answer-Centered feature Representation (ACR), which focuses on the characteristics of each of the potential answers. Using these two feature-representation approaches, we train machine-learning models to predict the correctness of a response and an answer. The trained models are used in our two proposed aggregation approaches: (1) The Response-Prediction (RP) approach aggregates the results of the group’s votes by exploiting the RCR feature-engineering approach; (2) The Answer-Prediction (AP) approach aggregates the results of the group’s votes by exploiting the ACR feature-engineering approach. To evaluate our methodology, we collected 2514 responses for different tasks. The results show a significant increase in the success rate compared to standard rule-based aggregation methods.
The outcome of a collective decision-making process, such as crowdsourcing, often relies on the procedure through which the perspectives of its individual members are aggregated. Popular aggregation ...methods, such as the majority rule, often fail to produce the optimal result, especially in high-complexity tasks. Methods that rely on meta-cognitive information, such as confidence-based methods and the Surprisingly Popular Option, had shown an improvement in various tasks. However, there is still a significant number of cases with no optimal solution. Our aim is to exploit meta-cognitive information and to learn from it, for the purpose of enhancing the ability of the group to produce a correct answer. Specifically, we propose two different feature-representation approaches: (1) Response-Centered feature Representation (RCR), which focuses on the characteristics of the individual response instances, and (2) Answer-Centered feature Representation (ACR), which focuses on the characteristics of each of the potential answers. Using these two feature-representation approaches, we train Machine-Learning (ML) models, for the purpose of predicting the correctness of a response and of an answer. The trained models are used as the basis of an ML-based aggregation methodology that, contrary to other ML-based techniques, has the advantage of being a "one-shot" technique, independent from the crowd-specific composition and personal record, and adaptive to various types of situations. To evaluate our methodology, we collected 2490 responses for different tasks, which we used for feature engineering and for the training of ML models. We tested our feature-representation approaches through the performance of our proposed ML-based aggregation methods. The results show an increase of 20% to 35% in the success rate, compared to the use of standard rule-based aggregation methods.
