In this paper we present the design, implementation, evaluation, and user experiences of the NoiseSpy application, our sound sensing system that turns the mobile phone into a low-cost data logger for ...monitoring environmental noise. It allows users to explore a city area while collaboratively visualizing noise levels in real-time. The software combines the sound levels with GPS data in order to generate a map of sound levels that were encountered during a journey. We report early findings from the trials which have been carried out by cycling couriers who were given Nokia mobile phones equipped with the NoiseSpy software to collect noise data around Cambridge city. Indications are that, not only is the functionality of this personal environmental sensing tool engaging for users, but aspects such as personalization of data, contextual information, and reflection upon both the data and its collection, are important factors in obtaining and retaining their interest.
•First hybrid Deep Learning approach applied to environmental and on-body sensor data.•Deep-learning is effective in human emotion classification.•Multi-model data fusion approach performs better ...than single modality classification.•Hybrid approach with combined CNN and LSTM exceed the performance of CNN alone.
The detection and monitoring of emotions are important in various applications, e.g., to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g., EEG and GSR), environmental data (e.g., audio and weather), videos (e.g., for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our approach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F-Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Furthermore, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)
In recent years, machine learning has developed rapidly, enabling the development of applications with high levels of recognition accuracy relating to the use of speech and images. However, other ...types of data to which these models can be applied have not yet been explored as thoroughly. Labelling is an indispensable stage of data pre-processing that can be particularly challenging, especially when applied to single or multi-model real-time sensor data collection approaches. Currently, real-time sensor data labelling is an unwieldy process, with a limited range of tools available and poor performance characteristics, which can lead to the performance of the machine learning models being compromised. In this paper, we introduce new techniques for labelling at the point of collection coupled with a pilot study and a systematic performance comparison of two popular types of deep neural networks running on five custom built devices and a comparative mobile app (68.5–89% accuracy within-device GRU model, 92.8% highest LSTM model accuracy). These devices are designed to enable real-time labelling with various buttons, slide potentiometer and force sensors. This exploratory work illustrates several key features that inform the design of data collection tools that can help researchers select and apply appropriate labelling techniques to their work. We also identify common bottlenecks in each architecture and provide field tested guidelines to assist in building adaptive, high-performance edge solutions.
Automatic recognition of human emotions is not a trivial process. There are many factors affecting emotions internally and externally. Expressing emotions could also be performed in many ways such as ...text, speech, body gestures or even physiologically by physiological body responses. Emotion detection enables many applications such as adaptive user interfaces, interactive games, and human robot interaction and many more. The availability of advanced technologies such as mobiles, sensors, and data analytics tools led to the ability to collect data from various sources, which enabled researchers to predict human emotions accurately. Most current research uses them in the lab experiments for data collection. In this work, we use direct and real time sensor data to construct a subject-independent (generic) multi-modal emotion prediction model. This research integrates both on-body physiological markers, surrounding sensory data, and emotion measurements to achieve the following goals: (1) Collecting a multi-modal data set including environmental, body responses, and emotions. (2) Creating subject-independent Predictive models of emotional states based on fusing environmental and physiological variables. (3) Assessing ensemble learning methods and comparing their performance for creating a generic subject-independent model for emotion recognition with high accuracy and comparing the results with previous similar research. To achieve that, we conducted a real-world study “in the wild” with physiological and mobile sensors. Collecting the data-set is coming from participants walking around Minia university campus to create accurate predictive models. Various ensemble learning models (Bagging, Boosting, and Stacking) have been used, combining the following base algorithms (K Nearest Neighbor KNN, Decision Tree DT, Random Forest RF, and Support Vector Machine SVM) as base learners and DT as a meta-classifier. The results showed that, the ensemble stacking learner technique gave the best accuracy of 98.2% compared with other variants of ensemble learning methods. On the contrary, bagging and boosting methods gave (96.4%) and (96.6%) accuracy levels respectively.
COVID-19 has shown a relatively low case fatality rate in young healthy individuals, with the majority of this group being asymptomatic or having mild symptoms. However, the severity of the disease ...among the elderly as well as in individuals with underlying health conditions has caused significant mortality rates worldwide. Understanding this variance amongst different sectors of society and modelling this will enable the different levels of risk to be determined to enable strategies to be applied to different groups. Long-established compartmental epidemiological models like SIR and SEIR do not account for the variability encountered in the severity of the SARS-CoV-2 disease across different population groups. The objective of this study is to investigate how a reduction in the exposure of vulnerable individuals to COVID-19 can minimise the number of deaths caused by the disease, using the UK as a case study. To overcome the limitation of long-established compartmental epidemiological models, it is proposed that a modified model, namely SEIR-v, through which the population is separated into two groups regarding their vulnerability to SARS-CoV-2 is applied. This enables the analysis of the spread of the epidemic when different contention measures are applied to different groups in society regarding their vulnerability to the disease. A Monte Carlo simulation (100,000 runs) along the proposed SEIR-v model is used to study the number of deaths which could be avoided as a function of the decrease in the exposure of vulnerable individuals to the disease. The results indicate a large number of deaths could be avoided by a slight realistic decrease in the exposure of vulnerable groups to the disease. The mean values across the simulations indicate 3681 and 7460 lives could be saved when such exposure is reduced by 10% and 20% respectively. From the encouraging results of the modelling a number of mechanisms are proposed to limit the exposure of vulnerable individuals to the disease. One option could be the provision of a wristband to vulnerable people and those without a smartphone and contact-tracing app, filling the gap created by systems relying on smartphone apps only. By combining very dense contact tracing data from smartphone apps and wristband signals with information about infection status and symptoms, vulnerable people can be protected and kept safer.
Urban spaces have a great impact on how people's emotion and behaviour. There are number of factors that impact our brain responses to a space. This paper presents a novel urban place recommendation ...approach, that is based on modelling in-situ EEG data. The research investigations leverages on newly affordable Electroencephalogram (EEG) headsets, which has the capability to sense mental states such as meditation and attention levels. These emerging devices have been utilized in understanding how human brains are affected by the surrounding built environments and natural spaces. In this paper, mobile EEG headsets have been used to detect mental states at different types of urban places. By analysing and modelling brain activity data, we were able to classify three different places according to the mental state signature of the users, and create an association map to guide and recommend people to therapeutic places that lessen brain fatigue and increase mental rejuvenation. Our mental states classifier has achieved accuracy of (%90.8). NeuroPlace breaks new ground not only as a mobile ubiquitous brain monitoring system for urban computing, but also as a system that can advise urban planners on the impact of specific urban planning policies and structures. We present and discuss the challenges in making our initial prototype more practical, robust, and reliable as part of our on-going research. In addition, we present some enabling applications using the proposed architecture.
The increasing level of air pollutants (e.g. particulates, noise and gases) within the atmosphere are impacting mental wellbeing. In this paper, we define the term ‘DigitalExposome’ as a conceptual ...framework that takes us closer towards understanding the relationship between environment, personal characteristics, behaviour and wellbeing using multimodal mobile sensing technology. Specifically, we simultaneously collected (for the first time) multi-sensor data including urban environmental factors (e.g. air pollution including: Particulate Matter (PM1), (PM2.5), (PM10), Oxidised, Reduced, Ammonia (NH3) and Noise, People Count in the vicinity), body reaction (physiological reactions including: EDA, HR, HRV, Body Temperature, BVP and movement) and individuals’ perceived responses (e.g. self-reported valence) in urban settings. Our users followed a pre-specified urban path and collected the data using a comprehensive sensing edge device. The data is instantly fused, time-stamped and geo-tagged at the point of collection. A range of multivariate statistical analysis techniques have been applied including Principle Component Analysis, Regression and Spatial Visualisations to unravel the relationship between the variables. Results showed that Electrodermal Activity (EDA) and Heart Rate Variability (HRV) are noticeably impacted by the level of Particulate Matter in the environment. Furthermore, we adopted Convolutional Neural Network (CNN) to classify self-reported wellbeing from the multimodal dataset which achieved an f1-score of 0.76.
Ubiquitous access to the internet is provided to students via WiFi to enhance their learning experience. Internet use is central to education, however when overused for distraction it can be ...problematic and negatively affect students’ academic performance and well-being.
A spatial-temporal analysis of actual behaviour on a University WiFi examines whether WiFi use enhances users’ experience of education. This study is an analysis of internet activities on WiFi, using data on more than two and a half million user sessions over an academic year. Principal Component Analysis (PCA) was performed to identify the most important dimensions of the users’ behavior. Distraction activity dominated. Over the year and relative to use for education, users were ten times more active in search, four times more active in social media and three times more active in shopping. The level of distraction activity may be problematic and negatively affect student wellbeing and academic performance. Increased education activity was linked to reduced social media activity. Over 75% of all user behaviour on the University WiFi was hosted by Google, Facebook and Amazon, enabling an extraordinary influence on a University population.
The research findings suggest that University WiFi is predominantly used for distraction activities and consequently expected educational benefits may be negatively impacted. Overuse of the internet for distraction is a potential harm. Universities should consider practical options to reduce internet distraction, such as education programs to develop awareness of the risks of Problematic Internet Use (PIU) and/or filtering, blocking or time limiting use of distraction-related websites on a University WiFi.
Display omitted
Today's mobile phone users are faced with large numbers of notifications on social media, ranging from new followers on Twitter and emails to messages received from WhatsApp and Facebook. These ...digital alerts continuously disrupt activities through instant calls for attention. This paper examines closely the way everyday users interact with notifications and their impact on users' emotion. Fifty users were recruited to download our application NotiMind and use it over a five-week period. Users' phones collected thousands of social and system notifications along with affect data collected via self-reported Positive and Negative Affect Schedule tests three times a day. Results showed a noticeable correlation between positive affective measures and keyboard activities. When large numbers of post and remove notifications occur, a corresponding increase in negative affective measures is detected. Our predictive model has achieved a good accuracy level using three different "in the wild" classifiers (F-measure 74%-78% within-subject model, 72%-76% global model). Our findings show that it is possible to automatically predict when people are experiencing positive, neutral, or negative affective states based on interactions with notifications. We also show how our findings open the door to a wide range of applications in relation to emotion awareness on social and mobile communication.
Abstract
The Internet of Things (IoT) and wearable computing are crucial elements of modern information systems and applications in which advanced features for user interactivity and monitoring are ...required. However, in the fields of pervasive gaming, IoT has had limited real-world applications. In this work, we present a prototype of a wearable platform for pervasive games that combines IoT with wearable computing to enable the real-time monitoring of physical activity. The main objective of the solution is to promote the utilization of gamification techniques to enhance the physical activity of users through challenges and quests. This aims to create a symbolic link between the virtual gameplay and the real-world environment without the requirement of a smartphone. With the integration of sensors and wearable devices by design, the platform has the capability of real-time monitoring the users’ physical activity during the game. The system performance results highlight the efficiency and attractiveness of the wearable platform for gamifying physical activity.
PDF
