Smartphones allow for prompting users with a short questionnaire about their current subjective experience, a technique often called Ecological Momentary Assessments. One of the biggest challenges ...for such studies is a lack of adherence, diminishing the benefits for both user and researcher. Being able to predict if a user is going to stop answering the questionnaire prompts would be beneficial for researchers and developers. This would allow for, for example, specifically addressing those users, or for over-sampling populations at higher risk of dropping out of a study. In this work, based on an observational study of the general population, we analyzed data from almost 1,000 users. The data include a large variety of sensor data from the users’ smartphones. We utilized machine learning to predict adherence on a day-to-day level, as well as predict adherence based on participant data after on-boarding. For day-to-day prediction, the best performing model was a model based on metadata features (days since first questionnaire was filled out, days since the last questionnaire was filled out, number of filled-out questionnaires, days since app installation), yielding an area under the precision–recall curve of 0.89. The inclusion of sensor data did not improve the model’s performance, indicating that the high cost of collecting and processing sensor data is not worth the benefits for predicting fill-out behavior. Predicting at sign-up if a user will adhere to a questionnaire prompt at least once was better than chance, but further studies are needed.
•User interactions with the ecological momentary assessment app predicted adherence.•Smartphone sensor data did not improve prediction performance.•Predicting if new users will fill out at a questionnaire was moderately possible.
•First hybrid Deep Learning approach applied to environmental and on-body sensor data.•Deep-learning is effective in human emotion classification.•Multi-model data fusion approach performs better ...than single modality classification.•Hybrid approach with combined CNN and LSTM exceed the performance of CNN alone.
The detection and monitoring of emotions are important in various applications, e.g., to enable naturalistic and personalised human-robot interaction. Emotion detection often require modelling of various data inputs from multiple modalities, including physiological signals (e.g., EEG and GSR), environmental data (e.g., audio and weather), videos (e.g., for capturing facial expressions and gestures) and more recently motion and location data. Many traditional machine learning algorithms have been utilised to capture the diversity of multimodal data at the sensors and features levels for human emotion classification. While the feature engineering processes often embedded in these algorithms are beneficial for emotion modelling, they inherit some critical limitations which may hinder the development of reliable and accurate models. In this work, we adopt a deep learning approach for emotion classification through an iterative process by adding and removing large number of sensor signals from different modalities. Our dataset was collected in a real-world study from smart-phones and wearable devices. It merges local interaction of three sensor modalities: on-body, environmental and location into global model that represents signal dynamics along with the temporal relationships of each modality. Our approach employs a series of learning algorithms including a hybrid approach using Convolutional Neural Network and Long Short-term Memory Recurrent Neural Network (CNN-LSTM) on the raw sensor data, eliminating the needs for manual feature extraction and engineering. The results show that the adoption of deep-learning approaches is effective in human emotion classification when large number of sensors input is utilised (average accuracy 95% and F-Measure=%95) and the hybrid models outperform traditional fully connected deep neural network (average accuracy 73% and F-Measure=73%). Furthermore, the hybrid models outperform previously developed Ensemble algorithms that utilise feature engineering to train the model average accuracy 83% and F-Measure=82%)
Depression and anxiety are leading causes of disability worldwide but often remain undetected and untreated. Smartphone and wearable devices may offer a unique source of data to detect moment by ...moment changes in risk factors associated with mental disorders that overcome many of the limitations of traditional screening methods.
The current study aimed to explore the extent to which data from smartphone and wearable devices could predict symptoms of depression and anxiety.
A total of
= 60 adults (ages 24-68) who owned an Apple iPhone and Oura Ring were recruited online over a 2-week period. At the beginning of the study, participants installed the
data acquisition app on their smartphone. The app continuously monitored participants' location (using GPS) and smartphone usage behavior (total usage time and frequency of use). The Oura Ring provided measures related to activity (step count and metabolic equivalent for task), sleep (total sleep time, sleep onset latency, wake after sleep onset and time in bed) and heart rate variability (HRV). In addition, participants were prompted to report their daily mood (valence and arousal). Participants completed self-reported assessments of depression, anxiety and stress (DASS-21) at baseline, midpoint and the end of the study.
Multilevel models demonstrated a significant negative association between the variability of locations visited and symptoms of depression (beta = -0.21,
= 0.037) and significant positive associations between total sleep time and depression (beta = 0.24,
= 0.023), time in bed and depression (beta = 0.26,
= 0.020), wake after sleep onset and anxiety (beta = 0.23,
= 0.035) and HRV and anxiety (beta = 0.26,
= 0.035). A combined model of smartphone and wearable features and self-reported mood provided the strongest prediction of depression.
The current findings demonstrate that wearable devices may provide valuable sources of data in predicting symptoms of depression and anxiety, most notably data related to common measures of sleep.
Smartphones enjoy high adoption rates around the globe. Rarely more than an arm’s length away, these sensor-rich devices can easily be repurposed to collect rich and extensive records of their users’ ...behaviors (e.g., location, communication, media consumption), posing serious threats to individual privacy. Here we examine the extent to which individuals’ Big Five personality dimensions can be predicted on the basis of six different classes of behavioral information collected via sensor and log data harvested from smartphones. Taking a machine-learning approach, we predict personality at broad domain (r
median = 0.37) and narrow facet levels (r
median = 0.40) based on behavioral data collected from 624 volunteers over 30 consecutive days (25,347,089 logging events). Our cross-validated results reveal that specific patterns in behaviors in the domains of 1) communication and social behavior, 2) music consumption, 3) app usage, 4) mobility, 5) overall phone activity, and 6) day- and night-time activity are distinctively predictive of the Big Five personality traits. The accuracy of these predictions is similar to that found for predictions based on digital footprints from social media platforms and demonstrates the possibility of obtaining information about individuals’ private traits from behavioral patterns passively collected from their smartphones. Overall, our results point to both the benefits (e.g., in research settings) and dangers (e.g., privacy implications, psychological targeting) presented by the widespread collection and modeling of behavioral data obtained from smartphones.
The vast majority of people worldwide have been impacted by coronavirus disease (COVID-19). In addition to the millions of individuals who have been infected with the disease, billions of individuals ...have been asked or required by local and national governments to change their behavioral patterns. Previous research on epidemics or traumatic events suggests that this can lead to profound behavioral and mental health changes; however, researchers are rarely able to track these changes with frequent, near-real-time sampling or compare their findings to previous years of data for the same individuals.
By combining mobile phone sensing and self-reported mental health data among college students who have been participating in a longitudinal study for the past 2 years, we sought to answer two overarching questions. First, have the behaviors and mental health of the participants changed in response to the COVID-19 pandemic compared to previous time periods? Second, are these behavior and mental health changes associated with the relative news coverage of COVID-19 in the US media?
Behaviors such as the number of locations visited, distance traveled, duration of phone usage, number of phone unlocks, sleep duration, and sedentary time were measured using the StudentLife smartphone sensing app. Depression and anxiety were assessed using weekly self-reported ecological momentary assessments of the Patient Health Questionnaire-4. The participants were 217 undergraduate students, with 178 (82.0%) students providing data during the Winter 2020 term. Differences in behaviors and self-reported mental health collected during the Winter 2020 term compared to previous terms in the same cohort were modeled using mixed linear models.
During the first academic term impacted by COVID-19 (Winter 2020), individuals were more sedentary and reported increased anxiety and depression symptoms (P<.001) relative to previous academic terms and subsequent academic breaks. Interactions between the Winter 2020 term and the week of the academic term (linear and quadratic) were significant. In a mixed linear model, phone usage, number of locations visited, and week of the term were strongly associated with increased amount of COVID-19-related news. When mental health metrics (eg, depression and anxiety) were added to the previous measures (week of term, number of locations visited, and phone usage), both anxiety (P<.001) and depression (P=.03) were significantly associated with COVID-19-related news.
Compared with prior academic terms, individuals in the Winter 2020 term were more sedentary, anxious, and depressed. A wide variety of behaviors, including increased phone usage, decreased physical activity, and fewer locations visited, were associated with fluctuations in COVID-19 news reporting. While this large-scale shift in mental health and behavior is unsurprising, its characterization is particularly important to help guide the development of methods to reduce the impact of future catastrophic events on the mental health of the population.
To predict and prevent mental health crises, we must develop new approaches that can provide a dramatic advance in the effectiveness, timeliness, and scalability of our interventions. However, ...current methods of predicting mental health crises (eg, clinical monitoring, screening) usually fail on most, if not all, of these criteria. Luckily for us, 77% of Americans carry with them an unprecedented opportunity to detect risk states and provide precise life-saving interventions. Smartphones present an opportunity to empower individuals to leverage the data they generate through their normal phone use to predict and prevent mental health crises.
To facilitate the collection of high-quality, passive mobile sensing data, we built the Effortless Assessment of Risk States (EARS) tool to enable the generation of predictive machine learning algorithms to solve previously intractable problems and identify risk states before they become crises.
The EARS tool captures multiple indices of a person's social and affective behavior via their naturalistic use of a smartphone. Although other mobile data collection tools exist, the EARS tool places a unique emphasis on capturing the content as well as the form of social communication on the phone. Signals collected include facial expressions, acoustic vocal quality, natural language use, physical activity, music choice, and geographical location. Critically, the EARS tool collects these data passively, with almost no burden on the user. We programmed the EARS tool in Java for the Android mobile platform. In building the EARS tool, we concentrated on two main considerations: (1) privacy and encryption and (2) phone use impact.
In a pilot study (N=24), participants tolerated the EARS tool well, reporting minimal burden. None of the participants who completed the study reported needing to use the provided battery packs. Current testing on a range of phones indicated that the tool consumed approximately 15% of the battery over a 16-hour period. Installation of the EARS tool caused minimal change in the user interface and user experience. Once installation is completed, the only difference the user notices is the custom keyboard.
The EARS tool offers an innovative approach to passive mobile sensing by emphasizing the centrality of a person's social life to their well-being. We built the EARS tool to power cutting-edge research, with the ultimate goal of leveraging individual big data to empower people and enhance mental health.
•Provide a systematic review of CAV based UTC studies in the last few decades.•Highlight the change of design philosophy for UTC.•Summarize the benefits and drawbacks of current CAV-based UTC ...methods.•Predict the future study directions.
Inefficient traffic control is pervasive in modern urban areas, which would exaggerate traffic congestion as well as deteriorate mobility, fuel economy and safety. In this paper, we systematically review the potential solutions that take advantage of connected and automated vehicles (CAVs) to improve the control performances of urban signalized intersections. We review the methods and models to estimate traffic flow states and optimize traffic signal timing plans based on CAVs. We summarize six types of CAV-based traffic control methods and propose a conceptual mathematical framework that can be specified to each of six three types of methods by selecting different state variables, control inputs, and environment inputs. The benefits and drawbacks of various CAV-based control methods are explained, and future research directions are discussed. We hope that this review could provide readers with a helpful roadmap for future research on CAV-based urban traffic control and draw their attention to the most challenging problems in this important and promising field.
PDF
