Increasingly, individuals identify with two or more racial or cultural, yet are sometimes externally misclassified, contributing to experiences of invisibility within U.S. society. Using ...computational techniques, we examined the transmission of cultural identity terms through time, providing some evidence for the changing representation of social identity. We examined the usage patterns of identity terms with the prefixes (mono‐, bi‐, multi‐), modifying the social identity terms: culture, ethnicity, and race (e.g., comparing monocultural, monoethnic, and monoracial). For bicultural and multicultural terms, those with ‐racial suffixes were the earliest used terms, while those with ‐cultural and ‐ethnic suffixes gained more popularity in recent years. We examined the evolution of the higher frequency social identity terms in lay sources (the NY Times, Reddit), and found that interracial and multicultural were the most popular over time, peaking recently. We examined the potential time lag in the sequence of identity terms among academic (PsycINFO, NIH, and NSF Databases), lay (the NY Times), and mixed sources (Google Books N‐Grams), demonstrating that newer terms (e.g., multicultural) are first used and gain prevalence in lay sources, then mixed sources, and eventually academic sources. The implications of these findings for research, public policy, and psychosocial experiences of individuals are discussed.
Most widely used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison,
models that operate directly on raw text (bytes or characters) have ...many benefits: They can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Because byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.
Inferring model parameters from experimental data is a grand challenge in many sciences, including cosmology. This often relies critically on high fidelity numerical simulations, which are ...prohibitively computationally expensive. The application of deep learning techniques to generative modeling is renewing interest in using high dimensional density estimators as computationally inexpensive emulators of fully-fledged simulations. These generative models have the potential to make a dramatic shift in the field of scientific simulations, but for that shift to happen we need to study the performance of such generators in the precision regime needed for science applications. To this end, in this work we apply Generative Adversarial Networks to the problem of generating weak lensing convergence maps. We show that our generator network produces maps that are described by, with high statistical confidence, the same summary statistics as the fully simulated maps.
Motion forecasting for autonomous driving is a challenging task because complex driving scenarios involve a heterogeneous mix of static and dynamic inputs. It is an open problem how best to represent ...and fuse information about road geometry, lane connectivity, time-varying traffic light state, and history of a dynamic set of agents and their interactions into an effective encoding. To model this diverse set of input features, many approaches proposed to design an equally complex system with a diverse set of modality specific modules. This results in systems that are difficult to scale, extend, or tune in rigorous ways to trade off quality and efficiency. In this paper, we present Wayformer, a family of simple and homogeneous attention based architectures for motion forecasting. Wayformer offers a compact model description consisting of an attention based scene encoder and a decoder. In the scene encoder we study the choice of early, late and hierarchical fusion of input modalities. For each fusion type we explore strategies to trade off efficiency and quality via factorized attention or latent query attention. We show that early fusion, despite its simplicity, is not only modality agnostic but also achieves state-of-the-art results on both Waymo Open Motion Dataset (WOMD) and Argoverse leaderboards, demonstrating the effectiveness of our design philosophy.
Motion forecasting has become an increasingly critical component of autonomous robotic systems. Onboard compute budgets typically limit the accuracy of real-time systems. In this work we propose ...methods of improving motion forecasting systems subject to limited compute budgets by combining model ensemble and distillation techniques. The use of ensembles of deep neural networks has been shown to improve generalization accuracy in many application domains. We first demonstrate significant performance gains by creating a large ensemble of optimized single models. We then develop a generalized framework to distill motion forecasting model ensembles into small student models which retain high performance with a fraction of the computing cost. For this study we focus on the task of motion forecasting using real world data from autonomous driving systems. We develop ensemble models that are very competitive on the Waymo Open Motion Dataset (WOMD) and Argoverse leaderboards. From these ensembles, we train distilled student models which have high performance at a fraction of the compute costs. These experiments demonstrate distillation from ensembles as an effective method for improving accuracy of predictive models for robotic systems with limited compute budgets.
Motion forecasting has become an increasingly critical component of autonomous robotic systems. Onboard compute budgets typically limit the accuracy of real-time systems. In this work we propose ...methods of improving motion forecasting systems subject to limited compute budgets by combining model ensemble and distillation techniques. The use of ensembles of deep neural networks has been shown to improve generalization accuracy in many application domains. We first demonstrate significant performance gains by creating a large ensemble of optimized single models. We then develop a generalized framework to distill motion forecasting model ensembles into small student models which retain high performance with a fraction of the computing cost. For this study we focus on the task of motion forecasting using real world data from autonomous driving systems. We develop ensemble models that are very competitive on the Waymo Open Motion Dataset (WOMD) and Argoverse leaderboards. From these ensembles, we train distilled student models which have high performance at a fraction of the compute costs. These experiments demonstrate distillation from ensembles as an effective method for improving accuracy of predictive models for robotic systems with limited compute budgets.
We built a Natural Language Processing (NLP) pipeline for each of Wikipedia languages through semi-supervised learning. Each pipeline consists of a language specific tokenizer, sentence segmenter, ...morphological analyzer, Part of Speech tagger, sentiment analysis, and Named Entity Recognition (NER) annotator. We automatically learn features (embedding) for each word in each language using continuous space language models, which capture syntactic and semantic characteristics of the language. We use these embeddings as features to train part of speech taggers with the help of human annotated datasets. To enable larger coverage of languages, we use these features with automatically-extracted annotations from Wikipedia to build a semi-supervised NER system. With strong prior (word embeddings) and simple statistical methods, we overcome the noise and bias introduced by the Wikipedia style guidelines. To demonstrate the quality of our work, we propose new evaluation metrics to accommodate to the large scale of languages we are targeting. Furthermore, all the pipelines are available to the community to use and study through the software package polyglot (available at http://polyglot-nlp.com).
PDF
