As an essential task for the architecture, engineering, and construction (AEC) industry, information processing and acquiring from unstructured textual data based on natural language processing (NLP) ...are gaining increasing attention. Although deep learning (DL) models for NLP tasks have been investigated for years, domain-specific pretrained DL models and their advantages are seldomly investigated in the AEC domain. Therefore, this work developed a large scale domain corpora and pretrained domain-specific language models for the AEC domain, and then systematically explores various transfer learning and fine-tuning techniques to explore the performance of pretrained DL models for various NLP tasks. First, both in-domain and close-domain Chinese corpora are developed. Then, two types of pretrained models, including static word embedding models and contextual word embedding models, are pretrained based on various domain corpora. Finally, several widely used DL models for NLP tasks are further trained and tested based on various pretrained models. The result shows that domain corpora can further improve the performance of static word embedding-based DL models and contextual word embedding-based DL models in text classification (TC) and named entity recognition (NER) tasks. Meanwhile, contextual word embedding-based DL models significantly outperform the static word embedding-based DL methods in TC and NER tasks, with maximum improvements of 8.1% and 3.8% in the F1 score, respectively. This research contributes to the body of knowledge in two ways: (1) demonstrating the advantages of domain corpora and pretrained DL models, and (2) opening the first domain-specific dataset and pretrained language models named ARCBERT for the AEC domain. Thus, this work sheds light on the adoption and application of pretrained models in the AEC domain.
•The first domain corpora for the architecture, engineering, and construction (AEC) domain are proposed.•Proposes the first domain-specific pretrained language model for typical natural language processing (NLP) tasks in AEC.•Systematical experiments are carried out to illustrate the effect of domain corpus and transfer learning techniques.•Proposed ARCBERT outperforms static word embedding-based methods in all typical NLP tasks, and increases F1 by up to 8.1%.•Improves the performance of deep learning models without increasing the effort of manual annotation.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
•This paper uses the BERT large language model and text mining method.•This paper proposes a firm-level alignment index between firm achievements and government development goals.•The capital markets ...have priced in this alignment index.
Using the BERT large language model and extensive textual data, this paper measures the alignment between firm achievements and government development goals. We prove that firms aligning closely with national goals are more favorably valued in the capital market. Our results offer valuable insights for policymakers and corporate leaders in strategy optimization within complex macroeconomic environments.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Semantic understanding enhancement methods and deep learning are popular areas of artificial intelligence research and have significant potential in natural language processing. The English ...translation is one of the typical application scenarios combining these two technologies. In order to thoroughly analyze the information contained in English texts and improve the accuracy of English text translation, this study proposes an unbalanced Bi-LSTM model. Firstly, the BERT model is used to vectorize the original English corpus and extract the preliminary semantic features. Then, the unbalanced Bi-LSTM network is used to increase the weight of the textual information containing important semantics to further improve the effect of the key features on the recognition of the English text and, at the same time, an attention mechanism that introduces the word vectors is used to widen the gap between the key textual information and the non-key information, so as to improve the effect of the English translation. The accuracy of English text translation can be significantly enhanced by comparing the classification effect with various models, as shown by the results. The accuracy of the model can reach over 90% in about 60 pieces of translation training, and the mean square average is only 1.52. Its translation effect has won the recognition of more than 50% of professionals. The model's ability to translate English is evident.
Sentiment Analysis Using Bert Model Manuel-Ilie, Dorca; Gabriel, Pitic Antoniu; George, Crețulescu Radu
International Journal of Advanced Statistics and IT&C for Economics and Life Sciences,
12/2023, Volume:
13, Issue:
1
Journal Article
Peer reviewed
Open access
The topic of this presentation entails a comprehensive investigation of our sentiment analysis algorithm. The document provides a thorough examination of its theoretical underpinnings, meticulous ...assessment criteria, consequential findings, and an enlightening comparative analysis. Our system makes a substantial contribution to the field of sentiment analysis by using advanced techniques based on deep learning and state-of-the-art architectures.
Regular polysemes are sets of ambiguous words that all share the same relationship between their meanings, such as CHICKEN and LOBSTER both referring to an animal or its meat. To probe how a ...distributional semantic model, here exemplified by bidirectional encoder representations from transformers (BERT), represents regular polysemy, we analyzed whether its embeddings support answering sense analogy questions similar to “is the mapping between CHICKEN (as an animal) and CHICKEN (as a meat) similar to that which maps between LOBSTER (as an animal) to LOBSTER (as a meat)?” We did so using the LRcos model, which combines a logistic regression classifier of different categories (e.g., animal vs. meat) with a measure of cosine similarity. We found that (a) the model was sensitive to the shared structure within a given regular relationship; (b) the shared structure varies across different regular relationships (e.g., animal/meat vs. location/organization), potentially reflective of a “regularity continuum;” (c) some high‐order latent structure is shared across different regular relationships, suggestive of a similar latent structure across different types of relationships; and (d) there is a lack of evidence for the aforementioned effects being explained by meaning overlap. Lastly, we found that both components of the LRcos model made important contributions to accurate responding and that a variation of this method could yield an accuracy boost of 10% in answering sense analogy questions. These findings enrich previous theoretical work on regular polysemy with a computationally explicit theory and methods, and provide evidence for an important organizational principle for the mental lexicon and the broader conceptual knowledge system.
Full text
Available for:
BFBNIB, FZAB, GIS, IJS, KILJ, NLZOH, NUK, OILJ, SAZU, SBCE, SBMB, UL, UM, UPUK
Live streaming danmaku can reflect both real-time interaction and user sentiment, two key characteristics of live streaming e-commerce. Using a sentiment analysis of live streaming danmaku comments, ...our study explores the mechanism through which user-user interaction influences live streaming sales. We collect real world data including live streaming danmaku, user stay time, sales from the Douyin live streaming e-commerce platform (the Chinese version of TikTok). Quantifying user-user interaction with BERT model, we then perform a sentiment analysis using the Baidu API. The results of our analysis demonstrate that entertainment-type and information-type user interaction have a significant positive effect on flow, including user stay time, number of danmaku, number of people sending danmaku. Our study also identifies that number of danmaku and number of people sending danmaku play a partial mediating role between user interactions and live streaming sales. Furthermore, we find that user sentiment plays a positive moderating role in the relationships between several flow variables and user purchase behavior. Theoretically, our study proposes a new method to empirically analyze user behavior in live streaming e-commerce. Practically, in light of these findings, we propose several optimization strategies to enhance the interaction functions of live streaming commerce, so as to enhance user experience and stimulate purchase intention.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Sentiment Analysis (SA) is one of the most active research areas in the Natural Language Processing (NLP) field due to its potential for business and society. With the development of language ...representation models, numerous methods have shown promising efficiency in fine-tuning pre-trained language models in NLP downstream tasks. For Vietnamese, many available pre-trained language models were also released, including the monolingual and multilingual language models. Unfortunately, all of these models were trained on different architectures, pre-trained data, and pre-processing steps; consequently, fine-tuning these models can be expected to yield different effectiveness. In addition, there is no study focusing on evaluating the performance of these models on the same datasets for the SA task up to now. This article presents a fine-tuning approach to investigate the performance of different pre-trained language models for the Vietnamese SA task. The experimental results show the superior performance of the monolingual PhoBERT model and ViT5 model in comparison with previous studies and provide new state-of-the-art performances on five benchmark Vietnamese SA datasets. To the best of our knowledge, our study is the first attempt to investigate the performance of fine-tuning Transformer-based models on five datasets with different domains and sizes for the Vietnamese SA task.
Stock comments published by experts are important references for accurate stock trends prediction. How to comprehensively and accurately capture the topic of expert stock comments is an important ...issue which belongs to text classification. The Bidirectional Encoder Representations from Transformers (BERT) pretrained language model is widely used for text classification, due to its high identification accuracy. However, BERT has some limitations. First, it only utilizes fixed length text, leading to suboptimal performance in long text information exploration. Second, it only relies on the features extracted from the last layer, resulting in incomprehensive classification features. To tackle these issues, we propose a multi-layer features ablation study of BERT model for accurate identification of stock comments’ themes. Specifically, we firstly divide the original text to meet the length requirement of the BERT model based on sliding window technology. In this way, we can enlarge the sample size which is beneficial for reducing the over-fitting problem. At the same time, by dividing the long text into multiple short texts, all the information of the long text can be comprehensively captured through the synthesis of the subject information of multiple short texts. In addition, we extract the output features of each layer in the BERT model and apply the ablation strategy to extract more effective information in these features. Experimental results demonstrate that compared with non-intercepted comments, the topic recognition accuracy is improved by intercepting stock comments based on sliding window technology. It proves that intercepting text can improve the performance of text classification. Compared with the BERT, the multi-layer features ablation study we present in the paper further improves the performance in the topic recognition of stock comments, and can provide reference for the majority of investors. Our study has better performance and practicability on stock trend prediction by stock comments topic recognition.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
Additive Manufacturing (AM) is gaining acceptance as a strategic manufacturing technique and technology for allowing new product development. Due to the lack of knowledge, design for additive ...manufacturing (DFAM) is now a major challenge in utilizing AM’s product innovation and manufacturing capabilities. The AM sector will benefit from developing an intuitive knowledge reasoning method by constructing a Knowledge Graph (KG). We presented Bidirectional Encoder Representations from the Transformers (BERT) model for collaborative entity/relation recognition to address the issue, allowing us to study and utilize the advantages of AM through knowledge reasoning for Fused Deposition Modeling (FDM) based DFAM. First, the model analyzes preprocessed text to find and extract entities. Then, the relation recognition procedure based on dependency parsing extracts the semantic relationships between the entities. To convert word segments into vectors and improve dependency parsing, we used Continuous-Bag-of-Words (CBOW) to process texts. Therefore, this allowed us to anticipate the probability output of the center word based on the n − 1 words around the input. The extracted knowledge is then visualized as a graph and stored in the Neo4j database. Following the methods above creates a KG for the FDM-based DFAM knowledge. It can be shown that BERT is a good option for handling knowledge-driven issues needing specialists by extracting the process knowledge from text data using our suggested model. We provide evidence demonstrating the model’s ability to set reasonable limitations on its predictions through a KG. Additionally, we use experiments and an application case study to demonstrate the effectiveness and competitiveness of our approach.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
The big data era enables automakers to mine users’ affective (Kansei) requirements for the car design. However, existing literature mostly applies text mining with users’ online comments, possibly ...leading to biased results since users without online comments were not considered. To fill in this gap, this paper proposes to jointly analyse users’ online commenting and offline usage big data, and develops a novel framework to efficiently fuse these two datasets for the Kansei engineering of the intelligent connected vehicle (ICV) functions. A behaviour-enhanced large language model is proposed to process users’ online comments; then, users’ Kansei requirements are further jointly analysed with their offline in-cabin behaviour data, by the proposed NLP-MDCEV (natural language process — multiple discrete-continuous extreme value) model, to understand user’s complex discrete and continuous choice decisions in the smart cockpit. In addition, the proposed framework aims to solve the problem of design tasks prioritization, where not all the Kansei requirements can be met if design resources are limited. The proposed framework is applied in the studied new energy vehicle company, with more than nine-months’ online comments and six-months’ offline usage data, where results suggest its merits of economic, efficient, and effective.
•Online comments and offline behaviours are mined for Kansei design requirements.•Entropy is used to evaluate the cost-benefit of customized text mining models.•Behaviour-enhanced BERT model is proposed to achieve a 98% F1-score in text mining.•NLP-boost MDCEV model is developed to bridge online-offline data for Kansei analysis.•Applied in a new energy vehicle company with real data.
Full text
Available for:
GEOZS, IJS, IMTLJ, KILJ, KISLJ, NLZOH, NUK, OILJ, PNG, SAZU, SBCE, SBJE, UILJ, UL, UM, UPCLJ, UPUK, ZAGLJ, ZRSKP
