As an essential task for the architecture, engineering, and construction (AEC) industry, information processing and acquiring from unstructured textual data based on natural language processing (NLP) ...are gaining increasing attention. Although deep learning (DL) models for NLP tasks have been investigated for years, domain-specific pretrained DL models and their advantages are seldomly investigated in the AEC domain. Therefore, this work developed a large scale domain corpora and pretrained domain-specific language models for the AEC domain, and then systematically explores various transfer learning and fine-tuning techniques to explore the performance of pretrained DL models for various NLP tasks. First, both in-domain and close-domain Chinese corpora are developed. Then, two types of pretrained models, including static word embedding models and contextual word embedding models, are pretrained based on various domain corpora. Finally, several widely used DL models for NLP tasks are further trained and tested based on various pretrained models. The result shows that domain corpora can further improve the performance of static word embedding-based DL models and contextual word embedding-based DL models in text classification (TC) and named entity recognition (NER) tasks. Meanwhile, contextual word embedding-based DL models significantly outperform the static word embedding-based DL methods in TC and NER tasks, with maximum improvements of 8.1% and 3.8% in the F1 score, respectively. This research contributes to the body of knowledge in two ways: (1) demonstrating the advantages of domain corpora and pretrained DL models, and (2) opening the first domain-specific dataset and pretrained language models named ARCBERT for the AEC domain. Thus, this work sheds light on the adoption and application of pretrained models in the AEC domain.
•The first domain corpora for the architecture, engineering, and construction (AEC) domain are proposed.•Proposes the first domain-specific pretrained language model for typical natural language processing (NLP) tasks in AEC.•Systematical experiments are carried out to illustrate the effect of domain corpus and transfer learning techniques.•Proposed ARCBERT outperforms static word embedding-based methods in all typical NLP tasks, and increases F1 by up to 8.1%.•Improves the performance of deep learning models without increasing the effort of manual annotation.
•This paper uses the BERT large language model and text mining method.•This paper proposes a firm-level alignment index between firm achievements and government development goals.•The capital markets ...have priced in this alignment index.
Using the BERT large language model and extensive textual data, this paper measures the alignment between firm achievements and government development goals. We prove that firms aligning closely with national goals are more favorably valued in the capital market. Our results offer valuable insights for policymakers and corporate leaders in strategy optimization within complex macroeconomic environments.
Live streaming danmaku can reflect both real-time interaction and user sentiment, two key characteristics of live streaming e-commerce. Using a sentiment analysis of live streaming danmaku comments, ...our study explores the mechanism through which user-user interaction influences live streaming sales. We collect real world data including live streaming danmaku, user stay time, sales from the Douyin live streaming e-commerce platform (the Chinese version of TikTok). Quantifying user-user interaction with BERT model, we then perform a sentiment analysis using the Baidu API. The results of our analysis demonstrate that entertainment-type and information-type user interaction have a significant positive effect on flow, including user stay time, number of danmaku, number of people sending danmaku. Our study also identifies that number of danmaku and number of people sending danmaku play a partial mediating role between user interactions and live streaming sales. Furthermore, we find that user sentiment plays a positive moderating role in the relationships between several flow variables and user purchase behavior. Theoretically, our study proposes a new method to empirically analyze user behavior in live streaming e-commerce. Practically, in light of these findings, we propose several optimization strategies to enhance the interaction functions of live streaming commerce, so as to enhance user experience and stimulate purchase intention.
Semantic understanding enhancement methods and deep learning are popular areas of artificial intelligence research and have significant potential in natural language processing. The English ...translation is one of the typical application scenarios combining these two technologies. In order to thoroughly analyze the information contained in English texts and improve the accuracy of English text translation, this study proposes an unbalanced Bi-LSTM model. Firstly, the BERT model is used to vectorize the original English corpus and extract the preliminary semantic features. Then, the unbalanced Bi-LSTM network is used to increase the weight of the textual information containing important semantics to further improve the effect of the key features on the recognition of the English text and, at the same time, an attention mechanism that introduces the word vectors is used to widen the gap between the key textual information and the non-key information, so as to improve the effect of the English translation. The accuracy of English text translation can be significantly enhanced by comparing the classification effect with various models, as shown by the results. The accuracy of the model can reach over 90% in about 60 pieces of translation training, and the mean square average is only 1.52. Its translation effect has won the recognition of more than 50% of professionals. The model's ability to translate English is evident.
Sentiment Analysis Using Bert Model Manuel-Ilie, Dorca; Gabriel, Pitic Antoniu; George, Crețulescu Radu
International Journal of Advanced Statistics and IT&C for Economics and Life Sciences,
12/2023, Volume:
13, Issue:
1
Journal Article
Peer reviewed
Open access
The topic of this presentation entails a comprehensive investigation of our sentiment analysis algorithm. The document provides a thorough examination of its theoretical underpinnings, meticulous ...assessment criteria, consequential findings, and an enlightening comparative analysis. Our system makes a substantial contribution to the field of sentiment analysis by using advanced techniques based on deep learning and state-of-the-art architectures.
Regular polysemes are sets of ambiguous words that all share the same relationship between their meanings, such as CHICKEN and LOBSTER both referring to an animal or its meat. To probe how a ...distributional semantic model, here exemplified by bidirectional encoder representations from transformers (BERT), represents regular polysemy, we analyzed whether its embeddings support answering sense analogy questions similar to “is the mapping between CHICKEN (as an animal) and CHICKEN (as a meat) similar to that which maps between LOBSTER (as an animal) to LOBSTER (as a meat)?” We did so using the LRcos model, which combines a logistic regression classifier of different categories (e.g., animal vs. meat) with a measure of cosine similarity. We found that (a) the model was sensitive to the shared structure within a given regular relationship; (b) the shared structure varies across different regular relationships (e.g., animal/meat vs. location/organization), potentially reflective of a “regularity continuum;” (c) some high‐order latent structure is shared across different regular relationships, suggestive of a similar latent structure across different types of relationships; and (d) there is a lack of evidence for the aforementioned effects being explained by meaning overlap. Lastly, we found that both components of the LRcos model made important contributions to accurate responding and that a variation of this method could yield an accuracy boost of 10% in answering sense analogy questions. These findings enrich previous theoretical work on regular polysemy with a computationally explicit theory and methods, and provide evidence for an important organizational principle for the mental lexicon and the broader conceptual knowledge system.
Stock comments published by experts are important references for accurate stock trends prediction. How to comprehensively and accurately capture the topic of expert stock comments is an important ...issue which belongs to text classification. The Bidirectional Encoder Representations from Transformers (BERT) pretrained language model is widely used for text classification, due to its high identification accuracy. However, BERT has some limitations. First, it only utilizes fixed length text, leading to suboptimal performance in long text information exploration. Second, it only relies on the features extracted from the last layer, resulting in incomprehensive classification features. To tackle these issues, we propose a multi-layer features ablation study of BERT model for accurate identification of stock comments’ themes. Specifically, we firstly divide the original text to meet the length requirement of the BERT model based on sliding window technology. In this way, we can enlarge the sample size which is beneficial for reducing the over-fitting problem. At the same time, by dividing the long text into multiple short texts, all the information of the long text can be comprehensively captured through the synthesis of the subject information of multiple short texts. In addition, we extract the output features of each layer in the BERT model and apply the ablation strategy to extract more effective information in these features. Experimental results demonstrate that compared with non-intercepted comments, the topic recognition accuracy is improved by intercepting stock comments based on sliding window technology. It proves that intercepting text can improve the performance of text classification. Compared with the BERT, the multi-layer features ablation study we present in the paper further improves the performance in the topic recognition of stock comments, and can provide reference for the majority of investors. Our study has better performance and practicability on stock trend prediction by stock comments topic recognition.
•A harmful news is defined as those explicit or implicit harmful speech in news text that harms people or affects readers’ perception.•A harmful news identification dataset is established and conduct ...a correlation analysis.•A BERT-based model which applies ensemble learning methods with a text sentiment analysis is proposed to identify harmful news.•Results show that the F1-score of the proposed model reaches 66.3%, an increase of 7.8% compared with that of the previous approach.
In an environment full of disordered information, the media spreads fake or harmful information into the public arena with a speed which is faster than ever before. A news report should ideally be neutral and factual. Excessive personal emotions or viewpoints should not be included. News articles ought not to be intentionally or maliciously written or create a media framing. A harmful news is defined as those explicit or implicit harmful speech in news text that harms people or affects readers’ perception. However, in the current situation, it is difficult to effectively identify and predict fake or harmful news in advance, especially harmful news. Therefore, in this study, we propose a Bidirectional Encoder Representation from Transformers (BERT) based model which applies ensemble learning methods with a text sentiment analysis to identify harmful news, aiming to provide readers with a way to identify harmful news content so as to help them to judge whether the information provided is in a more neutral manner. The working model of the proposed system has two phases. The first phase is collecting harmful news and establishing a development model for analyzing the correlation between text sentiment and harmful news. The second phase is identifying harmful news by analyzing text sentiment with an ensemble learning technique and the BERT model. The purpose is to determine whether the news has harmful intentions. Our experimental results show that the F1-score of the proposed model reaches 66.3%, an increase of 7.8% compared with that of the previous term frequency-inverse document frequency approach which adopts a Lagrangian Support Vector Machine (LSVM) model without using a text sentiment. Moreover, the proposed method achieves a better performance in recognizing various cases of information disorder.
Additive Manufacturing (AM) is gaining acceptance as a strategic manufacturing technique and technology for allowing new product development. Due to the lack of knowledge, design for additive ...manufacturing (DFAM) is now a major challenge in utilizing AM’s product innovation and manufacturing capabilities. The AM sector will benefit from developing an intuitive knowledge reasoning method by constructing a Knowledge Graph (KG). We presented Bidirectional Encoder Representations from the Transformers (BERT) model for collaborative entity/relation recognition to address the issue, allowing us to study and utilize the advantages of AM through knowledge reasoning for Fused Deposition Modeling (FDM) based DFAM. First, the model analyzes preprocessed text to find and extract entities. Then, the relation recognition procedure based on dependency parsing extracts the semantic relationships between the entities. To convert word segments into vectors and improve dependency parsing, we used Continuous-Bag-of-Words (CBOW) to process texts. Therefore, this allowed us to anticipate the probability output of the center word based on the n − 1 words around the input. The extracted knowledge is then visualized as a graph and stored in the Neo4j database. Following the methods above creates a KG for the FDM-based DFAM knowledge. It can be shown that BERT is a good option for handling knowledge-driven issues needing specialists by extracting the process knowledge from text data using our suggested model. We provide evidence demonstrating the model’s ability to set reasonable limitations on its predictions through a KG. Additionally, we use experiments and an application case study to demonstrate the effectiveness and competitiveness of our approach.
