Investigating the Difference of Fake News Source Credibility Recognition between ANN and BERT Algorithms in Artificial Intelligence

Abstract

Fake news permeating life through channels misleads people into disinformation. To reduce the harm of fake news and provide multiple and effective news credibility channels, the approach of linguistics is applied to a word-frequency-based ANN system and semantics-based BERT system in this study, using mainstream news as a general news dataset and content farms as a fake news dataset for the models judging news source credibility and comparing the difference in news source credibility recognition between ANN and BERT. The research findings show high similarity in the highest and lowest hit rates between the ANN system and the BERT system (Liberty Time had the highest hit rate, while ETtoday and nooho.net had the lowest hit rates). The BERT system presents a higher and more stable overall source credibility recognition rate than the ANN system (BERT 91.2% > ANN 82.75%). Recognizing news source credibility through artificial intelligence not only could effectively enhance people’s sensitivity to news sources but, in the long term, could cultivate public media literacy to achieve the synergy of fake news resistance with technology.

1. Introduction

1.1. Research Background and Motivation

Along with the rapid development of computer technology and broadband Internet, the emergence of blogs, collaboration platforms, and communities under Web2.0 has seen online media enter the self-media era with an information explosion. Changing the information-receiving model of the audience, they are no longer passive information recipients. Therefore, through the production, sharing, and communication characteristics of social media, they can become prosumers with creation and communication abilities to self-make and post content beyond mainstream media. Nevertheless, social media, in comparison with mainstream media, has a lack of gatekeepers that filter messages and reduce the information-posting threshold, which results in the rapid growth and spread of misrepresented fake news on social media.

The “toilet paper panic” was set off in February 2018, in Taiwan, when people heard the news of a price increase in toilet paper and rushed to purchase toilet paper in physical stores and online shopping platforms. The “toilet paper panic” was caused by RT MART INTERNATIONAL LIMITED, who, in order to promote toilet paper, actively sent the message “toilet paper increase 30%, store performance up 5 times” to cause people to rush to purchase toilet paper before the price increase, resulting in national panic. The Fair Trade Commission declared on 14 March 2018 that the promotion of RT MART INTERNATIONAL LIMITED using disinformation to mislead consumers’ panic and unbalance the market transaction order violated Article 25 of the Fair Trade Act, and they were fined 3.5 million dollars to eventually end the incident [1].

Typhoon Jebi devastated Kansai, Japan, in September 2018, resulting in international airports being closed and many visitors from Taiwan not being able to return. A great deal of criticism of the malpractice of the Taipei Economic and Cultural Office in Osaka appeared on the Internet and alleged that local embassy staff from China arranged tour buses to dispatch visitors from China. It was rumored that visitors from Taiwan could arrange to leave Kansai Airport as long as they admitted to being people from China. The incident being hyped without verification had people dissatisfied with the unit stationed abroad. Many media outlets criticized the Ministry of Foreign Affairs, resulting in Su Chii-Cherng, Taiwan’s head representative of the Taipei Economic and Cultural Office in Osaka, Japan, committing suicide due to too much pressure. After the incident in Kansai Airport, the National Communications Commission announced the maximal penalty of 2 million dollars for broadcasters harming public interests caused by violating fact checks, disrupting public order, or adversely affecting good social customs, according to the Radio and Television Act. The National Communications Commission also committed to taking positive action to strike disinformation in order to avoid information with specific objectives being expanded through communications to harm public interests, keep people’s trust in broadcast media, and protect the important value of broadcast media being the fourth estate [2].

Content farms, through advertising revenue, simply focus on news traffic and do not place stress on news content. Fake news from content farms, without being checked by news gatekeepers, presents far lower reliability than traditional news media. Fake news causes people to lose trust in news reports, so the media can no longer maintain their role and responsibility to provide truth and supervise the government. Through the hype and spread of media, fake news misleads the public, induces mass panic, and even affects social stability and national security. There are multiple and free news and public opinion outlets in Taiwan, but this could result in rumor spreading. Solving the problem of fake news becomes an issue requiring a great deal of attention. For this reason, news source credibility recognition is preceded by artificial intelligence in this study, which is expected to assist the audience, through technology, in enhancing the literacy of news sources and reducing the harmful effects of fake news.

1.2. Research Objective and Problem

Fake news refers to news articles that are purposively distorted and could be proven to be false [3]. The Reuters Institute for the Study of Journalism indicated in their report on a survey that there was no obvious boundary between fake news and true news, but rather the difference exists in terms of the degree of truth [4]. In the fiercely competitive media environment, there are many methods to address fake news. In addition to the request for media self-discipline, Audrey Tang, the Minister without Portfolio of the Executive Yuan, suggests including media literacy education in the formal curricula of schools to teach students to recognize fake news from elementary and high school levels. The effect of fake news could be fundamentally reduced simply by the cultivation of pupils’ media literacy.

To effectively cope with fake news, the first fact-check non-profit organization, Taiwan FactCheck Center, was collaboratively established by Taiwan Media Watch and The Association for Quality Journalism in July 2018, claiming to assist people in identifying the truthfulness of news from the standpoint of a fair third party. Aimed at the validation of fake news, the Taiwan FactCheck Center provided people with simple methods, including the confirmation of news sources, the provision of correct websites, the use of Google search to confirm pictures used in news content were altered or had a timing error, and the checking of news content with the rumor check application offered by Line. The fact check organization could effectively assist people in recognizing fake news. However, aimed at the validation of fake news, the fact check organization must rely on a large amount of manpower and consume a great deal of time for users’ manual reports of news or information, manual review, or the active detection and collection of disinformation through the organization. It could no longer cope with the currently rapid and tremendous amount of fake information.

The Taiwan FactCheck Center announced a fact check report in 2019 and drew attention to the report in a content farm that “sweet potato steamed with rice wine” was able to cure long-term soreness of lumbar strains and Ankylosing Spondylitis. The center interviewed Li Yu-Shan, the dietitian of Shin Kong Wu Huo-Shih Memorial Hospital, who reported that both rice wine and sweet potatoes could easily result in inflammation, which might worsen Ankylosing Spondylitis. Lin Jing-Chi, the attending physician in the Division of Rheumatology, Chang-Geng Medical Foundation Chiayi Chang-Geng Memorial Hospital, indicated that the factors in a lumbar strain and Ankylosing Spondylitis were completely different. Lumbar strain refers to a muscle sprain, which would be recovered by rest, while a non-steroidal anti-inflammatory drug (NSAID) and an immunomodulator would be clinically used for Ankylosing Spondylitis or a biological agent would be used for serious Ankylosing Spondylitis, which could not be cured simply by diet. Content farms spread fake news through various forms. People with a lack of media literacy mistakenly believing in false therapy might delay treatment and even endanger their health.

Fake news permeates life through various channels and misleads people to believe disinformation. There are currently many fake news recognition methods; however, in comparison with cultivating people’s media literacy and verifying news through fact-checking organizations, artificial intelligence could more effectively recognize news source credibility. For this reason, artificial intelligence is utilized in this study to recognize news source credibility in order to provide people with multiple and effective news credibility channels to reduce the harm caused by fake news. A word-frequency-based ANN system and a semantics-based BERT system are used by the models to judge news source credibility through deep learning, and the difference in the recognition of news source credibility between the ANN and BERT systems is compared. The studied problems are proposed below:

(1)

To discuss the recognition rate of the word-frequency-based ANN system on news source credibility.

(2)

To discuss the recognition rate of the semantics-based BERT system on news source credibility.

(3)

To compare the difference in the recognition rate of news source credibility between the word-frequency-based ANN system and the semantics-based BERT system.

2. Literacy Review

2.1. Fake News

To trace the origin of the term “fake news”, Leetaru [5], through the Google tool, found the appearance of fake news in a book two centuries ago, as well as two peaks in history during World War I and II, when political warfare among the belligerents encouraged national morale and struck the people of other countries through imaginary and self-serving news; nevertheless, relevant discussions over fake news reached their peak after the US election in 2016 [6].

There is not a consistent definition of fake news [7]. For example, the intellectual property of the United Nations Educational, Scientific, and Cultural Organization (UNESCO) reports on “Journalism, ‘Fake News’ & Disinformation” [8] discussed the terms “Fake News” and “Disinformation”. The Council of Europe report on “Information Disorder: Toward an interdisciplinary framework for research and policy making” [9] defined “INFORMATION DISORDER” based on three categories: Mis-information, dis-information, and mal-information. Guess, Nagler, and Tucker [10] regarded fake news as wrongly or intentionally misleading others through real article-like news. From the perspective of enterprise, the dissemination of fake news is aimed at advertising revenue. By synthesizing various researchers’ definitions of fake news, Leeder [11] pointed out the relationship of fake news with two types of incorrect information, namely inappropriate information and false information. Both were wrong information, but false information was intentionally disseminated or misled. In this case, fake news could be problematic when incorrect information is made to look like real news to intentionally deceive or mislead people. Cooke [12] indicated that inappropriate information was incomplete, ambiguous, and uncertain information, but might possibly be correct, which required judgment using the background context; false information, on the other hand, was wrong information, which might be intentionally disseminated with specific planning.

Fake news did not present specific patterns. Hunt [13] pointed out several basic patterns of fake news, including purely fictitious information, merely providing information with one-sided facts, true news with low quality, manipulative news with political intention, and reports presented as news with advertising and promotion motivations. Fake news misled readers with headlines, which could easily attract readers’ attention with partially or completely fake information content, for business profits or political objectives. Reuters Institute for the Study of Journalism applied a focus group approach in order to understand fake news in people’s minds, in which five fake news styles were presented, including (1) sarcasm, (2) lack of news value, (3) political manipulation, (4) advertising news, and (5) manipulated news [14].

Since the US election in 2016, fake news has been an issue emphasized by people from all walks of life. The website of Harvard Library attracted readers’ attention through the design of a visual infographic, showcasing five methods to interpret and stop fake news. The five interpretations included the following: (1) Pay attention to the data source, particularly being vigilant to information from strange domains; (2) check the URL of the news source: URLs of fake news websites were frequently designed as true, e.g., “.com.co”; (3) search clues from the appearance: Fake news websites are generally rough with unprofessional design and abuse capital letters; (4) seek other opinions by referring to other news data or rumor refutation websites for irritating and pondering news; and (5) utilize the function of browsers: Browsers can install plugins to block websites that have been exposed as fake news [15].

The International Federation of Library Associations (IFLA) also presented a data map on the subject of how to judge fake news, where eight principles are covered: (1) Consider the news source in order to analyze the goal and content matter of the news website; (2) check the author, to rapidly confirm the reliability and reality of an article’s authors; (3) check the date, as re-announced old news is not necessarily related to current events; (4) self-reflection, in order to reflect whether personal slants affect judgment; (5) see the intention beyond the text, which is commonly to purposively attract clicks with sensational headlines; (6) reference data by reading whether the evidence data for the news really support the news argument; (7) sarcastic speech: Extraordinary speeches might simply be sarcastic speeches, which should be reviewed by the website and confirmed by the author; and (8) ask experts, by inquiring to librarians or referring to rumor-refuting websites [16].

The flooding of fake news in Taiwan resulted in the emergence of fact-checking websites in recent years. “Taiwan FactCheck Center” co-organized by Taiwan Media Watch and The Association for Quality Journalism, exposes a great deal of fake news confirmed after careful validation. Such verified information is evaluated with four major indicators, namely “correct”, “wrong”, “partially wrong”, and “fact clarification”. Nonetheless, along with gradually complicated network information, some network information content without chapter and verse or lack of definite data could not be classified into any of the above indicators. The Taiwan FactCheck Center therefore added the fifth indicator of “evident insufficient” by referring to the measures of international authoritative fact check organizations to provide people with more accurate fact check indicators.

Fake news is presented in various forms, where content farms, in order to create traffic and make profits from online advertising, largely produce articles with various legal and illegal tactics; the article content is short of originality and the authenticity can seldom be confirmed. Since content farms do not actively manage the content, many articles are infringed and misappropriated, copied, and rewritten and mixed with excessive content and misinformation, becoming a hotbed for fostering fake news [17].

Concerning the prevention of fake news, Batchelor [18] regarded information literacy as the most practicable and effective method to strike fake news. Eva and Shea [19] stressed the potential and importance of critical thinking skills in fighting fake news. Andretta [20] indicated that the changeable digital environment results in individuals being exposed to large amounts of unfiltered and various types of information, making information validation more difficult, while the cultivation of information literacy-related criticism and analysis skills could help audiences in identifying the information source and combatting fake news. Nevertheless, while the promotion of the audience’s information literacy is the basis of coping with fake news, it cannot reinforce the audience’s judgment and literacy of news sources rapidly. Accordingly, a word-frequency-based ANN system and a semantics-based BERT system, through artificial intelligence, could assist audiences in recognizing fake news and effectively enhance audiences’ interpretation of news sources.

2.2. Artificial Neural Network (ANN)

Inspired by research on neurons in biology, a model imitating the operation of neurons in the brain was created, called the Artificial Neural Network (ANN), also named the Neural Network. It is the classification of supervised learning in the fields of machine learning and deep learning. The Artificial Neural Network basically consists of an input layer, a hidden layer, and an output layer. Similar to the operation of the biological neural system, when data are transmitted to the neural network, the input layer would largely receive nonlinear information, called neurons; each neuron has the activation function of judging the delivery of data to the next-layer neuron. The hidden layer mainly receives and calculates the weight and deviation value and transmits the results to the output layer. The output layer conducts the analysis, calculation, and weighing according to training rules to eventually form the output result.

In order to present human knowledge and behavior, the Artificial Neural Network would provide machines with decision-making and judgment abilities through large amounts of calculation and learning to find the best solution in any problem space. Backpropagation (BP) is the most common algorithm in the training of Artificial Neural Networks; the idea is to repeatedly calculate the result of each neuron and use the calculated deviation as the reference for adjusting the weight to achieve more accurate performance. As backpropagation requires a longer training time, Hinton et al. [21] proposed the Deep Belief Network (DBN) to solve the problem. The shortened training time and enhancement of technology and hardware of the Artificial Neural Network allowed the emergence of the Deep Neural Network (DNN) to break through the dilemma in past neural network layers and match various neural network layers of the Deep Neural Network, which was also called deep learning.

Based on the development of the Artificial Neural Network, the application of deep learning is gradually developed for machine learning. There are several types of deep learning structures presently, such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Long-Short Term Memory (LSTM) extended from Recurrent Neural Networks. Deep learning is broadly applied to various fields, such as in the field of computer vision [22] to deal with image classification, image recognition, and voice and music recognition [23]. The application to Nature Language Processing has been attempted in recent years [24,25,26] for text classification and prediction.

CNN and RNN are the most common neural network structures in text classification, and LSTM is the extension of RNN. CNN generally contains a series of convolutional layers and pooling layers as the hidden layer. CNN, in the text classification task, transforms various words in a sentence into the corresponding embedding to obtain the same vectors and form a matrix, or a 2D matrix that deals with matrices in the same way it deals with images, to build a model with CNN. CNN allows for simultaneously viewing a set of words with the context window to comprehend the sentence through word viewing and the combination of various consecutive words [27].

RNN is a time-sequence-related neural network and could precede the modeling and analysis of sequence data. In text data, language presents continuity; the sequence of each word would affect the semantics, meaning RNN is particularly suitable for dealing with text classification, as well as generating texts to predict the next word or character by reading previous context [28].

Although RNN presents a good ability and commonality, it cannot remember and address lengthy context. Hochreiter and Schmidhuber [29] therefore proposed Long-Short Term Memory. LSTM is a special time-recursive neural network suitable for predicting a time-sequential data model. In comparison with RNN, LSTM has a cell state to memorize long-term information and presents better performance with longer sequences. In order to deal with the lengthy context in text classification, LSTM can ignore irrelevant context and simply deal with the context required for the task to reduce the burden of using a vector expression to remember lengthy context [30]. In this case, it shows better general application, e.g., learning word translation, word generation, documentation summary, voice recognition, image analysis, image recognition, handwriting recognition, controlling Chatbots, predicting diseases, and predicting stocks.

The newest applications of ANN include tweet sentiment analysis with NLP [31,32], food safety [33], and other fields. Sitaula et al. [31] proposed three different feature extraction methods: FastText-based (ft), domain-specific (ds), and domain-agnostic (da), for the Nepali COVID-19-Related tweets. They also proposed three different convolution neural networks (CNNs) to implement the proposed features, and their dataset can be used as a benchmark to study the COVID-19-related sentiment analysis in the Nepali language. Shahi et al. [32] used two text representation methods, TF-IDF and FastText, and then combined them to achieve hybrid features to capture highly discriminating features. They also implemented nine widely used machine learning classifiers (Logistic Regression, Support Vector Machine, Naive Bayes, K-Nearest Neighbor, Decision Trees, Random Forest, Extreme Tree classifier, AdaBoost, and Multilayer Perceptron) based on the three feature representation methods: TF-IDF, FastText, and Hybrid. Gorbachev et al. [33] used an ANN with the structure of a “multilayer perceptron” (MLP) with the hyperbolic tangent (Tanh) as an activation function, and the values of the antiradical potential of 1315 items of food and agricultural raw materials were calculated. Based on the results of the study, it was found that the predicted antiradical potential (ARP) values depend not only on the type of raw materials and their method of processing, but also on a number of other environmental and technological factors that make it difficult to obtain accurate values.

2.3. Bidirectional Encoder Representations from Transformers (BERT)

BERT (Bidirectional Encoder Representations from Transformers), the deep learning model with Nature Language Processing proposed by Google in 2018, proved the importance of bidirectional pre-training in language models and largely reduced the training time for model use. BERT solves the one-way training of current language training models, e.g., ELMo and OpenAI GPT. BERT is also applied to different tasks, such as downstream tasks of sentence pairs in paraphrasing, hypothesis–premise pairs in entailment, SQuAD (Question–Passage Pairs in Question Answering), and a degenerate text–∅ pair in text classification or sequence tagging [34].

BERT stresses pre-training and fine-tuning; during pre-training, different pre-training tasks would precede model training of unlabeled data [34]. The training data of BERT are BooksCorpus (800 M words) and English Wikipedia (2500 M words). First, the input representation of text is preceded by Wordpiece Embedding to mark the first label of each sequence as [CLS], and the special symbol [SEP] is then used for separating two sentences and transforming them into token embedding, segment embedding, and position embedding for the successive pre-training task.

Two unsupervised tasks, Masked LM (MLM) and Next Sentence Prediction (NSP), are preceded by pre-training in Google. In comparison with Word2Vec and ELMo, BERT, with its deep bidirectional model, strengthens the function. In order to train the deep bi-direction, BERT inputs Tokens with random cover for the model, predicting such covered Tokens; such a process is termed Masked LM [34]. Moreover, BERT precedes Next Sentence Prediction through the corpus, allowing the model to comprehend sentence relationships in order to judge two sentences [35].

BERT could extract features through Transformer, which is composed of an encoder and decoder. The encoder enters the decoder through a multi-head attention mechanism and a fully connected feed-forward network. With the multi-head attention mechanism, fully connected feed-forward network, and pure fully connected layer (Linear), the Softmax function is eventually transformed into a probability value for the output. BERT, with MLM, NSP, and the encoder mechanism of Transformer, first transforms the token embedding of input text into three matrices of Q (Query), K (Key), and V (Value), precedes the dot product and computation of matrices through the multi-head attention mechanism, and finally, inputs the token embedding. Each Q, K, and V is a set of Heads, and the number of Heads could be set in the multi-head attention mechanism so that the model could simultaneously consider the vector of each word during training, and precede training in parallel, to better understand the tightness between words.

After completing the pre-training process, the model that could be applied to multiple Nature Language Processing tasks is first trained using the Encoder of Transformer, a large number of texts, and two pre-training tasks, employing the idea of transfer learning and then fine-tuning several downstream tasks. The self-attention mechanism in Transformer allows BERT to precede different downstream tasks, either single text or paired texts [34].

BERT, through the pre-training model, allows the model to better understand vocabulary and keywords and can more accurately correspond sentences to integrate the semantic information of the entire article. In terms of the application, BERT is currently applied to text classification [36], text abstracting [37], sequence labeling [38], Q&A [39], task searching [40], and dialogue robots [41]. The application of BERT is broadened through the pre-training process of Transformer.

The latest applications of BERT include text sentiment analysis [42], the protection of private health information [43], recommender systems [44], and other fields. Li et al. [42] proposed a comparative experiment on the Weibo text dataset collected during the COVID-19 epidemic, and their results showed that the performance of the proposed model was significantly improved compared with other similar models. Kang et al. [43] propose a filtered BERT-based method that predicts a masked word and validates the word again through a similarity filter to construct augmented sentences. Their results show that the augmentation method effectively improved the performance of the model in a limited data environment. Zhuang and Kim [44] proposed a customer recommender system that can quickly and effectively identify potential target customers for a hotel operator. Their proposed recommender system uses a fine-tuned BERT model to predict six criteria (value, service, location, room, cleanliness, and sleep quality) ratings on the TripAdvisor site

3. Research Method

The definition of fake news is difficult. From the viewpoint of information, Ireton and Posetti [45] considered that the information in fake news content could be divided into disinformation, misinformation, and mal-information. According to Allcott and Gentzkow’s [3] explanation of fake news, it is defined in this study as “purposively simulating the form similar to news to deliver false and wrong information”. Fake news in content farms is mostly infringed and misappropriated, copied, and re-written, resulting in the content lacking originality and the authenticity being unverifiable. For this reason, fake news in content farms is discussed in this study.

Fake news aims to mislead the public to acquire political or economic benefits, and fake news can be proven false. Nevertheless, there is no public or scaled traditional Chinese fake news database or dataset; manually checking false news cannot reach a sufficient scale for the training program. Linguistic methods, i.e., NLP text analysis and DNN deep learning, are therefore applied to this study, and content farms are used as a fake news dataset while mainstream news media, which is gatekept and checked by mainstream news, is used as a general news dataset, in order to develop a news source credibility system available for users’ evaluation.

3.1. Dataset Selection

This study concerns unproven news content on the Internet. By using unproven news on online news platforms and content farms as the data source, AI technology is applied to recognize the news credibility. The collection coverage and definition of datasets are explained below.

Regarding general news, four major newspapers, namely “United Daily News”, “Liberty Times”, “Apple Daily”, and “China Times”, in Taiwan constitute the data collection coverage as they have established good media reputations and complete editing systems.

Regarding fake news, four famous fake news content farms, including nooho.net, mission-tw.com, BuzzHand (buzzhand.com), and kknews.cc, constitute the data collection coverage. Content farms are selected as the fake news database sources, mainly because most content farms aim to create traffic and earn online advertising profits and largely produce articles with various legal or illegal tactics, and the content lacks originality and the authenticity is difficult to confirm. Since content farms do not actively manage the content, many articles uploaded by users might involve infringement and misappropriation and are even mixed with excessive content and false information. As a result, the accuracy and creditability of news content cannot be instantly ensured due to the lack of editing regulations and processes of news media.

Regarding the database schema field, headlines and news content are considered essential fields, while the author, source (including the name of the news website and the link), news release date, and news classification are not necessary.

A total of 1,422,632 pieces of news were collected from 1 January 2019 to 31 May 2021, and the relevant website contents are shown in

Table 1. News data collection.

Mainstream News	N	Content Farm News	N
Apple Daily	93,715	mission-tw.com	12,960
China Times	144,107	nooho.net	1839
Liberty Times	79,078	kknews.cc	42,577
United Daily News	610,572	buzzhand.com	719
ETtoday	340,804	qiqi.news	32,916
Central News Agency	57,309	Global Military	6036
Total	1,325,585	Total	97,047

.

3.2. System Design and Research Process

3.2.1. ANN

In the system design, Python and TensorFlow, combined with machine learning and deep learning algorithms, were used in this study for training and testing. Through the text analysis of Nature Language Processing (NLP), including word participles and parts of speech, chunks, and segmentation, DNN deep learning was combined for the multi-dimensional and multi-aspect analysis of data to design the system model for evaluating news credibility.

Regarding data preprocessing, the URLs of general online news and news in content farms were first established, and the database and fields required for data sheets were confirmed. Using Crawler to catch information labels and writing in the data sheet, a fixed time and frequency were set to regularly execute Crawler and save the required raw data into the database.

In regard to the exploratory data analysis, the steps and procedures were as follows:

(1)

Data cleaning: Capture required fields from the database and check the null field in each record; if present, the record is excluded. Export the selected data with the CSV format of UTF-8 BOM as the raw data for the program.

(2)

Feature extraction: Regard the article title and article text in the source file as the string, apply CKIP Word Segmentation and Named Entity Recognition developed by CKIP Chinese Lexical Knowledge Base for word segmentation and analysis, execute part-of-speech tagging (POS tagging) and name entity recognition (NER), and term frequency (TF) after each word segmentation is calculated as the feature parameter of the training model.

(3)

Modeling: Apply the Keras Sequential model (a linear stack model with multiple network layers) and designate the size parameters of the input layer, hidden layer, and output layer for the compilation model. The parameter details used for the model.add function are shown below.

Input layer: Characteristic dimension dim (130), nodes units (12), and activation function activation (ReLU).

Hidden layer: Nodes units (12), number of layers (10), and activation function activation (ReLU).

Output layer: Nodes units (1) and activation function activation (sigmoid).

(4)

Compilation model: The parameter details used for the model.compile function are shown below.

loss function binary_crossentropy, optimal algorithm RMSprop, learning rate (0.0005), and measurement index metrics (accuracy).

(5)

Training and validation model: The parameters used for the model.fit function are shown below.

batch size batch_size (250), number of training rounds epochs (200), and validation set splite ratio validation_split (0.2).

(6)

Data visualization: The data generated after completing the model training are presented as a graph with matplotlib and saved in the Result folder, as shown in Figure 1.

In the model accuracy distribution, acc denotes the correct rate of the training set and val_acc stands for the correct rate of the testing set. After 200 training runs, the highest correct rate in the experimental result achieves 82.75%.

In the model loss distribution, loss stands for the loss rate of the training set and val_loss represents the loss rate of the testing set. After 200 runs of training, the convergence approaches stability.

3.2.2. BERT

Different from traditional ANN, BERT is a representative model able to comprehend the context, which is a language representative model of Google with an unsupervised method, refined with a large number of unlabeled texts, and the structure is Encoder in Transformer.

Training in BERT contains two steps, namely pre-training and fine-tuning. In the pre-training stage, a large amount of text data is preceded by the training model with unsupervised learning by Google. In the fine-tuning stage, the model is fine-tuned with labeled data training, aiming at distinct tasks.

BERT deals with input news content as a series of sentences with punctuation. By inputting the [CLS] embedding sequence into BERT, another embedding sequence is output; the required parts are captured from the sequence and connected to a classifier that simply fine-tunes BERT and trains the classifier for training.

In terms of exploratory data analysis, the steps and procedures required for this study are described as follows:

(1)

Data preprocessing: Load tensorflow, tensorflow_dataset, transformers, and BertTokenizer from the library and apply vocab.txt corpus offered by Google, including encode_words and bert_encode, to transform the content word group in the training set into serial form and further transform it into a tensor.

(2)

Modeling: The applied model bert_model_chinese_wwm_ext is classified with TBbertForSequenceClassification, and bert_config.json is loaded and the checkpoint is set.

(3)

Compilation model: The used parameter details are shown below.

loss function SparseCategoricalCrossentropy, optimal algorithm Adam, learning rate (0.000001), measurement index SparseCategoricalAccuracy.

(4)

Training and validation model: The used parameter details contain the batch size batch_size (5), number of training rounds epochs (50), and validation set split ratio validation_split (0.1).

(5)

Data visualization: The data generated after completing model training are drawn as a curve with matplotlib and saved as an image in Results, as shown in Figure 2.

In the model accuracy distribution map, train_acc denotes the correct rate of the training set and test_val_acc is the correct rate of the test set. After 50 training runs, the highest correct rate in the experimental result is 91.2%.

In the model loss distribution map, train_loss represents the loss rate of the training set and test_val_loss is the loss rate of the test set. After 50 training runs, the convergence approaches stability.

4. Results

4.1. ANN

There is a total of 10,000 datasets in the ANN system, consisting of 8000 training sets and 2000 test sets. Among the 8000 training sets, true news and fake news constitute 4000 pieces each, and in the 2000 test sets, true news and fake news constitute 1000 pieces each. The modeling core parameters are dim (130), node (12), lr (0.0005), and hidden_layer (10); after 200 runs of training, the highest correct rate in the experimental results is 82.75%.

The trained model is saved, and test data with different data to the training set are collected, including 200 pieces of news data in the database and 200 pieces of news data beyond the database, which are further fed into the ANN training model; the system hit rate is 84.75%. The experimental results are shown in Table 1.

The ANN system is further applied to news credibility prediction in mainstream news platforms. The data sources of mainstream news platforms are from six mainstream news media outlets, with 200 pieces of data each for a total of 1200 pieces. The model training results reveal that the ANN system shows the following hit rates: Apple Daily, 82.0%; China Times, 71.0%; Liberty Times, 96.5%; United Daily News, 93.0%; ETtoday, 66.0%; and CNA, 88.5%. The training results reveal that the ANN system presents the highest hit rate for Liberty Times followed by United Daily News, and the lowest hit rate for ETtoday.

In terms of the news credibility prediction of content farms, the data sources of content farms are from six content farms, with 200 pieces of data each for a total of 1200 pieces. The model training results with the ANN system show the following hit rates: Mission-tw.com, 85.0%; nooho.net, 65.5%; kknews.cc, 87.0%; BuzzHand, 98.0%; qiqi.news, 86.0%; and Global Military, 87.0%. The training results reveal that the ANN system identified the highest hit rate for BuzzHand, followed by kknews.cc and Global Military, and the lowest hit rate for nooho.net (as shown in

Table 2. The result of the ANN method for mainstream news and content farm news.

Mainstream News
Test data source	Apple Daily	China Times	Liberty Times	United Daily News	ETtoday	Central News Agency
N	200	200	200	200	200	200
Hit rate	82.0%	71.0%	96.5%	93.0%	66.0%	88.5%
CI (95%)	76.57~87.43%	64.58~77.42%	93.90~99.10%	89.39~96.61%	59.30~72.70%	83.99~93.01%
Error	5.43%	6.42%	2.60%	3.61%	6.70%	4.51%
Content Farm News
Test data source	mission-tw.com	nooho.net	kknews.cc	buzzhand.com	qiqi.news	Global Military
N	200	200	200	200	200	200
Hit rate	85.0%	65.5%	87.0%	98.0%	86.0%	87.0%
CI (95%)	79.95~90.05%	58.78~72.22%	82.24~91.76%	96.02~99.18%	81.09~90.91%	82.24~91.76%
Error	5.05%	6.72%	4.76%	1.98%	4.91%	4.76%

).

4.2. BERT

The same 10,000 datasets are used as the training set, with the division ratio of 8:2 for the training set and the validation set, i.e., 8000 pieces for the training set and 2000 pieces for the validation set. The applied model bert_model_chinese_wwm_ext is classified with TBbertForSequenceClassification, and the core parameters contain 512 characters taken for each word processing, Optimizer (Adam), and learning rate lr (1 × 10⁻⁶); after 50 runs of training, the correct rate is up to 91.2%.

The trained models are saved, and test data, with different data to the training set, are collected, including 200 pieces of news data in the database and 200 pieces of news data beyond the database, which are further fed into the BERT training model; the system hit rate is 91.13%. The experimental results are shown in Table 2.

The BERT system is further applied to the news credibility prediction of mainstream news platforms; the data sources are from six mainstream news media, with 200 pieces of data each for a total of 1200 pieces. The model training results with the BERT system show the following hit rates: Apple Daily, 90.5%; China Times, 90.5%; Liberty Times, 100%; United Daily News, 96.5%; ETtoday, 89.0%; and CNA, 100%. In the training results, the BERT system presents the highest hit rate for Liberty Times followed by United Daily News, and the lowest hit rate for Ettoday.

Regarding the news credibility prediction of content farms, the data sources are six content farms, with 200 pieces of data each for a total of 1200 pieces. The model training results with the BERT system show the following hit rates: mission-tw.com, 98.5%; nooho.net, 79.0%; kknews.cc, 85.0%; BuzzHand, 89.0%; qiqi.news, 83.0%; and Global Military, 88.5%. The training results with the BERT system reveal the highest hit rate for mission-tw.com, followed by BuzzHand, and the lowest hit rate for nooho.net (as shown in

Table 3. The result of the BERT method for mainstream news and content farm news.

Mainstream News
Test data source	Apple Daily	China Times	Liberty Times	United Daily News	ETtoday	Central News Agency
N	200	200	200	200	200	200
Hit rate	90.5%	90.5%	100.0%	96.5%	89.0%	100.0%
CI (95%)	86.35~94.65%	86.35~94.65%	100.0%	93.90~99.10%	84.58~93.42%	100.0%
Error	4.15%	4.15%	0%	2.60%	4.42%	0%
Content Farm News
Test data source	mission-tw.com	nooho.net	kknews.cc	buzzhand.com	qiqi.news	Global Military
N	200	200	200	200	200	200
Hit rate	98.5%	79.0%	85.0%	89.0%	83.0%	88.5%
CI (95%)	100~96.78%	73.24~84.76%	79.95~90.05%	84.58~93.42%	77.69~88.31%	93.01~83.99%
Error	1.72%	5.76%	5.05%	4.42%	5.31%	4.51%

).

5. Discussion and Conclusions

This study aims to recognize news source credibility through artificial intelligence and provide people with multiple and effective news credibility channels to reduce the harm caused by fake news. A word-frequency-based ANN system and a semantics-based BERT system are utilized in this study to judge the source credibility of news through deep learning, and the differences in news source credibility recognition between the ANN and BERT systems are compared.

5.1. ANN Prediction Results

(1)

The ANN system presents a better source credibility recognition rate for mainstream news than content farms.

The experimental results with the ANN system reveal the true news hit rates of six mainstream news platforms, as follows: Apple Daily, 82.0%; China Times, 71.0%; Liberty Times, 96.5%; United Daily News, 93.0%; ETtoday, 66.0%; and CNA, 88.5%. The highest hit rate is 96.5% and the lowest hit rate is 66.0%. The fake news hit rate of six content farms shows mission-tw.com with 85.0%, nooho.net with 65.5%, kknews.cc with 87.0%, BuzzHand with 98.0%, qiqi.news with 86.0%, and Global Military with 87.0%, with the highest hit rate of 98.0% and the lowest hit rate of 65.5%.

To further analyze the above data, it is discovered that the ANN system presents better true news hit rates for mainstream news platforms than the fake news hit rates for content farms in the training set. This is possibly because the ANN system uses word-frequency-based recognition, which calculates the frequency of words in an article, and headlines and content on mainstream news platforms follow a fixed news writing format, the word expression is kept at a certain level, and the words used are careful with less tautology and pleonasm. In this case, the ANN system is more suitable for the prediction of mainstream news platforms than for content farms.

(2)

The ANN system found the lowest source credibility recognition rate for nooho.net.

Among all experimental results, prediction with the ANN system resulted in a mere 65.5% hit rate, the lowest source credibility recognition rate, for nooho.net. This may be due to the few training news parameter data of nooho.net, and the extremely low quantity of daily news results in a larger prediction amplitude. It is therefore not suitable for data analysis with this system and might affect the news source credibility prediction of the word-frequency-based ANN system.

5.2. BERT Prediction Results

(1)

The experimental results of the BERT system are almost identical to the test results of the saved model.

The experimental results with the BERT system show the highest correct rate of 91.2%, with the saved BERT number bert_30714. Two hundred pieces of news data in the non-training set collected from various websites are additionally collected to be fed into the BERT training model, and the average hit rate of 91.13% is almost the same as the experimental results. This is possibly because the semantics-based BERT system uses a bidirectional transformer to predict the hidden Token and predicts the downstream tasks for the model, learning the relationship between sentences, to result in almost the same experimental results as the test result of the saved model.

(2)

The BERT system has a better true news hit rate for mainstream news platforms than the fake news hit rate for content farms.

The experimental results with the BERT system reveal the true news hit rate of six mainstream news platforms, as follows: Apple Daily, 90.5%; China Times, 90.5%; Liberty Times, 100%; United Daily News, 96.5%; ETtoday, 89.0%; and CNA, 100%, where the highest hit rate is 100% and the lowest hit rate is 89.0%. The fake news hit rate of six content farms shows mission-tw.com with 98.5%, nooho.net with 79.0%, kknews.cc with 85.0%, BuzzHand with 89.0%, qiqi.news with 83.0%, and Global Military with 88.5%, where the highest hit rate is 98.5% and the lowest hit rate is 79.0%.

To further analyze the above data, it is discovered that the BERT system presents a better true news hit rate of mainstream news platforms than the fake news hit rate of content farms. It may be because the BERT system is semantics-based, and the news content expression of true news authors on mainstream news platforms obviously outperforms authors of content farms regarding semantics.

5.3. Comparison of System Prediction Results

To classify the data sources of mainstream news and content farm outlets and compare the experimental results of the ANN system and the BERT system, the following phenomena are discovered:

(1)

The ANN and BERT systems show high similarity in their hit rates.

In regard to the prediction of mainstream news source credibility, it is discovered that both the ANN system and the BERT system present the best hit rate for Liberty Times and the worst hit rate for ETtoday. Similarly, both the ANN system and the BERT system show the worst hit rate for nooho.net in the prediction of content farm source credibility.

(2)

In comparison with the ANN system, the BERT system presents a higher and more stable source credibility recognition rate.

The experimental results with ANN and BERT systems reveal that the ANN system appears to have the highest hit rate of 98.0% and the lowest hit rate of 65.5% across all data source predictions, with a system hit rate of 84.75%. The BERT system, on the other hand, presents the highest hit rate of 100% and the lowest hit rate of 79.0% across all data source predictions, with the system hit rate of 91.13%.

Further data analysis shows that the BERT system, in comparison to the ANN system, appears to have a higher hit rate across all data sources, and the hit rate of various platforms is relatively stable. This may be because the ANN system has a different data-processing mechanism than the BERT system; the ANN system is based on word frequency, while the BERT system is based on semantics, which could more easily learn the common composition among characters. In this case, the BERT system could maintain a higher source credibility recognition rate with relatively stable standards in the prediction process.

Summarizing the above data, it could be reasonably inferred that the semantics-based BERT system outperforms the word frequency-based ANN system on the effectiveness of source credibility prediction of mainstream news or content farms. Nevertheless, the existence of the news source credibility recognition system, either the ANN or BERT system, is an information alert for people. Using artificial intelligence to recognize news source credibility could effectively enhance people’s sensitivity to news sources, as well as cultivate people’s media literacy in the long term to resist fake news using technology.