skip to main content
research-article
Open access

User Opinion-Focused Abstractive Summarization Using Explainable Artificial Intelligence

Published: 13 December 2024 Publication History

Abstract

Recent methodologies have achieved good performance in objectively summarizing important information from fact-based datasets such as Extreme Summarization and CNN Daily Mail. These methodologies involve abstractive summarization, extracting the core content from an input text and transforming it into natural sentences. Unlike fact-based documents, opinion-based documents require a thorough analysis of sentiment and understanding of the writer’s intention. However, existing models do not explicitly consider these factors. Therefore, in this study, we propose a novel text summarization model that is specifically designed for opinion-based documents. Specifically, we identify the sentiment distribution of the entire document and train the summarization model to focus on major opinions that conform to the intended message while randomly masking minor opinions. Experimental results show that the proposed model outperforms existing summarization models in summarizing opinion-based documents, effectively capturing and highlighting the main opinions in the generated abstractive summaries.

1 Introduction

Text summarization involves transforming a source text into easily understandable and valuable information. Crafting a summary that effectively reflects the original text while remaining concise and comprehensible is a crucial task. Text summarization is generally divided into two methodologies: extractive and abstractive summarization. Extractive summarization [21] involves scoring sentences for importance and assembling a summary based on these scores. However, this method is limited in expression and may compromise the flow of context, as it relies solely on the existing text in the source document.
In contrast, abstractive summarization [20], a natural language generation methodology, generates new text rather than extracting and combining sentences. Abstractive summarization can produce summaries with a variety of words, including those that are not present in the original document, providing a more diverse and expressive summary compared to extractive summarization. Therefore, this study, aiming to achieve more refined and natural summaries, focuses on abstractive summarization.
Most existing abstractive summarization models primarily focus on objectively summarizing lengthy documents such as news articles (Extreme Summarization (XSUM) [22], CNN Daily Mail (CNN/DM) [13]). Indeed, language models such as bidirectional and auto-regressive transformer (BART) [16], T5 [26], and Pegasus [35], which effectively comprehend and summarize the key information in documents, have demonstrated high performance in fact-based document summarization, as illustrated in Table 1.
Table 1.
ArticleBUPA is an international health and care company with bases on three continents and more than eight million customers.BUPA began as The British United Provident Association in 1947 to preserve freedom of choice in health care.It believed that with a National Health Service being introduced a year later,there would still be a need for a complimentary service enabling people from all walks of life to afford the benefits of choice in where, when and by whom they were treated. Led by the growing public demand for health care and a lack of quality private accommodation BUPA initiated the Nuffield Homes Charitable Trust - later renamed Nuffield Hospitals….
SummaryBUPA was founded in 1947 in response to plans to establish the NHS. The company’s biggest base is in the UK but has customers in three continents. BUPA care homes cater for a number of conditions, including Parkinson’s
Table 1. Example of Summarization of Fact-Based Document (CNNDM Dataset)
Unlike fact-based documents where objective summarization is crucial, opinion-based documents require additional considerations for effective summarization [36]. Opinion-based documents exhibit the author’s writing intent, incorporating the essential opinion sentiment the author aims to express and opinion sentiments unrelated to the main intent. For instance, in a review that includes negative sentiments about the price alongside positive opinions, the negative aspect is not the primary focus but rather contributes to the review’s credibility, emphasizing the overall positive sentiment intended by the author.
The price may be a bit expensive, but the food is incredibly delicious, the atmosphere is great, and the staff are also very friendly.
Illustrating this with a review text as an example, although there is a negative opinion sentiment regarding the price, this serves as a mechanism to enhance the credibility of the writing rather than an overall condemnation of the restaurant being reviewed; the user’s intended message can be considered a positive opinion sentiment. This is a common expression method where the author mentions opposing sentiments to increase the credibility of the writing and emphasize the core opinion sentiment when expressing their opinion [14, 28, 31].
As confirmed by numerous studies, analyzing sentiment is a fundamental task in various natural language processing (NLP) studies. Understanding the sentiment conveyed through text is essential for businesses to gather customer insights or analyze public sentiment [12].
In essence, the process of generating a summary for opinion-based documents focuses on capturing the author’s writing intent by concentrating on the core opinion sentiment. The goal is to exclude opinion sentiments that are not significantly associated with the primary intent. Consequently, as depicted in Table 2, the correct summary (golden summary) of an opinion text includes only the essential writing intent, omitting opinion sentiments that are not closely related to the intent and incorporating content that corresponds to the core opinion sentiment.
Table 2.
ReviewI purchased this for my 6 year old Minecraft loving son. The directions are hard to follow and as an adult I had a hard time putting them together. After I built them they came apart easily and my son could not really play with them. This toy is better suited for an older child who is only interested in building things out of paper and not playing with the figures afterward. I ended up throwing it in the trash
Summaryflimsy and difficult to build.
SentimentNegative
Table 2. Example of Summarization of an Opinion-Based Document
Therefore, in this paper, we propose an advanced abstractive summarization model that explicitly considers opinion sentiment to efficiently perform text summarization for opinion-based documents. We specifically utilize a classification head attached to the encoder and the Shapley additive explanation (SHAP) explainer to derive the sentiment distribution of the original text [37]. Based on the sentiment distribution information, we extract the major opinion and minor opinion parts from the text. Finally, we incorporate random masking on the minor opinion part and integrate it into the encoder-decoder multi-attention operation during training, thereby emphasizing the major opinion sentiment [4].
We also propose a novel random masking method to ensure effective learning when handling minor opinions. Our method involves a different probabilistic masking technique that applies varying probabilistic masks to each learning process and attention head rather than a fixed mask. Furthermore, to facilitate the model’s ability to naturally emphasize major opinion parts, another novel incremental learning strategy is proposed. The proposed method, by incrementally learning to prioritize major opinion text, facilitates the creation of a model that, during inference, neglects minor opinion segments and concentrates solely on the major opinion, even in the absence of sentiment distribution data.
To the best of our knowledge, the proposed method is the first research explicitly considering both writing intent and sentiment distribution for summarizing opinion-based text in accordance with the writing intent. Additionally, our proposed methodology is an attachable modular approach, which can be seamlessly integrated into existing summarization models, showcasing notable advantages in efficiency and reusability. Consequently, we have validated the performance improvement of the summarization task for opinion-based documents when our proposed methodology is integrated into existing summarization models.
The remainder of this paper is structured as follows. Section 2 discusses various studies on abstractive summary models using pretrained language models (PLMs), and Section 3 outlines the proposed method. Section 4 describes the data used in the experiments and experimental results of our proposed method. Finally, Section 5 provides the concluding remarks and briefly discusses directions for future research.

2 Literature Review

2.1 PLM Based Abstractive Summarization Models

The most widely used abstractive summarization models are based on PLMs, such as bidirectional encoder representations from transformers (BERT) and generative pre-trained transformer (GPT). BERT [5, 30] is a model comprising the encoder part of the Transformer, which is pretrained on a large corpus to effectively learn the contextual representation of text. It can understand documents by learning bidirectional representations; however, it has limitations in its direct application to summarization models because it is structured with only an encoder. Meanwhile, GPT [3] is a PLM model that uses only the decoder part of the Transformer and exhibits strengths in various natural language tasks (e.g., text generation, question answering, and text classification). However, as an auto-regressive PLM model specialized in generation, it cannot learn bidirectional information and has limitations in understanding and summarizing text.
BART [16] is a PLM that combines the strengths of the two mentioned models and assumes the form of a seq2seq model by combining the encoder and decoder. It is pretrained by reconstructing text that has been corrupted by a noise function to reduce the loss between the decoder’s output and the original text. Consequently, BART employs a simultaneous approach of text infilling (replacing a text span with a single mask token) and sentence shuffling (randomly shuffling the order of sentences). This unique combination allows BART to utilize various noise types during training, surpassing the capabilities of existing auto-encoder models. Thus, various tasks, such as sequence classification, token classification, and machine translation, can be conducted through pre-learned BART, which shows excellent performance, particularly in abstractive summarization.
T5 [26] is a transformer-based PLM similar to BART. It reconstructs all NLP tasks into an integrated text-to-text format in which the inputs and outputs are always text strings, unlike BERT-style models, which can output only class labels or input ranges. It differs from the existing Transformer model in that relative position encoding is used for the input token, which is popular (the Transformer uses sinusoidal embedding and BERT uses position embedding). This means that learning occurs not through assigning a uniform encoding to each token’s input position and calculating attention, but by assigning relative positional encodings to tokens within an offset boundary during self-attention calculation.
Pegasus [35] uses the gap sensation generation (GSG) method, wherein sentences are selected based on their importance determined by the ROUGE score and individually masked. This approach assumes that aligning the pre-training objective with the text summarization process will yield improved performance. Similar to the MLM [27] method, GSG learns not by masking and predicting masked tokens but by masking and predicting important sentences. An important sentence refers to a sentence that can explain the overall context better than other sentences in a document.
However, the models described above primarily focus on fact-based documents, where objective summarization is prioritized. In contrast, summarizing opinion-based documents effectively demands attention to the document’s sentiment and authorial intent, aspects often overlooked by current models.
Several studies have focused on opinion-based texts. In [9], a model that generates abstractive summaries for each aspect was proposed. In an abstract summary of product reviews using a discourse structure, as in a graph-based study [11], a model generating an effective label description to learn a text representation via two different channels [37], and a model for generating summary statements by conducting aspect-to-aspect scoring using the Page Rank algorithm [24] were proposed. However, these studies are irrelevant to the research field covered by the proposed method because they focus on identifying sentiment by aspect [8], rather than generating summaries that explicitly consider sentiment.
There are studies targeting opinion-based text that focus on the degree of augmentation. In the research by Xu et al. [32] and Cheng et al. [4], additional attention is given to historical information, while in other studies by Xu et al. [34] and Xu et al. [33], personalized features are considered as supplementary information. However, these studies also do not explicitly consider opinion sentiment.
Therefore, to the best of our knowledge, our proposed methodology is the first to explicitly consider sentiment distribution for opinion-based text summarization.

2.2 XAI Methodologies Utilized in Our Proposed Study

In this study, SHAP was used to derive the sentiment distribution within a sentence. Based on this, the major opinion sentiment part on which the summarization model focuses was derived.
SHAP is the explainable artificial intelligence (XAI) [2] methodology used in this study. Figure 1 shows the entire process of SHAP, which is the original black box model \(f\), and the process of determining \(g\) with input values that provide variations to the data rather than putting the same input. SHAP is based on local explanations, enabling a global surrogate for the entire area of data. [18] proposed SHAP values as an “integrated measure” of characteristic importance. The conditional average of the SHAP value is calculated to define the simplified input by calculating the conditional average of \(f\), rather than the value of the existing model \(f\).
Fig. 1.
Fig. 1. SHAP framework.
SHAP values quantify the extent to which a specific feature characteristic contributes to changes in model predictions under specific conditions or contexts. \(E[f(z)]\) is predicted when no characteristic is known, called the base value, and SHAP values explain how the current result value \(f(x)\) originates from the base value (Figure 2).
Fig. 2.
Fig. 2. SHAP base values.
As discussed in numerous studies, SHAP has both advantages and disadvantages. Research indicates that SHAP is highly influenced by the correlations between variables when calculating Shapley values. Additionally, the computational complexity increases exponentially in high-dimensional data due to the consideration of all possible combinations of input variables in the Shapley value calculation.
However, SHAP values offer a model-agnostic approach, allowing application across different types of models. They consider both individual variable importance and overall variable importance. Despite its limitations, SHAP is widely utilized in various NLP tasks such as sentiment analysis, document classification, and document summarization [6, 7, 15]. In this study, we address the issue of variable correlation through a masking strategy. While the proposed research and its application objectives differ, it is noteworthy that Naretto et al. [23] also employed a masking strategy to compute the Shapley value in their study.

3 Method

Our proposed methodology adopts a fine-tuning approach, building upon a pretrained backbone architecture similar to existing summary models. It offers an efficient and versatile model that can be integrated into various existing summarization models. Figure 3 summarizes our proposed methodology.
Fig. 3.
Fig. 3. Summary of the proposed method.
The proposed methodology involves several distinct steps. First, the sentiment distribution of the input text is obtained by leveraging the classification head attached to the encoder and SHAP explainer. Second, major and minor opinion parts are extracted from the text considering the information derived from the sentiment distribution. Finally, random masking is applied to the minor opinion part and integrated into the encoder-decoder multi-attention operation during the training process. This inclusion of random masking helps prioritize and emphasize the major opinion sentiment during training. Details of each process are described in subsequent sections.

3.1 Derivation of Sentiment Distribution

First, the sentiment distribution of the input text is calculated using the classification head attached to the encoder and SHAP explainer. As mentioned previously, SHAP approximates a simplified input \({x}^{\prime}\) and a surrogate model \(g\) given an original input \(x\) and a complex black-box model \(f\). The simplified input \({x}^{\prime}\) is defined by the mapping function \(x=h_{x}(x)\). Assuming that \({z}^{\prime}\approx{x}^{\prime}\), \({z}\in\left\{0,1\right\}^{M}\) is represented as a binary variable (1: feature included in the model, 0: feature excluded from the model). We then train an explainable model \(g\) such that \(g({z}^{\prime})\approx f(h_{x}({x}^{\prime}))\). The explainable model \(g\) is approximated by a linear function of the binary variables, as shown below, where \(\phi_{i}\) denotes the importance score of the \(i\)th variable:
\begin{align*}g({z}^{\prime})=\phi_{0}+\sum_{i=1}^{M}\phi_{i}{z}^{\prime}_{i}.\end{align*}
In a real implementation, the result of tokenizing the dataset becomes the original input \(x\), and the classification head becomes the black-box model \(f\). By substituting \(x\) and \(f\) into the SHAP explainer function, we approximate the simplified input \({x}^{\prime}\) and surrogate model \(g\). Finally, the sentiment distribution of each token, \(\phi_{i}\), is calculated by the result of the SHAP explainer function. Figure 4 shows examples of sentiment distribution derived using the SHAP explainer.
Fig. 4.
Fig. 4. Example of sentiment distribution.
In this study, we employed the DeepExplainer among various SHAP explainers. SHAP explainer can be implemented in various forms, with TreeExplainer, KernelExplainer, and DeepExplainer being prominent examples. TreeExplainer is specialized for tree-based models (e.g., random forests, gradient boosting trees) and leverages the tree structure of the model for efficient and accurate computation of SHAP values. KernelExplainer is more versatile due to its minimal assumptions regarding the model, making it applicable to various models. Although it can be applied to non-linear or complex deep learning models, it suffers from high computational costs. DeepExplainer is optimized for deep learning models and provides SHAP values by analyzing the influence propagated through the layers of neural networks. It is particularly useful for handling non-linear relationships between model inputs and outputs, making it effective even in cases with complex internal mechanisms, such as transformer models. Considering these factors, we deemed DeepExplainer most suitable for alignment with the proposed research.

3.2 Extraction of Major and Minor Opinion Parts

Based on the sentiment distribution derived above, the input text is divided into major and minor opinion parts. In our proposed methodology, the “major opinion” part denotes the text segment where the opinion intended by the user is expressed strongly and prominently. Conversely, the “minor opinion” part is the text segment where the opinion expressed is secondary and understated. Essentially, the part with an explainer value based on sentiment distribution is defined as a major opinion part.
We derive the major and minor opinion parts using two methods. First, the explainer value derived from the sentiment distribution is used as a criterion. For the SHAP explainer, the explainer value can be calculated for positive and negative polarities. Therefore, the explainer value corresponding to the predicted polarity based on the classification head is used as the criterion. If the explainer value is within the top \(n\)%, it is considered a major opinion; otherwise, it is considered a minor opinion. In general, words that exhibit a polarity opposite to that intended by the user have a negative explainer value, and the explainer value of words unrelated to polarity converges to zero. Therefore, through this methodology, words expressing opposite polarities are likely to be derived as minor opinion segments. The pseudocode for the methodology expanded above is summarized in Algorithms 1 and 2.
The second method uses the absolute value of the explainer as a criterion. The rest is the same as in the methodology described above. However, if the absolute value is used as the judgement criterion, it is highly likely that a word irrelevant to the polarity where the explainer value converges to zero is derived as a minor opinion.

3.3 Masking Strategy

As previously mentioned, the proposed central learning strategy involves masking the minor opinion that deviates from the user’s intention. Simultaneously, the weight or emphasis on the major opinion part is increased within the encoder-decoder attention operation. This adjustment ensures that the generated summaries prioritize and emphasize the major opinion part, aligning with the original author’s intended viewpoint.
However, in this process, if a specific part of the minor opinion is collectively masked and the attention weight is given as zero, not only is part of the input text completely ignored, but the flow of the context also becomes unnatural.
Therefore, to avoid this problem, we randomly masked a certain percentage of minor opinions differently for each learning, rather than masking specific parts collectively. Additionally, the robustness of the learning process was increased using varying random masking for each attention head (Figure 5).
Fig. 5.
Fig. 5. Example of random masking strategy.
Our methodology aims to train a model capable of emphasizing the major opinion within a text solely based on the content, without relying on external data, such as sentiment distribution, during the inference phase. Therefore, a novel strategy for the summarization model to learn how to focus on the major opinion parts incrementally and naturally is presented in Figure 6.
Fig. 6.
Fig. 6. Masking strategy for incremental learning.
Specifically, the minor opinion part is initially masked at a high ratio to adapt to relatively easy problems first, and then the masking ratio is gradually reduced to learn to adapt to more difficult problems. Thus, it is possible to learn a model that effectively focuses on major opinion parts, even in the original text, without masking information.

3.4 Configuration of the Loss Function

Finally, in this study, the sentiment classification accuracy of the generated summary was also considered in learning to ensure that the generated summary could focus on major opinions that conform to the user’s intention. Specifically, by adding a classification head to the decoder output, the classification accuracy of the generated summary and the accuracy of the generated summary are reflected in the loss function as follows: \(Loss_{total}=Loss_{summaryGeneration}+\alpha Loss_{DecoderClassification}\). This aims to effectively create a summary that conforms to the user’s intentions by effectively maintaining the sentiment of the input text. Furthermore, we set the loss combination ratio \(\alpha\) to one to obtain the best performance in the experiment.

4 Experiment

4.1 Dataset

The proposed method was evaluated using the Amazon Review dataset, an opinion-based document dataset. The Amazon dataset not only contains golden summary information for the summarization task but also includes sentiment labels; therefore, it is almost the ideal dataset to verify the performance of the proposed methodology. It comprises 100 categories, and the four most widely used categories were selected as the experimental dataset. The statistics of this dataset are listed in Table 3.
Table 3.
DatasetTrainValidationTest
Toy79,0006,7966,775
Sport140,0007,7647,734
Home133,6008,6088,641
Movie178,69316,90916,890
Table 3. Dataset Statistics
Furthermore, as evident from Table 4 below, while the original text exhibits a relatively high proportion of neutral labels, the corresponding golden summary shows a decrease in neutral labels and an increase in the ratio of positive/negative labels. The Amazon dataset effectively demonstrates the specific traits of the opinion text we have discussed, making it an excellent choice for thoroughly evaluating the effectiveness of our proposed opinion text summarization approach.
Table 4.
DatasetLabelOriginal textGolden summary
ToyPos66%70%
Neu15%6%
Neg19%24%
SportPos61%66%
Neu20%9%
Neg19%25%
HomePos64%70%
Neu22%11%
Neg14%19%
MoviePos63%69%
Neu23%10%
Neg14%21%
Table 4. Ratio of Positive/Neutral/Negative Label

4.2 Experimental Setup

As previously mentioned, the proposed approach offers an attachable model that can be incorporated into existing summarization models. Therefore, we validated the performance improvement of the proposed methodology by applying it to the most widely used summarization models: BART and T5. We ensured the robustness of our proposed model’s performance improvement by comparing it against the most widely recognized summarization models as baselines.
In addition, we selected four benchmark models for performance comparison to demonstrate the superiority of the proposed method. The selected baselines are methodologies similar to our proposed approach, indicating that they incorporate attachable modules into existing summarization backbone architectures. By comparing these models with those that share similar improvement directions, we can objectively assess the performance enhancement of our proposed methodology. The seven models we used as baselines were: (1) Point Gathering Network [29], an Recurrent Neural Network-based abstractive summarization method with a copy mechanism; (2) C.Transformer [10], a variant of the Transformer equipped with a copy mechanism; (3) Hierarchical Model for Text Summarization and Sentiment Classification\(+\)Copy [19], a review summarization model that jointly improves review summarization and sentiment classification using a copy mechanism; (4) DualView [25], a dual-view model that improves review summarization and sentiment classification performance; (5) Transformer reasoning network for review summarization [32], a model that involves the historical information selectively to learn the specific representation; (6) Rating-boosted abstractive review summarization [34], a neural review-level attention model that effectively learns user preference embedding; and (7) Scalable implementation of remote scope promotion [33], an elaborately designed multi-task fine-tuning framework that fully leverages personalized information.
The generated summaries were evaluated using the Rouge Score, a commonly used evaluation metric for text summarization that measures the ROUGE-1, ROUGE-2, and ROUGE-L [17] scores of the generated summaries. Furthermore, the proposed methodology involves splitting the major and minor parts based on the prediction results of the classification head. Consequently, we must also validate its performance. Therefore, we present the inherent classification performance of the classification head embedded in the proposed methodology.
As mentioned previously, we propose two methodologies for splitting the major and minor parts based on the explainer value and absolute explainer value. Moreover, the proposed methodology employs an approach that induces natural learning by incrementally reducing the masking ratio. Therefore, we experimented with two masking strategies: (1) initially masking 85% of the minor opinions and decreasing the masking rate by 15% every epoch until reaching 10% and (2) initially masking 70% of the minor opinions and decreasing the masking rate by 10% every epoch until reaching 10%. We compared the performance of these options using an initial masking of 70% based on the absolute value, which showed optimal performance. The remaining options are presented in the ablation study to demonstrate their performances.
For BART-based approaches, the training parameters were set to a batch size of 256, AdamW optimizer [1] learning rate of 4e-4, and a warmup state of 0.025. The experiments were conducted on eight RTX 3090 GPUs in parallel. Similarly, for the T5-based approaches, the training parameters were set to a batch size of 128, an AdamW optimizer learning rate of 3e-4, and a warmup state of 0.025. Experiments were conducted on eight RTX 3090 GPUs in parallel, similar to the BART-based approaches.

4.3 Experimental Result

4.3.1 Abstractive Summarization Performance.

We present a comparison of the performance in the abstractive summarization task between our proposed methodology and the baselines in Table 5. As mentioned previously, the baselines included vanilla BART and vanilla T5, which represent the backbone architectures, as well as recently proposed methodologies similar to our proposed approach. This indicates that they incorporate attachable modules compatible with existing summarization backbone architectures. First, as evident from Table 5, our methodology demonstrated a significant performance improvement compared to vanilla BART and vanilla T5. This result validates the effectiveness of the attachable module designed based on our proposal. The module aims to split the major and minor opinion parts prioritizing the major opinion component during the abstractive summarization task, particularly for datasets that prioritize opinions.
Table 5.
DatasetMethodR-1R-2R-L
ToyPGNet14.836.1714.57
C.Transformer12.574.7612.36
HSSC\(+\)copy14.706.1814.46
Dual-View14.836.1714.57
TRNS18.878.7018.51
RARS18.045.0317.59
SRSP19.057.2818.63
Vanilla BART16.494.8214.99
Vanilla T517.615.8616.66
Ours (BART, Absolute value, 70%)18.035.6616.53
Ours (T5, Absolute value, 70%)19.307.1018.61
SportPGNet14.786.1314.58
C.Transformer13.735.1313.54
HSSC\(+\)copy14.645.9514.43
Dual-View15.396.4615.18
TRNS13.253.7813.04
RARS14.415.9316.11
SRSP18.266.6617.87
Vanilla BART15.444.7514.06
Vanilla T517.135.8916.20
Ours (BART, Absolute value, 70%)15.564.6614.09
Ours (T5, Absolute value, 70%)18.626.7017.93
HomePGNet14.826.2814.64
C.Transformer13.755.4413.58
HSSC\(+\)copy14.936.3414.75
Dual-View15.186.5715.00
TRNS14.604.9014.32
RARS16.674.6716.39
SRSP18.096.2818.08
Vanilla BART14.704.1713.27
Vanilla T517.095.8716.18
Ours (BART, Absolute value, 70%)15.064.2413.61
Ours (T5, Absolute value, 70%)18.106.6717.45
MoviePGNet12.285.1412.40
C.Transformer12.094.4611.81
HSSC\(+\)copy12.665.0612.39
Dual-View12.845.2212.57
TRNS11.722.8511.23
RARS15.045.3514.60
SRSP16.085.4715.57
Vanilla BART12.653.2611.58
Vanilla T513.313.9012.29
Ours (BART, Absolute value, 70%)13.573.9212.51
Ours (T5, Absolute value, 70%)13.924.3313.08
Table 5. Performance Comparison for the Abstractive Summarization
Bold indicates the best performance in each dataset.
Table 5 shows that the proposed methodology outperforms other recently proposed methods that incorporate attachable modules. This proves that our methodology is the most effective abstractive summarization approach for opinion-focused texts. In the experimental results, the proposed methodology achieved superior performance across all measures and datasets except for the Movie datasets. Additionally, the observed performance of certain methodologies demonstrated poorer results compared to the backbone architecture. This demonstrates that simply incorporating an additional module does not guarantee a performance improvement for opinion-focused datasets. This result substantiates the proposed necessity to incorporate modules tailored to the unique characteristics of opinion-focused datasets.
In addition, the superiority of the proposed methodology can be evaluated through its application of the SHAP explainer. The Shapley value, which forms the basis of the SHAP explainer, is a methodology heavily influenced by the correlation between variables. The masking strategy employed in the proposed methodology reduces the correlation between tokens, partially addressing the inherent limitations of the Shapley value in this aspect. Furthermore, in the proposed methodology, using transformers to learn interactions between tokens allowed for a more accurate reflection of the correlation between variables. This factor is believed to contribute to the superior performance of the proposed methodology over existing models.
On the other hand, the performance of the proposed methodology on the Movie dataset was lower compared to some other methodologies. Since the proposed methodology involves deriving major and minor sentiment parts from reviews and utilizing them for training by masking, it necessitates appropriate coherence between the sentiment of the entire document and the sentiment of each token. However, due to inconsistencies and a significant amount of ambiguous data between the overall sentiment and the sentiment of tokens, as exemplified below, we conclude that the performance of the proposed methodology was relatively lower.
‘the lucky ones is a pretty good movie. it’s not great but its not bad either. i was going to give it 3 stars but it deserves 4. i like how these three soldiers stories are told at once and they don’t get confusing or have some weird thing to do with each other like some movies do. i thought they did a good job making you feel the pain each person is going through too. i was hoping for a different ending but i don’t think it could have been too different at the same time. i would recommend this movie to everybody to see at least once.’ (Original label: Positive)
‘i have all of the cinematic titanic shows on DVD to date. i love most of them my favorite is war of the insects. i met the case last winter and admire them and their talents. i know humor can be subjective this one just didn’t click for me. the YouTube trailer covers most of what i found funny in this movie. i kind of prefer the studio movies as opposed to the live ones but note my favorite happens to be a live DVD. i m not slamming cinematic this is just not a movie i would recommend to others. I’ll still buy whatever the next release is.’ (Original label: Negative)

4.3.2 Qualitative Analysis.

In Tables 6 and 7, we provide examples of the summaries generated by BART, T5, and our proposed model for comparison. Analysis of these tables reveals that, like many opinion-based texts, the original sentences combine major and minor opinions, whereas the golden summary focuses solely on the major opinion.
Table 6.
ReviewThe n64 was nintendo’s 3rd console not counting the game boy and virtaul boy.It came out shortly after the sega saturn and sony playstation. If you do not have this system anymore or never have and collect systems and and games then i recommend this system the controller has always at first glace seemed like a gimmick and like myself probably made you think how are people supposed to use that controller. But it actually is not too bad! the system it s self is pretty simple yet cool looking and is for the most part really reliable.And having 4 controller ports is a plus too! though it would have been nice if nintendo released more mature games for the console there are some classic games I recommend the games below
SentimentPositive
Gold SummaryA pretty good system for it’s time…and even every now and then.
BART Summary (Baseline)Not fond of barbie but this adds to the fun.
T5 Summary (Baseline)Nintendo 3rd console.
Our SummaryGreat game system for the n64 and sony playstation.
Table 6. An Example of a Generated Summary (Positive Sentiment)
Table 7.
ReviewI thought this sounded like a pretty good game but it seemed to be for us at least more trouble than it was worth.the game controller itself is simple. it does not seem very heavy duty simply translucent blue lightweight plastic.the controller is not very comfortable in the hand like a regular video game controller is.the cartridge does not hook in very securely. to play the game you first have to load a cd into your computer.then while connected to the internet you need highspeed access you need to download more stuff.this took a looong time for us.it was not our access it was slow coming from their site.you also need to have 1 gb free…yes a whole gb! finally the game was running.while the game was running we were not able to adjust the volume on our computer.it was loud and annoying.the courses were nothing wonderful and you need the right cartridge for each car which does best on its own track.there are eight cartridges total.this was kind of annoying considering each car does not have any special chip or anything. the controller detects which cartridge is in it by which of the five pins the car triggers in the controller.so basically by buying additional cartridges you are simply buying a way to press down a couple of buttons.to top it all off the company reserves the right to terminate turbo driver online service after january 1 2010. so you can buy the unit and all the cartridges and after a year they may be useless. the game itself is ok to play. the kids played it for a little while but became quickly bored by it.the graphics are not great the controller is not great the game can be frustrating. it is a nice idea and is not a bad toy but i would not recommend it.
SentimentNegative
Gold SummaryNot a great game.
BART Summary (Baseline)A little bit of fun for the kids but not worth it
T5 Summary (Baseline)not a bad toy.
Our SummaryNot worth the money…frustrating and slow service.
Table 7. An Example of a Generated Summary (Negative Sentiment)
However, the summaries generated by the baselines include both major and minor opinion content, which is in clear contrast with the golden summary. As mentioned previously, this is because existing abstractive summarization models tend to summarize the entire text consistently without considering sentiment. Conversely, the summaries generated by our proposed model include only the major opinion segment, which is aligned with the golden summary. Based on these generated summary examples, we can demonstrate that our proposed model is effective in summarizing opinion-focused texts and that this factor contributes to achieving significantly higher Rouge scores than the baselines.

4.3.3 Ablation Study.

In the ablation study, we performed additional comparisons of the summarization performance based on the criteria for splitting the major and minor parts (i.e., explainer value-based and absolute explainer value-based, respectively) and the ratio of the initial masking (70%/85%). Furthermore, as mentioned previously, in our proposed methodology, we applied different random masking to each head to implement a robust model. To validate the effectiveness of this approach, experiments were conducted with an option that applied the same masking to all heads. Tables 8 and 9 present the results of the ablation experiments using BART and T5 as the backbone architectures, respectively.
Table 8.
DatasetMaskingABSABSABSEXPEXPEXP
RatioR-1R-2R-LR-1R-2R-L
Toy85%16.234.5016.7016.474.6515.00
70%18.035.6616.5315.954.5014.42
55%15.044.0115.4814.903.9014.20
40%14.953.9515.0114.843.8514.55
Sport85%16.105.2314.8515.564.6614.09
70%15.014.3813.6715.924.8114.55
55%14.854.2213.0214.804.1113.01
40%14.444.0112.9213.854.0212.95
Home85%15.004.3513.6015.985.0014.71
70%15.064.2413.6116.225.1014.96
55%14.864.0213.0514.343.9714.01
40%13.993.9513.4513.433.7413.44
Movie85%13.403.9012.3313.273.8712.27
70%13.573.9212.5113.003.5011.90
55%12.943.4512.1112.443.3011.40
40%12.873.2412.0112.323.1011.50
Table 8. Ablation Study (BART Backbone)
Table 9.
DatasetMaskingABSABSABSEXPEXPEXP
 RatioR-1R-2R-LR-1R-2R-L
Toy85%18.796.8918.2218.756.6718.12
70%19.307.1018.6118.966.7718.30
55%19.017.0018.3118.646.4318.01
40%18.856.9018.3418.346.3017.94
Sport85%18.576.7117.9218.546.7217.83
70%18.626.7017.9318.506.6317.83
55%18.346.5417.6418.326.5217.62
40%18.216.4217.3118.156.2217.34
Home85%18.006.6117.4317.976.4017.31
70%18.106.6717.4518.066.5017.43
55%17.946.5417.3017.646.3217.32
40%17.746.3217.0117.216.2117.11
Movie85%13.894.3113.0513.824.2412.95
70%13.924.3313.0813.854.2513.01
55%13.324.2212.9512.894.1112.84
40%13.014.0112.6512.444.0112.34
Table 9. Ablation Study (T5 Backbone)
Experimental results revealed that for the BART backbone, the performance results were significantly mixed across the datasets, whereas for the T5 backbone, relatively consistent performance comparisons were achieved. When considering T5 as the reference, using the criteria of absolute explainer value rather than explainer value achieved a relatively higher performance. Furthermore, it was evident that applying random masking consistently led to a higher overall performance than when it was not applied. However, the performance variation based on the ratio of initial masking was relatively small. Therefore, when replicating the proposed model, it is considered more efficient to prioritize the backbone model and splitting criteria than placing excessive emphasis on the initial masking ratio.

4.3.4 Classification Performance.

As explained previously, the proposed methodology focuses on the major opinion part of the text to generate summaries that align with the author’s intent. That is, it generates summaries that align with the sentiment of the original text. To further demonstrate the effectiveness of the proposed methodology, additional experiments were conducted to verify whether the generated summaries effectively reflect the sentiment of the original text. Specifically, in Table 10, we conducted an experimental evaluation to assess the classification performance of the summaries generated by the proposed model. Our findings indicate notable improvements when compared with the baselines, demonstrating the enhanced quality and effectiveness of the generated summaries.
Table 10.
DatasetMethodAccuracyF1-Score
ToyVanilla BART85.0191.32
Ours (BART)86.9392.47
Vanila T590.8794.91
Ours (T5)91.2695.15
SportVanilla BART75.9085.21
Ours (BART)77.0385.90
Vanila T587.8393.10
Ours (T5)91.2695.15
HomeVanilla BART76.5085.10
Ours (BART)80.7788.10
Vanila T585.791.52
Ours (T5)91.2695.15
MoiveVanilla BART76.4584.75
Ours (BART)81.0888.06
Vanila T583.3889.80
Ours (T5)91.2695.15
Table 10. Classification Performance of Generated Summary
Bold indicates the best performance in each dataset.
Experimental results indicate that the summaries generated by our proposed methodology were significantly superior to the baseline models across all datasets. This confirms that the summaries generated based on the proposed methodology effectively maintain the sentiment of the original text. Furthermore, this highlights the success of the proposed model in prioritizing key opinion segments.

4.3.5 Effectiveness of SHAP Value.

In the proposed methodology, major and minor sentiment parts are distinguished based on SHAP values and utilized for training. Therefore, this section verifies whether SHAP values are an effective measure for distinguishing major and minor sentiment parts in terms of the reliability of the proposed methodology.
Specifically, we compared the sentiment analysis results of the original sentences with the data extracted based on SHAP values, containing only tokens of major sentiment. We conducted experiments to determine whether sentiment scores are indeed more accurate and sharpened in sentences extracted using SHAP.
As shown in Table 11, when only major sentiment tokens were extracted based on SHAP values, it was observed that predictions were sharpened towards the ground truth distribution in both positive and negative cases. This indicates the effective extraction of major sentiment based on SHAP values and signifies that the SHAP values used in the proposed study are an effective methodology for distinguishing between major and minor sentiment parts.
Table 11.
DatasetLabelOriginal sentence (Pos:Neg)Major sentiment part (Pos:Neg)
ToyPositive0.7159:0.38410.9007:0.0993
Negative0.3681:0.63190.0799:0.9201
SportPositive0.5640:0.43600.8715:0.1285
Negative0.3383:0.66170.1015:0.8985
HomePositive0.6817:0.31830.9247:0.0753
Negative0.3218:0.67820.0882:0.9118
MoivePositive0.5731:0.42690.8374:0.1626
Negative0.3812:0.61880.1783:0.8217
Table 11. Comparison of Sentiment Prediction Distributions

5 Conclusion

Existing abstractive summary models have demonstrated state-of-the-art performance in summarizing fact-based documents, emphasizing the importance of objective information summaries. However, these models often cannot handle the unique characteristics of opinion-based documents, which requires an understanding of the intended message. Therefore, this paper proposes an abstractive summarization model that focuses on opinion-based documentation by considering the author’s intended message.
Specifically, we propose a summarization model that prioritizes the core intent using the SHAP explainer to identify the sentiment of the entire document. This method enables the extraction of a major opinion segment that accurately reflects the intended message. In addition, we present an incremental learning strategy that initially addresses easier problems with high masking ratios and gradually adapts to more challenging problems with low masking ratios.
Through experiments on opinion-based documents, we have demonstrated that our proposed model achieves improved summarization performance compared with existing summarization models. Furthermore, by presenting actual examples of the generated summaries, we have confirmed the effectiveness of the proposed method. Furthermore, we determined the optimal parameters through an ablation study. Validation of the classification performance based on the generated summaries further have demonstrated that the proposed model effectively delivers the author’s intended message.
However, a limitation of our proposed research is that we solely relied on the explainer value derived from the SHAP model as a reference. Notably, the SHAP methodology employed in this study has exhibited stability issues when applied to text data, and there are limitations in the Shapley value calculation process, which forms the foundation of the SHAP methodology due to the influence of correlations between variables. Therefore, in future studies, various improvements addressing these drawbacks of the SHAP technique could be proposed, and enhancing model robustness could be achieved by applying various XAI methodologies beyond SHAP.
Additionally, more effective end-to-end methods can be used in future studies rather than random masking, which requires hyperparameter tuning. Furthermore, the proposed methodology yielded diminished performance on datasets where the distribution of sentiment was ambiguous. Hence, future research could propose methodologies robust to dataset variations.

References

[1]
Omar Alqaryouti, Nur Siyam, Azza Abdel Monem, and Khaled Shaalan. 2019. Aspect-based sentiment analysis using smart government review data. Applied Computing and Informatics 20, 1/2 (2019), 142–161.
[2]
Plamen P. Angelov, Eduardo A. Soares, Richard Jiang, Nicholas I. Arnold, and Peter M. Atkinson. 2021. Explainable artificial intelligence: An analytical review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 11, 5 (2021), e1424.
[3]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, 1877–1901.
[4]
Xin Cheng, Shen Gao, Yuchi Zhang, Yongliang Wang, Xiuying Chen, Mingzhe Li, Dongyan Zhao, and Rui Yan. 2023. Towards personalized review summarization by modeling historical reviews from customer and product separately. arXiv:2301.11682. Retrieved from https://arxiv.org/abs/2301.11682
[5]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. Retrieved from https://arxiv.org/abs/1810.04805
[6]
Christine Dewi, Bing-Jun Tsai, and Rung-Ching Chen. 2022. Shapley additive explanations for text classification and sentiment analysis of internet movie database. In Proceedings of the Asian Conference on Intelligent Information and Database Systems. Springer, 69–80.
[7]
Arwa Diwali, Kawther Saeedi, Kia Dashtipour, Mandar Gogate, Erik Cambria, and Amir Hussain. 2023. Sentiment analysis meets explainable artificial intelligence: A survey on explainable sentiment analysis. IEEE Transactions on Affective Computing 15, 3 (2023), 837–846.
[8]
Hai Ha Do, Penatiyana W. C. Prasad, Angelika Maag, and Abeer Alsadoon. 2019. Deep learning for aspect-based sentiment analysis: A comparative review. Expert Systems with Applications 118 (2019), 272–299.
[9]
Lea Frermann and Alexandre Klementiev. 2019. Inducing document structure for aspect-based summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 6263–6273.
[10]
Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. Bottom-up abstractive summarization. arXiv:1808.10792. Retrieved from https://arxiv.org/abs/1808.10792
[11]
Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1602–1613.
[12]
Karthick Prasad Gunasekaran. 2023. Exploring sentiment analysis techniques in natural language processing: A comprehensive review. arXiv:2305.14842. Retrieved from https://arxiv.org/abs/2305.14842
[13]
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems.
[14]
Elvira Ismagilova, Emma Slade, Nripendra P. Rana, and Yogesh K. Dwivedi. 2020. The effect of characteristics of source credibility on consumer behaviour: A meta-analysis. Journal of Retailing and Consumer Services 53 (2020), 101736.
[15]
Enja Kokalj, Blaž Škrlj, Nada Lavrač, Senja Pollak, and Marko Robnik-Šikonja. 2021. BERT meets shapley: Extending SHAP explanations to transformer-based classifiers. In Proceedings of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, 16–21.
[16]
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv:1910.13461. Retrieved from https://arxiv.org/abs/1910.13461
[17]
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out. Association for Computational Linguistics, 74–81.
[18]
Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, 4765–4774.
[19]
Shuming Ma, Xu Sun, Junyang Lin, and Xuancheng Ren. 2018. A hierarchical end-to-end model for jointly improving text summarization and sentiment classification. arXiv:1805.01089. Retrieved from https://arxiv.org/abs/1805.01089
[20]
Kathleen McKeown. 1992. Text Generation. Cambridge University Press.
[21]
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31.
[22]
Reema Narayan, Usha Y. Nayak, Ashok M. Raichur, and Sanjay Garg. 2018. Mesoporous silica nanoparticles: A comprehensive review on synthesis and recent advances. Pharmaceutics 10, 3 (2018), 118.
[23]
Francesca Naretto, Roberto Pellungrini, Salvatore Rinzivillo, and Daniele Fadda. 2023. EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories. In Proceedings of the International Conference on Discovery Science. Springer, 325–340.
[24]
Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank Citation Ranking: Bringing Order to the Web. Technical Report. Stanford InfoLab.
[25]
Hou Pong Chan, Wang Chen, and Irwin King. 2020. A unified dual-view model for review summarization and sentiment classification with inconsistency loss. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval.
[26]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.
[27]
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2019. Masked language model scoring. arXiv:1910.14659. Retrieved from https://arxiv.org/abs/1910.14659
[28]
Joni Salminen, Chandrashekhar Kandpal, Ahmed Mohamed Kamel, Soon-gyo Jung, and Bernard J. Jansen. 2022. Creating and detecting fake reviews of online products. Journal of Retailing and Consumer Services 64 (2022), 102771.
[29]
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv:1704.04368. Retrieved from https://arxiv.org/abs/1704.04368
[30]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems.
[31]
Hui-Ju Wang. 2022. Understanding reviewer characteristics in online reviews via network structural positions. Electronic Markets 32, 3 (2022), 1311–1325.
[32]
Hongyan Xu, Hongtao Liu, Pengfei Jiao, and Wenjun Wang. 2021. Transformer reasoning network for personalized review summarization. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1452–1461.
[33]
Hongyan Xu, Hongtao Liu, Zhepeng Lv, Qing Yang, and Wenjun Wang. 2023. Sentiment-aware review summarization with personalized multi-task fine-tuning. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, 2826–2835.
[34]
Hongyan Xu, Hongtao Liu, Wang Zhang, Pengfei Jiao, and Wenjun Wang. 2021. Rating-boosted abstractive review summarization with neural personalized generation. Knowledge-Based Systems 218 (2021), 106858.
[35]
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the International Conference on Machine Learning. PMLR, 11328–11339.
[36]
Lei Zhang, Bing Liu, Suk Hwan Lim, and Eamonn O’Brien-Strain. 2010. Extracting and ranking product features in opinion documents. In Proceedings of the International Conference on Coling 2010: Posters, 1462–1470.
[37]
Xiaofei Zhu, Zhanwang Peng, Jiafeng Guo, and Stefan Dietze. 2023. Generating effective label description for label-aware sentiment classification. Expert Systems with Applications 213 (2023), 119194.

Index Terms

  1. User Opinion-Focused Abstractive Summarization Using Explainable Artificial Intelligence

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Intelligent Systems and Technology
      ACM Transactions on Intelligent Systems and Technology  Volume 15, Issue 6
      December 2024
      727 pages
      EISSN:2157-6912
      DOI:10.1145/3613712
      • Editor:
      • Huan Liu
      Issue’s Table of Contents

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 13 December 2024
      Online AM: 21 September 2024
      Accepted: 24 August 2024
      Revised: 07 May 2024
      Received: 18 September 2023
      Published in TIST Volume 15, Issue 6

      Check for updates

      Author Tags

      1. Abstractive summarization
      2. Opinion focused-document
      3. Intended message
      4. Explainable artificial intelligence
      5. Random masking
      6. Machine learning
      7. Artificial intelligence

      Qualifiers

      • Research-article

      Funding Sources

      • National Research Foundation of Korea (NRF)
      • Korean government (MSIT)

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 152
        Total Downloads
      • Downloads (Last 12 months)152
      • Downloads (Last 6 weeks)26
      Reflects downloads up to 12 Feb 2025

      Other Metrics

      Citations

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Full Access

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media