ali-bahrainian · MeMartijn · Oct 7, 2021 · Oct 7, 2021 · Oct 11, 2021 · Oct 11, 2021
diff --git a/.gitignore b/.gitignore
@@ -1 +1,8 @@
 .DS_Store
+data
+*.bak
+__init__.py
+.ipynb_checkpoints
+__pycache__
+logs/cats-train-full
+lda
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,15 @@
+FROM python:3.8
+
+WORKDIR /api
+
+RUN python3 -m pip install --upgrade pip
+
+COPY requirements.txt .
+
+RUN pip3 install -r requirements.txt
+
+COPY . .
+
+EXPOSE 5000
+
+CMD ["python3", "./api.py"]
diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
 This repository contains code related to the paper “CATS: Customizable Abstractive Topic-based Summarization” published at Transactions of Information Systems (TOIS) journal, 2021. 
 
 
-The code has been developed using python 2.7, and Tensorflow 1.4. This implementation is based on code releases related to Pointer-Generator Networks [here](https://github.com/abisee/pointer-generator) and the TextSum project. 
+The code has been developed using Python 3, and v1 compatible TensorFlow 2 code. This implementation is based on code releases related to Pointer-Generator Networks [here](https://github.com/abisee/pointer-generator) and the TextSum project. 
 
 
 #### Dataset
@@ -14,13 +14,13 @@ In order to obtain the dataset, we encourage users to download and preprocess th
 
 #### Using Topic Information:
 
-The LDA models used in our paper can be obtained from [here](https://drive.google.com/drive/folders/1M86uAM21Zx8Xn-W4TTi74t7nWCQr1f9v?usp=sharing). The current code release has been tested with the 150 topics pre-trained LDA model. You can make a reference to one of the provided LDA topic models in data.py in the TopicModel class. 
+The LDA models used in our paper can be obtained from [here](https://drive.google.com/drive/folders/1M86uAM21Zx8Xn-W4TTi74t7nWCQr1f9v?usp=sharing). The current code release has been tested with the 150 topics pre-trained LDA model. You can make a reference to one of the provided LDA topic models in data.py in the TopicModel class. By default, the LDA model is expected to be in a folder called `lda`, which contains `lda.model` and `dictionary.dic` available from the beforementioned URL.
 
 #### Train 
 In order to train the model you may run:
 
 ```
-python run_summarization.py --mode=train --data_path=/path/to/chunked/train_* --vocab_path=/path/to/vocab --log_root=/path/to/a/log/directory --exp_name=myexperiment
+python3 run_summarization.py --mode=train --data_path='data/chunked/train_*' --vocab_path='data/vocab' --log_root='logs' --exp_name=myexperiment
 ```
 
 This will create a subdirectory of your specified log_root called myexperiment where all checkpoints will be saved. Then the model will start training using the train_*.bin files as training data.
@@ -30,11 +30,29 @@ This will create a subdirectory of your specified log_root called myexperiment w
 As stated in the paper, no topic information were used at test time. In order to decode without topic information, we used the pointer-generator basic model code [here](https://github.com/abisee/pointer-generator). After downloading the code, you may decode using:
 
 ```
-python run_summarization.py --mode=decode --data_path=/path/to/chunked/val_* --vocab_path=/path/to/vocab --log_root=/path/to/a/log/directory --exp_name=myexperiment
+python3 run_summarization.py --mode=decode --data_path='data/chunked/val_*' --vocab_path='data/vocab' --log_root='logs' --exp_name=myexperiment
 ```
 
 Please note that one should run the above command using the same settings entered for the training job (plus any decode mode specific flags like beam_size).
 
 This will repeatedly load random examples from your specified datafile and generate a summary using beam search. The results will be printed to screen.
 
 If you would like to run evaluation on the entire validation or test set and obtain ROUGE scores, set the flag single_pass=1. This will go through the entire dataset in the same order, writing the generated summaries to file, and then running evaluation using pyrouge.   
+
+### CATS API
+To use the API, build the Docker container using the following command:
+
+```
+docker build -t cats-api-container . 
+```
+
+Then, run the Docker container with the following command:
+
+```
+docker run -d -p 5000:5000 cats-api-container 
+```
+
+This should expose the container on port 5000. To summarize one input text, you can send a POST-request to `/summarize`, structuring the request as follows:
+
+- The request must have form-data as body
+- The form-data object should include a key called `target`, with the value being the to-be-summarized text.
diff --git a/__init__.py b/__init__.py
diff --git a/api.py b/api.py
@@ -0,0 +1,47 @@
+from flask import Flask, request
+from encode import write_to_bin
+import uuid
+import os
+import shutil
+
+app = Flask(__name__)
+app.config["DEBUG"] = False
+
+@app.route('/', methods=['GET'])
+def home():
+    return "<h1>CATS: Customizable Abstractive Topic-based Summarization</h1><p>This site is the interface of an API to interact with an advanced topic-aware summarization model.</p>"
+
+@app.route('/summarize', methods=['POST'])
+def get_summary():
+    # Define settings
+    TARGET_INPUT_FOLDER = 'requests'
+    TRAINED_MODEL_FOLDER = 'logs'
+    TRAINED_MODEL_EXPERIMENT = 'cats-train-full'
+
+    # Generate other variables
+    request_uuid = request_uuid = str(uuid.uuid4())
+    input_file_location = os.path.join(TARGET_INPUT_FOLDER, request_uuid, 'test_000.bin')
+
+    # Get target text
+    input_text = request.form.get('target')
+
+    # Write target to binary for decoding
+    write_to_bin(input_text, input_file_location)
+
+    # Create summary
+    os.system(f"python3 run_summarization.py --mode=decode --data_path='{os.path.join(TARGET_INPUT_FOLDER, request_uuid)}/test_*' --vocab_path='data/vocab' --log_root='{TRAINED_MODEL_FOLDER}' --exp_name={TRAINED_MODEL_EXPERIMENT} --single_pass=True --single_input=True --decode_dir='{os.path.join('requests', request_uuid)}'")
+
+    # Read the summary file
+    with open(os.path.join(TARGET_INPUT_FOLDER, request_uuid, 'decoded', '000000_decoded.txt')) as f:
+        summary = f.read().splitlines() 
+
+    # Delete the request directory
+    shutil.rmtree(os.path.join(TARGET_INPUT_FOLDER, request_uuid))
+
+    # Return the summary
+    return {
+        'summary': summary
+    }
+
+if __name__ == '__main__':
+    app.run(host="0.0.0.0", port=5000)
diff --git a/attention_decoder.py b/attention_decoder.py
@@ -49,12 +49,12 @@ def attention_decoder(decoder_inputs, initial_state, encoder_states, enc_topicwo
     coverage: Coverage vector on the last step computed. None if use_coverage=False.
   """
   with variable_scope.variable_scope("attention_decoder") as scope:
-    batch_size = encoder_states.get_shape()[0].value # if this line fails, it's because the batch size isn't defined
-    attn_size = encoder_states.get_shape()[2].value # if this line fails, it's because the attention length isn't defined
+    batch_size = encoder_states.get_shape()[0] # if this line fails, it's because the batch size isn't defined
+    attn_size = encoder_states.get_shape()[2] # if this line fails, it's because the attention length isn't defined
 
     #tf.Print(attn_dist, [attn_dist])
-    print "-------------------------"
-    print "-------------------------"
+    print("-------------------------")
+    print("-------------------------")
 
     # Reshape encoder_states (need to insert a dim)
     encoder_states = tf.expand_dims(encoder_states, axis=2) # now is shape (batch_size, attn_len, 1, attn_size)
@@ -101,7 +101,7 @@ def masked_attention(e):
           """Take softmax of e then apply enc_padding_mask and re-normalize"""
           attn_dist = nn_ops.softmax(e) # take softmax. shape (batch_size, attn_length)
           attn_dist *= enc_padding_mask # apply mask
-          masked_sums = tf.reduce_sum(attn_dist, axis=1) # shape (batch_size)
+          masked_sums = tf.reduce_sum(input_tensor=attn_dist, axis=1) # shape (batch_size)
           return attn_dist / tf.reshape(masked_sums, [-1, 1]) # re-normalize
 
         if use_coverage and coverage is not None: # non-first step of coverage
@@ -146,13 +146,13 @@ def masked_attention(e):
       # Re-calculate the context vector from the previous step so that we can pass it through a linear layer with this step's input to get a modified version of the input
       context_vector, _, coverage = attention(initial_state, enc_topicwords_probs_batch, coverage) # in decode mode, this is what updates the coverage vector
     for i, inp in enumerate(decoder_inputs):
-      tf.logging.info("Adding attention_decoder timestep %i of %i", i, len(decoder_inputs))
+      tf.compat.v1.logging.info("Adding attention_decoder timestep %i of %i", i, len(decoder_inputs))
       if i > 0:
         variable_scope.get_variable_scope().reuse_variables()
 
       # Merge input and previous attentions into one vector x of the same size as inp
       input_size = inp.get_shape().with_rank(2)[1]
-      if input_size.value is None:
+      if input_size is None:
         raise ValueError("Could not infer input size from input: %s" % inp.name)
       x = linear([inp] + [context_vector], input_size, True)
 
@@ -169,7 +169,7 @@ def masked_attention(e):
 
       # Calculate p_gen
       if pointer_gen:
-        with tf.variable_scope('calculate_pgen'):
+        with tf.compat.v1.variable_scope('calculate_pgen'):
           p_gen = linear([context_vector, state.c, state.h, x], 1, True) # Tensor shape (batch_size, 1)
           p_gen = tf.sigmoid(p_gen)
           p_gens.append(p_gen)
@@ -227,14 +227,14 @@ def linear(args, output_size, bias, bias_start=0.0, scope=None):
       total_arg_size += shape[1]
 
   # Now the computation.
-  with tf.variable_scope(scope or "Linear"):
-    matrix = tf.get_variable("Matrix", [total_arg_size, output_size])
+  with tf.compat.v1.variable_scope(scope or "Linear"):
+    matrix = tf.compat.v1.get_variable("Matrix", [total_arg_size, output_size])
     if len(args) == 1:
       res = tf.matmul(args[0], matrix)
     else:
       res = tf.matmul(tf.concat(axis=1, values=args), matrix)
     if not bias:
       return res
-    bias_term = tf.get_variable(
-        "Bias", [output_size], initializer=tf.constant_initializer(bias_start))
+    bias_term = tf.compat.v1.get_variable(
+        "Bias", [output_size], initializer=tf.compat.v1.constant_initializer(bias_start))
   return res + bias_term
diff --git a/batcher.py b/batcher.py
@@ -16,7 +16,7 @@
 
 """This file contains code to process data into batches"""
 
-import Queue
+import queue
 from random import shuffle
 from threading import Thread
 import time
@@ -199,7 +199,7 @@ def init_encoder_seq(self, example_list, hps):
       self.enc_topicwords_batch[i, :] = ex.article_topic_vector[:]
       self.enc_topicwords_probs_batch[i, :] = ex.article_topic_word_probs[:]
       self.enc_lens[i] = ex.enc_len
-      for j in xrange(ex.enc_len):
+      for j in range(ex.enc_len):
         self.enc_padding_mask[i][j] = 1
 
     # For pointer-generator mode, need to store some extra info
@@ -236,7 +236,7 @@ def init_decoder_seq(self, example_list, hps):
     for i, ex in enumerate(example_list):
       self.dec_batch[i, :] = ex.dec_input[:]
       self.target_batch[i, :] = ex.target[:]
-      for j in xrange(ex.dec_len):
+      for j in range(ex.dec_len):
         self.dec_padding_mask[i][j] = 1
 
   def store_orig_strings(self, example_list):
@@ -266,8 +266,8 @@ def __init__(self, data_path, vocab, hps, single_pass):
     self._single_pass = single_pass
 
     # Initialize a queue of Batches waiting to be used, and a queue of Examples waiting to be batched
-    self._batch_queue = Queue.Queue(self.BATCH_QUEUE_MAX)
-    self._example_queue = Queue.Queue(self.BATCH_QUEUE_MAX * self._hps.batch_size)
+    self._batch_queue = queue.Queue(self.BATCH_QUEUE_MAX)
+    self._example_queue = queue.Queue(self.BATCH_QUEUE_MAX * self._hps.batch_size)
 
     # Different settings depending on whether we're in single_pass mode or not
     if single_pass:
@@ -282,12 +282,12 @@ def __init__(self, data_path, vocab, hps, single_pass):
 
     # Start the threads that load the queues
     self._example_q_threads = []
-    for _ in xrange(self._num_example_q_threads):
+    for _ in range(self._num_example_q_threads):
       self._example_q_threads.append(Thread(target=self.fill_example_queue))
       self._example_q_threads[-1].daemon = True
       self._example_q_threads[-1].start()
     self._batch_q_threads = []
-    for _ in xrange(self._num_batch_q_threads):
+    for _ in range(self._num_batch_q_threads):
       self._batch_q_threads.append(Thread(target=self.fill_batch_queue))
       self._batch_q_threads[-1].daemon = True
       self._batch_q_threads[-1].start()
@@ -309,9 +309,9 @@ def next_batch(self):
     """
     # If the batch queue is empty, print a warning
     if self._batch_queue.qsize() == 0:
-      tf.logging.warning('Bucket input queue is empty when calling next_batch. Bucket queue size: %i, Input queue size: %i', self._batch_queue.qsize(), self._example_queue.qsize())
+      tf.compat.v1.logging.warning('Bucket input queue is empty when calling next_batch. Bucket queue size: %i, Input queue size: %i', self._batch_queue.qsize(), self._example_queue.qsize())
       if self._single_pass and self._finished_reading:
-        tf.logging.info("Finished reading dataset in single_pass mode.")
+        tf.compat.v1.logging.info("Finished reading dataset in single_pass mode.")
         return None
 
     batch = self._batch_queue.get() # get the next Batch
@@ -324,11 +324,11 @@ def fill_example_queue(self):
 
     while True:
       try:
-        (article, abstract) = input_gen.next() # read the next example from file. article and abstract are both strings.
+        (article, abstract) = next(input_gen) # read the next example from file. article and abstract are both strings.
       except StopIteration: # if there are no more examples:
-        tf.logging.info("The example generator for this example queue filling thread has exhausted data.")
+        tf.compat.v1.logging.info("The example generator for this example queue filling thread has exhausted data.")
         if self._single_pass:
-          tf.logging.info("single_pass mode is on, so we've finished reading dataset. This thread is stopping.")
+          tf.compat.v1.logging.info("single_pass mode is on, so we've finished reading dataset. This thread is stopping.")
           self._finished_reading = True
           break
         else:
@@ -348,13 +348,13 @@ def fill_batch_queue(self):
       if self._hps.mode != 'decode':
         # Get bucketing_cache_size-many batches of Examples into a list, then sort
         inputs = []
-        for _ in xrange(self._hps.batch_size * self._bucketing_cache_size):
+        for _ in range(self._hps.batch_size * self._bucketing_cache_size):
           inputs.append(self._example_queue.get())
         inputs = sorted(inputs, key=lambda inp: inp.enc_len) # sort by length of encoder sequence
 
         # Group the sorted Examples into batches, optionally shuffle the batches, and place in the batch queue.
         batches = []
-        for i in xrange(0, len(inputs), self._hps.batch_size):
+        for i in range(0, len(inputs), self._hps.batch_size):
           batches.append(inputs[i:i + self._hps.batch_size])
         if not self._single_pass:
           shuffle(batches)
@@ -363,7 +363,7 @@ def fill_batch_queue(self):
 
       else: # beam search decode mode
         ex = self._example_queue.get()
-        b = [ex for _ in xrange(self._hps.batch_size)]
+        b = [ex for _ in range(self._hps.batch_size)]
         self._batch_queue.put(Batch(b, self._hps, self._vocab))
 
 
@@ -373,14 +373,14 @@ def watch_threads(self):
       time.sleep(60)
       for idx,t in enumerate(self._example_q_threads):
         if not t.is_alive(): # if the thread is dead
-          tf.logging.error('Found example queue thread dead. Restarting.')
+          tf.compat.v1.logging.error('Found example queue thread dead. Restarting.')
           new_t = Thread(target=self.fill_example_queue)
           self._example_q_threads[idx] = new_t
           new_t.daemon = True
           new_t.start()
       for idx,t in enumerate(self._batch_q_threads):
         if not t.is_alive(): # if the thread is dead
-          tf.logging.error('Found batch queue thread dead. Restarting.')
+          tf.compat.v1.logging.error('Found batch queue thread dead. Restarting.')
           new_t = Thread(target=self.fill_batch_queue)
           self._batch_q_threads[idx] = new_t
           new_t.daemon = True
@@ -393,14 +393,14 @@ def text_generator(self, example_generator):
     Args:
       example_generator: a generator of tf.Examples from file. See data.example_generator"""
     while True:
-      e = example_generator.next() # e is a tf.Example
+      e = next(example_generator) # e is a tf.Example
       try:
-        article_text = e.features.feature['article'].bytes_list.value[0] # the article text was saved under the key 'article' in the data files
-        abstract_text = e.features.feature['abstract'].bytes_list.value[0] # the abstract text was saved under the key 'abstract' in the data files
+        article_text = e.features.feature['article'].bytes_list.value[0].decode('utf-8') # the article text was saved under the key 'article' in the data files
+        abstract_text = e.features.feature['abstract'].bytes_list.value[0].decode('utf-8') # the abstract text was saved under the key 'abstract' in the data files
       except ValueError:
-        tf.logging.error('Failed to get article or abstract from example')
+        tf.compat.v1.logging.error('Failed to get article or abstract from example')
         continue
       if len(article_text)==0: # See https://github.com/abisee/pointer-generator/issues/1
-        tf.logging.warning('Found an example with empty article text. Skipping it.')
+        tf.compat.v1.logging.warning('Found an example with empty article text. Skipping it.')
       else:
         yield (article_text, abstract_text)
diff --git a/beam_search.py b/beam_search.py
@@ -20,7 +20,7 @@
 import numpy as np
 import data
 
-FLAGS = tf.app.flags.FLAGS
+FLAGS = tf.compat.v1.app.flags.FLAGS
 
 class Hypothesis(object):
   """Class to represent a hypothesis during beam search. Holds all the information needed for the hypothesis."""
@@ -102,13 +102,13 @@ def run_beam_search(sess, model, vocab, batch):
                      attn_dists=[],
                      p_gens=[],
                      coverage=np.zeros([batch.enc_batch.shape[1]]) # zero vector of length attention_length
-                     ) for _ in xrange(FLAGS.beam_size)]
+                     ) for _ in range(FLAGS.beam_size)]
   results = [] # this will contain finished hypotheses (those that have emitted the [STOP] token)
 
   steps = 0
   while steps < FLAGS.max_dec_steps and len(results) < FLAGS.beam_size:
     latest_tokens = [h.latest_token for h in hyps] # latest token produced by each hypothesis
-    latest_tokens = [t if t in xrange(vocab.size()) else vocab.word2id(data.UNKNOWN_TOKEN) for t in latest_tokens] # change any in-article temporary OOV ids to [UNK] id, so that we can lookup word embeddings
+    latest_tokens = [t if t in range(vocab.size()) else vocab.word2id(data.UNKNOWN_TOKEN) for t in latest_tokens] # change any in-article temporary OOV ids to [UNK] id, so that we can lookup word embeddings
     states = [h.state for h in hyps] # list of current decoder states of the hypotheses
     prev_coverage = [h.coverage for h in hyps] # list of coverage vectors (or None)
 
@@ -123,9 +123,9 @@ def run_beam_search(sess, model, vocab, batch):
     # Extend each hypothesis and collect them all in all_hyps
     all_hyps = []
     num_orig_hyps = 1 if steps == 0 else len(hyps) # On the first step, we only had one original hypothesis (the initial hypothesis). On subsequent steps, all original hypotheses are distinct.
-    for i in xrange(num_orig_hyps):
+    for i in range(num_orig_hyps):
       h, new_state, attn_dist, p_gen, new_coverage_i = hyps[i], new_states[i], attn_dists[i], p_gens[i], new_coverage[i]  # take the ith hypothesis and new decoder state info
-      for j in xrange(FLAGS.beam_size * 2):  # for each of the top 2*beam_size hyps:
+      for j in range(FLAGS.beam_size * 2):  # for each of the top 2*beam_size hyps:
         # Extend the ith hypothesis with the jth option
         new_hyp = h.extend(token=topk_ids[i, j],
                            log_prob=topk_log_probs[i, j],