Korean(한국어)

NLU-Benchmark support natural language understanding experimental code for Close Domain chatbot or Task-Oriented artificial intelligence assistant system. Experiments provided by NLU-Benchmark include:

• Intent Classification : classify intent of the utterance or sentence
• Entity Recognition : extract entity information from utterance or sentence

NLU-Benchmark is implemented based on Huggingface and PyTorch Lightning, and supports experiments with various language models.

• torch==1.9.0
• pytorch-lightning==1.4.1
• transformers>=4.7.0
• pandas
• sentencepiece
• scikit-learn
• seqeval

NLU-Benchmark supports the following datasets.

• After downloading datasets from above links, create a folder having each dataset name under the data folder and save downloaded datasets there, e.g., in case of ATIS dataset, save train, valid, test folders in data/atis.

• The following files are stored in the folders downloaded through above links.

  • seq.in : utterance sentences are stored.
  • label : intents about each sentences in seq.in are saved.
  • seq.out : sequence labels for each sentences in seq.in are saved. theses sequence labels follow BIO Format.

To perform intent classification and entity recognition experiments via NLU-Benchmark, you should build label dictionaries about each dataset.

If you run the following command, intent.vocab. a dictionary about intent labels, and entity.vocab, a dictionary about entity labels, are created.

• --data_name : dataset name to create label dictionary, e.g., atis, snips

python data/build_label_vocabs.py --data_name {$DATA_NAME}

Run intent classification experiment by the following command:

• --model_type : type of model, e.g., bert, albert
• --model_name : checkpoint name of language model you wanna run experiment, e.g., bert-base-cased, albert-base-v2
• --data_name : dataset name to use training and evaluating, e.g., atis, snips
• --do_train : set training mode
• --do_eval : set evaluating mode
• --num_train_epochs : number of epochs in training
• --gpu_id : GPU id to be used when performing experiment
• --batch_size : batch size at training

python run_intent_classification.py --model_type {$MODEL_TYPE} \
                                    --model_name {$MODEL_NAME} \
                                    --data_name {$DATA_NAME} \
                                    --do_train \
                                    --do_eval \
                                    --num_train_epochs 20 \
                                    --gpu_id 0 \
                                    --batch_size 32

Run entity recognition experiment by the following command:

• --model_type : type of model, e.g., bert, albert
• --model_name : checkpoint name of language model you wanna run experiment, e.g., bert-base-cased, albert-base-v2
• --data_name : dataset name to use training and evaluating, e.g., atis, snips
• --do_train : set training mode
• --do_eval : set evaluating mode
• --num_train_epochs : number of epochs in training
• --gpu_id : GPU id to be used when performing experiment
• --batch_size : batch size at training

python run_entity_recognition.py --model_type {$MODEL_TYPE} \
                                 --model_name {$MODEL_NAME} \
                                 --data_name {$DATA_NAME} \
                                 --do_train \
                                 --do_eval \
                                 --num_train_epochs 20 \
                                 --gpu_id 0 \
                                 --batch_size 32

Hyper parameters for experiments are as follows:

• If you look at the experimental results below, you can see that performance of the large size models have low performances. I think this problem can be solved by increasing the epoch and batch size.

• README (Version. EN)
• add CRF layers

• yuanxiaosc/BERT-for-Sequence-Labeling-and-Text-Classification
• huggingface/transformers
• monologg/KoBERT-NER

If you have any additional questions, please register an issue in this repository or contact sehunhu5247@gmail.com.