Keywords
Adapter
Pre-trained language model
Submitted : 2024-05-21
Accepted : 2024-06-05
Published : 2024-06-20
Abstract
We present an approach to classify medical text at a sentence level automatically. Given the inherent complexity of medical text classification, we employ adapters based on pre-trained language models to extract information from medical text, facilitating more accurate classification while minimizing the number of trainable parameters. Extensive experiments conducted on various datasets demonstrate the effectiveness of our approach.
References
Alishahi A, Chrupa?a G, Linzen T, 2019, Analyzing and Interpreting Neural Networks for NLP: A Report on the First BlackboxNLP Workshop. Natural Language Engineering, 25(4): 543–557.
Mujtaba G, Shuib L, Idris N, et al., 2019, Clinical Text Classification Research Trends: Systematic Literature Review and Open Issues. Expert Systems with Applications, 116: 494–520.
Kaurova O, Alexandrov M, Blanco X, 2011, Classification of Free Text Clinical Narratives (Short Review). Business and Engineering Applications of Intelligent and Information Systems, 2011: 124–135.
Hoang N, Patrick J, 2016, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, August 13–17, 2016: Text Mining in Clinical Domain: Dealing with Noise. San Francisco, 549–558.
Brown TB, Mann B, Ryder N, et al., 2020, Language Models are Few-Shot Learners. arXiv. https://doi.org/10.48550/arXiv.2005.14165
Radford A, Wu J, Child R, et al., 2019, Language Models are Unsupervised Multitask Learners. OpenAI blog, 1(8): 9.
Devlin J, Chang M-W, Lee K, et al., 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2–7, 2019: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Minneapolis, 4171–4186. Association for Computational Linguistics.
Houlsby N, Giurgiu A, Jastrzebski S, et al., 2019, Parameter-Efficient Transfer Learning for NLP. International Conference on Machine Learning, 2790–2799.
Hu E J, Shen Y, Wallis P, et al., 2021, LoRA: Low-rank Adaptation of Large Language Models. arXiv. https://doi.org/10.48550/arXiv.2106.09685
Zhang Q, Chen M, Bukharin A, et al., 2022, AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. arXiv. https://doi.org/10.48550/arXiv.2303.10512
Dettmers T, Pagnoni A, Holtzman A, et al., 2024, QLoRA: Efficient Finetuning of Quantized LLMs. arXiv. https://doi.org/10.48550/arXiv.2305.14314