TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW

November 2023
Vol-9, Issue-1
Paper ID: 22082
ISSN: 2395-4396
Downloads: 104
Download PDF

Abstract & Details

Research Area
Computer Science
Keywords
NLP tokenization strategies
Abstract
Tokenization is a natural language processing (NLP) preprocessing technique that involves breakingdown a text into individual tokens, often words or subwords. It has been the subject of extensive research in recent years, leading to the creation of numerous innovative methods and tools. The state of the art in NLP tokenization approaches is thoroughly reviewed in this study. We start by outlining the various tokenization techniques, including word, subword, and character-level tokenization. The benefits and drawbacks of various tokenization strategies, including rule-based, statistical, and neural network-based techniques, are then covered. The performance of various tokenization techniques and libraries is then compared.We also look at current developments in tokenization research,such as the use of unsupervised techniques and contextual data. Finally, we list many challenges in tokenization. We wrap up by examining prospective possibilities for tokenization research in the future and their effects on the larger discipline of NLP.

Author Information

# Name Institute / Affiliation
1 Dr. Bhavesh M. Patel Department of Computer Science, H.N.G.University
2 Mohammed Sule Department of Computer Science, H.N.G.University

How to Cite

Use the following formats to cite this article in your research.

APA Style
Patel, Dr. Bhavesh M. & Sule, Mohammed (2023). TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW. International Journal of Advance Research and Innovative Ideas In Education, 9(1), 1873-1892.
MLA Style
Patel, Dr. Bhavesh M., and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal of Advance Research and Innovative Ideas In Education, vol. 9, no. 1, 2023, pp. 1873-1892.
IEEE Style
Dr. Bhavesh M. Patel and Mohammed Sule, "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW," International Journal of Advance Research and Innovative Ideas In Education, vol. 9, no. 1, pp. 1873-1892, 2023.
Vancouver Style
Patel Dr. Bhavesh M., Sule Mohammed. TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW. International Journal of Advance Research and Innovative Ideas In Education. 2023;9(1):1873-1892.
Harvard Style
Patel, Dr. Bhavesh M. & Sule, Mohammed (2023) 'TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW', International Journal of Advance Research and Innovative Ideas In Education, 9(1), pp. 1873-1892.
Chicago Style
Patel, Dr. Bhavesh M. and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal of Advance Research and Innovative Ideas In Education 9, no. 1 (2023): 1873-1892.
Turabian Style
Patel, Dr. Bhavesh M. and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal of Advance Research and Innovative Ideas In Education 9, no. 1 (2023): 1873-1892.

Export Citation

Related Research

PReFace : AI -Assisted Plausible Facial Restoration and Damage Reconstruction System
Syeda Arbeena Kausar et al. 2026 Computer Science and Engineering (Computer Vision, Deep Learning and Image Processing)
Download PDF
Customer Churn Prediction Using Machine Learning
Dr. Roshni Khlhe et al. 2026 Machine Learning, Artificial Intelligence, Data Analytics, and Customer Churn Prediction.
Download PDF
Automated web vulnerability scanner
Jitendra Sahu et al. 2026 Computer science
Download PDF
BLOOD BANK DEMAND PREDICTION SYSTEM: MACHINE LEARNINGBASED BLOOD DEMAND FORECASTING
Uday Dharmapal Wankhede et al. 2026 Computer Science & Engineering
Download PDF