Call for Papers:Vol.12 Issue.2

Submission
Last date		 28-Apr-2026
Acceptance Status In One Day
Paper Publish In Two Days
Submit ManuScript

News & Updates

Submit Article

Dear Authors, Article publish in our journal for Volume-12,Issue-2. For article submission on below link: Submit Manuscript


Join As Board

Dear Reviewer, You can join our Reviewer team without given any charges in our journal. Submit Details on below link: Join As Board


Paper Publication Charges

IJARIIE APP
Download Android App

For Authors

Archives

IJARIIE Board

Downloads

Android App

Download IJARIIE APP

Authors

Title: :  TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW
PaperId: :  22082
Published in:   International Journal Of Advance Research And Innovative Ideas In Education
Publisher:   IJARIIE
e-ISSN:   2395-4396
Volume/Issue:    Volume 9 Issue 1 2023
DUI:    16.0415/IJARIIE-22082
Licence: :   IJARIIE is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Author NameAuthor Institute
Dr. Bhavesh M. PatelDepartment of Computer Science, H.N.G.University
Mohammed SuleDepartment of Computer Science, H.N.G.University

Abstract

Computer Science
NLP, tokenization strategies
Tokenization is a natural language processing (NLP) preprocessing technique that involves breakingdown a text into individual tokens, often words or subwords. It has been the subject of extensive research in recent years, leading to the creation of numerous innovative methods and tools. The state of the art in NLP tokenization approaches is thoroughly reviewed in this study. We start by outlining the various tokenization techniques, including word, subword, and character-level tokenization. The benefits and drawbacks of various tokenization strategies, including rule-based, statistical, and neural network-based techniques, are then covered. The performance of various tokenization techniques and libraries is then compared.We also look at current developments in tokenization research,such as the use of unsupervised techniques and contextual data. Finally, we list many challenges in tokenization. We wrap up by examining prospective possibilities for tokenization research in the future and their effects on the larger discipline of NLP.

Citations

Copy and paste a formatted citation or use one of the links to import into a bibliography manager and reference.

IJARIIE Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW" International Journal Of Advance Research And Innovative Ideas In Education Volume 9 Issue 1 2023 Page 1873-1892
MLA Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal Of Advance Research And Innovative Ideas In Education 9.1(2023) : 1873-1892.
APA Dr. Bhavesh M. Patel, & Mohammed Sule. (2023). TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW. International Journal Of Advance Research And Innovative Ideas In Education, 9(1), 1873-1892.
Chicago Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal Of Advance Research And Innovative Ideas In Education 9, no. 1 (2023) : 1873-1892.
Oxford Dr. Bhavesh M. Patel, and Mohammed Sule. 'TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW', International Journal Of Advance Research And Innovative Ideas In Education, vol. 9, no. 1, 2023, p. 1873-1892. Available from IJARIIE, https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf (Accessed : 20 August 2025).
Harvard Dr. Bhavesh M. Patel, and Mohammed Sule. (2023) 'TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW', International Journal Of Advance Research And Innovative Ideas In Education, 9(1), pp. 1873-1892IJARIIE [Online]. Available at: https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf (Accessed : 20 August 2025)
IEEE Dr. Bhavesh M. Patel, and Mohammed Sule, "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW," International Journal Of Advance Research And Innovative Ideas In Education, vol. 9, no. 1, pp. 1873-1892, Jan-Feb 2023. [Online]. Available: https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf [Accessed : 20 August 2025].
Turabian Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal Of Advance Research And Innovative Ideas In Education [Online]. volume 9 number 1 (20 August 2025).
Vancouver Dr. Bhavesh M. Patel, and Mohammed Sule. TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW. International Journal Of Advance Research And Innovative Ideas In Education [Internet]. 2023 [Cited : 20 August 2025]; 9(1) : 1873-1892. Available from: https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf
BibTex EndNote RefMan RefWorks

Number Of Downloads


Last download on 8/20/2025 3:05:31 PM

Save in Google Drive

Similar-Paper

TitleArea of ResearchAuther NameAction
A SURVEY ON AI POWERED SELF HEALING CODE DEBUGGERComputer EngineeringNandana Shibu Download
MULTIPLE DISEASE PREDICTION AND DRUG RECOMMENDATION SYSTEM USING MACHINE LEARNINGCOMPUTER ENGINEERING DR. J. AMUTHARAJ Download
SOLAR TRACKING SYSTEM WITH WEATHER FORECASTING AND AUTOMATED CLEANING SYSTEMArtificial Intelligence Electrical and Electronics EngineeringVanarasan S Download
ASD PredictionComputer scienceAbhilasha M Download
Music Recommendation based on Face Emotion Using Artificial Intelligence and Machine LearningInformation Science EngineeringDr. Sreenivasa Murthy V Download
SMART TRAFFIC AI MONITORING SYSTEM-AI/MLINFORMATION SCIENCE AND ENGINEERING PRIYANKA MT Download
SHADOW THE WEB BROWSERComputer Science and EngineeringLakshmi Narayan S Download
SECUHIRE – AI-POWERED INTERVIEW DETECTION SYSTEMComputer Engineering Tharun R Download
Early Detection Of Electrical Fault Line Using Artificial IntelligenceElectrical Engineering , Computer Science , Artificial IntelligenceRakshitha R Download
Real Time Object Detection Tracking using YOLO and Deep SORTInformation science and engineeringNandan M R Download
An AI-Integrated Intelligent Health Advisory System with Machine-Learning-Based Ayurvedic Formulation RecommendationInformation Science EngineeringSrinidhi G Download
Blockchain-Based Decentralized Medical Health Management System using Smart Contracts and IPFS Information Science and EngineeringVarun V Download
Precision Agriculture Using Machine Learning and IOTComputer Science and EngineeringRamesh B E Download
A Secure Blockchain Based Voting SystemComputer Science and EngineeringSagar K R Download
Chatbot based helpdesk for Government employee and departmentsComputer science & Engineering(Cyber Security Engineerting)Dr. Shanthi S Download
12