Call for Papers:Vol.11 Issue.3

Submission
Last date		 28-Jun-2025
Acceptance Status In One Day
Paper Publish In Two Days
Submit ManuScript

News & Updates

Submit Article

Dear Authors, Article publish in our journal for Volume-11,Issue-3. For article submission on below link: Submit Manuscript


Join As Board

Dear Reviewer, You can join our Reviewer team without given any charges in our journal. Submit Details on below link: Join As Board


Paper Publication Charges

IJARIIE APP
Download Android App

For Authors

Archives

IJARIIE Board

Downloads

Android App

Download IJARIIE APP

Authors

Title: :  TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW
PaperId: :  22082
Published in:   International Journal Of Advance Research And Innovative Ideas In Education
Publisher:   IJARIIE
e-ISSN:   2395-4396
Volume/Issue:    Volume 9 Issue 1 2023
DUI:    16.0415/IJARIIE-22082
Licence: :   IJARIIE is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Author NameAuthor Institute
Dr. Bhavesh M. PatelDepartment of Computer Science, H.N.G.University
Mohammed SuleDepartment of Computer Science, H.N.G.University

Abstract

Computer Science
NLP, tokenization strategies
Tokenization is a natural language processing (NLP) preprocessing technique that involves breakingdown a text into individual tokens, often words or subwords. It has been the subject of extensive research in recent years, leading to the creation of numerous innovative methods and tools. The state of the art in NLP tokenization approaches is thoroughly reviewed in this study. We start by outlining the various tokenization techniques, including word, subword, and character-level tokenization. The benefits and drawbacks of various tokenization strategies, including rule-based, statistical, and neural network-based techniques, are then covered. The performance of various tokenization techniques and libraries is then compared.We also look at current developments in tokenization research,such as the use of unsupervised techniques and contextual data. Finally, we list many challenges in tokenization. We wrap up by examining prospective possibilities for tokenization research in the future and their effects on the larger discipline of NLP.

Citations

Copy and paste a formatted citation or use one of the links to import into a bibliography manager and reference.

IJARIIE Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW" International Journal Of Advance Research And Innovative Ideas In Education Volume 9 Issue 1 2023 Page 1873-1892
MLA Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal Of Advance Research And Innovative Ideas In Education 9.1(2023) : 1873-1892.
APA Dr. Bhavesh M. Patel, & Mohammed Sule. (2023). TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW. International Journal Of Advance Research And Innovative Ideas In Education, 9(1), 1873-1892.
Chicago Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal Of Advance Research And Innovative Ideas In Education 9, no. 1 (2023) : 1873-1892.
Oxford Dr. Bhavesh M. Patel, and Mohammed Sule. 'TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW', International Journal Of Advance Research And Innovative Ideas In Education, vol. 9, no. 1, 2023, p. 1873-1892. Available from IJARIIE, https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf (Accessed : 29 May 2025).
Harvard Dr. Bhavesh M. Patel, and Mohammed Sule. (2023) 'TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW', International Journal Of Advance Research And Innovative Ideas In Education, 9(1), pp. 1873-1892IJARIIE [Online]. Available at: https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf (Accessed : 29 May 2025)
IEEE Dr. Bhavesh M. Patel, and Mohammed Sule, "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW," International Journal Of Advance Research And Innovative Ideas In Education, vol. 9, no. 1, pp. 1873-1892, Jan-Feb 2023. [Online]. Available: https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf [Accessed : 29 May 2025].
Turabian Dr. Bhavesh M. Patel, and Mohammed Sule. "TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW." International Journal Of Advance Research And Innovative Ideas In Education [Online]. volume 9 number 1 (29 May 2025).
Vancouver Dr. Bhavesh M. Patel, and Mohammed Sule. TOKENIZATION TECHNIQUES IN NLP:A COMPREHENSIVE REVIEW. International Journal Of Advance Research And Innovative Ideas In Education [Internet]. 2023 [Cited : 29 May 2025]; 9(1) : 1873-1892. Available from: https://ijariie.com/AdminUploadPdf/TOKENIZATION_TECHNIQUES_IN_NLP_A_COMPREHENSIVE_REVIEW_ijariie22082.pdf
BibTex EndNote RefMan RefWorks

Number Of Downloads


Last download on 5/29/2025 12:29:53 PM

Save in Google Drive

Similar-Paper

TitleArea of ResearchAuther NameAction
Detection of Phishing Website Using Gradient Boosting AlgorithmComputer Science and EngineeringYAWALKAR PRASAD PRAMOD Download
Property Dealing WebComputer Science EngineeringYash Chaudhari Download
Reinforcement Learning for the Evolution of Antimicrobial Nano formulationsmachine learningMadhusudan Download
SecuraVault: A secured blockchain based cloud storage systemComputer Engineering Anshika Jaiswal Download
Autoimmune Disease Detection in women Using Machine Learning Approachcomputer science EngineeringJ. L. V. S. Download
Medicine Overdose Detection System Using Machine LearningComputer Science EngineeringDr.Somashekhar B M Download
Home Price Prediction Using Machine LearningComputer Science & Engineering G Tushar Download
Heat diseases prediction using machine learningComputer EngineeringProf. Meghashree M B Download
GRIDSHIELD AIComputer EngineeringDr. Archana B Download
Diabetic Retinopathy Detection with AI InsightsComputer EngineeringJay Mahesh Gurav Download
Personalized Fitness Segmentation with Actionable InsightsMachine LearningAnju Tiwari Download
Sentiment-Based Machine Learning Approach for Mapping Citizen ProblemsComputer science and EngineeringDr. Madhu B K Download
Deligro: A Dual-Purpose Web Platform for Food Ordering and Leftover Food Donation ManagementComputer Science EngineeringRakesh Reddy K Download
AI-RATIONMITRA: SMART PUBLIC DISTRIBUTION THROUGH AI AND IOTComputer Science and EngineeringDr. Madhu B K Download
Driver Drowsiness Detection System Using OpenCV And IOTComputer Science And EngineeringAkshay Amrutkar Download
12