The Art of Tokenization: Breaking Down Text for AI
Demystifying NLP: From Text to Embeddings
I’m starting a series called “Demystifying NLP: From Text to Embeddings” on Towards Data Science. This series will provide an overview of text representation in modern NLP, progressing from basic concepts to more advanced techniques, and will include practical, hands-on examples.
In my first article, “The Art of Tokenization: Breaking Down Text for AI”, I explain how tokenization works in NLP and discuss various tokenization algorithms, including Byte-Pair Encoding.
Murilo Gustineli
Senior AI Software Engineer at Intel, Computer Science at Georgia Tech
My research interests include deep learning, computer vision, and NLP