Education
Urdu Transformer Chatbot - Custom NLP & Generative AI
A custom Urdu conversational chatbot built from scratch using PyTorch and an encoder–decoder Transformer architecture. The model uses multi-head attention, positional encoding, and token-by-token decoding to generate context-aware Urdu responses without relying on pretrained language models. A Streamlit or Gradio interface enables interactive conversations with Urdu text input and right-to-left rendering.
The challenge
Building conversational AI for Urdu presents challenges related to language normalization, vocabulary construction, tokenization, and generating coherent responses in a language with right-to-left writing. Using a pretrained multilingual model can simplify development, but it does not meet the goal of exploring a custom Transformer trained from scratch. The challenge was to build an end-to-end conversational pipeline that learns from Urdu dialogue data and supports interactive text generation.
Context
This project explores Natural Language Processing (NLP) and generative AI through a custom encoder–decoder Transformer implemented in PyTorch. The model is trained on an Urdu conversational dataset containing approximately 20,000 examples, with training, validation, and test splits. The project covers text preprocessing, vocabulary creation, model training, evaluation, and chatbot deployment through a lightweight interactive interface.
The solution
Developed a custom Transformer-based conversational model with an encoder, decoder, multi-head attention, positional encoding, and feed-forward layers. Urdu text is normalized and tokenized before being converted into numerical sequences for training. Teacher forcing is used during training, and the trained model generates responses token by token during inference. A Streamlit or Gradio frontend provides Urdu input, generated replies, conversation history, and configurable decoding strategies where implemented.
Design
The system is organized into distinct stages: Urdu text preprocessing, vocabulary and sequence preparation, Transformer encoding and decoding, response generation, and user interaction. Positional encoding provides sequence-order information, while attention mechanisms allow the model to learn relationships between tokens. The interface is designed for readable Urdu conversations, including right-to-left text presentation.
Architecture
- Framework: PyTorch
- Model: Custom encoder–decoder Transformer
- Attention: Multi-head attention
- Sequence Representation: Positional encoding and token embeddings
- Network Components: Encoder and decoder layers with feed-forward networks
- Training Method: Teacher forcing
- Optimizer: Adam, if used in the implementation
- Dataset: Urdu Dataset 20,000 from Kaggle
- Evaluation: BLEU, ROUGE-L, chrF, and perplexity, where implemented
- Interface: Streamlit or Gradio, depending on the actual application
Development
The development process includes Urdu text normalization, vocabulary construction, dataset splitting, sequence preparation, Transformer implementation, and model training. The model is trained using paired conversational examples, with validation data used to monitor performance and select a checkpoint if that selection process is implemented. During inference, the model generates a response sequentially. The interactive frontend connects user input to the inference pipeline and displays generated Urdu responses.
Results
The project delivers a custom Transformer-based Urdu chatbot architecture and an interactive workflow for generating conversational responses. It demonstrates the implementation of core NLP concepts, including attention, positional encoding, sequence-to-sequence learning, and autoregressive decoding. Quantitative performance should be reported using actual evaluation results, and example conversations should be selected from genuine model outputs rather than illustrative examples alone.