An experimental Thai-specialized LLM created through continued pretraining of Qwen2.5-7B using LoRA. A research project exploring affordable Thai language model adaptation.
Model Specifications
Base model
Qwen2.5-7B
Parameters
7.7B (bfloat16)
Trainable params
~2% (LoRA)
Training cost
< $10
LoRA config
r=64, alpha=128
Training data
3.7 GB+ Thai text
Training Data
Processed with SentencePiece tokenization and MinHash deduplication.
Evaluation
ThaiQA
Thai question answering
XNLI-th
Natural language inference
Wisesight
Sentiment analysis
Perplexity
Language modeling quality
Hardware Requirements
Roadmap
The pipeline is reproducible and could scale to 14B and 72B base models. This is exploratory — no guarantees on timeline or results.