From Limited Contexts to Rich Data

The efficacy of LLMs like GPT largely depends on the quality and diversity of their training datasets. However, for languages such as Twi, there is a noticeable scarcity of high-quality, diverse datasets. Preliminary research showed that datasets are often derived from limited contexts, such as religious texts, which do not provide the breadth for developing robust language models.

This hinders representation in multilingual LLMs and the development of downstream tools for education, healthcare, and civic tech.

Data Acquisition and Preprocessing

We source raw textual data from diverse domains, including literature, health, social media, and news articles. Texts are cleaned and aligned at the sentence level. To address domain imbalance, we supplement underrepresented categories with synthetically generated sentence pairs that simulate natural use cases.

Translation

Initial translations are generated using large language models (LLMs) such as NLLB-200 and Gemini 1.5, prompted with example-rich templates to improve fluency and contextual relevance. Google Translate will also be used to aid in the translation quality.

Expert Validation

Selected samples undergo final review via an interactive interface where linguists and fluent speakers evaluate translations based on clarity, correctness, and cultural fit.

• High-quality translations:
  At least 80% of validated sentence pairs are expected to score above 4 out of 5 in human adequacy and fluency evaluations.
• Improved translation performance:
  Fine-tuned multilingual models are projected to achieve +3 to +6 BLEU improvement over JW300 on general-domain Twi-English tasks.
• Domain balance:
  The final dataset will span ≥5 distinct domains, with no more than 30% from any single category.
• Expert agreement:
  Human evaluation via our interface is expected to yield a Cohen's K≥0.75.

The dataset will directly support the development of machine translation systems, voice assistants, and educational tools tailored to Twi speakers. In healthcare, it can improve patient-doctor communication through accurate translation of medical instructions and consent forms.

In governance, it can enable the delivery of public service announcements and legal information in a language that over 9 million Ghanaians understand. By grounding NLP systems in local linguistic realities, our work lays the foundation for inclusive AI in West Africa.

Our approach addresses key challenges in low-resource language development, namely, data scarcity, limited contextual diversity, and lack of verification pipelines. By integrating human expertise with automated translation and validation tools, we lay the groundwork for scalable, high-quality dataset creation.