03. Local Language Support

Natural language processing in African contexts requires addressing both orthographic variation and dialectal diversity. Yoruba, Hausa, and Igbo each present unique challenges that differ substantially from English or European language processing. Understanding these challenges enables informed decisions about model selection and deployment architecture.

Text normalization presents immediate difficulties. Yoruba uses diacritical marks (like ọ́, ẹ́, á) that frequently get lost in copy-paste operations and SMS transmission. Hausa employs both Latin and Arabic scripts (ajami), with significant population using the Arabic variant for religious and cultural content. Igbo tonal marks (like ị, ọ, ụ) serve grammatical functions that impact meaning when omitted.

Dialectal variation complicates corpus development. Yoruba contains distinct dialects (Yoruba proper, Egba, Ijesha, Ekiti) with vocabulary and tonal differences. Hausa dialects stratify by region and social context, with significant divergence between northern and southern varieties. Igbo shows massive variation across communities, leading some linguists to describe it as a dialect chain rather than a single language.

Vocabulary domain gaps create practical challenges. Existing NLP resources focus heavily on religious texts (especially for Hausa), Bible translations, and Wikipedia content. Agricultural, medical, and commercial vocabularies remain underrepresented, creating poor performance in these critical domains. Domain adaptation through targeted corpus collection becomes essential for production deployments.

# Text normalization for Yoruba diacritics
import unicodedata
import re

class YorubaNormalizer:
    # Yoruba diacritical marks
    YORUBA_COMBINING_MARKS = {
        '\u0301',  # acute accent
        '\u0300',  # grave accent
        '\u0308',  # diaeresis
        '\u0323',  # dot below
        '\u0304',  # macron
    }
    
    def __init__(self, preserve_tones: bool = True):
        self.preserve_tones = preserve_tones
    
    def normalize(self, text: str) -> str:
        # NFC normalization converts to composed form
        text = unicodedata.normalize('NFC', text)
        
        if not self.preserve_tones:
            # Remove tone marks while preserving base characters
            return ''.join(
                c for c in text 
                if c not in self.YORUBA_COMBINING_MARKS
            )
        
        # Remove extra whitespace
        text = re.sub(r'\s+', ' ', text)
        
        # Common encoding errors
        replacements = {
            'ẹ́': 'ẹ́',  # Verify correct form preserved
            'ọ́': 'ọ́',
            'ị': 'ị',
        }
        
        return text.strip()
    
    def detect_encoding_issues(self, text: str) -> list[dict]:
        """Identify likely encoding corruption in Yoruba text."""
        issues = []
        
        # Check for character removal that changes meaning
        problematic_pairs = [
            ('e', 'ẹ'), ('o', 'ọ'), ('i', 'ị'), ('u', 'ụ')
        ]
        
        for pos, char in enumerate(text):
            if char in [p[0] for p in problematic_pairs]:
                # Check if tonal context suggests missing diacritic
                context = text[max(0, pos-2):pos+3]
                if any(diac in context for diac in '̥́̀'):
                    issues.append({
                        'position': pos,
                        'char': char,
                        'context': context,
                        'severity': 'warning'
                    })
        
        return issues

# Usage
normalizer = YorubaNormalizer(preserve_tones=True)
corrupted = "E kaaro, bawo ni o se n san?"
corrected = normalizer.normalize(corrupted)
issues = normalizer.detect_encoding_issues(corrupted)

Evaluation metrics require localization beyond standard BLEU and accuracy scores. User satisfaction assessment must involve native speakers evaluating output quality. Error analysis should categorize failures by type: orthographic normalization, dialectal variation, domain shift, or cultural inappropriate responses.