Title page for ETD etd-09042008-133715

Document Type Master's Dissertation
Author Botha, Gerrit Reinier
Email gerrit.botha@gmail.com
URN etd-09042008-133715
Document Title Text-based language identification for the South African languages
Degree MEng
Department Electrical, Electronic and Computer Engineering
Advisor Name Title
Prof E Barnard Committee Chair
  • Nave Bayesian classification
  • support vector machine
  • n-gram statistics
  • text-based language identification
  • difference-in-frequency classification
Date 2008-04-09
Availability unrestricted

We investigate the factors that determine the performance of text-based language identification, with a particular focus on the 11 official languages of South Africa. Our study uses n-gram statistics as features for classification. In particular, we compare support vector machines, Nave Bayesian and difference-in-frequency classifiers on different amounts of input text and various values of n, for different amounts of training data. For a fixed value of n the support vector machines generally outperforms the other classifiers, but the simpler classifiers are able to handle larger values of n. The additional computational complexity of training the support vector machine classifier may not be justified in light of importance of using a large value of n, except possibly for small sizes of the input window when limited training data is available.

We find that it is more difficult to discriminate languages within language families then those across families. The accuracy on small input strings is low due to this reason, but for input strings of 100 characters or more there is only a slight confusion within families and accuracies as high as 99.4% are achieved. For the smallest input strings studied here, which consist of 15 characters, the best accuracy achieved is only 83%, but when the languages in different families are grouped together, this corresponds to a usable 95.1% accuracy.

The relationship between the amount of training data and the accuracy achieved is found to depend on the window size for the largest window (300 characters) about 400 000 characters are sufficient to achieve close-to-optimal accuracy, whereas improvements in accuracy are found even beyond 1.6 million characters of training data.

Finally, we show that the confusions between the different languages in our set can be used to derive informative graphical representations of the relationships between the languages.

University of Pretoria 2008

  Filename       Size       Approximate Download Time (Hours:Minutes:Seconds) 
 28.8 Modem   56K Modem   ISDN (64 Kb)   ISDN (128 Kb)   Higher-speed Access 
  dissertation.pdf 477.52 Kb 00:02:12 00:01:08 00:00:59 00:00:29 00:00:02

Browse All Available ETDs by ( Author | Department )

If you have more questions or technical problems, please Contact UPeTD.