IIT (BHU) System for Indo-Aryan Language Identification (ILI) at VarDial 2018

Abstract

Text Language Identification is a Natural Language Processing (NLP) task of identifying and recognizing a given language out of many different languages from a piece of text. In the present scenario, this task has become the basis and beginning step of various other NLP tasks, for example, Machine Translation, improving search relevance for a multilingual query, processing code-switched data etc. The biggest limitation of many Language Identification systems is not being able to differentiate between closely related languages. This paper describes our submission to the ILI 2018 shared-task, which includes the identification of 5 closely related Indo-Aryan languages. We used a word-level LSTM (Long Short-Term Memory) model, a specific type of Recurrent Neural Network model, for this task. Given a sentence, our model embeds each word of the sentence and convert into its trainable word embedding, feeds them into our LSTM network and finally predict the language. We obtained an F1 macro score of 0.836, ranking 5th in the task. © COLING 2018.All right reserved.