Lexifyd - Blog by Baskaran Sankaran

Character Frequency Analysis for Malayalam

Sat, 31 May 2025 01:22:19 +0530

As part of our efforts to build optimized, intuitive keyboard layouts for Indian languages, we’ve conducted detailed character frequency analysis for Malayalam. This post presents our findings—showing which characters, vowels, and consonants occur most frequently in real-world usage based on large Malayalam dataset—and briefly discusses how this data influenced the keyboard design.

For an explanation of our overall keyboard design methodology, including the rationale behind layout decisions, please refer to our Tamil keyboard design post: Designing a New Input Method for Tamil

Dataset Used for Frequency Analysis

To ensure broad linguistic coverage and reliability, we used Malayalam corpora by combining text data from multiple sources, including:

IndicCorp
Swathantra Malayalam corpus
Wikipedia Malayalam corpus
News articles, blogs, etc.

Here is the high-level statistics of the corpora used in our analysis:

Total words: 1.699 billion
Unique words: 25.345 million

This large and diverse dataset gives us a realistic view of character usage in modern written Malayalam.

Overall Character Frequency Heatmap

This heatmap illustrates the relative frequency of Malayalam characters based on their usage across a dataset. The top row represents the frequency of vowels, while the last column shows the frequency of pure consonants. The remaining cells capture the usage of consonant-vowel combinations. Color intensity indicates frequency, with darker shades signifying higher usage and lighter shades indicating lower frequency.

Characters such as അ (a), മ (ma), ന (na), and ക (ka) appear with consistently dark shades, across their rows, indicating they are among the most frequently used. These are foundational phonemes in Malayalam and are common across both spoken and written forms. Characters like പ (pa), ത (tha), and ല (la) show moderate frequency across their respective row, suggesting they are contextually important but not as dominant. Their usage may vary depending on the domain (e.g., literary vs. conversational text).

Vowel Frequency Heatmap

To gain deeper insights, we examined the frequency of Malayalam vowels both in their standalone and combined consonant-vowel forms. This analysis was derived by summing the columns of our overall character frequency chart, offering a clear view of vowel usage patterns. The results are visualized in the chart below.

Malayalam has a rich vowel system, but a handful of vowels—especially അ, ഇ, ഉ, and എ—occur most frequently. Further as expected, the shorter vowels are more frequent than their longer versions. Note that these counts include the usage frequency of vowels as well as the vowel sign glyphs in their consonant vowel forms. The last cell in the heat map actually refers to the frequency of the Chandrakala (Virama) character.

Consonant Frequency Heatmap

In addition to vowels, we also analyzed the frequency of Malayalam consonants. This was done by summing the rows of the overall character frequency chart, which highlights how often each consonant appears across different vowel combinations. The resulting data provides a clearer picture of consonant usage patterns in the language.

Among consonants, we can see a clear dominance of ക്, ന്, ത് യ്, and ര്. These findings informed our decisions in distributing consonants across the keyboard layout for minimizing the finger movements during typing. In line with the InScript keyboard layout convention, each aspirated consonant was assigned to a specific key, while its corresponding unaspirated counterpart was placed in the corresponding shift position.

Chillaksharam Frequency Heatmap

We also analyzed the frequency of chillaksharam (ചില്ലക്ഷരം)—the special consonant forms used at the end of syllables in Malayalam, such as ൽ, ൻ, ൾ, and ൿ. These characters are essential for accurate representation of the language and are frequently used in written text.

Following our keyboard design principles, each chillaksharam is placed in the Alt + Shift position of the corresponding base consonant key. For example, the character ൿ is mapped to the Alt + Shift position of the ക് key. Since mobile keyboards typically do not include an Alt key, these chillaksharam characters are made accessible through long-press gestures on their respective base keys. So, long-pressing the ക് key on a mobile keyboard would reveal ൿ, ensuring consistency and ease of access across platforms.

Impact on Keyboard Layout Design

Whether you're using a Malayalam phonetic keyboard or a standard Malayalam keyboard, these insights can guide better input method development. As part of our design, we excluded individual vowel signs and instead generate them dynamically from consonant-vowel combinations. This approach reduces complexity and prevents invalid character sequences.

These frequency trends helps in designing the layout for our Malayalam keyboard. High-frequency characters were assigned to primary home row or easy index finger positions, ensuring reduced typing effort and faster input for users. Low-frequency characters were positioned in secondary or long-press locations. As part of our design, we excluded individual vowel signs and instead generate them dynamically from consonant-vowel combinations. This approach reduces complexity and prevents invalid character sequences.

You can explore our optimized keyboard layout through the Varta Keyboard apps, available on Android and iOS, as well as through browser extensions for Chrome, Edge, and Safari.

Explore Frequency Analyses in Other Languages

We’ve performed similar analysis for other Indian languages as well. Explore them below:

Character Frequency Analysis for Hindi
Character Frequency Analysis for Kannada
Character Frequency Analysis for Tamil (includes design principles)
Character Frequency Analysis for Telugu

Character Frequency Analysis for Hindi

Fri, 30 May 2025 16:41:08 +0530

As part of our broader initiative to develop optimized and user-friendly keyboard layouts for various Indic languages, this blog post focusses on our efforts in Hindi. We start with a detailed character frequency analysis for Hindi. Using large-scale monolingual Hindi datasets collected from the web, this study focused on understanding the usage patterns of Hindi characters in real-world text. The insights gained from this analysis played a key role in designing a more intuitive and efficient Hindi keyboard layout, aimed at enhancing typing speed and improving text prediction capabilities on mobile platforms.

Dataset Overview

We collected multiple monolingual Hindi datasets from the web, including the following key corpora.

IITB Hindi Monolingual corpus
IndicCorp
Leipzig Hindi Datasets
Lindat HindEnCorp 0.5
IndicNLP News Articles
Samanantar v0.3 (En-Indic; Indic-Indic)
Classificataion Datasets (iNLTK, BBC Articles etc.)

After thorough data pre-processing and noise removal, the final dataset comprised approximately 8.92 billion total tokens and around 13.1 million unique tokens. Our character frequency analysis was conducted on this extensive and diverse dataset, providing a robust statistical foundation for our findings.

Detailed Character Frequency Analysis

We began by analyzing the frequency of all the characters in the Hindi alphabet, depicted as a heatmap as a matrix with vowels and consonants shown along the row and column axes respectively. The most frequenct characters are marged in darker colours, while the less frequent ones in progressively lighter shades.

Interestingly, unlike in Dravidian languages or Marathi, Hindi displayed a unique pattern: the vowel ए appeared more frequently than the alphabet-initial vowel अ in its standalone form. However, when it came to consonant-vowel combinations, the अ vowel was more dominant than ए, indicating a distinct usage trend in Hindi text.

Vowel Frequency Analysis

We then analyzed the frequency of Hindi vowels, both in their standalone form (e.g., अ, इ, उ) and when combined with consonants (e.g., का, कि, कु). This was done by summing the columns of our overall character frequency chart. The resulting data highlights which vowel sounds are most commonly used in Hindi text, offering valuable insights for layout prioritization and predictive typing.

The top three most frequently used vowels in consonant-vowel (CV) combinations are अ (a), आ (aa), and ए (e). These vowels form the core of Hindi phonetic structure and are heavily represented across the language. The virama (aka halant) sign (्), used to suppress the inherent vowel and form conjunct consonants, appears 3082.36M times (last cell in the heatmap) —making it the fourth most frequently used glyph in the dataset. This is a notable finding, especially when compared to Dravidian languages, where the virama is often as dominant as the अ CV form. In Hindi, while still highly frequent, it plays a slightly less central role in character composition. Vowels like इ (i), ई (ii), ओ (o), and अं (aṃ) also show significant usage, each contributing meaningfully to the overall character distribution. Characters such as ऋ (vR), औ (au), अः (aḥ), and अँ (aṅ) appear far less frequently, reflecting their more limited use in modern Hindi text.

Based on this analysis and also following our earlier convention for other languages, we laid out the अ, इ, उ, ए, and ओ short vowels in left-hand position of the home row and the long vowels (if applicable) in their corresponding shift key positions. This makes it easier for users to type the most frequent vowels with their dominant fingers.

Consonant Frequency Analysis

Next, we turned our attention to consonants. By summing the rows of the character frequency chart, we identified how frequently each consonant appears across different vowel combinations. This row-wise analysis reveals the most commonly used consonants in Hindi, which is crucial for optimizing key placement and improving typing efficiency.

The consonants क् (k), र् (r), न् (n), ह् (h), and स् (S) top the frequency chart, with क being the most dominant at 4498.56 million occurrences. These high-frequency consonants are central to Hindi phonology and appear across a wide range of words and contexts. Characters like म् (m), द् (d), य् (y), ल् (l), and प् (p) also show significant usage, each exceeding 1000 million occurrences. These contribute to the core structure of Hindi vocabulary. Aspirated and retroflex consonants such as झ् (jh), ञ् (ny), ङ् (ng), and घ् (gh) appear far less frequently, with ङ् being the least used at just 0.31 million.

Nukta Usage in Hindi Orthography

The Nukta (or nuqta, '़') is a diacritic mark used in Hindi to represent phonemes that are not native to Indic languages but have been borrowed from Arabic, Persian, English, and other foreign sources. In standard Hindi, the use of the Nukta is restricted to a specific set of consonants—seven of which are widely accepted and standardized. Our analysis focuses on the frequency of these seven Nukta-modified characters, as visualized in the accompanying heatmap.

The heatmap reveals significant variation in the usage of these characters. Among them, 'ड़' (ṛa) is the most frequently used, appearing approximately 192.87 million times. This is followed by 'ढ़' (ṛha) with 55.47 million instances, and 'ज़' (za) with 30.09 million. Other characters such as 'फ़' (fa), 'ख़' (kha), 'क़' (qa), and 'ग़' (gha) show moderate to low usage, ranging between 6 and 13 million occurrences.

Importantly, Nukta characters do not combine with the virama and therefore do not form conjuncts. However, they can take vowel signs to form syllabic units, as seen in words like सड़क, पड़ा, खिलाड़ी, and जुड़े. Due to this behavior, unlike regular consonants, Nukta characters are represented in their full CV (consonant-vowel) form—typically the akāra form—in the Varta keyboard layout. To maintain intuitive typing, these characters are mapped to the same key positions as their base consonants, accessible via Alt + Shift on desktop keyboards or through long press options on mobile keyboards.

Special Characters and Ligatures

Hindi, like many Indic scripts, includes a variety of ligatures and conjunct characters. To support commonly used consonant conjuncts in Hindi, we’ve assigned त्र् (tr), क्ष् (kss), श्र् (shr), and ज्ञ् (jny) to the Alt + Shift positions of specific keys on desktop keyboards. On mobile keyboards, where the Alt key is not available, these conjuncts can be accessed by long pressing the keys Y, U, I, and O, respectively. This design ensures that these frequently used clusters remain easily accessible across both desktop and mobile platforms.

Keyboard Design Implications

If you're exploring the Hindi script and its usage patterns, understanding the Hindi keyboard layout is essential. Whether you're using a Hindi phonetic keyboard or a traditional Hindi keyboard, character frequency insights can help optimize typing experiences and input methods. This analysis sheds light on how Hindi characters are used in real-world text, informing better design for digital tools and keyboards.

The insights from this frequency analysis directly inform our strategy for designing Varta Hindi phonetic keyboard. Frequently used characters are placed in more accessible positions (home row or index finger positions), while less common ones are assigned to secondary layers (e.g., Shift or long-press positions). This ensures a balance between comprehensive script coverage and ease of use. To simplify the design and reduce typing errors or invalid glyph combinations, we chose not to include individual vowel signs. Instead, the system automatically generates the correct vowel sign based on the consonant and following vowel, streamlining the input process.

You can explore our optimized keyboard layout through the Varta Keyboard apps, available on Android and iOS, as well as through browser extensions for Chrome, Edge, and Safari.

Explore Frequency Analyses in Other Languages

We’ve performed similar analysis for other Indian languages as well. Explore them below:

Character Frequency Analysis for Kannada
Character Frequency Analysis for M alayalam
Character Frequency Analysis for Tamil (includes design principles)
Character Frequency Analysis for Telug

Character Frequency Analysis for Kannada

Fri, 30 May 2025 16:40:46 +0530

Continuing with our initiative to develop optimized and user-friendly keyboard layouts for various Indic languages, we conducted a detailed character frequency analysis for Kannada. Using large-scale monolingual datasets collected from diverse web sources, this study aimed to uncover real-world usage patterns of Kannada characters to inform better design decisions for digital tools. This post lists the outcomes and our findings regarding the detailed character frequency analysis undertaken and how it guided the keyboard layout design for Kannada.

Dataset Overview

To ensure statistical robustness in our analysis, we collected a large Kannada dataset from multiple sources in the web, including from:

IndicCorp
Leipzig Kannada Corpus
Wikipedia articles
Samanantar v0.3 (En-Indic; Indic-Indic)
Kannada News Dataset

After data pre-processing and cleaning of the dataset, we arrive at following statistic for this:

Total tokens: 1.567 billion
Unique tokens: 14.825 million

This large and diverse dataset provided a solid foundation for analyzing character usage trends across Kannada as being used in modern daya.

Overall Character Frequency Analysis

The core of our analysis is a character frequency heatmap, where vowels are represented in columns and consonants in rows. Each cell in the matrix reflects the frequency of a specific vowel, consonant or consonant-vowel (CV) combination, with color intensity ranging from light yellow (low frequency) to dark purple (high frequency).

High-frequency vowels like ಅ (a), ಇ (i), and ಉ (u) appear prominently across many consonants, indicating their central role in Kannada phonology. At the same time, characters such as ನ (n), ಮ (m), and ರ (r) show consistently high usage across vowel combinations, suggesting their foundational presence in the language.

An interesting pattern observed in the heatmap is that hard consonants (such as ಗ (ga), ಡ (dda), and ದ (da)) generally appear more frequently than their soft counterparts (such as ಕ (ka), ಟ (tta), and ತ (ta)). This trend suggests a phonetic preference unique to Kannada, differing from other closely related languages like Telugu and Malayalam. These insights are particularly valuable for keyboard layout optimization, as they help prioritize the placement of more frequently used characters for improved typing efficiency.

Vowel Frequency Analysis

To better understand the distribution of vowel usage in Kannada, we analyzed the frequency of individual vowel sounds using a dedicated heatmap by summing the frequencies across the rows from the above heatmap. The results reveal clear patterns in how vowels are used across the language:

As in other Indic languages, the shorter vowels are more frequent than their longer counterparts. The vowels ಅ (a), ಇ (i), and ಆ (aa) are the frequently used either as standalone vowels or as vowel signs in a consonant-vowel (CV). Vowels like ಉ (u) and ಐ (e) also show significant overall usage. The ardhakshara (virama) sign occurs more than in billion instances (last cell in the heatmap) in this dataset, underscoring the frequent use of pure consonants in Kannada. This pattern aligns with other Dravidian languages such as Telugu and Malayalam, where the virama plays a similarly prominent role in forming consonant clusters and suppressing inherent vowels.

However, a notable divergence emerges in Kannada: the akāra CV form (first cell) exhibits a higher frequency than the virama, a trend not observed in other Dravidian languages, where virama usage often surpasses or is comparable to akāra CV combinations. This suggests a greater reliance on vowel-led syllables in Kannada, possibly reflecting phonotactic or orthographic preferences unique to the language.

Consonant Frequency Analysis

We then analyzed the frequency of the consonant by performing a row-wise sum over the frequency matrix. This gives us the frequency of each consonant across all its CV forms as captured in the heatmap below. The first cell captures the frequency of all the vowel forms and can be ignored from this analysis.

The consonants ರ್ (r), ದ್ (d), and ತ್ (t) are the most frequently used, highlighting their central role in Kannada phonology and word formation. Characters like ಕ್ (k) and ಗ್ (g) also show high usage, each exceeding 360M occurrences, indicating their importance in everyday vocabulary. Aspirated and retroflex consonants such as ಘ್ (gh), ಙ್ (ng), ಝ್ (jh), and ಞ್ (ny) appear very infrequently, with some registering near-zero usage. These are typically found in Sanskrit-derived or less common words.

As noted earlier, the hard consonants are slightly more frequent than their softer equivalents (except for ಕ್ (k) and ಪ್ (p)). However, in keeping with the conventions followed in both the Kannada InScript and Varta keyboard layouts for other Indic languages, we chose to place the soft consonants on the home or bottom row, while assigning the hard consonants to the top row. This arrangement maintains consistency across layouts and supports more intuitive typing patterns.

Special Characters and Ligatures

Kannada, like many Indic scripts, includes a variety of ligatures and conjunct characters. However, to support commonly used consonant conjuncts in Kannada, we’ve assigned ತ್ರ್ (tr), ಕ್ಷ್ (kss), ಶ್ರ್ (shr), and ಜ್ಞ್ (jny) to the Alt + Shift positions of specific keys on desktop keyboards. On mobile keyboards, where the Alt key is not available, these conjuncts can be accessed by long-pressing the keys Y, U, I, and O, respectively. This design ensures that these frequently used clusters remain easily accessible across both desktop and mobile platforms.

Keyboard Design Implications

Our character frequency analysis for Kannada offers valuable insights for anyone working with the Kannada keyboard layout. From native speakers to developers building Kannada phonetic keyboards, understanding which characters are most frequently used can enhance typing efficiency and user experience. This data-driven look at the Kannada keyboard supports smarter design and localization.

The insights from this frequency analysis plays a key role in designing the Varta Kannada keyboard. Frequently used characters are placed in more accessible positions (home row or index finger positions), while less common ones are assigned to secondary layers (e.g., Shift or long-press positions). This ensures a balance between comprehensive script coverage and ease of use. To improve usability, we opted not to include standalone vowel signs. Instead, vowel signs are generated automatically from consonant-vowel sequences, which simplifies the layout and minimizes input errors.

You can explore our optimized keyboard layout through the Varta Keyboard apps, available on Android and iOS, as well as through browser extensions for Chrome, Edge, and Safari.

Explore Frequency Analyses in Other Languages

We’ve performed similar analysis for other Indian languages as well. Explore them below:

Character Frequency Analysis for Hindi
Character Frequency Analysis for M alayalam
Character Frequency Analysis for Tamil (includes design principles)
Character Frequency Analysis for Telu

Character Frequency Analysis for Telugu

Fri, 30 May 2025 16:39:42 +0530

As part of our ongoing effort to create optimized and accessible keyboard layouts for Indic languages, we carried out an in-depth character frequency analysis for Telugu. Drawing from large-scale monolingual datasets sourced from across the web, this study focused on identifying real-world usage patterns of Telugu characters. This post lists the outcomes and our findings regarding the detailed character frequency analysis undertaken and how it guided the keyboard layout design for the language.

Dataset Summary

Our Telugu analysis is based on a large and diverse dataset compiled from multiple sources in the web including the following key resources.

IndicCorp
Leipzig Telugu Corpus
Samanantar v0.3 (En-Indic; Indic-Indic)
Telugu News Articles Dataset
Telugu Books Dataset

After pre-processing, this dataset contained over 1.45 billion total words and 12.3 million unique words, providing rich & diverse data for studying character frequency patterns in the language. This extensive dataset provides a strong statistical foundation for understanding character usage patterns in Telugu helping, us design an optimal keyboard layout.

Overall Character Frequency Heatmap

We created a heatmap to visualize the frequency of Telugu characters, where vowels are represented in columns and consonants in the rows. Each cell represents the frequency of a basic vowel, consonant or consonant-vowel (CV) combination, with darker shades indicating higher usage

Vowels such as అ (a) dominate the frequency spectrum, similar to how 'e' and 'a' are dominant in English. Consonants like క (ka), న (na), and ల (la) also show high usage, reflecting their foundational role in Telugu script. A number of characters, particularly aspirated or Sanskrit-derived forms, show very low or near-zero usage, indicating their limited role in everyday Telugu. We also note a unique pattern where long vowels like ఓ (oo) are more frequent than their short counterparts, a trend not commonly seen in other Indic scripts.

Vowel Frequency Analysis

By calculating the column-wise sum of the above character frequency matrix, we obtain the frequency heatmap of each vowel, either in its base form or as a vowel sign in a consonant-vowel (CV)/ consonant conjuncts.

As in many other languages, the short vowel forms—అ (a), ఇ (i), and ఎ (e)—are generally more frequent than their long counterparts. However, Telugu exhibits a unique pattern with the long vowels ఏ (ee) and ఓ (oo), which occur significantly more often than their corresponding short forms. Notably, ఓ (oo) is used nearly five times more frequently than ఒ (o), highlighting a distinct phonological preference in Telugu. Vowels like ఊ (U) and ఋ (vR) appear far less frequently, indicating limited usage in modern Telugu text. These insights are crucial for prioritizing vowel placement in keyboard layouts.

The final cell in the heatmap represents the frequency of the Telugu pollu sign, which denotes a pure consonant (i.e., without an inherent vowel). Interestingly, its distribution is comparable to that of the CV forms with the vowel అ (a)—found in the first column—indicating a similar usage pattern across consonants.

Consonant Frequency Analysis

We then started analysis of the Telugu consonants. By summing the rows of the character frequency chart, we obtained the frequency of each consonant, across its different vowel combinations. This row-wise analysis reveals the most commonly used consonants in Telugu.

The consonant forms of ర్ (r), న్ (na), and ಲ್ (la) emerge as the most frequently used in Telugu, underscoring their central role in the phonetic structure of the language. Following closely are క్ (ka), త్ (ta), and ప్ (pa), each with approximately 300 million occurrences. In contrast, aspirated and less commonly used consonants such as ఙ్ (nga), ఝ్ (jha), and ఢ్ (ḍha) appear only rarely, reflecting their limited presence in contemporary Telugu usage.

Special Characters and Ligatures

Telugu, like many Indic scripts, includes a variety of ligatures and conjunct characters. However, to support commonly used consonant conjuncts in Telugu, we’ve assigned త్ర్ (tr), క్ష్ (kss), శ్ర్ (shr), and జ్ఞ్ (jny) to the Alt + Shift positions of specific keys on desktop keyboards. On mobile keyboards, where the Alt key is not available, these conjuncts can be accessed by long-pressing the keys Y, U, I, and O, respectively. This design ensures that these frequently used clusters remain easily accessible across both desktop and mobile platforms.

Keyboard Design Implications

Our Telugu character frequency analysis is a must-read for those interested in refining the Telugu keyboard layout. With growing demand for intuitive Telugu phonetic keyboards and mobile-friendly Telugu keyboard solutions, this data helps developers to validate and appropriately align their design based on actual language usage patterns.

The insights from this frequency analysis directly inform our design strategy for the Varta Telugu phonetic keyboard. Frequently used characters are placed in more accessible positions (home row or index finger positions), while less common ones are assigned to secondary layers (e.g., Shift or long-press positions). This ensures a balance between comprehensive script coverage and ease of use. To streamline the typing experience, we decided not to include separate vowel signs. Instead, the correct sign is automatically generated from the consonant and vowel input, reducing errors and simplifying the keyboard design.

You can explore our optimized keyboard layout through the Varta Keyboard apps, available on Android and iOS, as well as through browser extensions for Chrome, Edge, and Safari.

Explore Frequency Analyses in Other Languages

We’ve performed similar analysis for other Indian languages as well. Explore them below:

Character Frequency Analysis for Hindi
Character Frequency Analysis for Kannada
Character Frequency Analysis for Malayalam
Character Frequency Analysis for Tamil (includes design principles)

Designing a New Input Method for Tamil

Wed, 22 Mar 2023 16:15:41 +0530

The present input methods for Tamil, while providing reasonable support for using the language in Computers and other devices, they have several shortcomings. We will explore the current input methods and their shortcomings briefly before proposing a new Input Method for Tamil that can be used in both keyboard based and touch devices.

Current Input Methods:

Presently there are 3 predominant input methods that are in wider use.

Tamil99
Murasu Anjal
Google Keyboard (GBoard)

The inadequacy of existing Input methods for Tamil and the idea for a new Input Method for the language has been around for several years [2]. More recently Elango Cheran published a detailed blog post [1] not only explaining the shortcomings of the current Input methods but also expounding his idea of exploiting the phonemic nature of the Tamil alphabets for the new Input method. Here is the brief summary of the key shortcomings of the current Input methods.

Unnatural Design: In Tamil Consonant Vowels such as 'க' and 'தை' are generated by the combination of pure consonants 'க்' and 'த்' with the vowels 'அ' and 'ஐ' respectively. Thus the Vowels and the Consonants, which form the basic units of the sounds (phonemes) in Tamil, should be the basis of designing a good Input method. Several Input methods including the Tamil99 follow the unnatural design of Vowels and Consonant vowels (CV) such as 'க', 'ங' and 'ச', as the basic units in the Keyboard. To be fair, this design came to be used, because these CV characters

Coming back to the Input methods, one of the unintended consequence of this design approach is that, this can produce illegal character sequences in Tamil such as with dangling consonant vowel modifiers such as '்', '

Transliteration Dependency: Murasu Anjal and other transliteration keyboards, rely on users being familiar with the English alphabet and actually inputting the Tamil words in transliterated English letters, which are then transliterated back to Tamil. This method is hugely popular due to the high English literacy among the Tamil speakers the population around the world. However, we believe this is doing more harm because the speakers no longer have to learn the script but only the sounds in the language. Secondly, there are multiple ways to represent a Tamil character in English, because the variations in the sounds in the two languages.

Non-adherence to QWERTY layout: Most of the Tamil keyboards do not adhere to the widely-used QWERTY layout in terms of the key positions reserved for punctuations and other symbols in the keyboard. These Input methods assign Tamil characters in these positions. Consequently, the bilingual users using QWERTY will find it difficult to switch back and forth between English and Tamil typing and they will be forced to learn and follow the different key positions for typing punctuations and symbols while using Tamil.

GBoard Design Incongruity: The Google Keyboard or GBoard for Tamil is the soft key layout launched for touch devices. It lays out the Vowels and Consonant Vowels ('அ' வரிசை such as 'க', 'ங', 'ச', ...) in a 9x4 matrix. The vowel character panel on the left changes every time a consonant vowel is pressed to show its other CV variations. The layout uses a simplistic sequential positional of characters in the alphabet, without any concern for either optimizing finger movements or the character frequency based layout design. Combined with the incongruity of ever changing vowel panel, GBoard's design choice is probably the least efficient Tamil key layout in use.

A New Input Method - Design Principles:

We want to design a new Input Method for Tamil that address the shortcomings in the existing one and also makes it easier to learn the new method with a short learning curve. Based on our research, we decided on the following design goals for the new Input method.

A design that adheres to and exploits the Phonemic nature of Tamil, taking the phonemes as the basic unit
Frequency analysis of base phonemes and consonant vowel combination in order to achieve an optimal design that speeds up touch typing in computers and equivalently reduces finger movement in touch devices
Intuitive arrangement of keys to make the learning easier that is consistent across different platforms and devices
Prevent any illegal character sequences in the output text
Maximize compatibility with the QWERTY keyboard to make the transition between English and Tamil typing seamless and easier.
Eliminate the forced requirement for the user to know other script/ language and instead facilitate typing in the Tamil script

It should be noted that, the same design principles could be used for designing better Input methods for other Abugida languages as well.

Designing the Input Method:

We started the design by identifying Tamil corpus data for doing usage frequency analysis of the characters in the language. We identified large enough corpora (approx. 537M words) from two different sources as below:

Kaggle - Tamil Language Corpus for NLP

Tamil Articles Corpus
Tamil New Corpus
Tamil Language Corpus

Github - Opensource Tamil Corpus

Wikipedia, TheHindu - 58M words in total

Frequency Analysis:

Our goal is to understand the usage frequency of basic phonemes as well as for the full set of Tamil alphabets including the consonant vowel characters. Once we understand the usage frequency of the phonemes and the full set of alphabets, we can exploit this information to design the keyboard layout. It should be noted that we have omitted the Sanskritized characters (வடமொழி எழுத்துக்கள்) 'ஜ்', 'ஶ்', 'ஷ்',

Among the top-10 most frequent characters are, we have 5 consonant vowels and 4 'அ' ending CVs and

Consonants: ம், ர், ல், க் and ன் - 349.13M
'அ' ending CVs: க, த, ப and வ - 306.19M
'உ' ending CV: து - 64.63M

Thus by using the base phonemes (vowels and pure consonants) for out keyboard layout, would result in a saving of nearly 43M keystrokes for this dataset. Now consider two more heatmaps i) by characters ending with vowel sounds (column-wise sum of the above heatmap) and ii) by the characters for each consonant-vowel series (row-wise sum).

Notice that the pure consonants (right-most cell) tend to be more frequent than any consonant vowel series. Here again by using the basic phonemes as the keys instead of the 'அ' ending consonant vowels, these pure consonants can be typed with a single key press as opposed to two presses, saving about 37M keystrokes on this dataset. Also notice that the vowels and CVs ending in short form vowel sound are much more frequent than their long form counterparts.

Using the frequency statistics of the Vowels (first two in the first heatmap) and the Consonant vowels above, we can design optimal keyboard layout to minimize the movement of fingers and to use the dominant fingers for the high frequency phonemes. The next section discusses the design decisions and explains our Tamil Phonemic keyboard layout.

Phonemic Keyboard Layout Design for Tamil:

As we mentioned earlier in our design goals, we want the new keyboard layout to be easier for the users to learn and use across different devices. We want to minimize Given the constraints of available keys and total required keys, we had to make certain design decisions in the character assignment to the key positions.

We want to use Home row of the keyboard for the Vowels and some high-frequency Consonants in the language
We also believe that the dominant index and middle finger keys in two non-Home rows should take precedence over the Home row keys with weaker fingers. We'll be using this later in optimizing the character assignment to the keys.
Given the high frequency of short form vowel ending characters, we have retained the short vowels in the Home row and assign the corresponding long form vowels to the same key in the Shift row. Following the wider convention, we've assigned the vowels on the left side of the keyboard.
Most of the Consonants are going to be assigned in the right hand side of the keyboard to exploit the dominant hand for the majority of the population. Note that this arrangement allows the CVs to be typed efficiently by a mix of both hands, without making the same hand/ finger to move to a different position for typing a single character.
Based on our observation #2 above, the frequent Consonants starting with 'க்' and 'த்' are assigned in the dominant finger positions in the Home row and the rows above and below. We assign the rest of the consonants to the weaker key positions in the decreasing frequency order.
We now specifically consider the case of மெல்லினம் (nasalized consonants), which are typically be followed by the corresponding வல்லினம் (plosive/ stop consonant) in the Tamil text. Thus, it made sense for us to place these nasalized consonants on the left side of the keyboard (above and below the home row) so that the following வல்லினம் can be typed with the right hand. We made an exception for 'ம்' and assign it to the dominant key position on the right side, due to its high frequency in both Consonant and CV forms.
On the Shift key layout, we assigned the Tamil numerals right below the roman numerals to make the typing intuitive and easier. Additionally, the Sanskritized consonants and other Tamil symbols are assigned on this Shift layout.

The screenshots of the new keyboard layout for the regular and Shift keys are below.

Input Method Analysis:

Based on the keyboard layouts for the three input methods, viz Anjal, Phonemic and Tamil99, we analyzed their efficiency and ease of typing in two ways. We first calculated the number of absolute keystrokes required to type the Tamil words in the above corpora of 537M words. To keep the analysis simple, we ignored the punctuations and any non-Tamil words/ characters for this. We also ignored the shift key here because the shift key is pressed simultaneously with the key following it. Here are the absolute number of keystrokes required for typing the above Tamil corpora by the 3 input methods.

Anjal : 4,470,795,879
Phonemic : 4,045,040,635
Tamil99 : 4,124,838,873

The Phonemic method requires the least number of keystrokes among the 3 methods; specifically it requires 80M fewer keystrokes than Tamil99. This is because, the pure consonants are usually frequent than their CV combination. Tami99 requires an additional keystroke '்' for typing the pure consonants. For Anjal, we used the standard transliteration mapping as suggested in the Sellinam app, thus requiring two keystrokes for each long vowel as well as for long CV combinations.

We then analysed the heatmap on the keyboard layouts of the three input methods to see which keys are typed in more frequently and their relative position in the keyboard. We plotted the heatmap on the 3 keyboard layouts separately for this analysis. As above, we ignored the punctuation marks and non-Tamiil words to keep this analysis simple. However, we considered the shift key in this analysis.

The layout will be easier for typing if the frequently typed keys are in the position of dominant fingers of both hands or in the Home row of the keyboard. The Anjal keyboard is clearly the least efficient option as the most of the frequently used keys are outside of the dominant finger positions of the keyboard.

Between the Phonemic and Tamil99 keyboards, the frequent keys are mostly located in the dominant finger position, which makes the typing easier. The dominant left and right index fingers positions (in all 3 rows) alone account for 61.22% and 46.32% of the overall typing in Phonemic and Tamil99 keyboards respectively. This difference of 15% is significant and makes the Phonemic layout a better (in terms of ease of use) option than the Tamil99 keyboard.

We then look at the percentage of typing for the keys in the Home row, which is the usual resting position for the hands when not typing. It thus has the advantage that the user will not have to move their hands from its resting position. The Phonemic layout accounts for 58.33% of the Home row typing, while the Tamil99 is slightly better with 61.83% of overall typing. We believe this small difference of Home row typing is far outweighed by the advantage gained in the Phonemic keyboard layout by the dominant index fingers across all the rows. In addition to the efficiency in typing, the new Phonemic keyboard layout offers other advantages over the Tamil99 keyboard as discussed earlier in this post.

Explore Frequency Analyses in Other Languages:

We’ve performed similar analysis for other Indian languages as well. Explore them below:

Character Frequency Analysis for Hindi
Character Frequency Analysis for Kannada
Character Frequency Analysis for Malayalam
Character Frequency Analysis for T elugu

References:

Redesigning an Input Method for an Abugida Script. Elango Cheran's Blog
Optimization of Tamil Phonetic Keyboard. Sendhil Kumar Cheran, Thuraiappah Vaseeharan and Elango Cheran. Tamil Internet Conference. 2004.

Native Language Proficiency in School Education in India

Sat, 04 Feb 2023 23:57:01 +0530

Native language proficiency could become a crucial factor in the education and academic success of students in early education. Research by UNESCO [1] or academic organizations has shown that students proficient in their native language tend to perform better academically and have a stronger sense of cultural identity and belonging.

However, despite the importance of native language proficiency, students in Indian schools are facing significant challenges in maintaining/ improving their native language skills. For example, English has replaced the vernacular languages as the preferred medium of instruction in early education in the urban areas, thereby reducing the native language to merely a single subject. Concerningly, this is also spreading to rural areas albeit slowly. This has actually led to a situation, where the Indian students are significantly found lacking in their native language proficiency.

Here we explore the relationship between native language proficiency and student success, particularly in early education. We also analyze the current status of language proficiency in Indian schools.

Language proficiency and Learning Quality in Schools

Assuming that typical schools will have a mix students from different socio-cultural backgrounds, we can expect their language proficiency to fall within a spectrum with one extreme corresponding to good English proficiency and the other to native language proficiency (in some rare cases, we might have students that are equally proficient in both languages). With English becoming the de facto medium of instruction in classrooms across the country, a student not so proficient in English will face significantly higher odds compared to a student that is proficient in English. There are few ways in which the native language proficiency and the learning quality in schools are interlinked to each other.

Identity and Cultural connect: For many students, their native language represents a connection to their family, community, and cultural heritage. Therefore, using their native language in school can help students feel valued and connected to their cultural identity, enhancing their overall sense of belonging and emotional/ mental well-being in school.

Better Academic Success: Studies have shown that students who are proficient in their native language tend to perform better academically than students who are not proficient in their native language. This is because language proficiency is closely tied to cognitive skills such as reading comprehension, problem-solving, and critical thinking. Students proficient in their native language can better understand and engage with the material taught, leading to improved learning outcomes.

However, as noted earlier, the official language of instruction is usually English for practical purposes. Thus, a child who is not very proficient in English will find it difficult to understand, participate in the class when the medium of instruction is English. Further, children lacking in English will even find it hard to follow the learning materials written in a language that is foreign (English) to her/ him. This can result in academic disinterest/ lack of engagement for children leading to decreased learning ability.

All other things being equal, two children with different levels of English proficiency might end up in drastically different academic performance. This can lead to a lack of confidence and decreased learning ability for children that doesn't have good English proficiency in the beginning. The early education system might thus be inadvertently introducing inequality in the learning among groups of children, which will likely perpetuate in the intermediate and high schools as well.

Ability to learn new languages: Students proficient in their native language can easily learn a second language, whether it is English or some other language. However, when students have good English proficiency, they do not typically show interest in mastering their native language and they grow up with low to average proficiency in their language. This is probably because of the huge variety of recent, relevant content available in English in social media, kids magazines etc. compared to their native language. While this is a subjective view based on anecdotal experience, the evidence for this can clearly be seen across urban India. A detailed study will likely explain this better and also identify further reasons.

Communication and Social Interactions: Students who are proficient in their native language are able to communicate more clearly and accurately with their peers, teachers, and other community members, leading to improved social skills and a stronger sense of belonging within the school community. Students who are not proficient in the language of instruction may need help participating in class discussions and may feel isolated from their peers.

In addition to communication within the school community, native language proficiency also enables the students to interact with the wider society. This includes communication with vendors, service providers, and other individuals outside of their immediate network, such as friends, family, and teachers.

Without native language skills, children proficient only in English may be isolated from the common non-English speaking population in their lives and may not be able to understand, appreciate, or empathize with these individuals. By promoting native language proficiency in schools, educators can help students develop the skills they need to effectively communicate and interact with others in their community, fostering a sense of belonging and connection to the wider society.

English vs Vernacular Languages in Indian Schools

As we noted earlier, English is fast replacing the vernacular languages as the preferred medium of instruction in schools across India. The trend was only in vogue in urban areas two decades back but is spreading to rural areas across different states in the country. There are different factors for this preference such as i) English being perceived as a 'global' language, ii) parental expectations, iii) peer pressure, iv) general societal trend and v) job opportunities for English proficiency.

This trend is leading to the vernacular languages being pushed to a secondary status in early education in Indian schools. It should be noted that this trend is also seen in the intermediate and higher education scenario. English is even routinely mentioned as the first language in many Education boards, whereas by definition this tag should be reserved for the primary language of the respective states.

The vernacular language doesn't get its due importance even as a second language in many instances due to various reasons such as quality of the pedagogy, minimal hours for the subject and lacklustre attitude of schools/ teachers/ students/ parents towards the subject. For example, since the marks scored in a language subject is not used in the college/ university admission process, the subject is given just enough importance for the students to pass.

Current state of Native language proficiency in Indian schools

We first look into to the medium of instruction question across states. The idea is to see how the preference for English as the medium of instruction varies across the states. The below chart shows student enrollment statistics (Source: 2019-20 UDISE+ Data [2]) in schools by the medium of instruction in different states. As can be seen all the Southern states, except for Karnataka, has over 50% students enrolling in English-medium. The Telugu states are particularly notable in that in both over 70% of the students prefer English-medium over Telugu. Northern states like Haryana and Punjab also appear to be transitioning to English over the state's majority language. Jammu & Kashmir and West Bengal appears to be outlier states, where majority preferring English and Bangla medium schools respectively.

We now examine specifically the current state of native language proficiency vis-à-vis English in Indian schools. The data for this was not readily available and so we looked into the news reports (Academic years 2019-20 thro' 2021-22) for the Class 10 results in different states and compared the pass percentage in the language (English and native language) papers. It should be noted that this is not a direct indicator of the language proficiency. However, this clearly is a reliable indicator in measuring the efficacy of language learning in a formal learning setting over a 10 year period. In all states except Punjab, the pass percentage in the native language trails that of English to varying degree though. In some cases like Maharashtra, the pass percentage for English varied depending on whether it is thought as first or second or third language and we had chosen the first and second language to make the comparison fair.

In any case, this seem to suggest the native language to be lagging behind English in terms of the learning efficacy.

Despite the importance of native language proficiency for student success, there are challenges and barriers that can impact the ability of students in Indian schools to maintain and develop their language skills. One significant challenge is the lower pass rates for certain native languages compared to English.

In Ahmedabad, the mother tongue of Gujarati medium students, 1.19 lakh students failed the exam, with a passing rate of only 82.15%. This is in contrast to the 94.73% passing rate for English, which is considered the students' first language. Purshottam Magarvadiya, a Gujarati teacher for the last 33 years, commented that "The 82% passing rate in the mother tongue is not acceptable and is mainly because students prefer to ignore the subject, thinking they will be able to pass the examination. The focus is on Science and Mathematics and not languages" (Times of India, 2022). [7]

In UP, there have been lower pass rates for the Hindi subject compared to English over the past few years (UP Board, 2019-20). In 2020, more than 5.27 lakh high school students failed in the Hindi examination, while 5.19 lakh students did not pass in English (Hindustan Times, 2020). In 2019, more than 5.74 lakh students failed in Hindi paper of UP Board high school exam compared to 5.02 lakh students who failed in English. This trend of lower pass rates for Hindi compared to English has continued in recent years. This presents a significant challenge to native language proficiency in the region. There may be several reasons for these lower pass rates, including students spending more time studying science subjects and not prioritizing the study of Hindi, or taking their native language lightly (Hindustan Times, 2020). [5]

Conclusion

In conclusion, native language proficiency of our children in the present school education is certainly becoming a major concern across different states and this is rightfully worrying the educators and language experts. As we saw, there are a number of factors that can cause this and we need to take preventive measures to address the serious issues. The new National Education Policy (NEP) announced in 2020 certainly realizes the gravity of this issue and is rightly prescribing native language based education to address the same.

Why should you learn your Native Language?

Wed, 21 Dec 2022 14:46:00 +0530

Gone are the days when we would smile cheerfully and wave to a neighbourhood aunt/ uncle on our way to school or play cricket with our friends in a ground or a dead-end street. Buying a little snack from a old lady in the street corner just outside the school on the way back has been a privilege for the previous generations in those days.

The digital age has ensured that we are more immersed on some Social media now, communicating with some virtual acquaintance rather than playing with physical friends. Sometimes, we can see children and young adults carefully trying to avoid communicating/ interacting with neighbours or service providers such as a apartment security guard, a postman, neighbourhood shop owner. Awkward silences and phone gazing are the norm these days be it in apartment lifts/ elevators or in a park walkway during morning walks, instead of a friendly hello and casual small talk with a long-term neighbourhood face.

This is particularly common in children and young adults today and we believe the lack of native language proficiency of our children is a critical reason in this. They feel and live an isolated life, and have interactions with a very small circle of friends and family. While this may seem inconspicuous now, we believe that we are not preparing them to face the larger world later in their life.

In this post we argue that, a firm grasp of ones native language can benefit in their many aspects of life from cultural understanding, easier social interaction, creativity and better mental acuity and even career advancement.

Cultural and Social Connections

Acquiring fluency in your native language can facilitate a deeper connection with your cultural heritage and ancestry. It can also provide you with a greater appreciation for the complexities and subtleties of your native language. Your ancestors' language can give you a better understanding of the unique stories, beliefs, and values that have been passed down through generations.

Being able to speak and understand your own regional language not only allows you to converse with family and friends but also provides an essential window into the culture's history and its current happenings. You will be able to learn more about the customs and traditions that define the culture. This knowledge can help foster tremendous respect and appreciation for the cultural heritage of your ancestors.

This becomes even more critical now as people migrate to different country/ continent for education, job or simply better opportunities. For such, their language is still a last and vital thread of connection to their roots.

Familiarity with your native language can foster a stronger sense of belonging and connection to the wider society that has people from different walks of live or from different regions with their own dialect/ slang. It can certainly help you to have deep and more meaningful interaction with native speakers. This can lead to a range of opportunities for enriching conversations and interactions that may not have been possible otherwise.

Native language proficiency will also enable you to access information from different media, such as news papers, magazines, books, films, television, digital media and other materials. Most of the Indian languages have vast amounts of literature and other written materials in varied domains that can intrigue, excite, engage and inspire our young minds for generations to come if only our children can read and write in their native tongue.

English or any other alien language can not help you learn/ understand and appreciate your culture and with your societal interactions. Your native tongue is the key to achieve this.

Image credit: Wikimedia

Appreciation for Other Cultures/ Languages

Learning your native language is an essential part of being able to appreciate and understand other cultures fully. By understanding the language spoken in different regions/ countries, you can gain insight into their culture and lifestyle. A better understanding and appreciation of other cultures can help you develop relationships with people from different backgrounds and build bridges between communities.

One of the interesting aspects of learning your native language is discovering the connections it has to other similar languages and cultures. The connections can be i) linguistic such as shared alphabet, common words, and grammatical aspect like sentence structure or ii) cultural such as shared folk tales, literature, and social customs. For example, if your native language is Kannada, you may find that it shares many similarities with languages such as Malayalam, Tamil, and Telugu and that they all have the same sentence structure with lots of vocabulary overlap.

Being able to understand and appreciate the cultural linkages between close languages/ cultures can broaden your perspective and enrich your overall understanding. For example, you might be able learn a sister language easily because of the similarities.

Improves employability

It is often said that the ability to communicate in multiple languages gives you an advantage in the job market.

This is especially true when it comes to speaking your native language. Knowing your native language can open up new career opportunities for you, as employers seek out people who can communicate with those who may speak that language. Having a good command of your native language can also give you an edge in certain professions.

Knowing your native language can be beneficial for career advancement in fields such as translation and interpretation, as it may make you a more attractive candidate to potential employers. Additionally, being proficient in more than one language can expand your professional opportunities, particularly in the global business world, by allowing you to reach new clients and markets. Additionally, speaking your native language may make it easier for you to advance in your career. For example, if you are looking to take on a management position or other leadership role, communicating effectively with colleagues in their native language can be an invaluable asset.

In short, there are many advantages attached to being involved with your love towards your regional language. Improving employability to deepening cultural connections is an invaluable skills in today's competitive job market.

Helps improve Mental Health and Acuity

Learning your native language has many benefits for your mental and physical health. For starters, it can help improve your memory and cognitive functions. Furthermore it can improve your communication skills. It's easier to express yourself accurately when you are familiar with the nuances of your language. It can also help you connect with people from your cultural background who share the same language. Studies have also shown that speaking a native language can help increase the speed at which we process information and form connections in our brains.

Also it can benefit your mental health. It can provide a sense of comfort and belongingness and connect with others on a deeper level. In fact, organizations such as UNESCO stress on the importance of native language as a medium of education in Schools and they have found this to help in better learning and reduced learning gaps.

Conclusion

Learning your native language can be a fun and rewarding experience! It's an opportunity to explore your roots and deepen your connection with your culture. In the process, you will also find new ways to express yourself, understand conversations around you better, or even pick up new words and phrases.

Learning your native language is a beneficial way to deepen your understanding your own culture as well as others. Not only can it help you build relationships with people from diverse social/ economical/ educational backgrounds, but it can also enrich your life through it's own literature, and cultural heritage and provide you with greater opportunities in life.

In addition, having a good command of your native language can provide a solid foundation for learning other languages or for pursuing academic or professional endeavors abroad. Even if you did not have the chance to learn your native language during childhood, it is never too late to start. There are numerous resources available to assist you in your language learning journey. Whether it's an online course, books, or conversations with family members and friends, there are plenty of ways to start exploring your native language today.

It may seem daunting at first, but if you approach it with an open mind and commit yourself to learn your native language over time, it can open up a world of possibilities for you.

Further Reads

Interested in learning more? Here are some additional references that you might find interesting.