DICO-JALF v.1.0: Diponegoro Corpus of Japanese Learners as a Foreign Language in Indonesia with AI Error Annotation and Human Supervision
Main Article Content
Keywords
corpus linguistics, error annotation, AI, ChatGPT, Indonesia, JFL
Abstract
There is a growing body of research in using AI for corrective feedback in foreign language teaching. However, few studies have specifically addressed the accuracy of AI analysis in learner corpus research. This study aims to create an AI-annotated corpus whose data were obtained from learners of Japanese as a Foreign Language (JFL) in Indonesia with human supervision; branded it as DICO-JALF v.1.0. The aim is to measure to what extent ChatGPT accurately annotates errors. A task was first administered to collect corpus data and metadata to build the corpus. The corpus was error-annotated using ChatGPT 4.0. Human annotators manually supervised the accuracy of AI-generated annotations. Regarding errors committed by learners, it is observed that incorrect lexical choices and forms dominate the cause of errors, while underuse and overuse are minimal. It can be concluded that ChatGPT demonstrated an average accuracy of 70% correct identification of errors. Regarding error rate, the verb is the category where errors are most frequent, which maybe driven by its conjugation, a feature absent in Indonesian, the L1 of the students. This suggests that Indonesian learners' acquisition of Japanese verbs needs greater emphasis. As compared to other similar studies, this is relatively low. However, it can be argued that one factor determining the accuracy of ChatGPT annotations, or any other LLM-based tool, is the complexity of the annotation scheme they adhere to. The corpus have been made available for download. The annotations shall be readable by a corpus query system that reads XML tags. This corpus serves as a foundational resource for future research on AI-assisted error analysis in JFL learning contexts in Indonesia.
Downloads
References
Aror, S., Lensun, S., Sompotan, A., & Pandi, H. (2024). Level of Difficulty Using Verbs Meaning ‘To Wear’ in Sentences by Students of the Japanese Language Education Study Program. KnE Social Sciences, 322–337. https://doi.org/10.18502/kss.v9i2.14860
Barus, M. B., & Pujiono, M. (2021). An errors analysis in using diathesis (態) in written text of Japanese Language Department senior students at universities in Medan. IZUMI, 10(2), 362–371. https://doi.org/10.14710/izumi.10.2.362-371
Budianto, F. (2023). Welcoming the opportunities: Deciphering contemporary mobility of Indonesian professionals to Japan. Intermestic Journal of International Studies, 7(2), 656. https://doi.org/10.24198/intermestic.v7n2.13
Forti, L. (2023). Learner corpora and the design of data-driven learning activities. In Proceedings of the EuroCALL 2023 Conference (pp. 139–144). Polytechnic University of Valencia. https://doi.org/10.4995/EuroCALL2023.2023.16959
Hasibuan, A., & Arfianty, R. (2019). Grammatical and lexical errors of Japanese sentence essay of stikes medistra Lubuk Pakam nurses as apprentices to Japan. ABDIMAS TALENTA Jurnal Pengabdian Kepada Masyarakat, 4(2), 259–266. https://doi.org/10.32734/abdimastalenta.v4i2.4058
Hayashishita, J.-R., Tanaka, D., & Ueyama, A. (2020). A linguistically-informed way of introducing Japanese verbs to second language learners. Journal of Japanese Linguistics, 36(1), 29–72. https://doi.org/10.1515/JJL-2019-2017
Heintz, K., Roh, Y., & Lee, J. (2022). Comparing the accuracy and effectiveness of Wordvice AI Proofreader to two automated editing tools and human editors. Science Editing, 9(1), 37–45. https://doi.org/10.6087/kcse.261
Holland, B. J. (2023). ChatGPT 3.5 and 4. In Advances in library and information science (ALIS) book series (pp. 316–340). https://doi.org/10.4018/978-1-6684-7693-2.ch016
Imamura, K., Saito, K., Sadamitsu, K., & Nishikawa, H. (2012). Grammar error correction using pseudo-Error sentences and domain adaptation. Meeting of the Association for Computational Linguistics, 2, 388–392. https://aclanthology.org/P12-2076/
Imamura, K., Saito, K., Sadamitsu, K., & Nishikawa, H. (2014). Particle error correction from small error data for Japanese learners. Journal of Natural Language Processing, 21(4), 941–963. https://doi.org/10.5715/jnlp.21.941
Park, J. (2019). An AI-based English grammar checker vs. human raters in evaluating EFL learners’ writing. Multimedia Assisted Language Learning, 22(1), 112–131. https://doi.org/10.15702/mall.2019.22.1.112
Karataş, F., Abedi, F. Y., Gunyel, F. O., Karadeniz, D., & Kuzgun, Y. (2024). Incorporating AI in foreign language education: An investigation into ChatGPT’s effect on foreign language learners. Education and Information Technologies. https://doi.org/10.1007/s10639-024-12574-6
Kasahara, S., Komachi, M., Nagata, M., & Matsumoto, Y. (2011). Error correcting Romaji-Kana conversion for Japanese language education. Workshop on Advances in Text Input Methods, 38–42. https://aclanthology.info/papers/W11-3506/w11-3506
Koyama, A., Kiyuna, T., Kobayashi, K., Arai, M., Mita, M., Oka, T., & Komachi, M. (2023). Construction of an Error-Tagged Evaluation Corpus for Japanese Grammatical error correction. Journal of Natural Language Processing, 30(2), 330–371. https://doi.org/10.5715/jnlp.30.330
Massey, P. A., Montgomery, C., & Zhang, A. S. (2023). Comparison of ChatGPT–3.5, ChatGPT-4, and orthopaedic resident performance on orthopaedic assessment examinations. Journal of the American Academy of Orthopaedic Surgeons. https://doi.org/10.5435/jaaos-d-23-00396
Nishina, K., Hodošček, B., Yagi, Y., & Abekawa, T. (2014). Construction of a learner corpus for Japanese language learners: Natane and Nutmeg. Acta Linguistica Asiatica, 4(2), 37–51. https://doi.org/10.4312/ala.4.2.37-51
OpenAI. (2024). ChatGPT (July-November 2024 version) [Large language model]. https://chat.openai.com/
Pan, M., Guo, K., & Lai, C. (2024). Using artificial intelligence chatbots to support English-as-a-foreign language students’ self-regulated reading. RELC Journal. https://doi.org/10.1177/00336882241264030
Pavlovič, M. (2020). Grammar errors by Slovenian learners of Japanese: Corpus analysis of writings on beginner and intermediate levels. Acta Linguistica Asiatica, 10(1), 87–104. https://doi.org/10.4312/ala.10.1.87-104
Pérez-Paredes, P., & Mark, G. (2022). What can corpora tell us about language learning? In Routledge eBooks (pp. 313–327). https://doi.org/10.4324/9780367076399-22
Poole, F. J., & Coss, M. (2024). Can ChatGPT reliably and accurately apply a rubric to L2 writing assessments? The devil is in the prompt(s). Journal of Technology for Chinese Language Teaching, 15(1), 1–24. http://www.tclt.us/journal/2024v15n1/poolecoss.pdf
Pandey, A. (2022). Improving Hindi POS Tagger Accuracy Through Domain Adaptation. Nepalese Linguistics, 79–85. https://doi.org/10.3126/nl.v35i01.46564
Prihantoro. (2022). SANTI-morf Dictionaries. Lexicography, 9(2), 175-193. https://doi.org/10.1558/lexi.23569
Prihantoro. (2025). The creation of the Indonesian TreeTagger for use in LancsBox and CQPweb (2). 13(2). https://doi.org/10.32714/ricl.13.01.11
Putri, M. A. (2020). Morphological errors on Japanese verb conjugation to passive form at third-year students of Japanese education study program of UNP. Proceedings of the Eighth International Conference on Languages and Arts, 258-263. Negeri Padang, Indonesia: Atlantis Press. https://doi.org/10.2991/assehr.k.200819.052
Reina, K., & Lee, J. (2023). An Analysis of Verb Errors Made by Japanese Learners of Korean. Studies in Linguistics, 67, 199–220. https://doi.org/10.17002/sil..67.202304.199
Safama, S. A., & Diner, L. (2022). Analisis kesulitan penggunaan partikel wa, no, ni, de pada siswa MAN 1 Kebumen. JLA (Jurnal Lingua Applicata), 6(1), 44. https://doi.org/10.22146/jla.75070
Sanosi, A. (2022). To err is human: comparing human and automated corrective feedback. Information Technologies and Learning Tools, 90(4), 149–161. https://doi.org/10.33407/itlt.v90i4.4980
Thewissen, J. (2013). Capturing L2 Accuracy Developmental Patterns: Insights From an Error‐Tagged EFL Learner Corpus. The Modern Language Journal, 97(S1), Article S1. https://doi.org/10.1111/j.1540-4781.2012.01422.x
Tyson, J. (2023). Shortcomings of ChatGPT. Journal of Chemical Education, 100(8), 3098–3101. https://doi.org/10.1021/acs.jchemed.3c00361
Umoro, A. L. (2023). Indonesian students’ motivation to pursue tertiary education in Japan. Japanese Research on Linguistics Literature and Culture, 5(2), 118–127. https://doi.org/10.33633/jr.v5i2.8282
Yang, J. C., & Akahori, K. (2013). Error analysis in Japanese writing and its implementation in a computer assisted language learning system on the world wide web. CALICO Journal, 15(1–3), 47–66. https://doi.org/10.1558/cj.v15i1-3.47-66
Yu, D., Li, L., Su, H., & Fuoli, M. (2024). Assessing the potential of LLM-assisted annotation for corpus-based pragmatics and discourse analysis. International Journal of Corpus Linguistics. https://doi.org/10.1075/ijcl.23087.yu
Yusuf, M., Taulia, T., & Mawaddah, A. (2024). An Analysis of Japanese Verb Conjugation Errors Among Students at Universitas Harapan Medan. Humanities & Language, 1(3), 183–190. https://doi.org/10.32734/w2gj8r54
