172 HmwkCheck:A Homework Auto-Checking System based on Arithmetic Operation Recognition using Smartphones Lingyu Zhang Yafeng Yin Kuang Yaming Honors School, State Key Laboratory for Novel Software Technology, Nanjing University Nanjing University Nanjing,China Nanjing,China 171240524@smail.nju.edu.cn yafeng@nju.edu.cn Lei Xie Sanglu Lu State Key Laboratory for Novel Software Technology, State Key Laboratory for Novel Software Technology, Nanjing University Nanjing University Nanjing,China Nanjing,China lxie@nju.edu.cn sanglu@nju.edu.cn ABSTRACT 1 INTRODUCTION The homework for low-grade pupils often contains simple arith- Homework is often adopted in education,and it can be used to metic problems,i.e.,four arithmetic operations.To evaluate the evaluate the teaching quality of teachers and learning quality of learning quality of pupils,teachers and parents often need to check students.However,correcting the homework manually can be time the homework manually,which is time and labor consuming.In this and labor consuming,especially for the homework assigned to- paper,we propose a homework auto-checking system HmwkCheck wards a larger number of students frequently,e.g.,the arithmetic which checks the four arithmetic operations automatically.Specifi- operation problems assigned for low-grade pupils.Therefore,the cally,HmwkCheck utilizes the embedded camera of a smartphone automatic homework checking approaches were proposed to reduce to capture the homework as an image,and then processes the human cost.With a scanned image,the optical character recogni- image in the smartphone to detect,segment and recognize both tion(OCR)[3]was a typical technology used to recognize printed printed characters and handwritten characters.We implement characters.By taking a picture of homework,image processing HmwkCheck in an Android smartphone.The experiment results [4]was used to detect and recognize handwritten answers.When show that HmwkCheck can check homework efficiently,i.e.,the av- given the image,the existing approaches usually send the image to erage precision,recall and F1-score of character recognition achieve a server for character recognition. 94.03%,93.41%and 93.72%,respectively. Different from the existing approaches,we aim to recognize both printed characters and handwritten answers in arithmetic opera- CCS CONCEPTS tions.In addition,considering the popularity of smartphones,we Human-centered computing-Ubiquitous and mobile com- aim to perform homework auto-checking locally in a smartphone puting systems and tools;Ubiquitous and mobile computing. Without transmitting images on the Internet,we can further pro- tect the user's privacy.As shown in Fig.1,the homework is four KEYWORDS arithmetic operation.At first,we use the embedded camera of a smartphone to capture an image of the homework.Then,we use Homework auto-checking;Arithmetic operation recognition;Image image processing to recognize the characters and calculate the processing:Smartphone arithmetic expression to check the homework. ACM Reference Format: Lingyu Zhang,Yafeng Yin,Lei Xie,and Sanglu Lu.2020.HmwkCheck:A Homework Auto-Checking System based on Arithmetic Operation Recogni- tion using Smartphones.In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers (Ubi- Comp/ISWC '20 Adjunct),September 12-16,2020,Virtual Event,Mexico.ACM. New York,NY,USA,4 pages.https://doi.org/10.1145/3410530.3414393 Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.Copyrights for third-party components of this work must be honored. For all other uses,contact the owner/author(s). UbiComp/ISWC'20 Adjunct,September 12-16,2020,Virtual Event,Mexico 2020 Copyright held by the owner/author(s) ACM1SBN978-1-4503-8076-8/20/09. https:/doi.org/10.1145/3410530.3414393 Figure 1:Homework auto-checking using smartphonesHmwkCheck: A Homework Auto-Checking System based on Arithmetic Operation Recognition using Smartphones Lingyu Zhang Kuang Yaming Honors School, Nanjing University Nanjing, China 171240524@smail.nju.edu.cn Yafeng Yin State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, China yafeng@nju.edu.cn Lei Xie State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, China lxie@nju.edu.cn Sanglu Lu State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, China sanglu@nju.edu.cn ABSTRACT The homework for low-grade pupils often contains simple arith￾metic problems, i.e., four arithmetic operations. To evaluate the learning quality of pupils, teachers and parents often need to check the homework manually, which is time and labor consuming. In this paper, we propose a homework auto-checking system HmwkCheck, which checks the four arithmetic operations automatically. Specifi￾cally, HmwkCheck utilizes the embedded camera of a smartphone to capture the homework as an image, and then processes the image in the smartphone to detect, segment and recognize both printed characters and handwritten characters. We implement HmwkCheck in an Android smartphone. The experiment results show that HmwkCheck can check homework efficiently, i.e., the av￾erage precision, recall and F1-score of character recognition achieve 94.03%, 93.41% and 93.72%, respectively. CCS CONCEPTS • Human-centered computing→Ubiquitous and mobile com￾puting systems and tools; Ubiquitous and mobile computing. KEYWORDS Homework auto-checking; Arithmetic operation recognition; Image processing; Smartphone ACM Reference Format: Lingyu Zhang, Yafeng Yin, Lei Xie, and Sanglu Lu. 2020. HmwkCheck: A Homework Auto-Checking System based on Arithmetic Operation Recogni￾tion using Smartphones. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers (Ubi￾Comp/ISWC ’20 Adjunct), September 12–16, 2020, Virtual Event, Mexico. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3410530.3414393 Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). UbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico © 2020 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-8076-8/20/09. https://doi.org/10.1145/3410530.3414393 1 INTRODUCTION Homework is often adopted in education, and it can be used to evaluate the teaching quality of teachers and learning quality of students. However, correcting the homework manually can be time and labor consuming, especially for the homework assigned to￾wards a larger number of students frequently, e.g., the arithmetic operation problems assigned for low-grade pupils. Therefore, the automatic homework checking approaches were proposed to reduce human cost. With a scanned image, the optical character recogni￾tion (OCR) [3] was a typical technology used to recognize printed characters. By taking a picture of homework, image processing [4] was used to detect and recognize handwritten answers. When given the image, the existing approaches usually send the image to a server for character recognition. Different from the existing approaches, we aim to recognize both printed characters and handwritten answers in arithmetic opera￾tions. In addition, considering the popularity of smartphones, we aim to perform homework auto-checking locally in a smartphone. Without transmitting images on the Internet, we can further pro￾tect the user’s privacy. As shown in Fig. 1, the homework is four arithmetic operation. At first, we use the embedded camera of a smartphone to capture an image of the homework. Then, we use image processing to recognize the characters and calculate the arithmetic expression to check the homework. Figure 1: Homework auto-checking using smartphones 172