172 HmwkCheck:A Homework Auto-Checking System based on Arithmetic Operation Recognition using Smartphones Lingyu Zhang Yafeng Yin Kuang Yaming Honors School, State Key Laboratory for Novel Software Technology, Nanjing University Nanjing University Nanjing,China Nanjing,China 171240524@smail.nju.edu.cn yafeng@nju.edu.cn Lei Xie Sanglu Lu State Key Laboratory for Novel Software Technology, State Key Laboratory for Novel Software Technology, Nanjing University Nanjing University Nanjing,China Nanjing,China lxie@nju.edu.cn sanglu@nju.edu.cn ABSTRACT 1 INTRODUCTION The homework for low-grade pupils often contains simple arith- Homework is often adopted in education,and it can be used to metic problems,i.e.,four arithmetic operations.To evaluate the evaluate the teaching quality of teachers and learning quality of learning quality of pupils,teachers and parents often need to check students.However,correcting the homework manually can be time the homework manually,which is time and labor consuming.In this and labor consuming,especially for the homework assigned to- paper,we propose a homework auto-checking system HmwkCheck wards a larger number of students frequently,e.g.,the arithmetic which checks the four arithmetic operations automatically.Specifi- operation problems assigned for low-grade pupils.Therefore,the cally,HmwkCheck utilizes the embedded camera of a smartphone automatic homework checking approaches were proposed to reduce to capture the homework as an image,and then processes the human cost.With a scanned image,the optical character recogni- image in the smartphone to detect,segment and recognize both tion(OCR)[3]was a typical technology used to recognize printed printed characters and handwritten characters.We implement characters.By taking a picture of homework,image processing HmwkCheck in an Android smartphone.The experiment results [4]was used to detect and recognize handwritten answers.When show that HmwkCheck can check homework efficiently,i.e.,the av- given the image,the existing approaches usually send the image to erage precision,recall and F1-score of character recognition achieve a server for character recognition. 94.03%,93.41%and 93.72%,respectively. Different from the existing approaches,we aim to recognize both printed characters and handwritten answers in arithmetic opera- CCS CONCEPTS tions.In addition,considering the popularity of smartphones,we Human-centered computing-Ubiquitous and mobile com- aim to perform homework auto-checking locally in a smartphone puting systems and tools;Ubiquitous and mobile computing. Without transmitting images on the Internet,we can further pro- tect the user's privacy.As shown in Fig.1,the homework is four KEYWORDS arithmetic operation.At first,we use the embedded camera of a smartphone to capture an image of the homework.Then,we use Homework auto-checking;Arithmetic operation recognition;Image image processing to recognize the characters and calculate the processing:Smartphone arithmetic expression to check the homework. ACM Reference Format: Lingyu Zhang,Yafeng Yin,Lei Xie,and Sanglu Lu.2020.HmwkCheck:A Homework Auto-Checking System based on Arithmetic Operation Recogni- tion using Smartphones.In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers (Ubi- Comp/ISWC '20 Adjunct),September 12-16,2020,Virtual Event,Mexico.ACM. New York,NY,USA,4 pages.https://doi.org/10.1145/3410530.3414393 Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.Copyrights for third-party components of this work must be honored. For all other uses,contact the owner/author(s). UbiComp/ISWC'20 Adjunct,September 12-16,2020,Virtual Event,Mexico 2020 Copyright held by the owner/author(s) ACM1SBN978-1-4503-8076-8/20/09. https:/doi.org/10.1145/3410530.3414393 Figure 1:Homework auto-checking using smartphones
HmwkCheck: A Homework Auto-Checking System based on Arithmetic Operation Recognition using Smartphones Lingyu Zhang Kuang Yaming Honors School, Nanjing University Nanjing, China 171240524@smail.nju.edu.cn Yafeng Yin State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, China yafeng@nju.edu.cn Lei Xie State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, China lxie@nju.edu.cn Sanglu Lu State Key Laboratory for Novel Software Technology, Nanjing University Nanjing, China sanglu@nju.edu.cn ABSTRACT The homework for low-grade pupils often contains simple arithmetic problems, i.e., four arithmetic operations. To evaluate the learning quality of pupils, teachers and parents often need to check the homework manually, which is time and labor consuming. In this paper, we propose a homework auto-checking system HmwkCheck, which checks the four arithmetic operations automatically. Specifically, HmwkCheck utilizes the embedded camera of a smartphone to capture the homework as an image, and then processes the image in the smartphone to detect, segment and recognize both printed characters and handwritten characters. We implement HmwkCheck in an Android smartphone. The experiment results show that HmwkCheck can check homework efficiently, i.e., the average precision, recall and F1-score of character recognition achieve 94.03%, 93.41% and 93.72%, respectively. CCS CONCEPTS • Human-centered computing→Ubiquitous and mobile computing systems and tools; Ubiquitous and mobile computing. KEYWORDS Homework auto-checking; Arithmetic operation recognition; Image processing; Smartphone ACM Reference Format: Lingyu Zhang, Yafeng Yin, Lei Xie, and Sanglu Lu. 2020. HmwkCheck: A Homework Auto-Checking System based on Arithmetic Operation Recognition using Smartphones. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers (UbiComp/ISWC ’20 Adjunct), September 12–16, 2020, Virtual Event, Mexico. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3410530.3414393 Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). UbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico © 2020 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-8076-8/20/09. https://doi.org/10.1145/3410530.3414393 1 INTRODUCTION Homework is often adopted in education, and it can be used to evaluate the teaching quality of teachers and learning quality of students. However, correcting the homework manually can be time and labor consuming, especially for the homework assigned towards a larger number of students frequently, e.g., the arithmetic operation problems assigned for low-grade pupils. Therefore, the automatic homework checking approaches were proposed to reduce human cost. With a scanned image, the optical character recognition (OCR) [3] was a typical technology used to recognize printed characters. By taking a picture of homework, image processing [4] was used to detect and recognize handwritten answers. When given the image, the existing approaches usually send the image to a server for character recognition. Different from the existing approaches, we aim to recognize both printed characters and handwritten answers in arithmetic operations. In addition, considering the popularity of smartphones, we aim to perform homework auto-checking locally in a smartphone. Without transmitting images on the Internet, we can further protect the user’s privacy. As shown in Fig. 1, the homework is four arithmetic operation. At first, we use the embedded camera of a smartphone to capture an image of the homework. Then, we use image processing to recognize the characters and calculate the arithmetic expression to check the homework. Figure 1: Homework auto-checking using smartphones 172
173 UbiComp/ISWC'20 Adjunct,September 12-16,2020,Virtual Event,Mexico Lingyu Zhang and Yafeng Yin,et al. Image Preprocessing Character Extraction Character Recognition Arithmetic Expression Gray processing Expression detection Separation for printed characters Calculation and handwritten characters Rule based Right Bilateral filtering Expression segmentation Printed character recognition calculation or Input mage Handwritten character Image binaryzation Character segmentation recognition Figure 2:System overview However,there are some challenges in the problem.Usually,the 3.1 Image Preprocessing picture taken by the smartphone is not as clear as the scanned image, As shown in Fig.1,the four arithmetic operations are in white and characters may have some deformation.Besides,the printed paper,and the picture taken by the smartphone is a RGB image.To characters and handwritten ones usually have different styles,even remove noises and distinguish characters with the background in for the same numeral.In addition,the computing power of a smart- the image,we first preprocess the image.As shown in Fig.2,we phone is limited.Therefore,to achieve the goal,we first introduce first process the raw image,i.e.,picture taken by the smartphone image preprocessing to distinguish the characters and background. with gray-scale image processing.Then,we use a bilateral filter Then,we extract each character based on horizontal projection and to remove noises while keeping the edges in the image.After that, vertical projection.After that,we separate the arithmetic expression we perform image binaryzation to separate the foreground (i.e., and recognize printed characters and handwritten ones individually. characters)and background,i.e.,the characters are in black color We also choose suitable image sizes to make the system work on a while the background is in white color.The preprocessed image smartphone. will be used for the following character extraction. 2 RELATED WORK 3.2 Character Extraction Four arithmetic operations including numerals and operators often To extract the characters,we need to detect the arithmetic expres- appear in the homework of low-grade pupils.To recognize and sion and separate each character.Specifically,with the binarized calculate arithmetic operations automatically,Meng et al.[5]used image,we use the horizontal projection of pixels in each row to a BP neural network and template matching to recognize printed detect the arithmetic expression.Supposed the coordinate of a pixel numerals and operators.To recognize handwritten-style characters, in an image is (xi,yj).i[1.wl.je[1,h].where wand h represent Jiang et al.[2]used the end-to-end learning technology to recognize the width and height of the image,respectively.If the number np of arithmetic operations in fixed-size or varied-size images,where black pixels (xi.yp),ie [1,w]in the pth row satisfies np ep.the the characters were CAPTCHA-style.Khalighi et al.[3]proposed a row is treated as 'Expression-Row',where ep is set to 2 by default. novel OCR system to recognize and calculate handwritten Persian By connecting the consecutive 'Expression-Rows,we can get the arithmetic expression.Instead of only recognizing numerals and arithmetic expressions,as shown in Fig.3(a).To further segment operators,Li et al.[4]proposed BAGS,an automatic homework the arithmetic expressions in same rows,e.g.,"28 x 14 =392"and grading system based on the pictures taken by smart phones.BAGS "38 x 40 1520",we introduce the vertical projection of each pixels detected answer areas in the answer sheets and recognized the in each column,where the principle of vertical projection is sim- handwritten characters(e.g.,words).In BAGS,the images were ilar to that of horizontal projection.After that,we can get each processed in another computers or servers instead of smartphones. arithmetic expression,as shown in Fig.3(b).In addition,to seg- Different from the existing work,we provide a homework auto- ment each character from the arithmetic expression,we repeatedly checking system based on images taken by smartphones,aiming use the vertical projection.Finally,we can extract each arithmetic to recognize both printed characters and handwritten characters expression and its corresponding characters,as shown in Fig.3(c). in arithmetic operations.Besides,our system runs on the easy-to- get smartphone locally without transmitting images or processing 840-420 604-2=302 38x40=1520 images offline. 38区40=|旧20 28×14=3923朗40=J520 (e)Vertical projection 3 SYSTEM DESIGN ↓ To check the homework consisted of four arithmetic operations, 840-420042=202 28x14=3238x40=1520☐ it is necessary to detect the characters,recognize the characters and verify the calculation of arithmetic expression.Therefore,the 28x14-39238x40=J520 28×14=392 36×40=1520 proposed system HmwkCheck consists of four components,i.e.. (a)Horizontal projection image preprocessing,character extraction,character recognition, arithmetic expression calculation,as shown in Fig.2. Figure 3:An example of character segmentation
UbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico Lingyu Zhang and Yafeng Yin, et al. Figure 2: System overview However, there are some challenges in the problem. Usually, the picture taken by the smartphone is not as clear as the scanned image, and characters may have some deformation. Besides, the printed characters and handwritten ones usually have different styles, even for the same numeral. In addition, the computing power of a smartphone is limited. Therefore, to achieve the goal, we first introduce image preprocessing to distinguish the characters and background. Then, we extract each character based on horizontal projection and vertical projection. After that, we separate the arithmetic expression and recognize printed characters and handwritten ones individually. We also choose suitable image sizes to make the system work on a smartphone. 2 RELATED WORK Four arithmetic operations including numerals and operators often appear in the homework of low-grade pupils. To recognize and calculate arithmetic operations automatically, Meng et al. [5] used a BP neural network and template matching to recognize printed numerals and operators. To recognize handwritten-style characters, Jiang et al. [2] used the end-to-end learning technology to recognize arithmetic operations in fixed-size or varied-size images, where the characters were CAPTCHA-style. Khalighi et al. [3] proposed a novel OCR system to recognize and calculate handwritten Persian arithmetic expression. Instead of only recognizing numerals and operators, Li et al. [4] proposed BAGS, an automatic homework grading system based on the pictures taken by smart phones. BAGS detected answer areas in the answer sheets and recognized the handwritten characters (e.g., words). In BAGS, the images were processed in another computers or servers instead of smartphones. Different from the existing work, we provide a homework autochecking system based on images taken by smartphones, aiming to recognize both printed characters and handwritten characters in arithmetic operations. Besides, our system runs on the easy-toget smartphone locally without transmitting images or processing images offline. 3 SYSTEM DESIGN To check the homework consisted of four arithmetic operations, it is necessary to detect the characters, recognize the characters and verify the calculation of arithmetic expression. Therefore, the proposed system HmwkCheck consists of four components, i.e., image preprocessing, character extraction, character recognition, arithmetic expression calculation, as shown in Fig. 2. 3.1 Image Preprocessing As shown in Fig. 1, the four arithmetic operations are in white paper, and the picture taken by the smartphone is a RGB image. To remove noises and distinguish characters with the background in the image, we first preprocess the image. As shown in Fig. 2, we first process the raw image, i.e., picture taken by the smartphone, with gray-scale image processing. Then, we use a bilateral filter to remove noises while keeping the edges in the image. After that, we perform image binaryzation to separate the foreground (i.e., characters) and background, i.e., the characters are in black color while the background is in white color. The preprocessed image will be used for the following character extraction. 3.2 Character Extraction To extract the characters, we need to detect the arithmetic expression and separate each character. Specifically, with the binarized image, we use the horizontal projection of pixels in each row to detect the arithmetic expression. Supposed the coordinate of a pixel in an image is (xi ,yj), i ∈ [1,w], j ∈ [1, h], where w and h represent the width and height of the image, respectively. If the number np of black pixels (xi ,yp ), i ∈ [1,w] in the pth row satisfies np > ϵp , the row is treated as ‘Expression-Row’, where ϵp is set to 2 by default. By connecting the consecutive ‘Expression-Rows’, we can get the arithmetic expressions, as shown in Fig. 3(a). To further segment the arithmetic expressions in same rows, e.g., “28 × 14 = 392” and “38 × 40 = 1520”, we introduce the vertical projection of each pixels in each column, where the principle of vertical projection is similar to that of horizontal projection. After that, we can get each arithmetic expression, as shown in Fig. 3(b). In addition, to segment each character from the arithmetic expression, we repeatedly use the vertical projection. Finally, we can extract each arithmetic expression and its corresponding characters, as shown in Fig. 3(c). Figure 3: An example of character segmentation 173
174 HmwkCheck:A Homework Auto-Checking System-. UbiComp/ISWC '20 Adjunct,September 12-16,2020,Virtual Event,Mexico Feature maps Feature maps Feature maps Feature maps Hidden units Hidden units Output 64@号×7 Lm=-2 12B×m-2)×0m2)32 Feature map 32a(-2)×2) 64a(k-2)×k2) 128面m-2Xw-2) Max-pool Max-pooling Flatten Dens Dense 3×3kg 2×2 kernel 3×3 kernel Figure 4:The CNN based character recognition 3.3 Character Recognition 15 classes,while for the CNN in 'handwritten-model',the input After character extraction,we will recognize each detected charac- image size of each character is'56x 56'(i.e,n=56)and the output ter.Due to the different styles of printed numerals and handwritten corresponds to 10 classes.For the segmented image containing the ones,e.g,the different styles of printed'9'and handwritten'9,we character,it is filled with background color to satisfy the above recognize the printed characters and handwritten ones separately. image size requirement.Then,the detected character will be sent to avoid the interference between them.It is worth noting that the to the corresponding CNN for further recognition. printed characters include numerals'0'-'9'and five operators'+ 3.4 Arithmetic Expression Calculation -,'x,','=',while the handwritten characters include numerals 0'-g When each character is recognized,we will calculate the arithmetic Firstly,we utilize the structure of the arithmetic expression,i.e. expression to verify whether the answer is right.Specifically,we 'p'=H'to separate printed characters and handwritten ones.Here, use the rules of four arithmetic operations to calculate the answer 'p'means the printed part consisted of numerals and operators,=' based on the recognized printed numerals and operators.Then,we is the equal sign,and'H'means the handwritten part consisted of compare the calculated answer with the handwritten one.If they numerals.Therefore,if we can recognize the equal sign'=',we can are the same,then the handwritten answer is right.Otherwise,the separate printed characters and handwritten ones in an arithmetic handwritten answer is treated as wrong. expression.To achieve this goal,we first use the 'printed-model' 4 SYSTEM IMPLEMENTATION AND to recognize the characters in an expression from left to right, PERFORMANCE EVALUATION while treating each character as a printed one by default.Once we recognize the'=',we will change to use the 'handwritten-model' In this paper,we aim to recognize four arithmetic operations and for the following handwritten character recognition,as shown in provide a homework auto-checking system running on a smart- Fig.5. phone for low-grade pupils.In the following subsections,we will In regard to the 'printed-model'and 'handwritten-model'used show the system implementation and evaluate the performance of for printed and handwritten character recognition respectively, HmwkCheck. they both use the convolutional neural network(CNN)shown in 4.1 System Implementation Fig.4.For the CNN in'printed-model',the input image size of each character is'28 x 28'(i.e,n =28)and the output corresponds to We implement HmwkCheck in an Android smartphone.As shown in Fig.6(a),HmwkCheck first takes a picture of the homework. Then,it preprocess the image,as the binarized image shown in egin Fig.6(b).After that,it extracts and recognizes the characters,and then calculates the arithmetic expression to verify whether the handwritten answer is right,as the colored text shown in Fig.6(b). No segmented The arithmetic expression with right answer is shown in green, image left? Yes while that with wrong answer is shown in red.Sometimes,the red No expression may be caused by the recognition error,ie.,the actual Get a segmented handwritten answer is right.Take the red expression in Fig.6(b)as Image an example,the handwritten'6'is wrongly recognized as '8',while the actual arithmetic expression is right.Neverthless,we make the found? potential wrong answer with red to raise concern. 4.2 Performance Evaluation Handw ritten Printed- End To evaluate the performance of HmwkCheck,we invite 20 volun- model model teers to perform the following experiments.As shown in Fig.1,the homework containing four arithmetic operations is in white paper. Figure 5:The process of recognizing characters The space between the expressions or characters is comparable to
HmwkCheck: A Homework Auto-Checking System... UbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico Figure 4: The CNN based character recognition 3.3 Character Recognition After character extraction, we will recognize each detected character. Due to the different styles of printed numerals and handwritten ones, e.g, the different styles of printed ‘9’ and handwritten ‘9’, we recognize the printed characters and handwritten ones separately, to avoid the interference between them. It is worth noting that the printed characters include numerals ‘0’-‘9’ and five operators ‘+’, ‘−’, ‘×’, ‘÷’, ‘=’, while the handwritten characters include numerals ‘0’-‘9’. Firstly, we utilize the structure of the arithmetic expression, i.e., ‘P’ = ‘H’, to separate printed characters and handwritten ones. Here, ‘P’ means the printed part consisted of numerals and operators, ‘=’ is the equal sign, and ‘H’ means the handwritten part consisted of numerals. Therefore, if we can recognize the equal sign ‘=’, we can separate printed characters and handwritten ones in an arithmetic expression. To achieve this goal, we first use the ‘printed-model’ to recognize the characters in an expression from left to right, while treating each character as a printed one by default. Once we recognize the ‘=’, we will change to use the ‘handwritten-model’ for the following handwritten character recognition, as shown in Fig. 5. In regard to the ‘printed-model’ and ‘handwritten-model’ used for printed and handwritten character recognition respectively, they both use the convolutional neural network (CNN) shown in Fig. 4. For the CNN in ‘printed-model’, the input image size of each character is ‘28 × 28’ (i.e, n = 28) and the output corresponds to Figure 5: The process of recognizing characters 15 classes , while for the CNN in ‘handwritten-model’, the input image size of each character is ‘56 × 56’ (i.e, n = 56) and the output corresponds to 10 classes. For the segmented image containing the character, it is filled with background color to satisfy the above image size requirement. Then, the detected character will be sent to the corresponding CNN for further recognition. 3.4 Arithmetic Expression Calculation When each character is recognized, we will calculate the arithmetic expression to verify whether the answer is right. Specifically, we use the rules of four arithmetic operations to calculate the answer based on the recognized printed numerals and operators. Then, we compare the calculated answer with the handwritten one. If they are the same, then the handwritten answer is right. Otherwise, the handwritten answer is treated as wrong. 4 SYSTEM IMPLEMENTATION AND PERFORMANCE EVALUATION In this paper, we aim to recognize four arithmetic operations and provide a homework auto-checking system running on a smartphone for low-grade pupils. In the following subsections, we will show the system implementation and evaluate the performance of HmwkCheck. 4.1 System Implementation We implement HmwkCheck in an Android smartphone. As shown in Fig. 6(a), HmwkCheck first takes a picture of the homework. Then, it preprocess the image, as the binarized image shown in Fig. 6(b). After that, it extracts and recognizes the characters, and then calculates the arithmetic expression to verify whether the handwritten answer is right, as the colored text shown in Fig. 6(b). The arithmetic expression with right answer is shown in green, while that with wrong answer is shown in red. Sometimes, the red expression may be caused by the recognition error, i.e., the actual handwritten answer is right. Take the red expression in Fig. 6(b) as an example, the handwritten ‘6’ is wrongly recognized as ‘8’, while the actual arithmetic expression is right. Neverthless, we make the potential wrong answer with red to raise concern. 4.2 Performance Evaluation To evaluate the performance of HmwkCheck, we invite 20 volunteers to perform the following experiments. As shown in Fig. 1, the homework containing four arithmetic operations is in white paper. The space between the expressions or characters is comparable to 174
175 UbiComp/ISWC '20 Adjunct,September 12-16,2020,Virtual Event,Mexico Lingyu Zhang and Yafeng Yin,et al. 5 CONCLUSION In this paper,we propose a homework auto-checking system HmwkCheck focusing on the recognition of four arithmetic operations for low- grade pupils.We use the embedded camera of a smartphone to take 1+31=83 明-64=26 a picture of the homework,and then process the image to recog- nize characters.After that,we calculate the arithmetic expression 5+2454 to check the handwritten answer.We implement HmwkCheck in an Android smartphone.The experiment results show that we can 51+31=82(位c10】 recognize characters accurately. ACKNOWLEDGMENTS (a) (b) This work is supported by National Natural Science Foundation of Figure 6:System implementation China under Grant Nos.61802169,61872174,61832008,61902175 61906085;JiangSu Natural Science Foundation under Grant Nos. that of the homework for low-grade pupils.Each volunteer calcu- BK20180325,BK20190293;the Key R&D Program of Jiangsu Province lated all the 40 expressions in a piece of paper and wrote the answers. under Grant No.BE2018116.This work is partially supported by Unless otherwise specified,we use the A4-sized white paper and Collaborative Innovation Center of Novel Software Technology and Samsung Note 8 smartphone to take pictures.To train the 'printed- model',we print out numerals and operators,use smartphone to Industrialization.Yafeng Yin is the corresponding author. take pictures,and get 360 images for each printed character.To REFERENCES train the 'handwritten-model',we use the public USPS data set [1] [1]Jonathan J.Hull.1994.A database for handwritten text recognition research.IEEE and randomly select 2150 images for each handwritten character. Transactions on Pattern Analysis and Machine Intelligence 16,5(1994),550-554. Considering that the calculation of arithmetic expression has cer- [2]Y.Jiang.H.Dong.and A.El Saddik.2018.Baidu Meizu Deep Learning Competition: tain rules which are fixed.we focus on the evaluation of character etic Operation Re IEEE Access6(2018).60128-60136 cognition Using End-to-End Learning OCR Technologies. recognition [3]S.Khalighi,P.Tirdad,H.R.Rabiee,and M.Parviz.2009.A Novel OCR System for In Fig.7,we show the recognition performance for printed char- Calculating Handwritten Persian Arithmetic Expressions.In 2009 International Conference on Machine Learning and Applications.755-758. acters,handwritten characters,and both of the them.For printed [4]Xiaoshuo Li,Tiezhu Yue,Xuanping Huang.Zhe Yang.and Gang Xu.2019.BAGS: characters,they have fixed typefaces,thus can be recognized well. An automatic homework grading system using the pictures taken by smart phones. i.e.,the precision,recall and F1-score achieve 96.22%,97.01%and CoRR abs/,1906.03767(2019.arXiv:1906.03767htp:arxiv.org/abs/1906.03767 [5]Wang Meng,Bian Guang-rong.Zheng Kai,He Ninghui,and Zhang Yan-lei.2017. 96.61%,respectively.The failure of recognizing some printed char- Recognition Technology for Four Arithmetic Operations.TELKOMNIKA Telecom- acters may be caused by the interference in the input image,e.g., munication Computing Electronics and Control 15,1(2017),306-313. smudges around the printed characters.For handwritten characters, different people can write the same character in different styles, thus the recognition performance of handwritten characters is not as good as that of printed characters,i.e.,the precision,recall and F1-score are 88.24%,84.34%and 86.25%,respectively.In addition, some illegible handwritten characters also decrease the recogni- tion performance.For example,two handwritten characters having overlap may be segmented as one character,the strokes in one character away from each other may be segmented as two charac- ters,which can lead to the recognition error.Overall,HmwkCheck can recognize characters efficiently,the average precision,recall and F1-score of character recognition achieve 94.03%,93.41%and 93.72%,respectively. 0 70 50 40 10 Printed Handwritten A Figure 7:Character recognition performance
UbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico Lingyu Zhang and Yafeng Yin, et al. Figure 6: System implementation that of the homework for low-grade pupils. Each volunteer calculated all the 40 expressions in a piece of paper and wrote the answers. Unless otherwise specified, we use the A4-sized white paper and Samsung Note 8 smartphone to take pictures. To train the ‘printedmodel’, we print out numerals and operators, use smartphone to take pictures, and get 360 images for each printed character. To train the ‘handwritten-model’, we use the public USPS data set [1] and randomly select 2150 images for each handwritten character. Considering that the calculation of arithmetic expression has certain rules which are fixed, we focus on the evaluation of character recognition. In Fig. 7, we show the recognition performance for printed characters, handwritten characters, and both of the them. For printed characters, they have fixed typefaces, thus can be recognized well, i.e., the precision, recall and F1-score achieve 96.22%, 97.01% and 96.61%, respectively. The failure of recognizing some printed characters may be caused by the interference in the input image, e.g., smudges around the printed characters. For handwritten characters, different people can write the same character in different styles, thus the recognition performance of handwritten characters is not as good as that of printed characters, i.e., the precision, recall and F1-score are 88.24%, 84.34% and 86.25%, respectively. In addition, some illegible handwritten characters also decrease the recognition performance. For example, two handwritten characters having overlap may be segmented as one character, the strokes in one character away from each other may be segmented as two characters, which can lead to the recognition error. Overall, HmwkCheck can recognize characters efficiently, the average precision, recall and F1-score of character recognition achieve 94.03%, 93.41% and 93.72%, respectively. Figure 7: Character recognition performance 5 CONCLUSION In this paper, we propose a homework auto-checking system HmwkCheck focusing on the recognition of four arithmetic operations for lowgrade pupils. We use the embedded camera of a smartphone to take a picture of the homework, and then process the image to recognize characters. After that,we calculate the arithmetic expression to check the handwritten answer. We implement HmwkCheck in an Android smartphone. The experiment results show that we can recognize characters accurately. ACKNOWLEDGMENTS This work is supported by National Natural Science Foundation of China under Grant Nos. 61802169, 61872174, 61832008, 61902175, 61906085; JiangSu Natural Science Foundation under Grant Nos. BK20180325, BK20190293; the Key R&D Program of Jiangsu Province under Grant No. BE2018116. This work is partially supported by Collaborative Innovation Center of Novel Software Technology and Industrialization. Yafeng Yin is the corresponding author. REFERENCES [1] Jonathan J. Hull. 1994. A database for handwritten text recognition research. IEEE Transactions on Pattern Analysis and Machine Intelligence 16, 5 (1994), 550–554. [2] Y. Jiang, H. Dong, and A. El Saddik. 2018. Baidu Meizu Deep Learning Competition: Arithmetic Operation Recognition Using End-to-End Learning OCR Technologies. IEEE Access 6 (2018), 60128–60136. [3] S. Khalighi, P. Tirdad, H. R. Rabiee, and M. Parviz. 2009. A Novel OCR System for Calculating Handwritten Persian Arithmetic Expressions. In 2009 International Conference on Machine Learning and Applications. 755–758. [4] Xiaoshuo Li, Tiezhu Yue, Xuanping Huang, Zhe Yang, and Gang Xu. 2019. BAGS: An automatic homework grading system using the pictures taken by smart phones. CoRR abs/1906.03767 (2019). arXiv:1906.03767 http://arxiv.org/abs/1906.03767 [5] Wang Meng, Bian Guang-rong, Zheng Kai, He Ninghui, and Zhang Yan-lei. 2017. Recognition Technology for Four Arithmetic Operations. TELKOMNIKA Telecommunication Computing Electronics and Control 15, 1 (2017), 306–313. 175