174 HmwkCheck:A Homework Auto-Checking System-. UbiComp/ISWC '20 Adjunct,September 12-16,2020,Virtual Event,Mexico Feature maps Feature maps Feature maps Feature maps Hidden units Hidden units Output 64@号×7 Lm=-2 12B×m-2)×0m2)32 Feature map 32a(-2)×2) 64a(k-2)×k2) 128面m-2Xw-2) Max-pool Max-pooling Flatten Dens Dense 3×3kg 2×2 kernel 3×3 kernel Figure 4:The CNN based character recognition 3.3 Character Recognition 15 classes,while for the CNN in 'handwritten-model',the input After character extraction,we will recognize each detected charac- image size of each character is'56x 56'(i.e,n=56)and the output ter.Due to the different styles of printed numerals and handwritten corresponds to 10 classes.For the segmented image containing the ones,e.g,the different styles of printed'9'and handwritten'9,we character,it is filled with background color to satisfy the above recognize the printed characters and handwritten ones separately. image size requirement.Then,the detected character will be sent to avoid the interference between them.It is worth noting that the to the corresponding CNN for further recognition. printed characters include numerals'0'-'9'and five operators'+ 3.4 Arithmetic Expression Calculation -,'x,','=',while the handwritten characters include numerals 0'-g When each character is recognized,we will calculate the arithmetic Firstly,we utilize the structure of the arithmetic expression,i.e. expression to verify whether the answer is right.Specifically,we 'p'=H'to separate printed characters and handwritten ones.Here, use the rules of four arithmetic operations to calculate the answer 'p'means the printed part consisted of numerals and operators,=' based on the recognized printed numerals and operators.Then,we is the equal sign,and'H'means the handwritten part consisted of compare the calculated answer with the handwritten one.If they numerals.Therefore,if we can recognize the equal sign'=',we can are the same,then the handwritten answer is right.Otherwise,the separate printed characters and handwritten ones in an arithmetic handwritten answer is treated as wrong. expression.To achieve this goal,we first use the 'printed-model' 4 SYSTEM IMPLEMENTATION AND to recognize the characters in an expression from left to right, PERFORMANCE EVALUATION while treating each character as a printed one by default.Once we recognize the'=',we will change to use the 'handwritten-model' In this paper,we aim to recognize four arithmetic operations and for the following handwritten character recognition,as shown in provide a homework auto-checking system running on a smart- Fig.5. phone for low-grade pupils.In the following subsections,we will In regard to the 'printed-model'and 'handwritten-model'used show the system implementation and evaluate the performance of for printed and handwritten character recognition respectively, HmwkCheck. they both use the convolutional neural network(CNN)shown in 4.1 System Implementation Fig.4.For the CNN in'printed-model',the input image size of each character is'28 x 28'(i.e,n =28)and the output corresponds to We implement HmwkCheck in an Android smartphone.As shown in Fig.6(a),HmwkCheck first takes a picture of the homework. Then,it preprocess the image,as the binarized image shown in egin Fig.6(b).After that,it extracts and recognizes the characters,and then calculates the arithmetic expression to verify whether the handwritten answer is right,as the colored text shown in Fig.6(b). No segmented The arithmetic expression with right answer is shown in green, image left? Yes while that with wrong answer is shown in red.Sometimes,the red No expression may be caused by the recognition error,ie.,the actual Get a segmented handwritten answer is right.Take the red expression in Fig.6(b)as Image an example,the handwritten'6'is wrongly recognized as '8',while the actual arithmetic expression is right.Neverthless,we make the found? potential wrong answer with red to raise concern. 4.2 Performance Evaluation Handw ritten Printed- End To evaluate the performance of HmwkCheck,we invite 20 volun- model model teers to perform the following experiments.As shown in Fig.1,the homework containing four arithmetic operations is in white paper. Figure 5:The process of recognizing characters The space between the expressions or characters is comparable toHmwkCheck: A Homework Auto-Checking System... UbiComp/ISWC ’20 Adjunct, September 12–16, 2020, Virtual Event, Mexico Figure 4: The CNN based character recognition 3.3 Character Recognition After character extraction, we will recognize each detected charac￾ter. Due to the different styles of printed numerals and handwritten ones, e.g, the different styles of printed ‘9’ and handwritten ‘9’, we recognize the printed characters and handwritten ones separately, to avoid the interference between them. It is worth noting that the printed characters include numerals ‘0’-‘9’ and five operators ‘+’, ‘−’, ‘×’, ‘÷’, ‘=’, while the handwritten characters include numerals ‘0’-‘9’. Firstly, we utilize the structure of the arithmetic expression, i.e., ‘P’ = ‘H’, to separate printed characters and handwritten ones. Here, ‘P’ means the printed part consisted of numerals and operators, ‘=’ is the equal sign, and ‘H’ means the handwritten part consisted of numerals. Therefore, if we can recognize the equal sign ‘=’, we can separate printed characters and handwritten ones in an arithmetic expression. To achieve this goal, we first use the ‘printed-model’ to recognize the characters in an expression from left to right, while treating each character as a printed one by default. Once we recognize the ‘=’, we will change to use the ‘handwritten-model’ for the following handwritten character recognition, as shown in Fig. 5. In regard to the ‘printed-model’ and ‘handwritten-model’ used for printed and handwritten character recognition respectively, they both use the convolutional neural network (CNN) shown in Fig. 4. For the CNN in ‘printed-model’, the input image size of each character is ‘28 × 28’ (i.e, n = 28) and the output corresponds to Figure 5: The process of recognizing characters 15 classes , while for the CNN in ‘handwritten-model’, the input image size of each character is ‘56 × 56’ (i.e, n = 56) and the output corresponds to 10 classes. For the segmented image containing the character, it is filled with background color to satisfy the above image size requirement. Then, the detected character will be sent to the corresponding CNN for further recognition. 3.4 Arithmetic Expression Calculation When each character is recognized, we will calculate the arithmetic expression to verify whether the answer is right. Specifically, we use the rules of four arithmetic operations to calculate the answer based on the recognized printed numerals and operators. Then, we compare the calculated answer with the handwritten one. If they are the same, then the handwritten answer is right. Otherwise, the handwritten answer is treated as wrong. 4 SYSTEM IMPLEMENTATION AND PERFORMANCE EVALUATION In this paper, we aim to recognize four arithmetic operations and provide a homework auto-checking system running on a smart￾phone for low-grade pupils. In the following subsections, we will show the system implementation and evaluate the performance of HmwkCheck. 4.1 System Implementation We implement HmwkCheck in an Android smartphone. As shown in Fig. 6(a), HmwkCheck first takes a picture of the homework. Then, it preprocess the image, as the binarized image shown in Fig. 6(b). After that, it extracts and recognizes the characters, and then calculates the arithmetic expression to verify whether the handwritten answer is right, as the colored text shown in Fig. 6(b). The arithmetic expression with right answer is shown in green, while that with wrong answer is shown in red. Sometimes, the red expression may be caused by the recognition error, i.e., the actual handwritten answer is right. Take the red expression in Fig. 6(b) as an example, the handwritten ‘6’ is wrongly recognized as ‘8’, while the actual arithmetic expression is right. Neverthless, we make the potential wrong answer with red to raise concern. 4.2 Performance Evaluation To evaluate the performance of HmwkCheck, we invite 20 volun￾teers to perform the following experiments. As shown in Fig. 1, the homework containing four arithmetic operations is in white paper. The space between the expressions or characters is comparable to 174