Fully Supervised LM Fully Supervised LM ESI AM ESLAM Num of Manually Labeled Training Tweets Num Labeled Tralning Tweets Figure 1:Effect of emoticons on accuracy of polarity classifica- Figure 4:Effect of emoticons on F-score of subjectivity classifi- tion. cation. ervised LM blue line corresponds to the performance of distantly super- vised LM.which also corresponds to the case of zero manu- ally labeled data.The red line is the results of ESLAM.We can find that ESLAM achieves better performance than the distantly supervised LM.With the increase of manually la- beled data,the performance gap between them will become larger and larger.This verifies our claim that it is not enough to use only the data of noisy labels for training. Num of Manually Labeled Training Tweets Figure 2:Effect of emoticons on F-score of polarity classification Distantly Supervised LM ★-ESLAM different number of manually labeled training data,respec- tively.The results are similar to those for polarity classi- 76 fication which once again verifies the effectiveness of our 74 ESLAM to utilize the noisy emoticon data.The good per- 72 formance of ESLAM also verifies that our url link based 32 .64 128256512788 method is effective to find objective tweets,which is a big Num of Manually Labeled Training Tweets challenge for most existing distantly supervised methods. Figure 5:Effect of manually labeled data on accuracy of polarity classification. ervised LM Distan Supervised LM 82 ★ESLAM 12 Num of Manually Labeled Training Tweets 128 25 512 8 Figure 3:Effect of emoticons on accuracy of subjectivity classifi- Num of Manually Labeled Training Tweets cation. Figure 6:Effect of manually labeled data on F-score of polarity classification. Effect of Manually Labeled Data Figure 7 and Figure 8 illustrate the accuracy and F-score We compare our ESLAM method to the distantly supervised of the two methods on subjectivity classification with differ- LM to verify whether the manually labeled data can provide ent number of manually labeled training data,respectively extra useful information for classification.Please note that The results are similar to those for polarity classification. the distantly supervised LM uses only the noisy emoticon data for training,while ESLAM integrates both manually Sensitivity to Parameters labeled data and the emoticon data for training. The parameter a in(1)plays a critical role to control the Figure 5 and Figure 6 illustrate the accuracy and F-score contribution between the manually labeled information and of the two methods on polarity classification with different noisy labeled information.To show the effect of this param- number of manually labeled training data,respectively.The eter in detail,we try different values for polarity classifica-32 64 128 256 512 768 60 65 70 75 80 85 Num of Manually Labeled Training Tweets Accuracy (in %) Fully Supervised LM ESLAM Figure 1: Effect of emoticons on accuracy of polarity classifica￾tion. 32 64 128 256 512 768 60 65 70 75 80 85 Num of Manually Labeled Training Tweets F−Score (in %) Fully Supervised LM ESLAM Figure 2: Effect of emoticons on F-score of polarity classification. different number of manually labeled training data, respec￾tively. The results are similar to those for polarity classi- fication which once again verifies the effectiveness of our ESLAM to utilize the noisy emoticon data. The good per￾formance of ESLAM also verifies that our url link based method is effective to find objective tweets, which is a big challenge for most existing distantly supervised methods. 32 64 128 256 512 768 68 70 72 74 76 78 80 Num of Manually Labeled Training Tweets Accuracy (in %) Fully Supervised LM ESLAM Figure 3: Effect of emoticons on accuracy of subjectivity classifi- cation. Effect of Manually Labeled Data We compare our ESLAM method to the distantly supervised LM to verify whether the manually labeled data can provide extra useful information for classification. Please note that the distantly supervised LM uses only the noisy emoticon data for training, while ESLAM integrates both manually labeled data and the emoticon data for training. Figure 5 and Figure 6 illustrate the accuracy and F-score of the two methods on polarity classification with different number of manually labeled training data, respectively. The 32 64 128 256 512 768 66 68 70 72 74 76 78 80 Num of Manually Labeled Training Tweets F−Score (in %) Fully Supervised LM ESLAM Figure 4: Effect of emoticons on F-score of subjectivity classifi- cation. blue line corresponds to the performance of distantly super￾vised LM, which also corresponds to the case of zero manu￾ally labeled data. The red line is the results of ESLAM. We can find that ESLAM achieves better performance than the distantly supervised LM. With the increase of manually la￾beled data, the performance gap between them will become larger and larger. This verifies our claim that it is not enough to use only the data of noisy labels for training. 0 32 64 128 256 512 768 70 72 74 76 78 80 82 84 Num of Manually Labeled Training Tweets Accuracy (in %) Distantly Supervised LM ESLAM Figure 5: Effect of manually labeled data on accuracy of polarity classification. 0 32 64 128 256 512 768 74 76 78 80 82 84 Num of Manually Labeled Training Tweets F−Score (in %) Distantly Supervised LM ESLAM Figure 6: Effect of manually labeled data on F-score of polarity classification. Figure 7 and Figure 8 illustrate the accuracy and F-score of the two methods on subjectivity classification with differ￾ent number of manually labeled training data, respectively. The results are similar to those for polarity classification. Sensitivity to Parameters The parameter α in (1) plays a critical role to control the contribution between the manually labeled information and noisy labeled information. To show the effect of this param￾eter in detail, we try different values for polarity classifica-