·1238· 工程科学学报.第41卷，第10期 works with the hashing trick /Proceedings of the 32nd Interna- [48]Gudovskiy D A,Rigazio L.ShiftCNN:generalized low-precision tional Conference on Machine Learning.Lille,2015:2285 architecture for inference of convolutional neural networks [J/ [34]Shi L,Feng S K,Zhu Z F.Functional hashing for compressing 0L].ArXie Preprint(2017-06-07)[2019-03-22].https:/ neural networks [J/OL].ArXir Preprint (2016-05-20)[2019- arxiv.org/abs/1706.02393 03-22].https://arxiv.org/abs/1605.06560 [49]Meller E,Finkelstein A,Almog U,et al.Same,same but dif- [35]Wu J R,Wang Y,Wu Z Y,et al.Deep k-Means:Re-training ferent-recovering neural network quantization error through weight and parameter sharing with harder cluster assignments for com- factorization [J/OL].ArXin Preprint (2019-02-05)[2019-03- pressing deep convolutions [J/OL].ArXie Preprint (2018-06- 22].https://arxiv.org/abs/1902.01917 24)[2019-03-22].htps:/ariv.org/ahs/1806.09228 [50]Xu Y H,Zhang S,Qi YY,et al.DNQ:Dynamic network [36]Lu Z Y,Sindhwani V,Sainath T N.Leaming compact recurrent quantization [J/OL].ArXie Preprint (2018-12-06)[2019-03- neural networks [J/OL].ArXir Preprint (2016-04-09)[2019- 22].https://arxiv.org/abs/1812.02375 03-22].htps:/axiw.org/abs/1604.02594 [51]Golub G H,Reinsch C.Singular value decomposition and least [37]Jin X J,Yang Y Z,Xu N.et al.WSNet:compact and efficient squares solutions /Linear Algebra.Springer,Berlin,1971: 134 networks through weight sampling J/OL ]ArXir Preprint [52]Jaderberg M,Vedaldi A,Zisserman A.Speeding up convolution- (2018-05-22)[2019-03-22].https:/aim.org/ahs/1711. al neural networks with low rank expansions [J/OL].ArYir Pre- 10067 pmiu(2014-05-15)[2019-03-22].https:/aiv.org/ahs/ [38] Yang YZ,Jojic N,Huan J.FSNet:Compression of deep convo- 1405.3866 lutional neural networks by filter summary J/OL].ArXir Pre- [53]Kim Y D,Park E,Yoo S,et al.Compression of deep convolu- prit(2019-02-13)[2019-03-22].htps:/axiv.org/ahs/ tional neural networks for fast and low power mobile applications 1902.03264 [J/0L].ArKiv Preprint(2016-02-24)[2019-03-22].ht- [39]Gupta S,Agrawal A,Gopalakrishnan K,et al.Deep leaming tps://arxiv.org/abs/1511.06530 with limited numerical precision [/OL].ArXir Preprint (2015- [54]Denil M,Shakibi B.Dinh L,et al.Predicting parameters in 02-09)[2019-03-22].https:/amiv.org/abs/1502.02551 deep leaming//Advances in Neural Information Processing Sys- [40]Jacob B,Kligys S,Chen B,et al.Quantization and training of tems.Lake Tahoe,2013:2148 neural networks for efficient integer-arithmetic-only inference// [55]Calvi GG,Moniri A,Mahfouz M,et al.Tucker tensor layer in Proceedings of the IEEE Conference on Computer Vision and Pat- fully connected neural networks [J/OL].ArXig Preprint (2019- tern Recognition.Salt Lake City.2018:2704 03-14)[2019-03-22].htps:/aiv.org/abs/1903.06133 [41] Courbariaux M,Bengio Y,David J P.BinaryConnect:training [56]Bucilua C,Caruana R,Niculescu-Mizil A.Model compression deep neural networks with binary weights during propagations// /Proceedings of the 12th ACM SIGKDD International Conference Adrances in Neural Information Processing Systems.Canada, on Knowledge Discovery and Data Mining.Philadelphia,2006: 2015:3123 535 [42]Courbariaux M,Hubara I,Soudry D,et al.Binarized neural net- [57]Ba J.Caruana R.Do deep nets really need to be deep?//Ad- works:training deep neural networks with weights and activations tances in Neural Information Processing Systems.Canada,2014: constrained to +1 or -1 [J/OL].ArXig Preprint (2016-03- 2654 17)[2019-03-22].https:/arxiv.og/abs/1602.02830 [58]Hinton G.Vinyals O,Dean J.Distilling the knowledge in a neu- [43]Rastegari M,Ordonez V,Redmon J,et al.XNOR-Net:Ima- ral network[J/OL].ArXin Preprint (2015-03-09)[2019-03- geNet classification using binary convolutional neural networks 22].https://arxiv.org/abs/1503.02531 [J/0L].ArXiv Preprint(2016-08-02）[2019-03-22].ht- [59]Romero A,Ballas N,Kahou S E,et al.FitNets:hints for thin tps://arxiv.org/abs/1603.05279 deep nets[J/PL].ArXin Preprint (2015-03-27)[2019-03- [44]Li Z F,Ni B B,Zhang WJ,et al.Performance guaranteed net- 22].https://arxiv.org/abs/1412.6550 work acceleration via high-order residual quantization /Proceed- [60]Chen T Q,Goodfellow I,Shlens J.Net2Net:accelerating leam- ings of the IEEE International Conference on Computer Vision. ing via knowledge transfer[J/OL].ArXie Preprint (2016-04- Venice,2017:2584 23)[2019-03-22].htps:/aiv.og/abs/1511.05641 [45]Hwang K.Sung W.Fixed-point feedforward deep neural network [61]Li ZZ,Hoiem D.Learing without forgetting./EEE Trans Pat- design using weights 1,0,and -1 //2014 IEEE Workshop tern Anal Mach Intell,2018,40(12):2935 on Signal Processing Systems (SiPS).Belfast,2014:1 [62]Zagoruyko S,Komodakis N.Paying more attention to attention: [46]Hou L,Yao Q M,Kwok J T.Loss-aware binarization of deep Improving the performance of convolutional neural networks via networks [J/OL].ArXin Preprint (2018-05-10)[2019-03- attention transfer J/OL ]ArXiv Preprint (2017-02-12) 22].https://arxiv.org/abs/1611.01600 [2019-03-22].htps:/axiv.org/abs/1612.03928 [47]Lee H,Battle A,Raina R,et al.Efficient sparse coding algo- [63]Mirzadeh S I,Farajtabar M,Li A,et al.Improved knowledge rithms /Adeances in Neural Information Processing Systems. distillation via teacher assistant:bridging the gap between student Canada,2007 and teacher [J/OL].ArXir Preprint (2019-02-09)[2019-03-工程科学学报,第 41 卷,第 10 期 works with the hashing trick / / Proceedings of the 32nd Interna鄄 tional Conference on Machine Learning. Lille, 2015: 2285 [34] Shi L, Feng S K, Zhu Z F. Functional hashing for compressing neural networks [J/ OL]. ArXiv Preprint (2016鄄鄄05鄄鄄20) [2019鄄鄄 03鄄鄄22]. https: / / arxiv. org / abs/ 1605. 06560 [35] Wu J R, Wang Y, Wu Z Y, et al. Deep k鄄Means: Re鄄training and parameter sharing with harder cluster assignments for com鄄 pressing deep convolutions [ J/ OL]. ArXiv Preprint (2018鄄鄄 06鄄鄄 24) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1806. 09228 [36] Lu Z Y, Sindhwani V, Sainath T N. Learning compact recurrent neural networks [J/ OL]. ArXiv Preprint (2016鄄鄄04鄄鄄09) [2019鄄鄄 03鄄鄄22]. https: / / arxiv. org / abs/ 1604. 02594 [37] Jin X J, Yang Y Z, Xu N, et al. WSNet: compact and efficient networks through weight sampling [ J/ OL ]. ArXiv Preprint (2018鄄鄄05鄄鄄22) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1711. 10067 [38] Yang Y Z, Jojic N, Huan J. FSNet: Compression of deep convo鄄 lutional neural networks by filter summary [ J/ OL]. ArXiv Pre鄄 print (2019鄄鄄02鄄鄄13) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1902. 03264 [39] Gupta S, Agrawal A, Gopalakrishnan K, et al. Deep learning with limited numerical precision [J/ OL]. ArXiv Preprint (2015鄄鄄 02鄄鄄09) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1502. 02551 [40] Jacob B, Kligys S, Chen B, et al. Quantization and training of neural networks for efficient integer鄄arithmetic鄄only inference / / Proceedings of the IEEE Conference on Computer Vision and Pat鄄 tern Recognition. Salt Lake City, 2018: 2704 [41] Courbariaux M, Bengio Y, David J P. BinaryConnect: training deep neural networks with binary weights during propagations / / Advances in Neural Information Processing Systems. Canada, 2015: 3123 [42] Courbariaux M, Hubara I, Soudry D, et al. Binarized neural net鄄 works: training deep neural networks with weights and activations constrained to + 1 or - 1 [ J/ OL]. ArXiv Preprint (2016鄄鄄 03鄄鄄 17) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1602. 02830 [43] Rastegari M, Ordonez V, Redmon J, et al. XNOR鄄鄄 Net: Ima鄄 geNet classification using binary convolutional neural networks [J/ OL]. ArXiv Preprint (2016鄄鄄 08鄄鄄 02) [2019鄄鄄 03鄄鄄 22]. ht鄄 tps: / / arxiv. org / abs/ 1603. 05279 [44] Li Z F, Ni B B, Zhang W J, et al. Performance guaranteed net鄄 work acceleration via high鄄order residual quantization / / Proceed鄄 ings of the IEEE International Conference on Computer Vision. Venice, 2017: 2584 [45] Hwang K, Sung W. Fixed鄄point feedforward deep neural network design using weights + 1, 0, and - 1 / / 2014 IEEE Workshop on Signal Processing Systems (SiPS). Belfast, 2014: 1 [46] Hou L, Yao Q M, Kwok J T. Loss鄄aware binarization of deep networks [J/ OL]. ArXiv Preprint (2018鄄鄄 05鄄鄄 10) [2019鄄鄄 03鄄鄄 22]. https: / / arxiv. org / abs/ 1611. 01600 [47] Lee H, Battle A, Raina R, et al. Efficient sparse coding algo鄄 rithms / / Advances in Neural Information Processing Systems. Canada, 2007 [48] Gudovskiy D A, Rigazio L. ShiftCNN: generalized low鄄precision architecture for inference of convolutional neural networks [ J/ OL]. ArXiv Preprint (2017鄄鄄06鄄鄄07) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1706. 02393 [49] Meller E, Finkelstein A, Almog U, et al. Same, same but dif鄄 ferent鄄recovering neural network quantization error through weight factorization [J/ OL]. ArXiv Preprint (2019鄄鄄02鄄鄄05) [2019鄄鄄03鄄鄄 22]. https: / / arxiv. org / abs/ 1902. 01917 [50] Xu Y H, Zhang S, Qi Y Y, et al. DNQ: Dynamic network quantization [J/ OL]. ArXiv Preprint (2018鄄鄄12鄄鄄06) [2019鄄鄄03鄄鄄 22]. https: / / arxiv. org / abs/ 1812. 02375 [51] Golub G H, Reinsch C. Singular value decomposition and least squares solutions / / Linear Algebra. Springer, Berlin, 1971: 134 [52] Jaderberg M, Vedaldi A, Zisserman A. Speeding up convolution鄄 al neural networks with low rank expansions [J/ OL]. ArXiv Pre鄄 print (2014鄄鄄05鄄鄄15) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1405. 3866 [53] Kim Y D, Park E, Yoo S, et al. Compression of deep convolu鄄 tional neural networks for fast and low power mobile applications [J/ OL]. ArXiv Preprint (2016鄄鄄 02鄄鄄 24) [2019鄄鄄 03鄄鄄 22]. ht鄄 tps: / / arxiv. org / abs/ 1511. 06530 [54] Denil M, Shakibi B, Dinh L, et al. Predicting parameters in deep learning / / Advances in Neural Information Processing Sys鄄 tems. Lake Tahoe, 2013: 2148 [55] Calvi G G, Moniri A, Mahfouz M, et al. Tucker tensor layer in fully connected neural networks [J/ OL]. ArXiv Preprint (2019鄄鄄 03鄄鄄14) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1903. 06133 [56] Bucilu觍 C, Caruana R, Niculescu鄄Mizil A. Model compression / / Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Philadelphia, 2006: 535 [57] Ba J, Caruana R. Do deep nets really need to be deep? / / Ad鄄 vances in Neural Information Processing Systems. Canada, 2014: 2654 [58] Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neu鄄 ral network[J/ OL]. ArXiv Preprint (2015鄄鄄03鄄鄄09) [2019鄄鄄03鄄鄄 22]. https: / / arxiv. org / abs/ 1503. 02531 [59] Romero A, Ballas N, Kahou S E, et al. FitNets: hints for thin deep nets[ J/ PL]. ArXiv Preprint (2015鄄鄄 03鄄鄄 27) [2019鄄鄄 03鄄鄄 22]. https: / / arxiv. org / abs/ 1412. 6550 [60] Chen T Q, Goodfellow I, Shlens J. Net2Net: accelerating learn鄄 ing via knowledge transfer[ J/ OL]. ArXiv Preprint (2016鄄鄄 04鄄鄄 23) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1511. 05641 [61] Li Z Z, Hoiem D. Learning without forgetting. IEEE Trans Pat鄄 tern Anal Mach Intell, 2018, 40(12): 2935 [62] Zagoruyko S, Komodakis N. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer [ J/ OL ]. ArXiv Preprint ( 2017鄄鄄 02鄄鄄 12 ) [2019鄄鄄03鄄鄄22]. https: / / arxiv. org / abs/ 1612. 03928 [63] Mirzadeh S I, Farajtabar M, Li A, et al. Improved knowledge distillation via teacher assistant: bridging the gap between student and teacher [J/ OL]. ArXiv Preprint (2019鄄鄄02鄄鄄09) [2019鄄鄄03鄄鄄 ·1238·