JS010862817B2 d2United States Patent ao)Patent No.:US 10,862,817 B2 Xue et al. (45)Date of Patent: Dec.8,2020 (54)CONGESTION CONTROL METHOD, (56) References Cited APPARATUS,AND SYSTEM U.S.PATENT DOCUMENTS (71)Applicants:Huawei Technologies Co.,Ltd. Shenzhen (CN):University of Science 9,391,911B1* 7/2016 Anderson …H04L1/1628 2010/0246400A1*9/20100 nishi H04L47/28 and Technology of China,Hefei (CN) 370/235 (72)Inventors:Kaiping Xue.Hefei (CN):Wenjia Wei. (Continued) Hefei (CN);Qing Xu,Shanghai (CN) FOREIGN PATENT DOCUMENTS (73)Assignees:Huawei Technologies Co.,Ltd., Shenzhen (CN);University of Science 贺 102318312A 1/2012 103581035A 22014 and Technology of China,Hefei(CN) (Continued) (*Notice: Subject to any disclaimer,the term of this OTHER PUBLICATIONS patent is extended or adjusted under 35 U.S.C.154(b)by 2 days. Ferlin et al.,"Revisiting Congestion Control for MultipathTCP with Shared Bottleneck Detection,"IEEE Infocom 2016-The 35th (21)Appl.No.:16/357,564 Annual IEEE International Conference on Computer Communica- tions,XP032930233,pp.1-9,Institute of Electrical and Electronics (22)Filed: Mar.19,2019 Engineers,New York,New York (Apr.2016). (Continued) (65) Prior Publication Data US2019/0215276A1Jul.11.2019 Primary Examiner-Phong La (74)Attorney,Agent,or Firm-Leydig,Voit Mayer, Related U.S.Application Data Ltd. (63)Continuation of application No. (57) ABSTRACT PCT/CN2016/099466,filed on Sep.20,2016. Embodiments of the present disclosure provide a congestion control method,an apparatus,and a system.The method (51)Int.Cl. includes receiving,by a first device,an acknowledgement H04L.12891 (2013.01) packet that is from a second device and that includes an ECE H04L12801 (2013.01) identifier.Based on an indication of the ECE identifier the (Continued) first device starts monitoring a plurality of sub streams on a (52)U.S.C. plurality of transmission paths between the first device and CPC… H04L47/41(2013.01);H04L45/24 the second device,and grouping,into one set,sub streams (2013.01):Ⅱ04L47/11(2013.01):I04L that meet a congestion control condition.Additionally,intra- 47/115(2013.01)月 set congestion control is performed on the sub streams in the (Continued) set.Separately congestion control is performed on a sub stream that is not in the set.The congestion control method (58)Field of Classification Search is applied to a scenario in which data transmission is CP℃，.H04L47/41;H04L47/193;H04L47/1; performed between the first device and the second device by H04L47/2482:H04L47/115:H04L using the MPTCP. 45/24:H04L43/0829:H04L43/0852 (Continued) 6 Claims,9 Drawing Sheets Routing device 12 Transmit Receive end 10 end 11
US010862817B2 ( 12 ) United States Patent ( 10 ) Patent No .: US 10,862,817 B2 Xue et al . ( 45 ) Date of Patent : Dec. 8 , 2020 ( 54 ) CONGESTION CONTROL METHOD , ( 56 ) References Cited APPARATUS , AND SYSTEM U.S. PATENT DOCUMENTS ( 71 ) Applicants : Huawei Technologies Co. , Ltd. , Shenzhen ( CN ) ; University of Science and Technology of China , Hefei ( CN ) 9,391,911 B1 * 2010/0246400 A1 * 7/2016 Anderson 9/2010 Onishi HO4L 1/1628 HO4L 47/28 370/235 ( Continued ) ( 72 ) Inventors : Kaiping Xue , Hefei ( CN ) ; Wenjia Wei , Hefei ( CN ) ; Qing Xu , Shanghai ( CN ) FOREIGN PATENT DOCUMENTS ( 73 ) Assignees : Huawei Technologies Co. , Ltd. , Shenzhen ( CN ) ; University of Science and Technology of China , Hefei ( CN ) CN CN 102318312 A 1/2012 103581035 A 2/2014 ( Continued ) ( * ) Notice : OTHER PUBLICATIONS Subject to any disclaimer , the term of this patent is extended or adjusted under 35 U.S.C. 154 ( b ) by 2 days . ( 21 ) Appl . No .: 16 / 357,564 Ferlin et al . , “ Revisiting Congestion Control for MultipathTCP with Shared Bottleneck Detection , ” IEEE Infocom 2016 — The 35th Annual IEEE International Conference on Computer Communica tions , XP032930233 , pp . 1-9 , Institute of Electrical and Electronics Engineers , New York , New York ( Apr. 2016 ) . ( Continued ) ( 22 ) Filed : Mar. 19 , 2019 ( 65 ) Prior Publication Data US 2019/0215276 A1 Jul . 11 , 2019 Primary Examiner - Phong La ( 74 ) Attorney , Agent , or Firm - Leydig , Voit & Mayer , Ltd. Related U.S. Application Data ( 63 ) Continuation of application PCT / CN2016 / 099466 , filed on Sep. 20 , 2016 . No. ( 51 ) Int . Ci . H04L 12/891 ( 2013.01 ) H04L 12/801 ( 2013.01 ) ( Continued ) ( 52 ) U.S. Ci . CPC H04L 47/41 ( 2013.01 ) ; H04L 45/24 ( 2013.01 ) ; H04L 47/11 ( 2013.01 ) ; H04L 47/115 ( 2013.01 ) ; ( Continued ) ( 58 ) Field of Classification Search CPC HO4L 47/41 ; H04L 47/193 ; H04L 47/11 ; H04L 47/2482 ; H04L 47/115 ; HO4L 45/24 ; H04L 43/0829 ; H04L 43/0852 ( Continued ) ( 57 ) ABSTRACT Embodiments of the present disclosure provide a congestion control method , an apparatus , and a system . The method includes receiving , by a first device , an acknowledgement packet that is from a second device and that includes an ECE identifier . Based on an indication of the ECE identifier the first device starts monitoring a plurality of sub streams on a plurality of transmission paths between the first device and the second device , and grouping , into one set , sub streams that meet a congestion control condition . Additionally , intra set congestion control is performed on the sub streams in the set . Separately congestion control is performed on a sub stream that is not in the set . The congestion control method is applied to a scenario in which data transmission is performed between the first device and the second device by using the MPTCP . 6 Claims , 9 Drawing Sheets Routing device 12 000 000 000 Transmit end 10 Receive end 11
US10,862,817B2 Page 2 (51)Int.C. FOREIGN PATENT DOCUMENTS H04L12/707 (2013.01) H04L12807 (2013.01) 103973587A 8/2014 H04L12/851 (2013.01) 104796350A 72015 wo 2015174901A1 11/2015 H04L12/26 (2006.01) (52)U.S.C1. OTHER PUBLICATIONS CPC....... H04L47193(2013.01):H04L472483 2013.01):H04L47/27(2013.01:H04L Braden et al.."Recommendations on Queue Management and 43/0829(2013.01H04L43/0852(2013.01) Congestion Avoidance in the Internet,"RFC 2309,The Internet (58)Field of Classification Search Society,XP055604783,Internet Engineering Task Force.Reston, USPC 370/236 Virginia (Apr.1998). See application file for complete search history. Floyd et al.."Random Early Detection Gateways for Congestion Avoidance,"IEEE/ACM Transactions on Networking vol.I.No.4. pp.1-22,Institute of Electrical and Electronics Engineers,New (56) References Cited York,New York (Aug.1993). Hassayoun et al.,"Dynamic Window Coupling for Multipath Con- U.S.PATENT DOCUMENTS gestion Control,"Proceedings of the 19th IEEE International Con- ference on Network Protocols (ICNP).pp.1-12.Institute of Elec- 2011/0292801A112/2011 Turnbull et al. trical and Electronics Engineers.New York,New York (2011). 2012/0054583A1◆3/2012Pak. H03M13/6306 Xue et al.,"Migrating Unfairness Among Subflows in MPTCP With 714/776 Network Coding for Wired-Wireless Networks",IEEE Transactions 2013/0182567A1◆ 7/2013 Scheffenegger .......H04L 47/12 370/23 on Vehicular Technology,vol.66,No.1,pp.798-809,Institute of Electrical and Electronics Engineers.New York.New York (Jan. 2014/0164641A1.6/2014Ye H04L47/11 2017)- 709/235 2015/0043345A12/2015 Testicioglu Ramakrishnan et al.,"The Addition of Explicit Congestion Notifi- .H04L47/24 370/232 cation (ECN)to IP,"Request for Comments:3168,pp.1-63 (Sep. 2015/0085665A13/2015 Kompella H04L47/20 2001). Xu et al.,"Delay-based Congestion Control for MPTCP:draft-xu- 370/236 2015/0138978A1◆ 5/2015 Ishikawa......... H04L45/74 mptcp-congestion-control-04.txt,"Multipath TCP.Internet Draft. Intended status:Standard Track (Jul.3,2016). 370)39 2016/0014030A1◆ 1/2016 Thyni...H04L47/115 Alizadeh et al.,"Data Center TCP(DCTCP),"SIGCOMM ACM. 370/237 New Delhi.India (Aug.30-Sep.3.2010). 2016/0088645A1◆ 3/2016 Burhan H04W7202 Raiciu et al."Coupled Congestion Control for Multipath Transport 370/349 Protocols,"Request for Comments:6356.ISSN:2070-1721.Inter- 2017/0180261A1◆6/2017Ma H04L.5/0055 net Engineering Task Force,Reston,Virginia (Oct.2011). 2017/0366374A1◆1220170sug4H04L47/14 20180220325A1◆8/2018Lcc H04W7210 cited by examiner
US 10,862,817 B2 Page 2 FOREIGN PATENT DOCUMENTS CN CN WO 103973587 A 104796350 A 2015174901 A1 8/2014 7/2015 11/2015 ( 51 ) Int . Ci . H04L 12/707 ( 2013.01 ) H04L 12/807 ( 2013.01 ) H04L 12/851 ( 2013.01 ) H04L 12/26 ( 2006.01 ) ( 52 ) U.S. CI . CPC H04L 47/193 ( 2013.01 ) ; H04L 47/2483 ( 2013.01 ) ; H04L 47/27 ( 2013.01 ) ; HO4L 43/0829 ( 2013.01 ) ; H04L 43/0852 ( 2013.01 ) ( 58 ) Field of Classification Search USPC 370/236 See application file for complete search history . OTHER PUBLICATIONS ( 56 ) References Cited U.S. PATENT DOCUMENTS 2011/0292801 Al 2012/0054583 A1 * 12/2011 Turnbull et al . 3/2012 Park 2013/0182567 A1 * 7/2013 Scheffenegger Braden et al . , “ Recommendations on Queue Management and Congestion Avoidance in the Internet , ” RFC , The Internet Society , XP055604783 , Internet Engineering Task Force , Reston , Virginia ( Apr. 1998 ) . Floyd et al , “ Random Early Detection Gateways for Congestion Avoidance , ” IEEE / ACM Transactions on Networking vol . 1 , No. 4 , pp . 1-22 , Institute of Electrical and Electronics Engineers , New York , New York ( Aug. 1993 ) . Hassayoun et al . , “ Dynamic Window Coupling for Multipath Con gestion Control , ” Proceedings of the 19th IEEE International Con ference on Network Protocols ( ICNP ) , pp . 1-12 , Institute of Elec trical and Electronics Engineers , New York , New York ( 2011 ) . Xueetal , “ Migrating UnfairnessAmong Subflowsin MPTCPWith Network Coding for Wired - Wireless Networks ” , IEEE Transactions on Vehicular Technology , vol . 66 , No. 1 , pp . 798-809 , Institute of Electrical and Electronics Engineers , New York , New York ( Jan. 2017 ) . Ramakrishnan et al . , “ The Addition of Explicit Congestion Notific cation ( ECN ) to IP , ” Request for Comments : 3168 , pp . 1-63 ( Sep. 2001 ) . Xuet al . , « Delay - based Congestion Control for MPTCP ; draft - xu . mptcp - congestion - control - 04.txt ” Multipath TCP , Internet Draft , Intended status : Standard Track ( Jul . 3 , 2016 ) . Alizadeh et al . , “ Data Center TCP ( DCTCP ) , ” SIGCOMM ACM , New Delhi , India ( Aug. 30 - Sep . 3 , 2010 ) . Raiciu et al . , “ Coupled Congestion Control for Multipath Transport Protocols , ” Request for Comments : 6356 , ISSN : 2070-1721 , Inter net Engineering Task Force , Reston , Virginia ( Oct. 2011 ) . 2014/0164641 A1 * 6/2014 Ye 2015/0043345 A1 * 2/2015 Testicioglu 2015/0085665 A1 * HO3M 13/6306 714/776 HO4L 47/12 370/231 HO4L 47/11 709/235 H04L 47/24 370/232 HO4L 47/20 370/236 H04L 45/74 370/235 HO4L 47/115 370/237 HO4W 72/02 370/349 HO4L 5/0055 H04L 47/14 HO4W 72/10 3/2015 Kompella 2015/0138978 A1 * 5/2015 Ishikawa 2016/0014030 A1 * 1/2016 Thyni 2016/0088645 A1 * 3/2016 Burhan 2017/0180261 A1 * 2017/0366374 A1 * 2018/0220325 A1 * 6/2017 Ma 12/2017 Osuga 8/2018 Lee * cited by examiner
U.S.Patent Dec.8,2020 Sheet 1 of 9 US10,862,817B2 Routing device 12 Transmit Receive end 10 end 11 FIG.1 Radio frequency Mobile phone circuit 110 Audio -il Speaker 151 Memory circuit 150 Microphone 152 120 Processor 160 Power supply 170 Input unit 130 Display unit 140 Touch panel 131 Display panel Another input 141 device 132 FIG 2
U.S. Patent Dec. 8 , 2 2020 Sheet 1 of 9 US 10,862,817 B2 Routing device 12 Transmit end 10 Receive end 11 FIG . 1 Radio frequency Mobile phone circuit 110 Speaker 151 Audio Memory circuit 150 120 + Microphone 152 Processor 160 Power supply 170 Input unit 130 Display unit 140 Touch panel 131 Another input device 132 Display panel 141 FIG . 2
U.S.Patent Dec.8,2020 Sheet 2 of 9 US10,862,817B2 Router Memory Processor Interface 220 210 230 FIG.3
U.S. Patent Dec. 8 , 2 2020 Sheet 2 of 9 US 10,862,817 B2 Router Memory 220 Processor 210 Interface 230 FIG . 3
U.S.Patent Dec.8,2020 Sheet 3 of 9 US10,862,817B2 First device Routing device Second device S101.A first device sends a first data packet to a routing device S102.The routing device receives the first data packet S103.The routing device determines that an average queue length of a data packet queue is greater than or equal to a first preset threshold and is less than or equal to a second preset threshold 工 S104.The routing device makes a congestion mark for the first data packet,and adds the first data packet to the data packet queue S105.The routing device sends the first data packet to a second device S106.The second device receives the first data packet S107.The second device makes an ECE mark for a first acknowledgement packet S108.The second device sends the first acknowledgement packet to the first device S109.The first device receives the first acknowledgement packet S110.The first device determines a first sub stream set S111.The first device performs congestion control on sub streams in the first sub stream set FIG.4
U.S. Patent Dec. 8 , 2 2020 Sheet 3 of 9 US 10,862,817 B2 First device Routing device Second device S101 . A first device sends a first data packet to a routing device S102 . The routing device receives the first data packet S103 . The routing device determines that an average queue length of a data packet queue is greater than or equal to a first preset threshold and is less than or equal to a second preset threshold S104 . The routing device makes a congestion mark for the first data packet , and adds the first data packet to the data packet queue $ 105 . The routing device sends the first data packet to a second device S106 . The second device receives the first data packet S107 . The second device makes an ECE mark for a first acknowledgement packet S108 . The second device sends the first acknowledgement packet to the first device S109 . The first device receives the first acknowledgement packet S110 . The first device determines a first sub stream set S111 . The first device performs congestion control on sub streams in the first sub stream set FIG . 4
U.S.Patent Dec.8,2020 Sheet 4 of 9 US10,862,817B2 X Y RTT, RTT. RTT, 2 T 2 T+ RTT 2 2 FIG.5
U.S. Patent Dec. 8 , 2 2020 Sheet 4 of 9 US 10,862,817 B2 X Y RTT RTT . T RTT , 2 T 2 . T + RTT 2 2 FIG . 5
U.S.Patent Dec.8,2020 Sheet 5 of 9 US10,862,817B2 First device Routing device Second device S101.A first device sends a first data packet to a routing device S102.The routing device receives the first data packet S103.The routing device determines that an average queue length of a data packet queue is greater than or equal to a first preset threshold and is less than or equal to a second preset threshold S104.The routing device makes a congestion mark for the first data packet,and adds the first data packet to the data packet queue S105.The routing device sends the first data packet to a second device S106.The second device receives the first data packet S107.The second device makes an ECE mark for a first acknowledgement packet S108.The second device sends the first acknowledgement packet to the first device S109.The first device receives the first acknowledgement packet S110.The first device determines a first sub stream set S112.The first device sends a second data packet to the routing device TO TO TO FIG.6B FIG.6B FIG.6B FIG.6A
U.S. Patent Dec. 8 , 2 2020 Sheet 5 of 9 US 10,862,817 B2 First device Routing device Second device S101 . A first device sends a first data packet to a routing device S102 . The routing device receives the first data packet S103 . The routing device determines that an average queue length of a data packet queue is greater than or equal to a first preset threshold and is less than or equal to a second preset threshold S104 . The routing device makes a congestion mark for the first data packet , and adds the first data packet to the data packet queue $ 105 . The routing device sends the first data packet to a second device S106 . The second device receives the first data packet S107 . The second device makes an ECE mark for a first acknowledgement packet S108 . The second device sends the first acknowledgement packet to the first device S109 . The first device receives the first acknowledgement packet $ 110 . The first device determines a first sub stream set S112 . The first device sends a second data packet to the routing device TO FIG . 6B TO FIG . 6B TO FIG . 6B FIG . 6A
U.S.Patent Dec.8,2020 Sheet 6 of 9 US10,862,817B2 CONT. CONT. CONT. FROM FROM FROM FIG.6A FIG.6A FIG.6A S113.The routing device receives the second data packet S114.The routing device determines that an average queue length of a data packet queue is greater than or equal to the first preset threshold and is less than or equal to the second preset threshold S115.The routing device makes a congestion mark for the second data packet,and adds the second data packet to the data packet queue S116.The routing device sends the second data packet to the second device S117.The second device receives the second data packet S118.The second device makes an ECE mark for a second acknowledgement packet S119.The second device sends the second acknowledgement packet to the first device S120.The first device receives the second acknowledgement packet S121.The first device determines a second sub stream set S111a.The first device performs congestion control on sub streams in the second sub stream set FIG.6B
U.S. Patent Dec. 8. 2 2020 Sheet 6 of 9 US 10,862,817 B2 CONT . CONT . CONT . FROM FROM FROM FIG . 6A FIG . 6A FIG . 6A Y S113 . The routing device receives the second data packet S114 . The routing device determines that an average queue length of a data packet queue is greater than or equal to the first preset threshold and is less than or equal to the second preset threshold S115 . The routing device makes a congestion mark for the second data packet , and adds the second data packet to the data packet queue S116 . The routing device sends the second data packet to the second device S117 . The second device receives the second data packet S118 . The second device makes an ECE mark for a second acknowledgement packet S119 . The second device sends the second acknowledgement packet to the first device S120 . The first device receives the second acknowledgement packet S121 . The first device determines a second sub stream set Silla . The first device performs congestion control on sub streams in the second sub stream set FIG . 6B
U.S.Patent Dec.8,2020 Sheet 7 of 9 US10,862,817B2 First device Sending module 310 Receiving module 320 Determining module 330 Control module 340 FIG 7 First device Storage module 410 Processing module 420 Communications module 430 FIG.8
U.S. Patent Dec. 8 , 2 2020 Sheet 7 of 9 US 10,862,817 B2 First device Sending module 310 Receiving module 320 Determining module 330 Control module 340 FIG . 7 First device Storage module 410 Processing module 420 Communications module 430 FIG . 8
U.S.Patent Dec.8,2020 Sheet 8 of 9 US10,862,817B2 Routing device Sending module 510 Determining module 530 Receiving module 520 FIG.9 Routing device Storage module 610 Processing module 620 Communications module 630 FIG.10
U.S. Patent Dec. 8 , 2 2020 Sheet 8 of 9 US 10,862,817 B2 Routing device Sending module 510 Determining module 530 Receiving module 520 FIG . 9 Routing device Storage module 610 Processing module 620 Communications module 630 FIG . 10