1038191 - New highest bandwidth peak in China data center

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task, P2)

Description

[Yousheng Li [:yli]]

Attachment: Peak-Bandwidth-China-20140714.png

Last year we report an bandwidth cost bug: Bug 897456
On that time, the peak of the bandwidth is about 90Mb.
But the peak is up to about 120Mb recently.

Not like US, the bandwidth traffic in China is cost by peak number.
Like the attachment, when the real time traffic is up to 120Mb, the traffic will be cut.
When that happened, it will lost packet, and the quality of the network will become poor.

I don't know why the peak jump from 90Mb to 120Mb (or more), Could you help to check which service make this happen? 

On my testing, looks like the xpi files are not using cdn on AMO service.
If we uploading server logs from China to US during the bandwidth peak time, could we change the uploading time?

Comment 1

[Arzhel Younsi [:XioNoX]]

Attachment: pek1-traffic-day.png

According to our trending, most of the traffic comes from the Load Balancers, and according to them, most of the traffic is for addons.mozilla.org (and it seems to match the pattern of your screenshot).

I'm moving this bug to WebOps so they can look deeper at what's going on.

Comment 2

[Yousheng Li [:yli]]

Hi Arzhel,
Thanks a lot!

Comment 3

[Peter Radcliffe [:pir]]

Dropping priority so this doesn't keep paging before webops/addons folks are awake.

Comment 4

[Jake Maul [:jakem]]

WebOps doesn't actually handle AMO... that's part of Cloud Services. Kicking this over to the AMO Ops folks. :)

Comment 5

[Jeremy Orem [:oremj]]

Here are the top 10 urls rolled up at different depths:

URL DEPTH = 1
/pt-PT: 88151838 (0.03%)
/zh-tw: 97563141 (0.03%)
/pt-BR: 110034314 (0.04%)
/ar: 113124117 (0.04%)
/ja: 117211907 (0.04%)
/zh-cn: 137933954 (0.05%)
/zh-TW: 284432268 (0.09%)
/en-US: 649486946 (0.22%)
/zh-CN: 2468552606 (0.82%)
/blocklist: 296297592192 (98.37%)

URL DEPTH = 2
/blocklist/1: 69302871 (0.02%)
/ar/firefox: 69521664 (0.02%)
/zh-tw/firefox: 74657631 (0.02%)
/ja/firefox: 87298546 (0.03%)
/zh-cn/firefox: 111386494 (0.04%)
/zh-TW/firefox: 180670091 (0.06%)
/blocklist/2: 357009899 (0.12%)
/en-US/firefox: 512657348 (0.17%)
/zh-CN/firefox: 2158984948 (0.72%)
/blocklist/3: 295871279422 (98.31%)

URL DEPTH = 3
/zh-CN/firefox/files: 133112930 (0.04%)
/zh-CN/firefox/users: 150348830 (0.05%)
/blocklist/2/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D: 153364975 (0.05%)
/en-US/firefox/addon: 181929145 (0.06%)
/blocklist/2/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D: 198806242 (0.07%)
/zh-CN/firefox/addon: 272173831 (0.09%)
/zh-CN/firefox/collections: 1182189579 (0.39%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D: 4178037708 (1.39%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D: 4404993414 (1.47%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D: 287177279542 (95.56%)

URL DEPTH = 4
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0: 2858096734 (0.95%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0: 3129950081 (1.05%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.6.0: 3225466534 (1.08%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.0: 3245811996 (1.08%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/31.0: 3492679221 (1.17%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0: 4006417723 (1.34%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1: 6163899025 (2.06%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0: 9861967734 (3.29%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1: 21155027253 (7.06%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0: 209843065347 (70.08%)

URL DEPTH = 5
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0/Thunderbird: 2853066279 (0.96%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0/Fennec: 3129847915 (1.05%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.6.0/Firefox: 3148840674 (1.05%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.0/Firefox: 3245673150 (1.09%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/31.0/Firefox: 3492123837 (1.17%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0/Firefox: 4006278876 (1.34%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1/Firefox: 6151402861 (2.06%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0/Firefox: 9860856965 (3.30%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1/Firefox: 21148640330 (7.08%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox: 209807981316 (70.29%)

URL MAX DEPTH
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/4/4/1: 916596617 (0.30%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release/Windows_NT%206.1/default/default/1/1/new: 970936210 (0.32%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/2/2/1: 977411460 (0.32%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/3/3/1: 985386465 (0.33%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/3/3/1: 1091415049 (0.36%)
/zh-CN/firefox/collections/mozilla/featured-personas: 1126788103 (0.37%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release/Windows_NT%205.1/default/default/1/1/new: 1163203596 (0.39%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/2/2/1: 1223526378 (0.41%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/1/1/new: 2483278921 (0.82%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/1/1/new: 3061257900 (1.02%)

Comment 6

[Jeremy Orem [:oremj]]

This all appears to be legitimate blocklist traffic.

Comment 7

[Yousheng Li [:yli]]

Hi Jeremy,
I have some questions:
1.The number after url is the traffic? Bit or byte?
2.What's the period time of the result?
3.How about the PV?

Comment 8

[Jeremy Orem [:oremj]]

(In reply to Yousheng Li [:yli] from comment #7)
> Hi Jeremy,
> I have some questions:
> 1.The number after url is the traffic? Bit or byte?
Byte

> 2.What's the period time of the result?
24 hours, this was 7/16

> 3.How about the PV?
Page Views
URL DEPTH = 1
/ar: 11807 (0.42%)
/pt-BR: 12472 (0.45%)
/zh-cn: 14904 (0.54%)
/collections: 18421 (0.66%)
/zh-TW: 33766 (1.21%)
/: 46785 (1.68%)
/en-US: 53138 (1.91%)
/firefox: 84186 (3.03%)
/zh-CN: 206473 (7.43%)
/blocklist: 2162603 (77.81%)

URL DEPTH = 2
/ar/firefox: 7128 (0.26%)
/zh-tw/firefox: 8546 (0.31%)
/zh-cn/firefox: 12010 (0.44%)
/collections/mozilla: 12416 (0.46%)
/it/: 18232 (0.67%)
/zh-TW/firefox: 21009 (0.77%)
/en-US/firefox: 45366 (1.67%)
/firefox/downloads: 77157 (2.84%)
/zh-CN/firefox: 174628 (6.42%)
/blocklist/3: 2158618 (79.32%)

URL DEPTH = 3
/zh-CN/firefox/files: 12649 (0.48%)
/en-US/firefox/addon: 20341 (0.77%)
/zh-CN/firefox/users: 26592 (1.01%)
/firefox/downloads/latest: 32073 (1.22%)
/zh-CN/firefox/addon: 33099 (1.26%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D: 40753 (1.55%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D: 43294 (1.65%)
/firefox/downloads/file: 45069 (1.71%)
/zh-CN/firefox/collections: 56583 (2.15%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D: 2073557 (78.90%)

URL DEPTH = 4
/zh-CN/firefox/users/login?to=: 26591 (1.08%)
/zh-CN/firefox/collections/mine: 26691 (1.08%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0: 27883 (1.13%)
/zh-CN/firefox/collections/mozilla: 27973 (1.13%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0: 28934 (1.17%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0: 30763 (1.25%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1: 44508 (1.80%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0: 71234 (2.89%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1: 152746 (6.19%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0: 1515071 (61.42%)

URL DEPTH = 5
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.0/Firefox: 23448 (0.95%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/31.0/Firefox: 25227 (1.02%)
/zh-CN/firefox/users/login?to=/zh-CN: 26590 (1.07%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0/Thunderbird: 27834 (1.12%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0/Firefox: 28933 (1.17%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0/Fennec: 30762 (1.24%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1/Firefox: 44418 (1.79%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0/Firefox: 71226 (2.88%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1/Firefox: 152700 (6.17%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox: 1514818 (61.18%)

URL MAX DEPTH
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release/Windows_NT%205.1/default/default/1/1/new: 8398 (0.31%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/2/2/1: 8834 (0.32%)
/collections/mozilla/featured-personas: 12416 (0.45%)
/zh-CN/firefox/search: 13019 (0.47%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/1/1/new: 17955 (0.65%)
/it: 18770 (0.68%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/1/1/new: 22107 (0.80%)
/zh-CN/firefox/users/login?to=/zh-CN/firefox/collections/mine/favorites: 26498 (0.96%)
/zh-CN/firefox/collections/mine/favorites: 26691 (0.97%)
/zh-CN/firefox/collections/mozilla/featured-personas: 26731 (0.97%)

Comment 9

[Yousheng Li [:yli]]

Hi Jeremy,

Looks like the problem caused by the blocklist requests.
Could we limit the peak of the bandwidth? (Ideally we hope the peak can be limit to 60Mb)
Did we limit the rate of the request? If we set, what's the value of it?
Is it possible we just proxy the AMO websites, and keep the blocklist requests back to US?

Comment 10

[Jeremy Orem [:oremj]]

We are working on decoupling blocklist from amo in bug 981663.

Comment 11

[Yousheng Li [:yli]]

Thanks, so when this changes will be online?
The mana wiki said: 2014-07-24 1400PDT.
Is it correct?

Comment 12

[Jason Thomas [:jason]]

We are waiting on two dependencies to be completed before we proceed, see bug 1020320 comment 13 for more information.

Comment 13

[Yousheng Li [:yli]]

(In reply to Jason Thomas [:jason] from comment #12)
> We are waiting on two dependencies to be completed before we proceed, see
> bug 1020320 comment 13 for more information.

Hi Jason,
Do you know when the bugs on comment 13 can be fixed?
And what's the plan for blocklist.addons.mozilla.org?
Thanks!

Comment 14

[Jason Thomas [:jason]]

I've asked for a update in bug 1020320.

Comment 15

[Jason Thomas [:jason]]

bug 1020320 is now completed. I checked the pek1 zlb logs and blocklist traffic is being redirected to blocklist.addons.mozilla.org. :yli how do the bandwidth graphs look now?

Comment 16

[Yousheng Li [:yli]]

Hi Jason, 

The peak bandwidth of pek1 datacenter is dropped.

Thanks for all!