New highest bandwidth peak in China data center

RESOLVED FIXED

Status

Cloud Services
Operations: Marketplace
P2
normal
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: yli, Assigned: oremj)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

4 years ago
Created attachment 8455360 [details]
Peak-Bandwidth-China-20140714.png

Last year we report an bandwidth cost bug: Bug 897456
On that time, the peak of the bandwidth is about 90Mb.
But the peak is up to about 120Mb recently.

Not like US, the bandwidth traffic in China is cost by peak number.
Like the attachment, when the real time traffic is up to 120Mb, the traffic will be cut.
When that happened, it will lost packet, and the quality of the network will become poor.

I don't know why the peak jump from 90Mb to 120Mb (or more), Could you help to check which service make this happen? 

On my testing, looks like the xpi files are not using cdn on AMO service.
If we uploading server logs from China to US during the bandwidth peak time, could we change the uploading time?
Assignee: network-operations → arzhel
Created attachment 8455996 [details]
pek1-traffic-day.png

According to our trending, most of the traffic comes from the Load Balancers, and according to them, most of the traffic is for addons.mozilla.org (and it seems to match the pattern of your screenshot).

I'm moving this bug to WebOps so they can look deeper at what's going on.
Component: NetOps: Other → WebOps: Other
QA Contact: jbarnell → nmaul
(Reporter)

Comment 2

4 years ago
Hi Arzhel,
Thanks a lot!
Assignee: arzhel → server-ops-webops
Dropping priority so this doesn't keep paging before webops/addons folks are awake.
Severity: major → normal
Priority: -- → P2

Comment 4

4 years ago
WebOps doesn't actually handle AMO... that's part of Cloud Services. Kicking this over to the AMO Ops folks. :)
Assignee: server-ops-webops → server-ops-amo
Component: WebOps: Other → Server Operations: AMO Operations
Product: Infrastructure & Operations → mozilla.org
QA Contact: nmaul → oremj
(Assignee)

Updated

4 years ago
Assignee: server-ops-amo → oremj
(Assignee)

Comment 5

4 years ago
Here are the top 10 urls rolled up at different depths:

URL DEPTH = 1
/pt-PT: 88151838 (0.03%)
/zh-tw: 97563141 (0.03%)
/pt-BR: 110034314 (0.04%)
/ar: 113124117 (0.04%)
/ja: 117211907 (0.04%)
/zh-cn: 137933954 (0.05%)
/zh-TW: 284432268 (0.09%)
/en-US: 649486946 (0.22%)
/zh-CN: 2468552606 (0.82%)
/blocklist: 296297592192 (98.37%)

URL DEPTH = 2
/blocklist/1: 69302871 (0.02%)
/ar/firefox: 69521664 (0.02%)
/zh-tw/firefox: 74657631 (0.02%)
/ja/firefox: 87298546 (0.03%)
/zh-cn/firefox: 111386494 (0.04%)
/zh-TW/firefox: 180670091 (0.06%)
/blocklist/2: 357009899 (0.12%)
/en-US/firefox: 512657348 (0.17%)
/zh-CN/firefox: 2158984948 (0.72%)
/blocklist/3: 295871279422 (98.31%)

URL DEPTH = 3
/zh-CN/firefox/files: 133112930 (0.04%)
/zh-CN/firefox/users: 150348830 (0.05%)
/blocklist/2/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D: 153364975 (0.05%)
/en-US/firefox/addon: 181929145 (0.06%)
/blocklist/2/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D: 198806242 (0.07%)
/zh-CN/firefox/addon: 272173831 (0.09%)
/zh-CN/firefox/collections: 1182189579 (0.39%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D: 4178037708 (1.39%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D: 4404993414 (1.47%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D: 287177279542 (95.56%)

URL DEPTH = 4
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0: 2858096734 (0.95%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0: 3129950081 (1.05%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.6.0: 3225466534 (1.08%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.0: 3245811996 (1.08%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/31.0: 3492679221 (1.17%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0: 4006417723 (1.34%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1: 6163899025 (2.06%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0: 9861967734 (3.29%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1: 21155027253 (7.06%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0: 209843065347 (70.08%)

URL DEPTH = 5
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0/Thunderbird: 2853066279 (0.96%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0/Fennec: 3129847915 (1.05%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.6.0/Firefox: 3148840674 (1.05%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.0/Firefox: 3245673150 (1.09%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/31.0/Firefox: 3492123837 (1.17%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0/Firefox: 4006278876 (1.34%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1/Firefox: 6151402861 (2.06%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0/Firefox: 9860856965 (3.30%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1/Firefox: 21148640330 (7.08%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox: 209807981316 (70.29%)

URL MAX DEPTH
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/4/4/1: 916596617 (0.30%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release/Windows_NT%206.1/default/default/1/1/new: 970936210 (0.32%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/2/2/1: 977411460 (0.32%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/3/3/1: 985386465 (0.33%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/3/3/1: 1091415049 (0.36%)
/zh-CN/firefox/collections/mozilla/featured-personas: 1126788103 (0.37%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release/Windows_NT%205.1/default/default/1/1/new: 1163203596 (0.39%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/2/2/1: 1223526378 (0.41%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/1/1/new: 2483278921 (0.82%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/1/1/new: 3061257900 (1.02%)
(Assignee)

Comment 6

4 years ago
This all appears to be legitimate blocklist traffic.
(Reporter)

Comment 7

4 years ago
Hi Jeremy,
I have some questions:
1.The number after url is the traffic? Bit or byte?
2.What's the period time of the result?
3.How about the PV?
(Assignee)

Comment 8

4 years ago
(In reply to Yousheng Li [:yli] from comment #7)
> Hi Jeremy,
> I have some questions:
> 1.The number after url is the traffic? Bit or byte?
Byte

> 2.What's the period time of the result?
24 hours, this was 7/16

> 3.How about the PV?
Page Views
URL DEPTH = 1
/ar: 11807 (0.42%)
/pt-BR: 12472 (0.45%)
/zh-cn: 14904 (0.54%)
/collections: 18421 (0.66%)
/zh-TW: 33766 (1.21%)
/: 46785 (1.68%)
/en-US: 53138 (1.91%)
/firefox: 84186 (3.03%)
/zh-CN: 206473 (7.43%)
/blocklist: 2162603 (77.81%)

URL DEPTH = 2
/ar/firefox: 7128 (0.26%)
/zh-tw/firefox: 8546 (0.31%)
/zh-cn/firefox: 12010 (0.44%)
/collections/mozilla: 12416 (0.46%)
/it/: 18232 (0.67%)
/zh-TW/firefox: 21009 (0.77%)
/en-US/firefox: 45366 (1.67%)
/firefox/downloads: 77157 (2.84%)
/zh-CN/firefox: 174628 (6.42%)
/blocklist/3: 2158618 (79.32%)

URL DEPTH = 3
/zh-CN/firefox/files: 12649 (0.48%)
/en-US/firefox/addon: 20341 (0.77%)
/zh-CN/firefox/users: 26592 (1.01%)
/firefox/downloads/latest: 32073 (1.22%)
/zh-CN/firefox/addon: 33099 (1.26%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D: 40753 (1.55%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D: 43294 (1.65%)
/firefox/downloads/file: 45069 (1.71%)
/zh-CN/firefox/collections: 56583 (2.15%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D: 2073557 (78.90%)

URL DEPTH = 4
/zh-CN/firefox/users/login?to=: 26591 (1.08%)
/zh-CN/firefox/collections/mine: 26691 (1.08%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0: 27883 (1.13%)
/zh-CN/firefox/collections/mozilla: 27973 (1.13%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0: 28934 (1.17%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0: 30763 (1.25%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1: 44508 (1.80%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0: 71234 (2.89%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1: 152746 (6.19%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0: 1515071 (61.42%)

URL DEPTH = 5
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/24.0/Firefox: 23448 (0.95%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/31.0/Firefox: 25227 (1.02%)
/zh-CN/firefox/users/login?to=/zh-CN: 26590 (1.07%)
/blocklist/3/%7B3550f703-e582-4d05-9a08-453d09bdfdc6%7D/24.6.0/Thunderbird: 27834 (1.12%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/26.0/Firefox: 28933 (1.17%)
/blocklist/3/%7Baa3c5121-dab2-40e2-81ca-7ea25febc110%7D/30.0/Fennec: 30762 (1.24%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/27.0.1/Firefox: 44418 (1.79%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/28.0/Firefox: 71226 (2.88%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/29.0.1/Firefox: 152700 (6.17%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox: 1514818 (61.18%)

URL MAX DEPTH
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release/Windows_NT%205.1/default/default/1/1/new: 8398 (0.31%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/2/2/1: 8834 (0.32%)
/collections/mozilla/featured-personas: 12416 (0.45%)
/zh-CN/firefox/search: 13019 (0.47%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%206.1/MozillaOnline/2014.6/1/1/new: 17955 (0.65%)
/it: 18770 (0.68%)
/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/30.0/Firefox/20140605174243/WINNT_x86-msvc/zh-CN/release-cck-mozillaonline/Windows_NT%205.1/MozillaOnline/2014.6/1/1/new: 22107 (0.80%)
/zh-CN/firefox/users/login?to=/zh-CN/firefox/collections/mine/favorites: 26498 (0.96%)
/zh-CN/firefox/collections/mine/favorites: 26691 (0.97%)
/zh-CN/firefox/collections/mozilla/featured-personas: 26731 (0.97%)
(Reporter)

Comment 9

4 years ago
Hi Jeremy,

Looks like the problem caused by the blocklist requests.
Could we limit the peak of the bandwidth? (Ideally we hope the peak can be limit to 60Mb)
Did we limit the rate of the request? If we set, what's the value of it?
Is it possible we just proxy the AMO websites, and keep the blocklist requests back to US?
Flags: needinfo?(oremj)
(Assignee)

Comment 10

4 years ago
We are working on decoupling blocklist from amo in bug 981663.
Depends on: 981663
Flags: needinfo?(oremj)
(Reporter)

Comment 11

4 years ago
Thanks, so when this changes will be online?
The mana wiki said: 2014-07-24 1400PDT.
Is it correct?
We are waiting on two dependencies to be completed before we proceed, see bug 1020320 comment 13 for more information.
(Assignee)

Updated

4 years ago
Group: mozilla-employee-confidential
Component: Server Operations: AMO Operations → Operations: Marketplace
Product: mozilla.org → Mozilla Services
Version: other → unspecified
(Reporter)

Comment 13

4 years ago
(In reply to Jason Thomas [:jason] from comment #12)
> We are waiting on two dependencies to be completed before we proceed, see
> bug 1020320 comment 13 for more information.

Hi Jason,
Do you know when the bugs on comment 13 can be fixed?
And what's the plan for blocklist.addons.mozilla.org?
Thanks!
I've asked for a update in bug 1020320.
bug 1020320 is now completed. I checked the pek1 zlb logs and blocklist traffic is being redirected to blocklist.addons.mozilla.org. :yli how do the bandwidth graphs look now?
Flags: needinfo?(yli)
(Reporter)

Comment 16

3 years ago
Hi Jason, 

The peak bandwidth of pek1 datacenter is dropped.

Thanks for all!
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Flags: needinfo?(yli)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.