Closed Bug 989239 Opened 11 years ago Closed 11 years ago

spin up ftp8.dmz.scl3

Categories

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Product:
Infrastructure & Operations Graveyard
Component:
WebOps: Product Delivery
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: rbryce, Assigned: nmaul)

References

Details

(Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/232] )

I am not to worried about this issue, but all evening ftp2.dmz has been alerting for high load. I can't seem to find anything wrong. The cpu load fluctuatesd between the WARN and CRIT thresholds. The http.workers are the ones causing the load, but the server doesn't seem to be affected by these load spikes. Seems to be doing its job just fine. This may very well subside before WebOps sees this bug. Call this tracking for now.
Summary: ftp2.dmz intermittened load spikes → ftp2.dmz intermittent load spikes
This seems to show up as 500 responses in some requests we make to stage.m.o, which we've seen in the last two beta releases (29.0b3 on March 28, 29.0b4 on April 1). For 29.0b4 more than just ftp2 had issues - ftp4, ftp5 and ftp6 joined in the fun (CC ashish who fixed it). The update verify work we did then kicked off at 1940 Pacific, running until 0715 the next day, but mostly done by 0220. Looking at ganglia, the background level of traffic seems to be up about 50% in the last week or so (Week 13) then spikes higher when the releases come along. We have a bug about being smarter about what is downloaded (avoided repeated requests), and I'll check if we added a lot more d/l requests by turning on new jobs. Is all well with the cluster ? Are the usual hosts all online ? Perhaps we need to grow it soon.
Depends on: 993307
Happening again, load well past 120. CPU isn't especially utilised, possibly NFS issues.
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/232]
Let's use this for spinning up the already-online-but-idle ftp8.dmz.scl3. @Storage: could you add 10.22.74.129 to the access list for these 2 mounts: 10.22.74.157:/pvtbuilds 10.22.74.157:/ftp_stage Permissions level(s) should be the same as ftp1.dmz.scl3 (10.22.74.48). Thanks!
Assignee: server-ops-webops → server-ops-storage
Severity: minor → normal
Component: WebOps: Product Delivery → Server Operations: Storage
Product: Infrastructure & Operations → mozilla.org
QA Contact: nmaul → cshields
Summary: ftp2.dmz intermittent load spikes → spin up ftp8.dmz.scl3
Added 10.22.74.129 to policies pvtbuilds, root, stage, modeled after the other ftp clients
Assignee: server-ops-storage → server-ops-webops
Component: Server Operations: Storage → WebOps: Product Delivery
Product: mozilla.org → Infrastructure & Operations
QA Contact: cshields → nmaul
This seems to be working nicely, puppet is happy, a few random curl tests work fine. Added to zeus and it's handling traffic. Hopefully this takes care of the problem with other nodes overloading sometimes.
Assignee: server-ops-webops → nmaul
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
This box needs a netflow so it can reach metrics-logger1.private.scl3.mozilla.com on port 22 for log transfer.
Let's talk to BI/DW and make sure they're not using this data... they shouldn't be. They should be using logs from Zeus, not from the end servers. @anurag: it looks to me like the FTP servers are sending you log data for ... releases.mozilla.com. I'm not sure why, though... that's not part of our product delivery pipeline these days. Possibly just legacy stuff. Can you confirm whether or not you're using any data from the ftpX.dmz.scl3.mozilla.com nodes?
Status: RESOLVED → REOPENED
Flags: needinfo?(aphadke)
Resolution: FIXED → ---
Jake - r.m.c is used to measure number of adis and completed/partial downloads for bing addons. We have a business contract with Microsoft that mandates us to send them a monthly report for all partial/complete downloads. This is done via a apache setting flag which I believe needed custom config and is only available for r.m.c via custom config.
Flags: needinfo?(aphadke)
Can you confirm you're getting r.m.c data from Zeus and not somewhere else?
Flags: needinfo?(aphadke)
ls /mnt/metrics-logger1/logs/*/releases.mozilla.com/access_log_2014-06-19* /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-00.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-01.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-02.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-03.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-04.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-05.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-06.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-07.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-08.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-09.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-10.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-11.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-12.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-13.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-14.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-15.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-16.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-17.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-18.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-19.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-20.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-21.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-22.gz /mnt/metrics-logger1/logs/ftp1.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-23.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-00.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-01.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-02.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-03.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-04.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-05.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-06.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-07.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-08.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-09.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-10.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-11.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-12.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-13.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-14.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-15.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-16.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-17.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-18.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-19.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-20.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-21.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-22.gz /mnt/metrics-logger1/logs/ftp2.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-23.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-00.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-01.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-02.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-03.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-04.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-05.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-06.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-07.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-08.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-09.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-10.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-11.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-12.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-13.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-14.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-15.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-16.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-17.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-18.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-19.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-20.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-21.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-22.gz /mnt/metrics-logger1/logs/ftp4.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-23.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-00.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-01.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-02.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-03.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-04.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-05.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-06.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-07.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-08.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-09.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-10.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-11.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-12.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-13.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-14.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-15.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-16.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-17.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-18.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-19.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-20.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-21.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-22.gz /mnt/metrics-logger1/logs/ftp5.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-23.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-00.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-01.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-02.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-03.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-04.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-05.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-06.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-07.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-08.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-09.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-10.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-11.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-12.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-13.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-14.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-15.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-16.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-17.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-18.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-19.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-20.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-21.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-22.gz /mnt/metrics-logger1/logs/ftp6.dmz.scl3.mozilla.com/releases.mozilla.com/access_log_2014-06-19-23.gz these are the locations for r.m.c data
Flags: needinfo?(aphadke)
Ugh. Okay, I'm going to close this bug back out, and open a new bug for us to work together and change things so you're reading logs for this from Zeus. That is bug 1028307.
Status: REOPENED → RESOLVED
Closed: 11 years ago → 11 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.