602076 - Figure out why cb-xserve04 is checking for updates ~2K times/day

Status: RESOLVED FIXED

(Camino Graveyard :: General, defect)

Description

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Stuart, I can't keep track of when we're supposed to check for updates.  Do we do it at startup, or only every 24 hours of runtime?

I ask because I noticed today that 63.245.210.43 is pinging us between 1800-2000 times per day.  Earlier, I blacklisted the IP in bug 591868, apparently without checking to see who it was.  It's MoCo; specifically, it's cb-xserve04.

What's even stranger/worse is that cb-xserve01, cb-miniosx01, and cb-minibinus01 only ping us 1-2 times per day, which is what I'd expect based on what I think I remember about how update works.

On cb-xserve04, though, it looks like we're pinging every startup; a build every ~8 minutes means ~180 builds/day; each build does 1 MozillaAlive run and 10 Ts runs; on average, that would be 1980 pings every 24 hours.

cb-xserve01 is doing an identical set of builds and tests (only slower; a build every ~12 minutes), but on 10.4.

I can get more log info and other stuff later; I just wanted to file this now.

Comment 1

[Stuart Morgan]

It's every 24 hours of runtime, or on startup if it's been at least 24 hours. If user prefs are getting nuked before each launch we'd always ping on startup.

We could prevent that by writing the defaults value manually after nuking other prefs.

Comment 2

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

I'm still confused as to why we're getting 11 per build from cb-x4 and only 1 per day from cb-x1 and the others.

I pulled random log files (the latest 1.9.2 builds from x1 http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1286332260.1286332868.17875.gz and x4 http://tinderbox.mozilla.org/showlog.cgi?log=Camino/1286332440.1286332926.18053.gz), and both have 

Found profile.
Creating clean profile ...
Deleting /Users/caminobld/Library/Application Support/Camino ...
profile dir = /Users/caminobld/Library/Application Support/Camino
prefs.js    = /Users/caminobld/Library/Application Support/Camino/prefs.js
Setting browser.dom.window.dump.enabled to true
…
once per build.

That is http://hg.mozilla.org/users/alqahira_ardisson.org/madhatter/annotate/07ecbe88c159/build-seamonkey-util.pl#l2333 (and all tinder-configs have that set, http://mxr.mozilla.org/seamonkey/search?string=CleanProfile&find=&findi=&filter=tinderbox-configs%2Fcamino&hitlimit=&tree=seamonkey (that's CVS trunk, but those are set the same in CAMINO_2_1_M1_9_2_BRANCH tinder-configs).

So all the boxen should be behaving identically (or, at the very least, the 2 Xserves doing "release"-style builds should), so I'm puzzled by the lack of identical behavior.  Based on what you said and based on the settings/tinderbox behavior, I'd expect 1 ping per build, during the Alive test right after the profile is created.  Neither Xserve is doing that. :P

Comment 3

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Stuart figured this out, as usual :)

[4:56pm] smorgan: ardissone: so what does our test script on the bots do with prefs?
[4:57pm] ardissone: deletes the profile folder, if found, once per build, and adds some prefs to a new one it creates
[4:57pm] ardissone: http://hg.mozilla.org/users/alqahira_ardisson.org/madhatter/annotate/07ecbe88c159/build-seamonkey-util.pl#l2333
[4:57pm] smorgan: Nothing with OS-level prefs?
[4:57pm] smorgan: org.mozilla.camino.plist?
[4:57pm] ardissone: no
[4:58pm] smorgan: weird... sounds like something is messing with those prefs on that machine
[4:58pm] smorgan: Is the file writeable?
[4:58pm] ardissone: the plist?
[4:59pm] smorgan: yes
[4:59pm] ardissone: heh
[4:59pm] ardissone: -rw------
[4:59pm] ardissone: so, no :P
[4:59pm] ardissone: hmm
[5:00pm] ardissone: but neither is cb-x1's
[5:00pm] smorgan: That's it then
[5:00pm] smorgan: Wait, that's writeable
[5:00pm] smorgan: unless the owner is wrong
[5:00pm] ardissone: oh,
[5:00pm] smorgan: What does
[5:00pm] smorgan: defaults read org.mozilla.camino SULastCheckTime
[5:01pm] smorgan: say?
[5:01pm] ardissone: caminobld:caminobld on cb-x1, caminobld:staff on x4 (bad)
[5:01pm] sauron: caminobld@cb-xserve04 bash$ defaults read org.mozilla.camino SULastCheckTime
[5:01pm] sauron: 2010-03-29 18:04:58 -0700
[5:01pm] sauron: caminobld@cb-xserve01 bash$ defaults read org.mozilla.camino SULastCheckTime
[5:01pm] sauron: 2010-10-06 03:22:28 -0700
[5:01pm] ardissone: so
[5:01pm] ardissone: since march 29 :P
[5:02pm] smorgan: Yeah, that's bad news
[5:02pm] smorgan: Maybe it's crashing or something before prefs get written?
[5:02pm] ardissone: possibly, since quits are not clean
[5:02pm] ardissone: and it's really fast
[5:03pm] smorgan: If we never quit cleanly, there's no guarantee prefs will persist
[5:03pm] smorgan: On a slow machine we'd have a better chance of hitting the periodic sync
[5:03pm] ardissone: mm
[5:04pm] smorgan: So we should probably just turn off update checks on that machine
[5:05pm] sauron: http://www.ardisson.org/smokey/moz/perf.html?size=50&days=2&avg=1&autoscale=1 is not appreciably slower than cb-x1
[5:06pm] ardissone: and the Alive test always waits n seconds
[5:07pm] smorgan: What's n?
[5:07pm] ardissone: 15
[5:08pm] smorgan: Huh
[5:08pm] smorgan: I'm surprised that's not enough
[5:09pm] smorgan: Still, probably best to just disable it on bots
[5:10pm] ardissone: we set that as a Gecko pref?
[5:15pm] smorgan: No, SUEnableAutomaticChecks -> NO
[5:15pm] ardissone: ok
[5:16pm] • ardissone adds that to the list of defaults to write on setting up a box
[5:19pm] ardissone: that's a real BOOL, right, not a string like breakpad? :P
[5:20pm] smorgan: I believe it's a real BOOL
[5:21pm] ardissone: k
[5:22pm] ardissone: all tinderbox'd


I updated http://wiki.caminobrowser.org/IT:Setting_up_Camino_Tinderboxen#Software_Update with the info after running it on all four boxen.

Comment 4

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

…and there have been no pings from any tinderbox so far today :)