1465996 - l10n-check is failing in Linux coverage builds (No space left on device)

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Marco Castelluccio [:marco]]

Attachment: live_backing.tar.xz

It started with https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=0dc7bfc7b0c1de8d390cea05f0248fbadeb4d5fc.

Comment 1

[Marco Castelluccio [:marco]]

Here's the relevant part of the log:
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32)
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> rsync: write failed on "/builds/worker/workspace/build/src/obj-firefox/dist/l10n-stage/firefox/libxul.so": No space left on device (28)
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> rsync: connection unexpectedly closed (548 bytes received so far) [sender]
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> /builds/worker/workspace/build/src/toolkit/locales/l10n.mk:107: recipe for target 'unpack' failed
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> make[5]: *** [unpack] Error 12
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> make[5]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/browser/locales'
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> Makefile:124: recipe for target 'installers-x-test' failed
> [task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> make[4]: *** [installers-x-test] Error 2
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  l10n-check> Makefile:145: recipe for target 'l10n-check' failed
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  l10n-check> make[3]: *** [l10n-check] Error 2
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  l10n-check> /builds/worker/workspace/build/src/browser/build.mk:36: recipe for target 'l10n-check' failed
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  l10n-check> make[2]: *** [l10n-check] Error 2
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  /builds/worker/workspace/build/src/build/moz-automation.mk:89: recipe for target 'automation/l10n-check' failed
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  make[1]: *** [automation/l10n-check] Error 2
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  client.mk:154: recipe for target 'build' failed
> [task 2018-05-31T22:34:03.903Z] 22:34:03     INFO -  make: *** [build] Error 2
> [task 2018-05-31T22:34:03.939Z] 22:34:03     INFO -  375 compiler warnings present.
> [task 2018-05-31T22:34:04.114Z] 22:34:04     INFO -  Notification center failed: Install notify-send (usually part of the libnotify package) to get a notification when the build finishes.
> [task 2018-05-31T22:34:04.180Z] 22:34:04    ERROR - Return code: 2

Comment 2

[Gregory Szorc [:gps]]

The meaningful failure is:

[task 2018-05-31T22:34:03.902Z] 22:34:03     INFO -  l10n-check> rsync: write failed on "/builds/worker/workspace/build/src/obj-firefox/dist/l10n-stage/firefox/libxul.so": No space left on device (28)

Does this task produce a lot of data or something? If so, we may want to give it a special worker with more disk space. Because it isn't worth increasing our storage allocation for all workers because we would have to pay a bit of money for unused storage.

Comment 3

[Marco Castelluccio [:marco]]

It does produce bigger object files, but I've never seen it fail before because it was running out of space.

Comment 4

[Chris Manchester (limited bugmail, email directly)]

I'd expect turning off l10n-check on these builds to be an acceptable solution.

Comment 5

[Gregory Szorc [:gps]]

The gecko-3-b-linux workers have a 120 GB EBS volume and have "diskspaceThreshold" set to 30,000,000,000. I believe that means they are supposed to guarantee 30GB of storage is available before allowing a task to run.

Can individual tasks declare how much storage they need?

Comment 6

[Gregory Szorc [:gps]]

I'm all for disabling l10n-check on these builds. That should be done.

But I think we're playing roulette with the build phase that experiences the out of space error. This could presumably have occurred during any build phase and is not specified to l10n-check. (Unless l10n-check is copying gigabytes of files around. But I don't see evidence of that. Unless the .so files are really large.)

Comment 7

[Marco Castelluccio [:marco]]

(In reply to Gregory Szorc [:gps] from comment #6)
> I'm all for disabling l10n-check on these builds. That should be done.
> 
> But I think we're playing roulette with the build phase that experiences the
> out of space error. This could presumably have occurred during any build
> phase and is not specified to l10n-check. (Unless l10n-check is copying
> gigabytes of files around. But I don't see evidence of that. Unless the .so
> files are really large.)

IIRC libxul is huge in the coverage build.

Comment 8

[Gregory Szorc [:gps]]

Attachment: Bug 1465996 - Disable l10n-check in code coverage builds;

Because we don't care about them for this build configuration.

Review commit: https://reviewboard.mozilla.org/r/248324/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/248324/

Comment 9

[Marco Castelluccio [:marco]]

Should we disable them for the Windows and Mac coverage builds too? Or are the l10n checks only done on Linux?

Comment 10

[(not currently active) Ted Mielczarek]

Comment on attachment 8982388 [details]
Bug 1465996 - Disable l10n-check in code coverage builds;

https://reviewboard.mozilla.org/r/248324/#review254734

This is totally sensible. As marco says, we should probably do this for all code coverage builds.

Comment 11

[Gregory Szorc [:gps]]

AFAICT we already disable l10n-check on other platforms. This is another great example of how the current structure of mozconfigs make things really hard to follow.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c3afc68aab2c
Disable l10n-check in code coverage builds; r=ted

Comment 13

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/c3afc68aab2c