1089953 - Hamachi nightlies are broken on every B2G branch except master

Status: VERIFIED FIXED

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Something broke these today between the 4am PT and 4pm nightlies.

https://treeherder.mozilla.org/ui/logviewer.html#?job_id=82970&repo=mozilla-b2g30_v1_4

17:43:54 INFO - mkdir -p `dirname out/target/product/hamachi/fota/partial/update.zip` || true
17:43:59 ERROR - Traceback (most recent call last):
17:43:59 INFO - File "tools/update-tools/build-flash-fota.py", line 129, in <module>
17:43:59 INFO - main()
17:43:59 INFO - File "tools/update-tools/build-flash-fota.py", line 123, in main
17:43:59 INFO - build_flash_fota(parser.parse_args())
17:43:59 INFO - File "tools/update-tools/build-flash-fota.py", line 56, in build_flash_fota
17:43:59 INFO - output_zip, update_bin)
17:43:59 INFO - File "/builds/slave/b2g_m-b30_14_ham_ntly-00000000/build/tools/update-tools/update_tools.py", line 1019, in build_flash_fota
17:43:59 INFO - flash_zip.write_updater_script(self.build_flash_script())
17:43:59 INFO - File "/builds/slave/b2g_m-b30_14_ham_ntly-00000000/build/tools/update-tools/update_tools.py", line 1053, in build_flash_script
17:43:59 INFO - self.generator.DeleteFilesRecursive(["/"+d])
17:43:59 INFO - AttributeError: 'EdifyGenerator' object has no attribute 'DeleteFilesRecursive'
17:43:59 INFO - make: *** [out/target/product/hamachi/fota/partial/update.zip] Error 1
17:43:59 INFO - real 3m33.799s
17:43:59 INFO - user 6m20.938s
17:43:59 INFO - sys 0m40.307s
17:43:59 INFO - 
17:43:59 INFO - > Build failed! <
17:43:59 INFO - Build with |./build.sh -j1| for better messages
17:43:59 INFO - If all else fails, use |rm -rf objdir-gecko| to clobber gecko and |rm -rf out| to clobber everything else.
17:43:59 ERROR - Return code: 2
17:43:59 FATAL - failed to create complete update
17:43:59 FATAL - Running post_fatal callback...
17:43:59 FATAL - Exiting 2

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Regression from bug 1061134.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

b2g34 as well

Comment 3

[Andreas Pehrson [:pehrsons]]

This is a symptom of having a new version of B2G, an old version of platform_build and building the targets gecko-update-fota or gecko-update-fota-full.

There are two reasons it is failing for hamachi:

1. Hamachi build.sh builds the fota packages by default. Other devices do not. (rather masking the issue on other devices)

2. The B2G repo is not revisioned in the manifests in gecko. So when you build with the b2g30 manifest in gecko (http://hg.mozilla.org/releases/mozilla-b2g30_v1_4/file/599f2ea93566/b2g/config/hamachi/sources.xml), you get master mozilla-b2g/B2G but an old platform_build.


How do we solve 2?

A quick and dirty fix would be to update the platform_build revision across all devices in the old b2g-gecko repos.

I think a proper fix could be to code bug 1061134 more defensively to fall back to the old way of doing DeleteFilesRecursive() in case it's missing. Then we could wait until mozilla-b2g/B2G is properly versioned before removing the fallback.

John, how far did you get with your proposal to get versioning of mozilla-b2g/B2G? Is there a bug for tracking?

Alexandre/Gabriele, your thoughts on this?

Comment 4

[:gerard-majax]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #3)
> This is a symptom of having a new version of B2G, an old version of
> platform_build and building the targets gecko-update-fota or
> gecko-update-fota-full.
> 
> There are two reasons it is failing for hamachi:
> 
> 1. Hamachi build.sh builds the fota packages by default. Other devices do
> not. (rather masking the issue on other devices)
> 
> 2. The B2G repo is not revisioned in the manifests in gecko. So when you
> build with the b2g30 manifest in gecko
> (http://hg.mozilla.org/releases/mozilla-b2g30_v1_4/file/599f2ea93566/b2g/
> config/hamachi/sources.xml), you get master mozilla-b2g/B2G but an old
> platform_build.
> 
> 
> How do we solve 2?

No idea, this has already bitten us in the past (because of patches I landed).

> 
> A quick and dirty fix would be to update the platform_build revision across
> all devices in the old b2g-gecko repos.
> 
> I think a proper fix could be to code bug 1061134 more defensively to fall
> back to the old way of doing DeleteFilesRecursive() in case it's missing.
> Then we could wait until mozilla-b2g/B2G is properly versioned before
> removing the fallback.
> 
> John, how far did you get with your proposal to get versioning of
> mozilla-b2g/B2G? Is there a bug for tracking?
> 
> Alexandre/Gabriele, your thoughts on this?

I've just checked on my Nexus S build tree, and it's there: DeleteFilesRecursive() is available from your previous patch in build/tools/releasetools/edify_generator.py.

Comment 5

[:gerard-majax]

Andreas, for this specific usecase, maybe we can just uplift your patch that adds DeleteFilesRecursive() ?

Comment 6

[Andreas Pehrson [:pehrsons]]

Ah, I see, hamachi (and other ICS devices I guess) fetches the platform_build branch corresponding to their gecko version, rather than the gonk version.

To fix all builds though, we'd need to update all the way from v1.0.0 to v2.1. Feasible?

Comment 7

[:gerard-majax]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #6)
> Ah, I see, hamachi (and other ICS devices I guess) fetches the
> platform_build branch corresponding to their gecko version, rather than the
> gonk version.
> 
> To fix all builds though, we'd need to update all the way from v1.0.0 to
> v2.1. Feasible?

I don't know, you should ask mwu for how to properly handle this.

Comment 8

[Ryan VanderMeulen [:RyanVM]]

(In reply to Andreas Pehrson [:pehrsons] (Telenor) from comment #3)
> John, how far did you get with your proposal to get versioning of
> mozilla-b2g/B2G? Is there a bug for tracking?

Bug 1048854

Comment 9

[Gabriele Svelto [:gsvelto] (PTO)]

(In reply to Alexandre LISSY :gerard-majax from comment #7)
> I don't know, you should ask mwu for how to properly handle this.

Same here, I haven't been hacking the build system for a while and I'm not sure I can help with this.

Comment 10

[Michael Wu [:mwu]]

For the short term, update_tools.py can be fixed to handle devices without the proper support. I'm not sure that update_tools.py belongs in B2G though - it should probably either be in platform_build or gonk-misc.

Comment 11

[Andreas Pehrson [:pehrsons]]

Attachment: Be backwards compatible if DeleteFilesRecursive() missing from edify generator

Here's a patch that adds backwards compatibility to update-tools.py.

Gabriele, are you up for review like last time? Or who would be better suited? mwu?

Comment 12

[Gabriele Svelto [:gsvelto] (PTO)]

Comment on attachment 8513486 [details] [review]
Be backwards compatible if DeleteFilesRecursive() missing from edify generator

I can review this and it looks good but I have a question. In the edify_generator.py you're emitting only one delete_recursive() call with a comma-separated list of files to delete. Here you're emitting one delete_recursive() call per file. Is there a specific reason for that? If there isn't I'd like for both this compatibility function and the original to emit identical code.

Comment 13

[Andreas Pehrson [:pehrsons]]

(In reply to Gabriele Svelto [:gsvelto] from comment #12)
> Comment on attachment 8513486 [details] [review]
> Be backwards compatible if DeleteFilesRecursive() missing from edify
> generator
> 
> I can review this and it looks good but I have a question. In the
> edify_generator.py you're emitting only one delete_recursive() call with a
> comma-separated list of files to delete. Here you're emitting one
> delete_recursive() call per file. Is there a specific reason for that? If
> there isn't I'd like for both this compatibility function and the original
> to emit identical code.

That was simply to not change existing behavior of update_tools.py - printing "Cleaning <folder>" to the user before actually cleaning it. See: https://github.com/mozilla-b2g/B2G/pull/386/files

Comment 14

[Gabriele Svelto [:gsvelto] (PTO)]

Comment on attachment 8513486 [details] [review]
Be backwards compatible if DeleteFilesRecursive() missing from edify generator

OK, thanks for the clarification, LGTM.

Comment 15

[Andreas Pehrson [:pehrsons]]

Thanks!

Since the patch says "TODO: Remove after bug 1048854 has been fixed.", I'm adding a dependency on that bug so we have some sense of tracking for a future revert.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Master: https://github.com/mozilla-b2g/B2G/commit/4c2e6673c0230c4d2828a64bc7bedc31e6b385ae

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Green!