Closed Bug 1305139 Opened 8 years ago Closed 7 years ago

[tcmigration] schedule checksums related tasks in the nightly graph

Categories

(Firefox Build System :: Task Configuration, task)

Product:
Firefox Build System
Component:
Task Configuration
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: jlund, Assigned: mtabara)

References

Details

Attachments

(6 files, 1 obsolete file)

WIP - first iteration on checksums
57 bytes, text/x-github-pull-request
mozilla
: feedback+
jlund
: feedback+
Details | Review
Add checksums generation logic in beetmoverscript.
97 bytes, text/plain
Details
Bug 1305139 - bump beetmoverscript and tweak according to checksums.
59 bytes, text/x-review-board-request
mozilla
: review+
Details
Bug 1305139 - schedule checksums related tasks in nightly graph.
59 bytes, text/x-review-board-request
mozilla
: review+
mtabara
: review+
lizzard
: approval-mozilla-aurora+
Details
fix-date2
4.02 KB, patch
Callek
: feedback+
Details | Diff | Splinter Review
attempt to download props from beetmover task
6.35 KB, patch
Details | Diff | Splinter Review 
in bbot world, we generate a checksum file for fennec nightlies and sign it on the spot during the Nightly build job:

04:23:36     INFO -  python /builds/slave/m-cen-and-api-15-ntly-00000000/build/tools/release/signing/signtool.py --cachedir /builds/slave/m-cen-and-api-15-ntly-00000000/build/signing_cache -t /builds/slave/m-cen-and-api-15-ntly-00000000/token -n /builds/slave/m-cen-and-api-15-ntly-00000000/nonce -c /builds/slave/m-cen-and-api-15-ntly-00000000/build/tools/release/signing/host.cert -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing4.srv.releng.scl3.mozilla.com:9100 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing5.srv.releng.scl3.mozilla.com:9100 -H gpg:sha2signcode:osslsigncode:signcode:mar:jar:emevoucher:signing6.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing2.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing3.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing4.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing6.srv.releng.scl3.mozilla.com:9100 -H dmgv2:mac-v2-signing7.srv.releng.scl3.mozilla.com:9100 -f gpg '../../../dist//fennec-52.0a1.en-US.android-arm.checksums'


In tc world we don't pass our signing token to the build job. So similar to how we sign the apk with signingscript, we should create another task in the nightly graph that signs the checksums artifact
mtabara agreed to pick this up at the nightly migration mtg yesterday.
Assignee: nobody → mtabara
Aiui,

- We generate a checksums file for android in the build
- We can now sign multiple file types in a signing task, so we could sign that checksums file.  This is what we're doing for linux{32,64}.

*However*,

- The checksums file from the build has un-pretty names.  We don't get pretty names until the beetmover task.
- The linux checksums file in buildbot contains the detached sig in the checksums. This file doesn't exist until after the first signing task.
- The signing task changes the binary for all non-linux builds, so the checksum in the file is wrong once we sign the binary.

The solution we are thinking:

After the beetmover task, we have all the artifacts needed to generate the checksums file: the pretty names, the signed binary or detached sig.  So, post-beetmover,

1. checksums task: this downloads the beetmoved bits and generate a checksums artifact.  Since we'll need this for all platforms, this task type should be able to support all platforms.
2. checksums signing task: like a build or repack signing task, but it signs the checksums file and uploads the checksums sigfile.
3. checksums beetmover task: this task takes the checksums file and signature and beetmoves them to the right place.
This is actually quite a good bug for me as I've been dealing with workers code side, puppet side but haven't touched too much the in-tree graph side. Good opportunity to plug in and have some fun with the transforms :)

Thanks for clarifications :aki.
(In reply to Aki Sasaki [:aki] from comment #2)
> 1. checksums task: this downloads the beetmoved bits and generate a
> checksums artifact.  Since we'll need this for all platforms, this task type
> should be able to support all platforms.

Another approach is to allow for beetmoverscript to generate the checksums file.
This blurs or even violates the one-purpose-per-task principle, but it makes a lot of sense to me.
beetmoverscript with generate_checksums=True in the task definition could generate the checksums from the on-disk bits, or even the on-disk chain of trust artifacts, and it knows the pretty names.

So:

1) beetmover with generate_checksums=True - beetmoves all the current bits to s3, plus generates a pretty name checksums file that it copies to its artifact_dir
2) signing task for the checksums file
3) beetmover task with generate_checksums=False - beetmoves the checksums file + sig

The most complicated parts of that are a) making beetmover generic, since the transforms currently assume they're for build or test tasks, and b) adding checksums capability to beetmoverscript.

I prefer this approach over comment 0, but both ways work.
(In reply to Aki Sasaki [:aki] from comment #5)
> I prefer this approach over comment 0, but both ways work.

This should read comment 2.
(In reply to Aki Sasaki [:aki] from comment #5)
> beetmoverscript with generate_checksums=True in the task definition could
> generate the checksums from the on-disk bits, or even the on-disk chain of
> trust artifacts, and it knows the pretty names.

We may not even care about having a generate_checksums boolean; we can assume generate_checksums is always true.

beetmover task #1:
- gets downloaded bits with verified checksums from cot
- renames them with pretty names
- generates a checksums file
- beetmoves the pretty named files (not the checksums file)
- copies the checksums file to the artifact_dir, uploaded as a tc artifact

checksums signing task:
- downloads the checksums file, signs it, uploads the sig

beetmover task #2:
- gets downloaded checksums file + sig with verified checksums from cot
- renames them with pretty names - noop?
- generates another checksums file
- beetmoves the original checksums file + sig
- copies the new checksums file to artifact_dir, uplaoded as a tc artifact  <- this can be ignored

So the boolean isn't needed, but might save us a few cycles and kb of disk.
Summary: schedule a fennec nightly checksums signing task → tc: schedule a nightly checksums signing task
* WIP - first iteration on checksums - more specifically - first part of Aki's https://bugzilla.mozilla.org/show_bug.cgi?id=1305139#c7

I still need to add testing and tweak the usage of "generate_checksums" which may be useless in the end. I need to see how I separate in beetmover logic the first "normal beetmover" from the one that's to beetmove the actual signed file. Might need to keep that to toggle the logic between them.

The way I see this, we'll have:

1.1 One beetmover task per platform that will depend on both the build and the signed one (so scriptworker will grab both its artifacts from "build" tasks but also the signed ones from "signed" tasks)
1.2 One beeetmover task per locale (with dep for only the signed tasks as we're not beetmoving but signed artifacts in this case)

2. The signing task - one per each of the above

3. The final beetmove-to-S3 beetmover task - one per each of the above.

Please let me know what you think of this.
Attachment #8828406 - Flags: feedback?(jlund)
Attachment #8828406 - Flags: feedback?(aki)
Attachment #8828406 - Flags: feedback?(aki) → feedback+
Attachment #8828406 - Flags: feedback?(jlund) → feedback+
Comment on attachment 8830996 [details]
Bug 1305139 - bump beetmoverscript and tweak according to checksums.

https://reviewboard.mozilla.org/r/107654/#review108806
Attachment #8830996 - Flags: review?(aki) → review+
Status update: beetmover now generates the checksums per each task. 
Left to be done: schedule tasks in the nightly graph to grab the checksums, pretty name it, sign it and upload it accordingly.
Both :jlund and me met today to discuss the in-tree graph changes that are needed to accommodate checksums generation. 

Facts: 
a) * with beetmoverscript >= 0.1.6 *all* beetmover tasks will generate, regardless, a `target.checksums` file containing the digests for files beetmoved to S3
b) there are no longer "signed" vs "unsigned" task-based, at least from beetmoverscript side. The two manifests are now merged into a single mapping manifest/per platform(repacks are still separated though). And since "the new world" (scriptworker CoT) the files that need to be beetmoved are no longer controled from within the beetmoverscript manifests, but instead from its dependency inputs (hence the taskgraph in-tree definition). That means beetmoverscript is now generic and will upload to S3 whatever scriptworker finds in its upstreamArtifacts. The same artifacts consitute the subjects for checksums generation. 

The initial proposal was the following:
1. BM1 - a task that depends on both `build` and `signing` tasks (to download all artifacts that are needed to constitute the checksums file)

2. a SBM1 - a signing task consuming the `target.checksums` file and returning a signed `target.checksums` + `target.checksums.asc`

3. a follow-up BM2 depending on SBM1, consuming the checksums file + its detached signature, pretty names them and then (naturally) uploads them to S3.

Digging deepeer within [1], we realized we could apply the same "fact b)" optimization to the in-tree definitions as well. That is, instead of cloning the existing list of artifacts and define separate checksums ones + redownloading the artifacts again during task 1, we could instead merge the "unsigned" + signed" task definitions from within the tree. That means we'll shrink our beetmover total number of tasks by half and we'll have a single beetmover task for en-US/platform, that moves both the unsigned and signed artifacts to S3 and also generate the checksums.

To sum this up, instead of layer 1 from above, we'd:
1.1 Merge the existing "beetmover build ..." and "beetmover signining ..." tasks within the tree into a single one.
1.2. Reuse that for checksums - (already done since beetmoverscript==0.1.6)

Other benefits:
* we can get rid of update_manifest variable that no longer makes sense to seaparate  the "unsigned"/"signed" manifests
* subset of beetmover tasks from the graph shrinks by half
* cleaner code within [1] - as we'll no longer need all those "singning" if conditions

Other side effects:
* we may need to tweak the balrog task definition too to point to the newly created task (merge between "unsigned" + "signed" beetmover tasks), rather than presently "signed" one,

[1]: https://dxr.mozilla.org/mozilla-central/source/taskcluster/taskgraph/transforms/beetmover.py
I am not sure if this is a good place, but I wonder if this discussion has something to do with lacking chacksums file for linux builds of Aurora 53.0?
(In reply to kbks from comment #16)
> I am not sure if this is a good place, but I wonder if this discussion has
> something to do with lacking chacksums file for linux builds of Aurora 53.0?

In the last two weeks we've switched the nightlies from buildbot to Taskcluster. Initially for mozilla-central, and now, as part of train riding, on aurora as well.

The checksums are still a missing piece in this nightly puzzle. Apologies for that. 
They are currently being generated but are not yet signed, hence we're not yet uploading them. I'm working heavily on wrapping this up in the next few days.

Hope this helps.
Hello,
Checksums are still not generated for nightly linux fr.
Ok, I backlogged a bit and have read that you are already working on it. Sorry for my last comment.
(In reply to Anthony Ruhier from comment #21)
> Ok, I backlogged a bit and have read that you are already working on it.
> Sorry for my last comment.

No worries. Apologies again for the delays in making this happen, am working to wrap this as soon as possible, we should have them signed and uploaded in a day or two.
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #22)
> (In reply to Anthony Ruhier from comment #21)
> > Ok, I backlogged a bit and have read that you are already working on it.
> > Sorry for my last comment.
> 
> No worries. Apologies again for the delays in making this happen, am working
> to wrap this as soon as possible, we should have them signed and uploaded in
> a day or two.

Cool, that is great news!
Status: NEW → ASSIGNED
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

https://reviewboard.mozilla.org/r/109176/#review110340

::: taskcluster/taskgraph/transforms/beetmover.py:282
(Diff revision 1)
> -                signing_kind = dependency
> +                signing_task = dependency
>              else:
> -                build_kind = dependency
> +                build_task = dependency
>  
> -        if signing_kind:
> -            taskid_to_beetmove = "<" + str(signing_kind) + ">"
> +        signing_taskid = "<" + str(signing_task) + ">"
> +        build_taskid = "<" + str(build_task) + ">"

to be clear, these are not taskID's here.

The signing_task = dependency above sets it to a string that corresponds to the name of the dep.

This goes through magic as the stuff rolls through taskgraph code that pops out the other side (after optimization) to be replaced to a taskid though.

I'm not sure offhand a good alternate name for this...

::: taskcluster/taskgraph/transforms/beetmover_l10n.py:28
(Diff revision 1)
>  
>              group = 'tc-BM-L10n'
>  
>              # add the locale code
> -            if 'signing' in label:
> -                symbol = 'S{}'.format(locale)
> +            symbol = 'S{}'.format(locale)

nit: where we only have one beetmover task per locale now, I'm a fan of `locale` as the symbol rather than `S<locale>`
Attachment #8832940 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

https://reviewboard.mozilla.org/r/109176/#review110340

> to be clear, these are not taskID's here.
> 
> The signing_task = dependency above sets it to a string that corresponds to the name of the dep.
> 
> This goes through magic as the stuff rolls through taskgraph code that pops out the other side (after optimization) to be replaced to a taskid though.
> 
> I'm not sure offhand a good alternate name for this...

thanks, renamed those.
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

https://reviewboard.mozilla.org/r/109176/#review110560

::: taskcluster/taskgraph/transforms/checksums_signing.py:29
(Diff revision 3)
> +    basestring,
> +    {Required('task-reference'): basestring})
> +
> +checksums_signing_description_schema = Schema({
> +    Required('dependent-task'): object,
> +    Required('depname', default='build'): basestring,

change this default to `beetmover` and you're reference to `<beetmover>` later on will just work.

::: taskcluster/taskgraph/transforms/checksums_signing.py:58
(Diff revision 3)
> +        treeherder.setdefault('platform',
> +                              "{}/opt".format(dep_th_platform))
> +        treeherder.setdefault('tier', 1)
> +        treeherder.setdefault('kind', 'build')
> +
> +        label = job.get('label', "checksumssigning-{}".format(dep_job.label))

just to call it out, this creates the really ugly string:

`checksumssigning-beetmover-ar-signing-l10n-linux-nightly-1/opt`

in the case of l10n.  This is bad, but I'm the one who proliferated that format of "prepend, and move on" So I'm not blocking here, but is some cleanup we should do. (I don't know offhand a good way to fix this issue as a whole)
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

https://reviewboard.mozilla.org/r/109176/#review110566

To validate the task_reference bits, you can use `./mach taskgraph optimized` (instead of `./mach taskgraph full`). Optimized only operates on what tasks would get run with whatever is in parameters, so even if it passes once could fail in tree (try, android nightly, etc) in theory. But since this code only is used on desktop I don't think you need to worry too much about the optimized matrix of choices.

::: taskcluster/taskgraph/transforms/checksums_signing.py:60
(Diff revisions 3 - 4)
>          treeherder.setdefault('tier', 1)
>          treeherder.setdefault('kind', 'build')
>  
>          label = job.get('label', "checksumssigning-{}".format(dep_job.label))
> +        dependent_kind = str(dep_job.kind)
> +        dependencies = {dependent_kind: dep_job.label}

the dependent_kind key here is used to expand the task_ref stuff in <>. So it doesn't really matter what you do as long as it aligns.

::: taskcluster/taskgraph/transforms/checksums_signing.py:72
(Diff revisions 3 - 4)
>          if dep_job.attributes.get('locale'):
>              treeherder['symbol'] = 'tc-cs({})'.format(dep_job.attributes.get('locale'))
>              attributes['locale'] = dep_job.attributes.get('locale')
>  
>          upstream_artifacts = [{
>              "taskId": {"task-reference": "<beetmover>"},

which it needs to align with this.

::: taskcluster/taskgraph/transforms/beetmover_checksums.py:51
(Diff revision 4)
> +def make_beetmover_checksums_description(config, jobs):
> +    for job in jobs:
> +        dep_job = job['dependent-task']
> +
> +        treeherder = job.get('treeherder', {})
> +        treeherder.setdefault('symbol', 'tc-BMcs(N)')

Lets use "tc-BM(checksums)" imo
Attached patch fix-date (obsolete) — — Splinter Review 
This passes ./mach taskgraph tasks and ./mach taskgraph optimized.
It doesn't modify the treeherder symbol, per #tcmigration
Attachment #8833404 - Flags: feedback?(bugspam.Callek)
Comment on attachment 8833404 [details] [diff] [review]
fix-date

Review of attachment 8833404 [details] [diff] [review]:
-----------------------------------------------------------------

Using a desktop-nightly (aurora) parameters.yml it still fails `optimize`

 0:04.89 Generating optimized task graph
Traceback (most recent call last):
...
  File "/home/callek/mozilla/hg/mozilla-central/taskcluster/taskgraph/optimize.py", line 55, in recurse
    return TASK_REFERENCE_PATTERN.sub(repl, val['task-reference'])
  File "/home/callek/mozilla/hg/mozilla-central/taskcluster/taskgraph/optimize.py", line 48, in repl
    raise KeyError("task '{}' has no dependency named '{}'".format(label, key))
KeyError: u"task 'checksumssigning-beetmover-ar-signing-l10n-linux-nightly-1/opt' has no dependency named 'beetmover'"

Specifically this is because of:
(In reply to Justin Wood (:Callek) from comment #34)
> Comment on attachment 8832940 [details]
> Bug 1305139 - Add all three layers of checksums. .
> 
> ::: taskcluster/taskgraph/transforms/checksums_signing.py:60
> (Diff revisions 3 - 4)
> >          treeherder.setdefault('tier', 1)
> >          treeherder.setdefault('kind', 'build')
> >  
> >          label = job.get('label', "checksumssigning-{}".format(dep_job.label))
> > +        dependent_kind = str(dep_job.kind)
> > +        dependencies = {dependent_kind: dep_job.label}
> 
> the dependent_kind key here is used to expand the task_ref stuff in <>. So
> it doesn't really matter what you do as long as it aligns.
> 
> ::: taskcluster/taskgraph/transforms/checksums_signing.py:72
> (Diff revisions 3 - 4)
> >          if dep_job.attributes.get('locale'):
> >              treeherder['symbol'] = 'tc-cs({})'.format(dep_job.attributes.get('locale'))
> >              attributes['locale'] = dep_job.attributes.get('locale')
> >  
> >          upstream_artifacts = [{
> >              "taskId": {"task-reference": "<beetmover>"},
> 
> which it needs to align with this.
> 

Everything else in here looks great though
Attachment #8833404 - Flags: feedback?(bugspam.Callek) → feedback-
Attached patch fix-date2 — — Splinter Review 

        Attachment #8833404 -
        Attachment is obsolete: true

        Attachment #8833417 -
        Flags: feedback?(bugspam.Callek)

        Attachment #8833417 -
        Flags: feedback?(bugspam.Callek) → feedback+

    
    

    
      
  

    
    

    
  
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

https://reviewboard.mozilla.org/r/109176/#review110810

as mentioned in IRC, we may want to only grab the balrog props for en-US, not multi as well.

        Attachment #8832940 -
        Flags: review?(aki) → review+

    
    

    
      
  

    
    

    
  
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

https://reviewboard.mozilla.org/r/109176/#review110812

    
    

    
  
https://hg.mozilla.org/projects/date/rev/4ae02b6e1f3f
Bug 1305139 - fix balrog props deps in beetmover checksums tasks. r=bustage r=aki

    
    

    
  


  
Landed this to try to get balrog_props working on Friday evening.  If this works, we should get review for it.  If it doesn't let's back it out.

If this works, instead of the deepcopy, maybe we should look for a key that starts with "beetmover", and make sure there's only one ?  We can also make sure the length of the dependencies is 2.

    
    

    
  
Until a few days ago there were checksum files and gpg sigs for Linux AMD64 firefox-53.0a2 builds, there still are for win mac builds and there still are for firefox-52.0a2 -- has something recently broken?

https://download-installer.cdn.mozilla.net/pub/firefox/nightly/latest-mozilla-aurora/

    
    

    
  
(In reply to Chris Croome from comment #51)
> Until a few days ago there were checksum files and gpg sigs for Linux AMD64
> firefox-53.0a2 builds, there still are for win mac builds and there still
> are for firefox-52.0a2 -- has something recently broken?
> 
> https://download-installer.cdn.mozilla.net/pub/firefox/nightly/latest-
> mozilla-aurora/

In the last two-three weeks we've switched the nightlies from buildbot to Taskcluster. Initially for mozilla-central, and now, as part of train riding, on aurora as well. The checksums were a missing piece in this nightly puzzle when we flipped the switch. Apologies for that. We've added the missing logic and we have them nicely generated & signed on `date` project branch. We'll do the merge and uplift to aurora tonight most likely.

Hope this helps.

    
    

    
  
I’m taking this opportunity to ask about two related things:

1) Just discovered the download-installer.cdn.mozilla.net, why isn’t it used for Nightly links (here for instance: https://www.mozilla.org/en-US/firefox/nightly/all/) ?

2) I was on the verge of asking what is this `date` branch that I’ve also discovered some minutes ago while browsing the download server, for which they are indeed checksums?

Thanks!

    
    

    
  
(In reply to Bruno Pagani from comment #53)
> I’m taking this opportunity to ask about two related things:
> 
> 1) Just discovered the download-installer.cdn.mozilla.net, why isn’t it used
> for Nightly links (here for instance:
> https://www.mozilla.org/en-US/firefox/nightly/all/) ?

Hm, I have no idea to be honest. But in any case, that's a question for :pmac or follow-up with your comment in bug 937865.

> 2) I was on the verge of asking what is this `date` branch that I’ve also
> discovered some minutes ago while browsing the download server, for which
> they are indeed checksums?

It's just a project branch that we use as "staging" tree to test our changes before we rollout to mozilla-central.

    
    

    
      
  

    
    

    
  
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

https://reviewboard.mozilla.org/r/109176/#review111858

        Attachment #8832940 -
        Flags: review+

    
    

    
  
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #56)
> Comment on attachment 8832940 [details]
> Bug 1305139 - schedule checksums related tasks in nightly graph.
> 
> https://reviewboard.mozilla.org/r/109176/#review111858

ship it

    
    

    
  
Still looking around, I’ve discovered you’ve started PGP signing the tarball themselves. When did that started? With the move from buildbot to taskcluster?

    
    

    
  
(Which makes the sums useless for my usage BTW…)

    
    

    
  
Pushed by mtabara@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/861eb35d257b
schedule checksums related tasks in nightly graph. r=aki,Callek

    
    

    
  
(In reply to Bruno Pagani from comment #58)
> Still looking around, I’ve discovered you’ve started PGP signing the tarball
> themselves. When did that started? With the move from buildbot to
> taskcluster?

Yes, it's part of the migration to increase security.

    
  
Keywords: leave-open

    
    

    
  
Pushed by cbook@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f94097684230
followup for flake8 failures. r=Tomcat

    
    

    
  
Since I’m not sure this is worth opening a new bug, I’m asking about this here:

The sig and tarball at https://download-installer.cdn.mozilla.net/pub/${_name}/nightly/latest-mozilla-aurora doesn’t match.
File 	firefox-53.0a2.en-US.linux-x86_64.tar.bz2 	62M 	08-Feb-2017 09:37
File 	firefox-53.0a2.en-US.linux-x86_64.tar.bz2.asc 	836 	08-Feb-2017 09:37

Neither do they here at https://archive.mozilla.org/pub/firefox/nightly/2017/02/2017-02-08-08-41-06-mozilla-aurora/.

But they do here:
https://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-aurora/

The .asc file is the same in all cases, but the tarball changes. The one matching the sig:
705c1c062934606663a4ff6e9935dbb8b68ea196e9da8727b67c1ea3b7cd5a0739bd1ab5bcd91d0a7d8d42067d1b567c855e6cfe06c5d7da6562b009d397ed6b  firefox-53.0a2.en-US.linux-x86_64.tar.bz2

The other:
72dbf22d8ab2636ea040aa1e4d42a74ad35206e1b639bbf551d5fa40f0f9fb1f65f0165d526ffa76096174bd09814d42ef354b1c55bc165e9674ebcd444de4ea  firefox-53.0a2.en-US.linux-x86_64.tar.bz2

Should I open a new bug for this?

    
  
Summary: tc: schedule a nightly checksums signing task → [tcmigration] schedule checksums related tasks in the nightly graph

    
    

    
  
All checksums seem to be in place now with the last set of nightlies.
We need to uplift this code to mozilla-aurora and we-re done.

    
    

    
  
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

Approval Request Comment
[Feature/Bug causing the regression]: 
Not a regression, it's adding functionality - Bug 1305139.

[User impact if declined]: 
all developers requesting checksums in downstream systems.

[Is this code covered by automated tests?]: 
not that I know of. I'm adding a bunch of tasks in the decision task.

[Has the fix been verified in Nightly?]: 
Yes, it's working properly in nightly.

[Needs manual test from QE? If yes, steps to reproduce]:
 no.
[List of other uplifts needed for the feature/fix]:
Firstly, all patches from bug 1334167 need to be uplifed from central to aurora
https://hg.mozilla.org/mozilla-central/rev/f31ddcc843e8
https://hg.mozilla.org/mozilla-central/rev/f3ebbf9445ab
https://hg.mozilla.org/mozilla-central/rev/ae6c320b791b
https://hg.mozilla.org/mozilla-central/rev/3cb163fd021d
https://hg.mozilla.org/mozilla-central/rev/d33d1836e354
https://hg.mozilla.org/mozilla-central/rev/e74dc930625c
Then, these both patches from my bug:
* https://hg.mozilla.org/mozilla-central/rev/861eb35d257b
* https://hg.mozilla.org/mozilla-central/rev/f94097684230 (fixing linter follow-up patch)

[Is the change risky?]:
no
[Why is the change risky/not risky?]:
-
[String changes made/needed]:
-
Requesting approval for adding tasks with respect to checksums generation on aurora. There's also a follow-up patch to fix some linters. The patches were tested on nightly on mozilla-central and work as expected.

        Attachment #8832940 -
        Flags: approval-mozilla-aurora?

    
    

    
  
If the work in bug 1334167 should land first, can you request uplift there or if it's not part of the build, land whatever needs to be landed on aurora? Thanks!
Flags: needinfo?(mtabara)

    
    

    
  
Hi Liz,

I saw you already asked :dustin for uplifts of those patches in bug 1334167. I spoke with him yesterday and he seemed fine about it so let's wait for his reply in the bug. His patches change the import path for a function I'm using so that's why I need his patches landed first. The changes were already on central when I rebased so that's why I ended up using those. If :dustin thinks it's better to let the changes ride the trains normally, I could find my way around it in some other way, no problems.
Flags: needinfo?(mtabara)

    
    

    
  
Mihai: can you push on this to get the uplift done, for both parts (incl. bug 1334167) if necessary?
Flags: needinfo?(mtabara)

    
    

    
  
(In reply to Chris Cooper [:coop] from comment #70)
> Mihai: can you push on this to get the uplift done, for both parts (incl.
> bug 1334167) if necessary?

Sure. I'll ping :dustin if I can do the uplifts on his behalf, if he hasn't that already bug 1334167.
Flags: needinfo?(mtabara)

    
    

    
  
Status update: I got :dustin's blessing to uplift the pre-req patches from bug 1334167 to aurora. I requested the approval there. Once that's done and landed, I'll resume to uplift this bugs' checksums patches should they have RelMan's approval too.

    
    

    
  
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #68)
> If the work in bug 1334167 should land first, can you request uplift there
> or if it's not part of the build, land whatever needs to be landed on
> aurora? Thanks!

I've uplifted the pre-requirement patches from bug 1334167 to aurora so I guess we're now ready to uplift the checksums patches as well. Can you please bless the approval? :)
Flags: needinfo?(lhenry)

    
    

    
  
Comment on attachment 8832940 [details]
Bug 1305139 - schedule checksums related tasks in nightly graph.

Ready to go now that work from bug 1334167 has landed.
Fly, and be free!
Flags: needinfo?(lhenry)

        Attachment #8832940 -
        Flags: approval-mozilla-aurora? → approval-mozilla-aurora+

    
    

    
  
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #74)
> Comment on attachment 8832940 [details]
> Bug 1305139 - schedule checksums related tasks in nightly graph.
> 
> Ready to go now that work from bug 1334167 has landed.
> Fly, and be free!

Thank you! \o/

Uplifted the checksums patches to aurora as well:
https://hg.mozilla.org/releases/mozilla-aurora/rev/7d2e654d186f2e2cc27317355fd4679935926a2a
https://hg.mozilla.org/releases/mozilla-aurora/rev/53ceb8410c64d589103ba0a8eb54bcd2432e6f1b

The next set of aurora nightlies should have the checksums. I'll leave this open until we verified that everything is in place.
Thank you all for patience and apologies once again for the delays.

    
    

    
  
We can now finally close this as the latest aurora nightlies confirmed the existence of checksums:
e.g. 

* http://archive.mozilla.org/pub/firefox/nightly/2017/02/2017-02-23-08-41-36-mozilla-aurora/firefox-53.0a2.en-US.linux-i686.checksums
* http://archive.mozilla.org/pub/firefox/nightly/2017/02/2017-02-23-08-41-36-mozilla-aurora-l10n/firefox-53.0a2.ach.linux-i686.checksums

Thank you all for patience and apologies once again for the delays.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED

    
  
Status: RESOLVED → VERIFIED

    
    

    
  
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open

    
  
Product: TaskCluster → Firefox Build System

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: