If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

l10n repackage tasks have duplicate build artifacts

REOPENED
Unassigned

Status

Release Engineering
General Automation
REOPENED
2 months ago
2 months ago

People

(Reporter: pmoore, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 months ago
In a task, such as https://tools.taskcluster.net/groups/L0GyoyuZTL2m55NggEnGGg/tasks/CKeH8b-cTgKISyQ6FoT5Eg/details there is an artifact section like this:

      "artifacts": [
        {
          "path": "public/build/target.installer.exe",
          "expires": "2018-07-14T00:49:49.913722Z",
          "type": "file",
          "name": "public/build/mk/target.installer.exe"
        },
        {
          "path": "public/build/target.complete.mar",
          "expires": "2018-07-14T00:49:49.913722Z",
          "type": "file",
          "name": "public/build/mk/target.complete.mar"
        },
        {
          "path": "public/build/target.stub-installer.exe",
          "expires": "2018-07-14T00:49:49.913722Z",
          "type": "file",
          "name": "public/build/mk/target.stub-installer.exe"
        },
        {
          "path": "public/build",
          "expires": "2018-07-14T00:49:49.913722Z",
          "type": "directory",
          "name": "public/build"
        }
      ],

This results in duplicate artifacts, like these:

public/build/mk/target.complete.mar
public/build/mk/target.installer.exe
public/build/mk/target.stub-installer.exe
public/build/target.complete.mar
public/build/target.installer.exe
public/build/target.stub-installer.exe

The "directory" artifact here should simply be removed. Alternatively, the file artifacts should be deleted, and the directory artifact should be changed to use "name": "public/build/mk". My preference would be the first option, as explicitly listing artifacts introduces less risk that suddenly large unexpected artifacts suddenly get introduced unintentionally.

It appears these tasks are coming from the date project branch of gecko.

Comment 1

2 months ago
This was on `date` only, and I think I may have fixed it. This was because generic-worker enforces `public/build` as an artifact directory, and complains if its missing, therefore I set the output folder to `public/build`, the l10n artifact paths explicitly specified put the artifacts in `{locale}` folders though (from the output at base of folder). I've adjusted the logic for now to output directly to the locale specific folder, and to find those files there.
Status: NEW → RESOLVED
Last Resolved: 2 months ago
Resolution: --- → FIXED
Summary: l10n repacks have duplicate build artifacts → l10n repackage tasks have duplicate build artifacts
(Reporter)

Comment 2

2 months ago
(In reply to Justin Wood (:Callek) from comment #1)
> This was because generic-worker enforces `public/build` as an artifact directory

No it doesn't. That must be something in-tree adding it.
(Reporter)

Updated

2 months ago
See Also: → bug 1380978

Comment 3

2 months ago
(In reply to Pete Moore [:pmoore][:pete] from comment #2)
> (In reply to Justin Wood (:Callek) from comment #1)
> > This was because generic-worker enforces `public/build` as an artifact directory
> 
> No it doesn't. That must be something in-tree adding it.

to be more verbose I meant the in tree taskgraph enforces public/build/ as an artifact directory for generic worker, and changing it is convoluted at present.
(Reporter)

Comment 4

2 months ago
(In reply to Justin Wood (:Callek) from comment #1)
> I think I may have fixed it.

It looks like the task is still including the individual files and the folder. For example, from yesterday:

https://tools.taskcluster.net/groups/dKUeaMgPTbu7R6tFSLqaXA/tasks/CXRtIdmVQVqeNEAsWGvblw/details


  "artifacts": [
    {
      "path": "public/build/mk/target.installer.exe",
      "expires": "2018-07-18T20:50:20.046804Z",
      "type": "file",
      "name": "public/build/mk/target.installer.exe"
    },
    {
      "path": "public/build/mk/target.complete.mar",
      "expires": "2018-07-18T20:50:20.046804Z",
      "type": "file",
      "name": "public/build/mk/target.complete.mar"
    },
    {
      "path": "public/build/mk/target.stub-installer.exe",
      "expires": "2018-07-18T20:50:20.046804Z",
      "type": "file",
      "name": "public/build/mk/target.stub-installer.exe"
    },
    {
      "path": "public/build",
      "expires": "2018-07-18T20:50:20.046804Z",
      "type": "directory",
      "name": "public/build"
    }
  ]
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 5

2 months ago
Admittedly, this doesn't result in duplicate artifacts thanks to the idempotency of the APIs, but it does mean the artifact publish occurs twice, and there is redundancy in the task definition, even if in the end, only one artifact is persisted.
You need to log in before you can comment on or make changes to this bug.