1306159 - Inconsistent values for PULSE_DATA_INGESTION_SOURCES between environments

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Infrastructure, defect, P1)

Description

[Ed Morley [:emorley]]

There are a few inconsistencies in the value of PULSE_DATA_INGESTION_SOURCES:
* On SCL3+Heroku stage, there's a `treeherder-test` exchange - but not on prod. Is this used?
* The value for Heroku stage matches that of SCL3 prod (and not SCL3 stage). I'm presuming it should instead match SCL3 stage?

SCL3 stage:

[
  {
    "exchange": "exchange/taskcluster-treeherder-staging/v1/jobs",
    "destinations": ["tc-treeherder-stage"],"projects": ["#"]
  },
  {
    "exchange": "exchange/treeherder-test/jobs",
    "destinations": ["treeherder-staging"],"projects": ["#"]
  }
]

Heroku stage:

[
  {
    "exchange": "exchange/taskcluster-treeherder/v1/jobs",
    "destinations": ["tc-treeherder"], "projects": ["#"]
  },
  {
    "exchange": "exchange/treeherder-test/jobs",
    "destinations": ["treeherder-staging"], "projects": ["#"]
  }
]

SCL3 prod:

[
  {
    "exchange": "exchange/taskcluster-treeherder/v1/jobs",
    "destinations": ["tc-treeherder"],"projects": ["#"]
  }
]

Heroku prod:

[
  {
    "exchange": "exchange/taskcluster-treeherder/v1/jobs",
    "destinations": ["tc-treeherder"],"projects": ["#"]
  }
]

Comment 1

[Cameron Dawson [:camd]]

We could easily omit the treeherder-test.  I used it a lot while developing, etc, but don't expect to use it again.  If we ever DO want to do so on stage, we could easily re-add it.  So I'm for removing it and making them all consistent.

Comment 2

[Ed Morley [:emorley]]

Also this part?

> * The value for Heroku stage matches that of SCL3 prod (and not SCL3 stage).
> I'm presuming it should instead match SCL3 stage?

Another question too:

Is the taskcluster-treeherder-staging exchange a superset of the taskcluster-treeherder exchange, or does it contain a different set of jobs? ie should stage listen to both? Perhaps this comes down the the discussion we had recently about having both a "test environment for us to use, that's as close to prod as possible", plus a "test environment for people developing projects that interact with treeherder"?

Comment 3

[Cameron Dawson [:camd]]

As I understand it, taskcluster-treeherder staging and prod both have the same jobs.  It's just that they can try out new processes on their stage, of course.  

So my inclination is what I think you're getting at: Set all our instances (stage and prod) to the prod exchange for taskcluster-treeherder.  If and when we want to try out some change that's been pushed to taskcluster-treeherder stage, we can switch our stage to point to that and do the test.

If someone has a job they want to show on TH stage and not prod, that's what the exclusion profiles are for. :)

Comment 4

[Cameron Dawson [:camd]]

To be clear, I think they all should be set to:

[
  {
    "exchange": "exchange/taskcluster-treeherder/v1/jobs",
    "destinations": ["tc-treeherder"],"projects": ["#"]
  }
]

Comment 5

[Ed Morley [:emorley]]

Sounds good :-)

$ ths config:set PULSE_DATA_INGESTION_SOURCES='[{"exchange": "exchange/taskcluster-treeherd er/v1/jobs", "destinations": ["tc-treeherder"], "projects": ["#"]}]'
Setting PULSE_DATA_INGESTION_SOURCES and restarting treeherder-stage... done, v374
 !    Release command executing: this config change will not be available until the command succeeds.
PULSE_DATA_INGESTION_SOURCES: [{"exchange": "exchange/taskcluster-treeherder/v1/jobs", "destinations": ["tc-treeherder"], "projects": ["#"]}]

-> Heroku stage/prod and SCL3 prod all now have identical PULSE_DATA_INGESTION_SOURCES values.

(I won't bother changing SCL3 stage)