Closed Bug 1460984 Opened 8 years ago Closed 7 years ago

I seem to be missing data while doing queries on activedata

Categories

(Testing Graveyard :: ActiveData, enhancement)

Product:
Testing Graveyard
Component:
ActiveData
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: jmaher, Assigned: ekyle)

Details

my goal is to take fresh oranges: https://charts.mozilla.org/FreshOranges/index.html and given a start revision, platform, branch, testname, find the last 20 revisions and show the job to retrigger and total pass/fail status. I start with a revision: https://activedata.allizom.org/tools/query.html#query_id=1pA_tMVM this gives me a date: 08-May-2018 15:48:00 I then use the date in another query to get the 19 revisions prior to that revision: https://activedata.allizom.org/tools/query.html#query_id=nj4R_x56 This returns a list of 20 revisions, but the original revision I queried wit hthe timestamp isn't three. Even adjust up 1 minute doesn't help. What I get is a list of revisions, that when mapped to treeherder look like this: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&fromchange=e22518b7790c&filter-searchStr=test-linux32%2Fopt-mochitest-devtools-chrome-e10s-2&tochange=e0b652659732 which is 38 revisions, not 20 (+- 1) so somehow revisions are missing from activedata. Given the 20 revisions from the above query: "data":[# revision, # date, # job count ["e0b652659732",1525792260,217771], ["7cd5006f1365",1525790280,221438], ["eb036f55167d",1525789920,1178788], ["e6dea32cd53b",1525788900,737987], ["dba34d8471d2",1525787520,1239435], ["abd2b3aec711",1525786680,213283], ["44c76dad59b6",1525784760,746982], ["d86d76dd906f",1525783440,1773641], ["5b74b76204d2",1525780140,1245330], ["5f85b4722429",1525776540,2], ["064597ea4795",1525771740,754313], ["e640d678c988",1525768920,1773809], ["f1f7e0069c2a",1525766640,716452], ["54a8664e402a",1525763220,712715], ["4dc436553912",1525758720,1613317], ["fb3186662dda",1525752780,145894], ["5382bedfa843",1525747020,1291672], ["634a505843de",1525734420,1239080], ["73bf33ab67f4",1525732383,750632], ["e22518b7790c",1525731444,392] ] what revisions are missing? It ends up looking like this: ["e0b652659732",1525792260,217771], ["7cd5006f1365",1525790280,221438], ["eb036f55167d",1525789920,1178788], ["e6dea32cd53b",1525788900,737987], ["dba34d8471d2",1525787520,1239435], ["abd2b3aec711",1525786680,213283], ["44c76dad59b6",1525784760,746982], ["d86d76dd906f",1525783440,1773641], ["5b74b76204d2",1525780140,1245330], ["5f85b4722429",1525776540,2], ["064597ea4795",1525771740,754313], ["e640d678c988",1525768920,1773809], - ["f1f7e0069c2a",1525766640,716452], 00630f737536 - ["54a8664e402a",1525763220,712715], - ["4dc436553912",1525758720,1613317], ["fb3186662dda",1525752780,145894], f1f7e0069c2a 00630f737536 54a8664e402a 4dc436553912 b57df5aa1534 c00426f6bc7d 292cf80054b4 ["5382bedfa843",1525747020,1291672], 0910ef8fa28d f9296d2a3287 b006cfda5d9e 45642b2381b5 346cf7a79bf8 96c17e4d2d9e 44fb2f695bd7 ["634a505843de",1525734420,1239080], ["73bf33ab67f4",1525732383,750632], e31a502239b9 ["e22518b7790c",1525731444,392] as you can see many of these are missing or out of order. Could it be I am querying incorrectly?
:ekyle, can you help me figure this out?
Flags: needinfo?(klahnakoski)
(In reply to Joel Maher ( :jmaher ) (UTC-4) (back May 21st) from comment #0) > I start with a revision: > https://activedata.allizom.org/tools/query.html#query_id=1pA_tMVM > > this gives me a date: > 08-May-2018 15:48:00 > > I then use the date in another query to get the 19 revisions prior to that > revision: > https://activedata.allizom.org/tools/query.html#query_id=nj4R_x56 The first query uses "repo.changeset.date", while the second uses the "repo.push.date". The former is always less than the latter. https://activedata.allizom.org/tools/query.html#query_id=Sz7RCi3m I will look at the corsss-compare with Treeherder shortly.
Flags: needinfo?(klahnakoski)
On the subject of the apparent missing revisions the problem is filtering by "repo.push.date", but sorting by "repo.changeset.date"; the changeset.date can be days before the push date, and the "limit":20 is not going back far enough. This can also explain the lack-of-order you are seeing in the missing data. This may be more what you expect: https://activedata.allizom.org/tools/query.html#query_id=L_2HYhof The "repo.changeset.date" is the date that the changeset was registered with hg on some branch; it is the same not matter what branch you look at. The repo.push.date is the date the changeset arrived on the particular branch. My apologies for not getting back to you sooner.
thanks Kyle, that makes a lot of sense and I am glad to have this in a better state. Here is what I get now: +a314710b0acd: test-macosx64/debug-mochitest-chrome-3: 1(XX) +4d511f7fc5b5: test-macosx64/debug-mochitest-chrome-3: 1(XX) +3d5b6c62a386: test-macosx64/debug-mochitest-chrome-3: 1(XX) +a2dfc3fd6c0c: test-macosx64/debug-mochitest-chrome-3: 1(XX) +da8925b18399: test-macosx64/debug-mochitest-chrome-3: 1(XX) +a5347b2c7f0c: test-macosx64/debug-mochitest-chrome-3: 1(XX) - b214af1d2690 (this is a backout with DONTBUILD - but missing in the repo list) +f266f3f133a9: test-macosx64/debug-mochitest-chrome-3: 1(XX) +7ae884246635: test-macosx64/debug-mochitest-chrome-3: 1(XX) +7614c38e5c0f: test-macosx64/debug-mochitest-chrome-3: 1(XX) +d5ec7b9c344a: test-macosx64/debug-mochitest-chrome-3: 1(XX) +87d810949bdf: test-macosx64/debug-mochitest-chrome-3: 0(XX) +1f0b7e9a6450: test-macosx64/debug-mochitest-chrome-3: 1(XX) +f5184707be08: test-macosx64/debug-mochitest-chrome-3: 1(XX) +f7b8d0d06d29: test-macosx64/debug-mochitest-chrome-3: 1(XX) +9d111a95cfbb: test-macosx64/debug-mochitest-chrome-3: 0(XX) +63b24cc63eca: test-macosx64/debug-mochitest-chrome-3: 1(XX) +aab1afd86d77: test-macosx64/debug-mochitest-chrome-3: 0(XX) +e46cb2927af9: test-macosx64/debug-mochitest-chrome-3: 1(XX) +f61158077bd0: test-macosx64/debug-mochitest-chrome-3: 1(XX) +31044a37a94f: test-macosx64/debug-mochitest-chrome-3: 1(XX) and this corresponds to: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-searchStr=osx%20debug%20mochitest-chrome-3&tochange=a314710b0acd&fromchange=31044a37a94f66cbb384e2f141a9b7a183441de0 in the above list 1 revision is missing and 2 revisions have 0 reported tests (9d111a95cfbb and aab1afd86d77). I verified in the logs for these revisions that the test test_arrowpanel.xul did run and get a test-ok status. Kyle, can you help me: 1) confirm/resolve the missing revision (I suspect DONTBUILD doesn't count in your table which is ok) 2) the 2 revisions that do not have the test reference.
Flags: needinfo?(klahnakoski)
rev 9d111a95cfbb can bee seen in TH: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=9d111a95cfbbe2dea1427d22ab342a584d9c8af6&filter-searchStr=osx%20debug%20mochitest-chrome-3&selectedJob=179516942 from there we see test-macosx64/debug-mochitest-chrome-3 was run in task CMu9e7RETrmmhAQhnMh5DQ looking for task in activedata shows the whole task is missing: https://activedata.allizom.org/tools/query.html#query_id=uFmFwpwt The task failed and finished at 2018-05-21T15:34:19.492Z: https://tools.taskcluster.net/groups/RKMVbGTSS3OKoqslVDS1PA/tasks/CMu9e7RETrmmhAQhnMh5DQ/runs/0 Let's look for other tasks that end near that time; notice the _id is tc.1567295...: https://activedata.allizom.org/tools/query.html#query_id=TEvdg5Vu Checking Amazon S3, we can see the task picked up by the pulse listener in the very next block tc.1567295: https://s3-us-west-2.amazonaws.com/active-data-task-cluster-logger/tc.1567296%3A156728763.json.gz I will now run the ETL locally on this block (tc.1567295) to see what happened. Stay tuned.
Processing locally, there does not seem to be a problem. I should have checked S3 for this nomralized artifact before I ran the ETL locally: https://s3.console.aws.amazon.com/s3/object/active-data-task-cluster-normalized/tc.1567296%253A156728763.json.gz?region=us-west-2&tab=overview&prefixSearch=tc.1567296 I confirmed the tc.1567296 does not exist in the test_result S3 bucket. Running ETL now...
Still running the unittest ETL. In the meantime, I talked to jmaher: He would like to know how many data points are missing so his tool can decide if it has enough to make a reasonable report. We can look at the "treeherder" table to get a sense of what should exist; and compare what what we really have: https://activedata.allizom.org/tools/query.html#query_id=cNpXRYJl unfortunately, this table is filled with the same ETL code that failed to provide the unittest data in a timely manner. The hope is the separate pipelines (unitttest and treeherder) are not both significantly delayed.
I have confirmed there is no problem in the ETL; it can process both the task block and the unittests inside of it: ETL processed to S3 (erased during testing, will be back at 15:00EDT): https://s3-us-west-2.amazonaws.com/active-data-test-result/tc.1567296%3A156728763.75.json.gz Now in ActiveData: https://activedata.allizom.org/tools/query.html#query_id=uxaY_AMi
Flags: needinfo?(klahnakoski)
Monitoring the delay in the data has no easy solution. I made an issue for it: https://github.com/klahnakoski/ActiveData-ETL/issues/41
I think there is no problem anymore.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
Product: Testing → Testing Graveyard
You need to log in before you can comment on or make changes to this bug.