1500995 - 'mach test-info' broken after ActiveData upgrade

Assignee

Description

•

6 years ago

$ ./mach test-info netwerk/test/unit/test_simple.js --verbose
===== netwerk/test/unit/test_simple.js =====
Found netwerk/test/unit/test_simple.js in source control.
netwerk/test/unit/test_simple.js found in manifest netwerk/test/unit/xpcshell.ini
  flavor: xpcshell
Querying ActiveData...
2018-10-22 10:38:02.532934
{"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["netwerk/test/unit/test_simple.js", "netwerk/test/unit/test_simple.js", "test_simple.js"]}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*test\\_simple\\.js.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-15"}}}, {"lt": {"run.timestamp": {"date": "2018-10-22"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"}
2018-10-22 10:38:03.017720
<Response [400]>
Error running mach:

    ['test-info', 'netwerk/test/unit/test_simple.js', '--verbose']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

HTTPError: 400 Client Error: BAD REQUEST for url: https://activedata.allizom.org/query

  File "/home/gbrown/src/testing/mach_commands.py", line 515, in test_info
    self.set_test_name()
  File "/home/gbrown/src/testing/mach_commands.py", line 645, in set_test_name
    data = self.submit(query)
  File "/home/gbrown/src/testing/mach_commands.py", line 682, in submit
    response.raise_for_status()
  File "/home/gbrown/src/third_party/python/requests/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

Kyle Lahnakoski [:ekyle]

Comment 1

•

6 years ago

needinfo me so I do not forget

Flags: needinfo?(klahnakoski)

Kyle Lahnakoski [:ekyle]

Comment 2

•

6 years ago

gbrown, please verify this is no longer a problem.

Kyle Lahnakoski [:ekyle]

Updated

•

6 years ago

Flags: needinfo?(klahnakoski) → needinfo?(gbrown)

Geoff Brown [:gbrown]

Assignee

Comment 3

•

6 years ago

Getting further, but still failing:

$ ./mach test-info netwerk/test/unit/test_simple.js --verbose
===== netwerk/test/unit/test_simple.js =====
Found netwerk/test/unit/test_simple.js in source control.
netwerk/test/unit/test_simple.js found in manifest netwerk/test/unit/xpcshell.ini
  flavor: xpcshell
Querying ActiveData...
2018-10-23 08:24:54.644148
{"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["netwerk/test/unit/test_simple.js", "netwerk/test/unit/test_simple.js", "test_simple.js"]}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*test\\_simple\\.js.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-16"}}}, {"lt": {"run.timestamp": {"date": "2018-10-23"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"}
2018-10-23 08:25:02.682551
<Response [200]>
Found records matching 'netwerk/test/unit/test_simple.js' in ActiveData.

Test results for netwerk/test/unit/test_simple.js on mozilla-central,mozilla-inbound,autoland between 2018-10-16 and 2018-10-23
2018-10-23 08:25:02.734695
{"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.test": "netwerk/test/unit/test_simple.js"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-16"}}}, {"lt": {"run.timestamp": {"date": "2018-10-23"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "count"}, {"aggregate": "sum", "default": 0, "name": "failures", "value": {"case": [{"then": 1, "when": {"eq": {"result.ok": "F"}}}]}}]}
2018-10-23 08:25:04.219446
<Response [400]>
Error running mach:

    ['test-info', 'netwerk/test/unit/test_simple.js', '--verbose']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

HTTPError: 400 Client Error: BAD REQUEST for url: http://activedata.allizom.org/query

  File "/home/gbrown/src/testing/mach_commands.py", line 517, in test_info
    self.report_test_results()
  File "/home/gbrown/src/testing/mach_commands.py", line 713, in report_test_results
    data = self.submit(query)
  File "/home/gbrown/src/testing/mach_commands.py", line 682, in submit
    response.raise_for_status()
  File "/home/gbrown/src/third_party/python/requests/requests/models.py", line 840, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

Flags: needinfo?(gbrown)

Kyle Lahnakoski [:ekyle]

Comment 4

•

6 years ago

The query no longer fails, but the 

> "groupby": ["build.platform", "build.type", "run.type"]

is still double counting

Flags: needinfo?(klahnakoski)

Joel Maher ( :jmaher ) (UTC -8)

Comment 5

•

6 years ago

checking in here Kyle, is this getting closer to being fixed?

Kyle Lahnakoski [:ekyle]

Comment 6

•

6 years ago

this is now working.  :gbrown, please confirm and close

Flags: needinfo?(klahnakoski) → needinfo?(gbrown)

Joel Maher ( :jmaher ) (UTC -8)

Comment 7

•

6 years ago

$ ./mach test-info testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html --verbose
===== testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html =====
Found testing\web-platform\tests\encoding\legacy-mb-korean\euc-kr\euckr-decode-ksc_5601.html in source control.
testing\web-platform\tests\encoding\legacy-mb-korean\euc-kr\euckr-decode-ksc_5601.html not found in any test manifest!
Querying ActiveData...
2018-11-05 12:38:58.747000
{"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["testing\\web-platform\\tests\\encoding\\legacy-mb-korean\\euc-kr\\euckr-decode-ksc_5601.html", "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"]}}, {"regexp": {"result.test": ".*testing\\\\web\\-platform\\\\tests\\\\encoding\\\\legacy\\-mb\\-korean\\\\euc\\-kr\\\\euckr\\-decode\\-ksc\\_5601\\.html.*"}}, {"regexp": {"result.test": ".*testing\\/web\\-platform\\/tests\\/encoding\\/legacy\\-mb\\-korean\\/euc\\-kr\\/euckr\\-decode\\-ksc\\_5601\\.html.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"}
2018-11-05 12:39:08.960000
<Response [200]>
Unable to find matching records in ActiveData; using testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html!

Test results for testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05
2018-11-05 12:39:09.082000
{"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.test": "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "count"}, {"aggregate": "sum", "default": 0, "name": "failures", "value": {"case": [{"then": 1, "when": {"eq": {"result.ok": "F"}}}]}}]}
2018-11-05 12:39:15.959000
<Response [200]>
No test result data found.
2018-11-05 12:39:15.966000
{"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.ok": "T"}}, {"eq": {"result.test": "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "average", "name": "average", "value": "result.duration"}, {"aggregate": "min", "name": "min", "value": "result.duration"}, {"aggregate": "max", "name": "max", "value": "result.duration"}, {"aggregate": "count"}]}
2018-11-05 12:39:23.198000
<Response [200]>

Test durations for testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05
No test durations found.
2018-11-05 12:39:23.204000
{"limit": 1000, "from": "unittest", "where": {"and": [{"eq": {"result.test": "testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "select": ["build.platform", "build.type", "run.type", "run.name"], "format": "list"}
2018-11-05 12:39:23.890000
<Response [200]>

Test tasks for testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05
No test tasks found.

Bugzilla quick search for 'testing\web-platform\tests\encoding\legacy-mb-korean\euc-kr\euckr-decode-ksc_5601.html,testing/web-platform/tests/encoding/legacy-mb-korean/euc-kr/euckr-decode-ksc_5601.html':




so this still doesn't work :(

Kyle Lahnakoski [:ekyle]

Comment 8

•

6 years ago

maybe this is part of the ingestion backlog.  I am working on increasing this ingestion speed.

Flags: needinfo?(klahnakoski)

Kyle Lahnakoski [:ekyle]

Updated

•

6 years ago

Flags: needinfo?(gbrown)

Geoff Brown [:gbrown]

Assignee

Comment 9

•

6 years ago

Another example, that breaks in a different way:

$ ./mach test-info test_simple.js --verbose
===== test_simple.js =====
Found netwerk/test/unit/test_simple.js in source control.
Build configuration changed. Regenerating backend.
No handlers could be found for logger "mozbuild.frontend.reader"
netwerk/test/unit/test_simple.js found in manifest netwerk/test/unit/xpcshell.ini
  flavor: xpcshell
Querying ActiveData...
2018-11-05 11:02:40.400358
{"limit": 10, "where": {"and": [{"or": [{"in": {"result.test": ["netwerk/test/unit/test_simple.js", "test_simple.js"]}}, {"regexp": {"result.test": ".*netwerk\\/test\\/unit\\/test\\_simple\\.js.*"}}, {"regexp": {"result.test": ".*test\\_simple\\.js.*"}}]}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "from": "unittest", "groupby": ["result.test"], "format": "list"}
2018-11-05 11:02:46.457527
<Response [200]>
Found records matching 'netwerk/test/unit/test_simple.js' in ActiveData.

Test results for netwerk/test/unit/test_simple.js on mozilla-central,mozilla-inbound,autoland between 2018-10-29 and 2018-11-05
2018-11-05 11:02:46.503368
{"from": "unittest", "format": "list", "limit": 100, "where": {"and": [{"eq": {"result.test": "netwerk/test/unit/test_simple.js"}}, {"in": {"build.branch": ["mozilla-central", "mozilla-inbound", "autoland"]}}, {"gt": {"run.timestamp": {"date": "2018-10-29"}}}, {"lt": {"run.timestamp": {"date": "2018-11-05"}}}]}, "groupby": ["build.platform", "build.type", "run.type"], "select": [{"aggregate": "count"}, {"aggregate": "sum", "default": 0, "name": "failures", "value": {"case": [{"then": 1, "when": {"eq": {"result.ok": "F"}}}]}}]}
2018-11-05 11:03:02.023607
<Response [200]>
Error running mach:

    ['test-info', 'test_simple.js', '--verbose']

The error occurred in the implementation of the invoked mach command.

This should never occur and is likely a bug in the implementation of that
command. Consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: u'failures'

  File "/home/gbrown/src/testing/mach_commands.py", line 517, in test_info
    self.report_test_results()
  File "/home/gbrown/src/testing/mach_commands.py", line 724, in report_test_results
    failures = record['failures']



This is curious in that the query succeeds and returns results for the count column, but not for 'failures':

{
	"from":"unittest",
	"format":"list",
	"limit":100,
	"where":{"and":[
		{"eq":{"result.test":"netwerk/test/unit/test_simple.js"}},
		{"in":{"build.branch":["mozilla-central","mozilla-inbound","autoland"]}},
		{"gt":{"run.timestamp":{"date":"2018-10-29"}}},
		{"lt":{"run.timestamp":{"date":"2018-11-05"}}}
	]},
	"groupby":["build.platform","build.type","run.type"],
	"select":[
		{"aggregate":"count"},
		{
			"aggregate":"sum",
			"default":0,
			"name":"failures",
			"value":{"case":[{"then":1,"when":{"eq":{"result.ok":"F"}}}]}
		}
	]
}

"run"	"build"	"count"
	"{\"platform\":\"win64\",\"type\":\"debug\"}"	447
	"{\"platform\":\"maxosx64\",\"type\":\"debug\"}"	441
	"{\"platform\":\"linux32\",\"type\":\"debug\"}"	423
...

Kyle Lahnakoski [:ekyle]

Comment 10

•

6 years ago

New version deployed

:gbrown, please test again.  :jmaher is not getting any data, maybe you can see what's still wrong

Flags: needinfo?(klahnakoski)

Geoff Brown [:gbrown]

Assignee

Comment 11

•

6 years ago

Discussed on irc this morning. It looks like the only remaining issue is https://github.com/mozilla/ActiveData/issues/101.

Geoff Brown [:gbrown]

Assignee

Updated

•

6 years ago

Priority: -- → P3

Geoff Brown [:gbrown]

Assignee

Comment 12

•

5 years ago

I'll just work around the activedata issue to get this working again.

Assignee: nobody → gbrown

Geoff Brown [:gbrown]

Assignee

Updated

•

5 years ago

Priority: P3 → P1

Pulsebot

Comment 13

•

5 years ago

Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fb4be273bff4
Allow for missing 'failures' data in 'mach test-info'; r=me,a=test-only

Cristina Coroiu [:ccoroiu]

Comment 14

•

5 years ago

bugherder

https://hg.mozilla.org/mozilla-central/rev/fb4be273bff4

Status: NEW → RESOLVED

Closed: 5 years ago

status-firefox66: --- → fixed

Resolution: --- → FIXED

Target Milestone: --- → mozilla66

Bugzilla

Quick Search

'mach test-info' broken after ActiveData upgrade

Categories

(Testing :: General, defect, P1)

Tracking

(firefox66 fixed)

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Crash Data

Security

(public)

User Story

Description

Comment 1

Comment 2

Updated

Comment 3

Comment 4

Comment 5

Comment 6

Comment 7

Comment 8

Updated

Comment 9

Comment 10

Comment 11

Updated

Comment 12

Updated

Comment 13

Comment 14