1687732 - Perma OSX testing/mozbase/mozproxy/tests/test_recording.py::test_record_and_replay TEST-UNEXPECTED-FAIL | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

Status: RESOLVED FIXED

(Testing :: Mozbase, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: abutkovits [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=327261928&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TlLlNio9RWywrJsC5LQuvA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-01-20T15:51:53.757Z]  0:46.58 /Users/cltbld/tasks/task_1611156228/checkouts/gecko/testing/mozbase/mozproxy/tests/test_recording.py
[task 2021-01-20T15:51:53.758Z]  0:46.58 ============================= test session starts ==============================
[task 2021-01-20T15:51:53.758Z]  0:46.58 platform darwin -- Python 2.7.16, pytest-3.6.2, py-1.5.4, pluggy-0.6.0 -- /Users/cltbld/tasks/task_1611156228/checkouts/gecko/obj-x86_64-apple-darwin18.6.0/_virtualenvs/python-test/bin/python
[task 2021-01-20T15:51:53.758Z]  0:46.58 rootdir: /Users/cltbld/tasks/task_1611156228/checkouts/gecko, inifile: /Users/cltbld/tasks/task_1611156228/checkouts/gecko/config/mozunit/mozunit/pytest.ini
[task 2021-01-20T15:51:53.758Z]  0:46.58 collecting ... collected 1 item
[task 2021-01-20T15:51:53.758Z]  0:46.58 
[task 2021-01-20T15:51:53.758Z]  0:46.58 testing/mozbase/mozproxy/tests/test_recording.py::test_record_and_replay TEST-UNEXPECTED-FAIL
[task 2021-01-20T15:51:53.760Z]  0:46.58 
[task 2021-01-20T15:51:53.761Z]  0:46.58 =================================== FAILURES ===================================
[task 2021-01-20T15:51:53.761Z]  0:46.58 ____________________________ test_record_and_replay ____________________________
[task 2021-01-20T15:51:53.761Z]  0:46.58 
[task 2021-01-20T15:51:53.761Z]  0:46.58 args = ()
[task 2021-01-20T15:51:53.761Z]  0:46.58 recording_file = '/Users/cltbld/tasks/task_1611156228/checkouts/gecko/testing/mozbase/mozproxy/tests/files/new_recoding.zip'
[task 2021-01-20T15:51:53.761Z]  0:46.58 config = {'app': 'firefox', 'binary': 'firefox', 'host': '127.0.0.1', 'obj_path': '/var/folders/9w/gjfgmclx0fg6_2c_b07v2l4m000017/T/tmpyw2rEh', ...}
[task 2021-01-20T15:51:53.761Z]  0:46.58 obj_path = '/var/folders/9w/gjfgmclx0fg6_2c_b07v2l4m000017/T/tmpyw2rEh'
[task 2021-01-20T15:51:53.761Z]  0:46.58 
[task 2021-01-20T15:51:53.761Z]  0:46.58     def test_record_and_replay(*args):
[task 2021-01-20T15:51:53.761Z]  0:46.58         # test setup
[task 2021-01-20T15:51:53.762Z]  0:46.58         recording_file = os.path.join(here, "files", "new_recoding.zip")
[task 2021-01-20T15:51:53.762Z]  0:46.58 
[task 2021-01-20T15:51:53.762Z]  0:46.58         # Record part
[task 2021-01-20T15:51:53.762Z]  0:46.58         config = {
[task 2021-01-20T15:51:53.762Z]  0:46.58             "playback_tool": "mitmproxy",
[task 2021-01-20T15:51:53.762Z]  0:46.58             "recording_file": recording_file,
[task 2021-01-20T15:51:53.762Z]  0:46.58             "playback_version": "5.1.1",
[task 2021-01-20T15:51:53.762Z]  0:46.58             "platform": mozinfo.os,
[task 2021-01-20T15:51:53.762Z]  0:46.58             "run_local": "MOZ_AUTOMATION" not in os.environ,
[task 2021-01-20T15:51:53.762Z]  0:46.58             "binary": "firefox",
[task 2021-01-20T15:51:53.762Z]  0:46.58             "app": "firefox",
[task 2021-01-20T15:51:53.763Z]  0:46.58             "host": "127.0.0.1",
[task 2021-01-20T15:51:53.763Z]  0:46.58             "record": True,
[task 2021-01-20T15:51:53.763Z]  0:46.58         }
[task 2021-01-20T15:51:53.763Z]  0:46.58 
[task 2021-01-20T15:51:53.763Z]  0:46.58         with tempdir() as obj_path:
[task 2021-01-20T15:51:53.763Z]  0:46.58             config["obj_path"] = obj_path
[task 2021-01-20T15:51:53.763Z]  0:46.58             record = get_playback(config)
[task 2021-01-20T15:51:53.763Z]  0:46.58             assert record is not None
[task 2021-01-20T15:51:53.763Z]  0:46.58 
[task 2021-01-20T15:51:53.763Z]  0:46.58             try:
[task 2021-01-20T15:51:53.763Z]  0:46.58 >               record.start()
[task 2021-01-20T15:51:53.763Z]  0:46.58 
[task 2021-01-20T15:51:53.764Z]  0:46.58 testing/mozbase/mozproxy/tests/test_recording.py:46:```

Comment 1

[Intermittent Failures Robot]

1 failures in 3488 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1014-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-01-18&endday=2021-01-24&tree=all

Comment 2

[Intermittent Failures Robot]

1 failures in 4322 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-02-08&endday=2021-02-14&tree=all

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 4

[Cosmin Sabou [:CosminS]]

https://treeherder.mozilla.org/logviewer?job_id=333555863&repo=mozilla-central

Comment 5

[Intermittent Failures Robot]

1 failures in 4257 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-03-15&endday=2021-03-21&tree=all

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 7

[Alexandru Michis [:malexandru]]

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=337733669&repo=autoland&lineNumber=1446

Comment 8

[Alexandru Michis [:malexandru]]

Started appearing on tier 1 jobs.

Comment 9

[Intermittent Failures Robot]

1 failures in 3706 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1014-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-04-26&endday=2021-05-02&tree=all

Comment 10

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 11

[Sandor Molnar[:smolnar]]

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=345503874&repo=mozilla-central&lineNumber=1172

Comment 12

[Intermittent Failures Robot]

1 failures in 2224 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1014-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-07-12&endday=2021-07-18&tree=all

Comment 13

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 14

[Cristian Tuns]

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=353600402&repo=mozilla-beta&lineNumber=1549

Comment 15

[Intermittent Failures Robot]

1 failures in 3626 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform and build breakdown:

• macosx1015-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-10-04&endday=2021-10-10&tree=all

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 17

[Alexandru Michis [:malexandru]]

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=357526705&repo=autoland&lineNumber=3732

Comment 18

[Intermittent Failures Robot]

3 failures in 2723 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• windows10-64: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-11-08&endday=2021-11-14&tree=all

Comment 19

[Intermittent Failures Robot]

6 failures in 3259 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 3

Platform and build breakdown:

• windows10-64: 5
  • opt: 5
• linux1804-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2021-11-15&endday=2021-11-21&tree=all

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 21

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=369192358&repo=mozilla-central

Comment 22

[Intermittent Failures Robot]

1 failures in 2992 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2022-02-21&endday=2022-02-27&tree=all

Comment 23

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 24

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=401225472&repo=autoland

Comment 25

[Intermittent Failures Robot]

1 failures in 2607 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-01-02&endday=2023-01-08&tree=all

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Comment 27

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=420225050&repo=autoland

Comment 28

[Intermittent Failures Robot]

1 failures in 3880 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-06-19&endday=2023-06-25&tree=all

Comment 29

[Intermittent Failures Robot]

2 failures in 2956 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 2

Platform and build breakdown:

• windows11-64: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-07-03&endday=2023-07-09&tree=all

Comment 30

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 31

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=426654919&repo=mozilla-release

Comment 32

[Intermittent Failures Robot]

1 failures in 2218 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-08-21&endday=2023-08-27&tree=all

Comment 33

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 34

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=430535701&repo=autoland

Comment 35

[Intermittent Failures Robot]

1 failures in 3726 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-09-25&endday=2023-10-01&tree=all

Comment 36

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 37

[Treeherder Bug Filer]

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=433798781&repo=autoland

Comment 38

[Intermittent Failures Robot]

1 failures in 3657 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-10-23&endday=2023-10-29&tree=all

Comment 39

[Intermittent Failures Robot]

1 failures in 3495 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1

Platform and build breakdown:

• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-11-13&endday=2023-11-19&tree=all

Comment 40

[Intermittent Failures Robot]

7 failures in 3623 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 2

Platform and build breakdown:

• windows11-64: 2
  • opt: 2
• macosx1015-64: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-11-20&endday=2023-11-26&tree=all

Comment 41

[Sandor Molnar[:smolnar]]

@Serg, can you look at this python failure?
it turned perma after bug 1863411 landed.
Caught it so late because the test doesn't run so often.

Comment 42

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1863411

Comment 43

[Intermittent Failures Robot]

22 failures in 4108 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 14
• mozilla-central: 8

Platform and build breakdown:

• macosx1015-64: 20
  • opt: 20
• windows11-64: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-11-27&endday=2023-12-03&tree=all

Comment 44

[Intermittent Failures Robot]

5 failures in 3671 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2

Platform and build breakdown:

• macosx1015-64: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-12-04&endday=2023-12-10&tree=all

Comment 45

[Intermittent Failures Robot]

7 failures in 4224 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-release: 1
• autoland: 4
• mozilla-central: 2

Platform and build breakdown:

• macosx1015-64: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-12-11&endday=2023-12-17&tree=all

Comment 46

[[:sergesanspaille]]

I'm investigating this one, looks like the taskcluster-built python cannoit find the required certificates...

Comment 47

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1863411

Comment 48

[Intermittent Failures Robot]

5 failures in 3349 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 2
• mozilla-central: 2

Platform and build breakdown:

• macosx1015-64: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-12-18&endday=2023-12-24&tree=all

Comment 49

[Intermittent Failures Robot]

1 failures in 1162 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• macosx1015-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2023-12-25&endday=2023-12-31&tree=all

Comment 50

[Intermittent Failures Robot]

9 failures in 2274 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 4

Platform and build breakdown:

• macosx1015-64: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2024-01-01&endday=2024-01-07&tree=all

Comment 51

[Intermittent Failures Robot]

17 failures in 3217 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 7
• mozilla-beta: 1

Platform and build breakdown:

• macosx1015-64: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2024-01-08&endday=2024-01-14&tree=all

Comment 52

[Intermittent Failures Robot]

17 failures in 3599 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-release: 1
• mozilla-central: 6

Platform and build breakdown:

• macosx1015-64: 16
  • opt: 16
• windows11-64: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2024-01-15&endday=2024-01-21&tree=all

Comment 53

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Should the tests be disabled until this gets resolved?

Comment 54

[Intermittent Failures Robot]

8 failures in 4034 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1
• autoland: 5
• mozilla-central: 2

Platform and build breakdown:

• macosx1015-64: 8
  • opt: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2024-01-22&endday=2024-01-28&tree=all

Comment 55

[Intermittent Failures Robot]

15 failures in 3952 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 6
• oak: 1

Platform and build breakdown:

• macosx1015-64: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2024-01-29&endday=2024-02-04&tree=all

Comment 56

[Intermittent Failures Robot]

3 failures in 4009 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1

Platform and build breakdown:

• macosx1015-64: 3
  • opt: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2024-02-05&endday=2024-02-11&tree=all

Comment 57

[[:sergesanspaille]]

Attachment: Bug 1687732 - Bundle certifi certificates in taskcluster's Python on OSX r=glandium

Instead of trying to find the system certificates, something that may
depend on the running system, use the certifi package and have ssl point
to it.

Comment 58

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b48cab2cc9dc
Bundle certifi certificates in taskcluster's Python on OSX r=jmaher

Comment 59

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b48cab2cc9dc

Comment 60

[Intermittent Failures Robot]

4 failures in 3125 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2
• mozilla-central: 1
• oak: 1

Platform and build breakdown:

• macosx1015-64: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1687732&startday=2024-02-12&endday=2024-02-18&tree=all