talos on mozharness results in a lot of errors while installing PyYAML to the virtualenv

RESOLVED FIXED

Status

Release Engineering
Mozharness
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: jmaher, Assigned: jyeo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment, 3 obsolete attachments)

(Reporter)

Description

4 years ago
https://tbpl.mozilla.org/php/getParsedLog.php?id=25958737&full=1&branch=birch#error0

01:25:06     INFO -    Running setup.py install for PyYAML
01:25:06     INFO -      checking if libyaml is compilable
01:25:06     INFO -      gcc -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/tools/python27/include/python2.7 -c build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c -o build/temp.macosx-10.4-x86_64-2.7/check_libyaml.o
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:2:18: error: yaml.h: No such file or directory
01:25:06     INFO -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c: In function 'main':
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: 'yaml_parser_t' undeclared (first use in this function)
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: for each function it appears in.)
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:5: error: expected ';' before 'parser'
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:6: error: 'yaml_emitter_t' undeclared (first use in this function)
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:6: error: expected ';' before 'emitter'
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:8: warning: implicit declaration of function 'yaml_parser_initialize'
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:8: error: 'parser' undeclared (first use in this function)
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:9: warning: implicit declaration of function 'yaml_parser_delete'
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:11: warning: implicit declaration of function 'yaml_emitter_initialize'
01:25:06    ERROR -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:11: error: 'emitter' undeclared (first use in this function)
01:25:06  WARNING -      build/temp.macosx-10.4-x86_64-2.7/check_libyaml.c:12: warning: implicit declaration of function 'yaml_emitter_delete'
01:25:06  WARNING -      libyaml is not found or a compiler error: forcing --without-libyaml
01:25:06     INFO -      (if libyaml is installed correctly, you may need to
01:25:06     INFO -       specify the option --include-dirs or uncomment and
01:25:06     INFO -       modify the parameter include_dirs in setup.cfg)
01:25:06     INFO -    Running setup.py install for talos
01:25:06     INFO -      Installing PerfConfigurator script to /builds/slave/talos-slave/test/build/venv/bin
01:25:06     INFO -      Installing talos script to /builds/slave/talos-slave/test/build/venv/bin
01:25:06     INFO -      Installing talos-results script to /builds/slave/talos-slave/test/build/venv/bin
01:25:06     INFO -  Successfully installed datazilla httplib2 mozcrash mozdevice mozfile mozhttpd mozinfo mozlog mozprocess oauth2 PyYAML talos

It appears to be setup correctly and it functions for our purposes.  We need to find a way to add --without-libyaml to the installation in the virtualenv.

Updated

4 years ago
Blocks: 713055
(Assignee)

Comment 1

4 years ago
Two ways to fix this:
- Install the libyaml c library.
- Do `pip install PyYAML --global-option='--without-libyaml'` before installing talos
- some other way to pass --without-libyaml to `pip install talos` step?

Comment 2

4 years ago
we could, at least in theory, move away from YAML for talos configuration; IIRC, JSON is supported and working now

Comment 3

4 years ago
(In reply to Jeff Hammel [:jhammel] from comment #2)
> we could, at least in theory, move away from YAML for talos configuration;
> IIRC, JSON is supported and working now

That would be great as any of the options that jyeo mentions sound hacky (with all due respect).
(Assignee)

Comment 4

4 years ago
Created attachment 785141 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML
Assignee: nobody → yshun
Status: NEW → ASSIGNED
(Assignee)

Updated

4 years ago
Attachment #785141 - Flags: review?(aki)
Comment on attachment 785141 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML

I'm not a big fan of tuples for this type of thing, as the order is never intuitive.  Pretty sure the python style guide agrees with me.

I prefer {'module': ..., 'url': ..., 'options': [...]}.  However, this will require a lot of backporting since we already use the {modulename: url} construct in a decent number of places.

If this error is a massive problem that needs immediate fixing, I'm ok with this  (marking r+ with this caveat).  If we can live with it a bit longer to fix it in a better way, I would lean towards either getting rid of yaml or going with the above dict approach.
Attachment #785141 - Flags: review?(aki) → review+
(Assignee)

Comment 6

4 years ago
Created attachment 786319 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML v2

Using dicts instead of tuples
Attachment #785141 - Attachment is obsolete: true
Attachment #786319 - Flags: review?(aki)
(Assignee)

Comment 7

4 years ago
Created attachment 786325 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML v3

Change {'module1': 'path/to/module'} to {'name': module1, 'url': 'path/to/module'}
Attachment #786319 - Attachment is obsolete: true
Attachment #786319 - Flags: review?(aki)
(Assignee)

Updated

4 years ago
Attachment #786325 - Flags: review+
(Assignee)

Updated

4 years ago
Attachment #786325 - Flags: review+ → review?(aki)
Comment on attachment 786325 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML v3

Awesome, thanks for doing this!

>             if isinstance(module, dict):
>-                (module, module_url) = module.items()[0]
>+                module_name = module.get('name', None)

I'm currently leaning towards making module_name required... install_module() seems to need it.
What do you think?

If so, we should self.fatal() here if module_name is None.
Attachment #786325 - Flags: review?(aki) → review+
(Assignee)

Comment 9

4 years ago
Created attachment 786399 [details] [diff] [review]
Pass global options to install_module, add --without-libyaml as global-option for PyYAML v4

> I'm currently leaning towards making module_name required... install_module() seems to need it.
> What do you think?

Yep, I was thinking about that too. Better to be defensive.
Attachment #786325 - Attachment is obsolete: true
Attachment #786399 - Flags: review?(aki)
Attachment #786399 - Flags: review?(aki) → review+
Merged to production.
(Assignee)

Comment 11

4 years ago
I don't see any warnings related to libyaml. https://tbpl.mozilla.org/php/getParsedLog.php?id=26306899&tree=Mozilla-Central&full=1

FIXED. :)
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering

Updated

3 years ago
Component: General Automation → Mozharness
You need to log in before you can comment on or make changes to this bug.