1428363 - [mozlog] Reduce number of unused dependencies for log-parsing-only consumers of mozlog

Status: NEW

(Testing :: Mozbase, enhancement, P3)

Description

[Ed Morley [:emorley]]

In bug 1428049, mozterm was added as a dependency of mozlog. This is causing pip check to fail on Travis, unless we add mozterm to our requirements file:
https://travis-ci.org/mozilla/treeherder/jobs/325473107#L564 

However Treeherder only uses mozlog for structured logging parsing/support, and does not need any of the mach or terminal integration, and as such mozterm will be unused.

Would it be viable to make the mozterm dependency optional? (Perhaps adding as an extra_requires?)

It also appears that as of bug 1421799, blessings isn't used anywhere in mozlog - so it could be removed from mozlog's setup.py (and also anywhere else in the mozilla-central use-from-src-package plumbing).

Comment 1

[Andrew Halberstadt [:ahal]]

mozlog v3.7 (released this morning) should depend on mozterm properly, so you can likely just update your requirements to that.

I think we should still get mozlog to depend on blessings, as that will provide colors to the formatter. Mozterm is designed to work without blessings, you just won't get any formatting.

Comment 2

[Ed Morley [:emorley]]

I'm meaning the opposite - I'd prefer to not have the dependency, or at least some way for log parsing only consumers to have a package they can use that doesn't pull in two others.

Comment 3

[Ed Morley [:emorley]]

I don't know if you saw the above? :-)

Comment 4

[Andrew Halberstadt [:ahal]]

Oh sorry, I misinterpreted your request :).

Tbh, explicitly requiring mozterm was kind of the whole point of making that module in the first place. We usually make 'blessings' an optional dependency, but this lead to a lot of code like:

if blessings:
    <do stuff with styling>
else:
    <do stuff without styling>

This creates code duplication and complexity. The point of mozterm is that it's guaranteed to be installed, and will produce valid strings even if blessings isn't. So making it an optional dependency imo gets us back to the original problem where formatters need to test for it. Fwiw, I consider a dependency on mozterm to be exactly the same as depending on another mozbase package (the only difference is that mozterm lives under a different directory).

That said, we could make 'blessings' an optional dependency, though it would mean users don't get coloured output by default.. though they'd still get it when using mach, so that's probably not a huge deal.

Comment 5

[Andrew Halberstadt [:ahal]]

Just to clarify, mozterm is a *very* lightweight wrapper around blessings. If you are writing to a non-terminal, or if blessings isn't installed, you can still do stuff like:

term = mozterm.Terminal()
print(term.red('foobar'))

This will print 'foobar' without the red formatting. So mozterm is more about providing an abstraction around the terminal + non-terminal cases.

Comment 6

[Ed Morley [:emorley]]

Perhaps what is really needed is a mozlog-core that can be used by anything that needs to process structured logs but not actually contain the CLI parts? Treeherder would then use that, and mozlog can depend on whatever it likes for in-tree uses :-)

Comment 7

[Ed Morley [:emorley]]

There probably aren't many other out of tree consumers of mozlog other than Treeherder though, so I'd understand if it wasn't worth it.

It just feels like death by 1000 cuts for how many dependencies we have (albeit likely because they are more noticeable than for JS deps due to the lack of an official lockfile solution, pipfile WIPs aside):
https://github.com/mozilla/treeherder/blob/master/requirements/common.txt

Comment 8

[Andrew Halberstadt [:ahal]]

Yeah, fair enough. I think what you're asking would be to move the formatters out of the main mozlog module. Though that feels like it would make life harder for most other consumers of mozlog.

Just curious, what is the main problem you're trying to solve here? Have you heard of the 'hashin' package? It can generate requirements.txt complete with hashes for all dependencies and platforms automatically.

Comment 9

[Ed Morley [:emorley]]

(In reply to Andrew Halberstadt [:ahal] from comment #8)
> Just curious, what is the main problem you're trying to solve here? Have you
> heard of the 'hashin' package? It can generate requirements.txt complete
> with hashes for all dependencies and platforms automatically.

Individually an extra package isn't a big deal in the slightest - it's just that in every extra package is another dependency that:
* may or may not be compatible with Python 3 (or in the future Python 3.latest)
  -> worth noting that every single other Treeherder dependency was Python 3 compatible 12+ months before mozlog was
* adds to deployment/CI/local development environment bootstrap times (even with caches, it increases transfer time from S3 etc)
* is another package that results in pyup.io bot update PRs that someone has to spend time reviewing
* can often transiently pull in many packages (some of the mozbase related packages transiently pull in half of the mozbase suite when most of the other packages end up not being directly used)

In this case I suspect it's not worth doing anything about it for now, but I'd like mozbase packages to carefully consider the impact of too many dependencies in the future.

Comment 10

[Ed Morley [:emorley]]

s/in every extra package/in aggregate, every extra package/