1137044 - Updating skia is under-documented

Status: RESOLVED INACTIVE

(Core :: Graphics, defect)

Description

[Mike Hommey [:glandium]]

gfx/skia/moz.build has a bug I want to fix. Its header says to go away unless I'm debugging generate_mozbuild.py.

Ok. Fine.

$ python generate_mozbuild.py 
sh: 1: ./gyp_skia: not found
Failed to generate sources for linux
sh: 1: ./gyp_skia: not found
Failed to generate sources for mac
sh: 1: ./gyp_skia: not found
Failed to generate sources for android
sh: 1: ./gyp_skia: not found
Failed to generate sources for win
Traceback (most recent call last):
  File "generate_mozbuild.py", line 467, in <module>
    main()
  File "generate_mozbuild.py", line 463, in main
    write_mozbuild(includes, separated_sources)
  File "generate_mozbuild.py", line 427, in write_mozbuild
    write_sources(f, sources['mac'], 4)
KeyError: 'mac'

So, running generate_mozbuild.py doesn't work with what's in the tree. Ok.

README_MOZILLA talks about an update.sh script... which is nowhere to be found. It was removed in bug 777614. Maybe that wasn't intentional.

Anyways. update.sh doesn't do anything to pull skia, you need to have a clone already. Ok.

No mention of the repository in README_MOZILLA, but one can guess from a google^Wyahoo search. So I clone skia.

README_MOZILLA says: "- Copy the entire trunk/ directory from a Skia clone to mozilla-central/gfx/skia"

But there is no trunk/ directory. That very much looks like those instructions are stuck in the age of svn. Ok.

So, I copy the whole skia working tree for the mentioned git commit (thankfully, that, at least, is mentioned) under gfx/skia/trunk.

$ python generate_mozbuild.py 
Traceback (most recent call last):
  File "./gyp_skia", line 29, in <module>
    import gyp
ImportError: No module named gyp
Failed to generate sources for linux
Traceback (most recent call last):
  File "./gyp_skia", line 29, in <module>
    import gyp
ImportError: No module named gyp
Failed to generate sources for mac
Traceback (most recent call last):
  File "./gyp_skia", line 29, in <module>
    import gyp
ImportError: No module named gyp
Failed to generate sources for android
Traceback (most recent call last):
  File "./gyp_skia", line 29, in <module>
    import gyp
ImportError: No module named gyp
Failed to generate sources for win
Traceback (most recent call last):
  File "generate_mozbuild.py", line 467, in <module>
    main()
  File "generate_mozbuild.py", line 463, in main
    write_mozbuild(includes, separated_sources)
  File "generate_mozbuild.py", line 427, in write_mozbuild
    write_sources(f, sources['mac'], 4)
KeyError: 'mac'

Still no luck.

So now I have to read:
https://sites.google.com/site/skiadocs/user-documentation/downloading

to figure out that I need to use gclient. Ok.

$ python generate_mozbuild.py 
GYP_GENERATORS is "dump_mozbuild"
Updating projects from gyp files...
Traceback (most recent call last):
  File "./gyp_skia", line 161, in <module>
    res = gyp.main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 527, in main
    return gyp_main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 503, in gyp_main
    options.circular_check)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 91, in Load
    generator = __import__(generator_name, globals(), locals(), generator_name)
ImportError: No module named dump_mozbuild
Failed to generate sources for linux
GYP_GENERATORS is "dump_mozbuild"
Updating projects from gyp files...
Traceback (most recent call last):
  File "./gyp_skia", line 161, in <module>
    res = gyp.main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 527, in main
    return gyp_main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 503, in gyp_main
    options.circular_check)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 91, in Load
    generator = __import__(generator_name, globals(), locals(), generator_name)
ImportError: No module named dump_mozbuild
Failed to generate sources for mac
GYP_GENERATORS is "dump_mozbuild"
Updating projects from gyp files...
Traceback (most recent call last):
  File "./gyp_skia", line 161, in <module>
    res = gyp.main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 527, in main
    return gyp_main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 503, in gyp_main
    options.circular_check)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 91, in Load
    generator = __import__(generator_name, globals(), locals(), generator_name)
ImportError: No module named dump_mozbuild
Failed to generate sources for android
GYP_GENERATORS is "dump_mozbuild"
Updating projects from gyp files...
Traceback (most recent call last):
  File "./gyp_skia", line 161, in <module>
    res = gyp.main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 527, in main
    return gyp_main(args)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 503, in gyp_main
    options.circular_check)
  File "/home/glandium/gecko/gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/__init__.py", line 91, in Load
    generator = __import__(generator_name, globals(), locals(), generator_name)
ImportError: No module named dump_mozbuild
Failed to generate sources for win
Traceback (most recent call last):
  File "generate_mozbuild.py", line 467, in <module>
    main()
  File "generate_mozbuild.py", line 463, in main
    write_mozbuild(includes, separated_sources)
  File "generate_mozbuild.py", line 427, in write_mozbuild
    write_sources(f, sources['mac'], 4)
KeyError: 'mac'

Turns out dump_mozbuild.py needs to be copied in gfx/skia/trunk/third_party/externals/gyp/pylib/gyp/generator/ or something.

Can someone please fix this documentation so that people who want to fix generate_mozbuild.py actually can?

Comment 1

[(not currently active) Ted Mielczarek]

I had a similar issue, I'm working on my iOS port and needed to change the skia moz.build but couldn't get the generator to work.

I think this is why I filed bug 1130343, I'd like us to just have a mach command to update third-party projects from upstream.

Comment 2

[George Wright (:gw280) (needinfo me!)]

Yeah, the first paragraph in README_MOZILLA was inadvertently left in there. It should be removed. The rest of the instructions are correct though (clone skia from upstream, copy it to gfx/skia/trunk, run gyp_mozbuild).

If dump_mozbuild.py needed to be copied that's a new requirement since the last update. Unfortunately there's a lot of code churn upstream and gyp_mozbuild just calls the Skia buildsystem itself to get a list of files that should be built for each platform, and then uses that list to generate moz.build. If they do anything that changes the upstream buildsystem, that potentially breaks us, which is what I think has happened here.

Unfortunately we have nobody working full time on maintaining Skia anymore (as I'm now on the e10s team) so some of this stuff has bitrotted :(

Comment 3

[George Wright (:gw280) (needinfo me!)]

It's a start: https://hg.mozilla.org/integration/mozilla-inbound/rev/2877cc2ddca8

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2877cc2ddca8

Comment 5

[Mason Chang [Inactive] [:mchang]]

I suspect this is done? If so, please close.

Comment 6

[Lee Salzman [:lsalzman]]

(In reply to Mason Chang [:mchang] from comment #5)
> I suspect this is done? If so, please close.

Our documentation is still a mess for this, so unfortunately this is still not done.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

The leave-open keyword is there and there is no activity for 6 months.
:davidb, maybe it's time to close this bug?

Comment 8

[David Bolter [:davidb] (NeedInfo me for attention)]

Lee is this worth leaving open?

Comment 9

[Lee Salzman [:lsalzman]]

Unfortunately the process of updating Skia is somewhat nightmarish right now and can't be automated, since a clean Skia just doesn't work and has to be modified in different ways every time we upgrade. The README_MOZILLA contains the basic rules for what a clean update might look like, which I think is as best we can do for now. I will just retire this bug as inactive for now, and maybe some day in the future when the process of updating Skia becomes less hellish we can revisit what it would mean to document that process.