1052217 - Index binary files

Status: RESOLVED FIXED

(Webtools Graveyard :: DXR, defect)

Description

[Erik Rose [:erik][:erikrose]]

At the moment, DXR indexes and browses only text files. For MXR parity, we should be able to at least browse images. It wouldn't be bad to at least acknowledge the existence of other sorts of binary files as well. Search for "Make this no longer conditional on is_text" in the code for the first line you'll have to change. This will require some UI design work, but if we take only images as a first iteration, we could defer much of that. We'll store the images as either the binary type in elasticsearch (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html#binary) or as attachments (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-attachment-type.html), which is experimental but gives us some free indexing of metadata.

Comment 1

[Peter Elmers [:new_one]]

I took some time to work on an initial implementation of image preview.
Demo: http://104.236.51.218:8000/code/source/
Branch: https://github.com/pelmers/dxr/tree/es_img_preview
Diff w/ es: https://github.com/pelmers/dxr/compare/mozilla:es...es_img_preview?expand=1

Currently this doesn't store the images in the ES index and we just retrieve them as files off the disc, but is this the right direction for the feature?

Comment 2

[Erik Rose [:erik][:erikrose]]

Thanks, Peter! I think we should store the images—at least under a certain, settable size—in ES. That way, we won't need any shared filesystem at all for the production systems. (At least, we won't until we introduce the ability to pull old revisions out of version control: bug 962108.)

Comment 3

[Erik Rose [:erik][:erikrose]]

If you open a PR (even if it's not ready to merge yet), I can more easily comment on individual lines.

Comment 5

[[github robot]]

Commits pushed to es at https://github.com/mozilla/dxr

https://github.com/mozilla/dxr/commit/936ce19e1937aade62385da9aafe5526e9065afa
Index binary files and show image previews (Bug #1052217)

Binary files are stored in the elastic search index under the 'IMAGE'
doctype and retrieved on demand. A bit of Javascript on browse pages
adds the image preview.

https://github.com/mozilla/dxr/commit/c2e7c8dce81199a01a3869755433479ea3d686d2
Index and thumbnail image files. Addresses bug 1052217. Close #370.

Still to come: indexing just directory entries for non-image binary files.

Comment 6

[Erik Rose [:erik][:erikrose]]

Oops. This should remain open until we index (just) file listings for all binary files. They should be greyed out or something; you shouldn't be able to click into them and see nothing useful.

Comment 7

[Peter Elmers [:new_one]]

I went ahead and did a bit of work on the followup of putting binary files into the file listings. PR at https://github.com/mozilla/dxr/pull/386

Comment 10

[(not currently active) Ted Mielczarek]

This bug is why binary files don't show up in dxr at all right now, right? I was looking at https://dxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/win32 and really confused since that directory only contains binary files and dxr shows nothing.

For mxr-parity it'd be nice if we could also at least click through to the hg log for binary files.

Comment 11

[Erik Rose [:erik][:erikrose]]

Ted, correct. We'll get that merged in in the next few weeks, and it will land with the rest of the "es" branch, which fubar is staging as we speak.

Being able to get at VCS-hosted stuff for binary files is a good idea. I'll file that separately: bug 1142076. I took the liberty of CCing you.

Comment 12

[[github robot]]

Commits pushed to es at https://github.com/mozilla/dxr

https://github.com/mozilla/dxr/commit/d84963acf04e7ea4b13617c7dfed5aa114791e7f
Bug 1052217: index binary files

Now index all binary files, not just images. Arbitrary binary files are
shown in faded italics on the browse view and are not clickable.

https://github.com/mozilla/dxr/commit/53846ceacf612c4973950d9f1e3b638bebdfa763
Merge browsing of binary files. Fixes bug 1052217. Close #386.

Comment 14

[[github robot]]

Commits pushed to master at https://github.com/mozilla/dxr

https://github.com/mozilla/dxr/commit/936ce19e1937aade62385da9aafe5526e9065afa
Index binary files and show image previews (Bug #1052217)

https://github.com/mozilla/dxr/commit/c2e7c8dce81199a01a3869755433479ea3d686d2
Index and thumbnail image files. Addresses bug 1052217. Close #370.

https://github.com/mozilla/dxr/commit/d84963acf04e7ea4b13617c7dfed5aa114791e7f
Bug 1052217: index binary files

https://github.com/mozilla/dxr/commit/53846ceacf612c4973950d9f1e3b638bebdfa763
Merge browsing of binary files. Fixes bug 1052217. Close #386.