Open Bug 444284 Opened 11 years ago Updated 2 days ago

Sync search plugins

Categories

(Firefox :: Sync, enhancement, P3)

enhancement

Tracking

()

Future

People

(Reporter: stazybohorn, Unassigned)

References

(Blocks 2 open bugs)

Details

(Whiteboard: [sync-engine-addition][fxsync])

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1a1pre) Gecko/2008070603 Minefield/3.1a1pre (like Firefox/3.0)
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1a1pre) Gecko/2008070603 Minefield/3.1a1pre (like Firefox/3.0) ID:2008070603

It would be convenient if weave sync-ed search plugins and their history.


Reproducible: Always
Target Milestone: -- → Future
This bug is particularly valuable because it is so hard to find search engines from Firefox's UI: the "get more" link to addons.mozilla.org but the majority and good ones are in mycroft.mozdev.org.
Component: Weave → Needs Triage
Product: Mozilla Labs → Weave
Target Milestone: Future → ---
QA Contact: weave → needstriage
Target Milestone: --- → Future
There are some add-ons for easy cook new search engines with some mouse clicks.
Blocks: 530399
Assignee: nobody → edilee
Target Milestone: Future → 1.2
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Component: Needs Triage → Server
QA Contact: needstriage → server
Assignee: edilee → nobody
Component: Server → Sync
QA Contact: server → sync
Target Milestone: 1.2 → Future
This is a mass change. Every comment has "assigned-to-new" in it.

I didn't look through the bugs, so I'm sorry if I change a bug which shouldn't be changed. But I guess these bugs are just bugs that were once assigned and people forgot to change the Status back when unassigning.
Status: ASSIGNED → NEW
Please incorporate search plugin & search keyword sync into Firefox Sync. This is the most time-consuming aspect of setting up Firefox on new computers (along with add-ons, but that's another issue.)
Duplicate of this bug: 575065
Duplicate of this bug: 633964
Duplicate of this bug: 635596
Summary: option to sync search plugins and their history → Sync search plugins and their history
Whiteboard: [sync-engine-addition]
I agree with Martin.

I want give an idea to developers about this: If search plugins were saved at 'Places' like bookmarks (that have the same features about assign keyworks and allow '%s' to perform searchs), sync will already work without new efforts on sync system (that involve changes in firefox and changes in the sync server software, I suppose); Can be do it this work, move search plugins to Places?
Search plugins history can be saved at History, inside Places, too.
A remaining problem can be the saved icon and additional information about the search plugin (more on opensearch.org), maybe saving the original url from what plugin was downloaded can be used to download again the plugin xml file when after sync the bookmark/search-plugin.
I'm proposing this without full knowledge of the internal operation, I'm sorry if something have no sense.

For other users like me: just now I use more the keyworks instead of the search-box to perform searches in differents engines, for this I save plugins search-url in my bookmarks (under Quick Searchs default folder, but can be any folder) and I have already sync'ed my 'search bookmarks' (like a replace of search engines). 
The only fault about this tip, is that you will not have 'search suggestions' like in the search box.

I hope this helps.
Apparently there has been a lot of recent activity in the search engine service. Unfocused suggests waiting until bug 699856 lands before we attempt to support syncing search engines.

For anyone looking at implementing this, you'll be interested in toolkit/components/search/nsSearchService.js. See also the XML files in the 'searchplugins' directory of the app/profile. Some considerations for implementation include differences in parameters between release channels. See bug 722352. This may preclude syncing the XML files verbatim.

Also, from what I could tell, search engine history is stored in the forms database, which is already syncable. So, I think we just need to sync the search engine providers and history should just work.
Depends on: jsonSearchSvc
(In reply to Gregory Szorc [:gps] from comment #13)
> Apparently there has been a lot of recent activity in the search engine
> service. Unfocused suggests waiting until bug 699856 lands before we attempt
> to support syncing search engines.

bug 699856 is step one of multiple steps to remove the use of SQLite in the search service, and make the API async (since it depends on IO!). There's not much use in waiting for it alone, but I don't think you need to wait for it at all - it seems mostly tangential to doing search engine sync.

> For anyone looking at implementing this, you'll be interested in
> toolkit/components/search/nsSearchService.js. See also the XML files in the
> 'searchplugins' directory of the app/profile. Some considerations for
> implementation include differences in parameters between release channels.
> See bug 722352. This may preclude syncing the XML files verbatim.

I think you'll basically always want to avoid syncing app-shipped search plugins, so you shouldn't need to worry about build-specific parameters. Rather than syncing XML files, it would be simpler to sync the JSON cache that's the result of parsing them.

Search service changes will be required for this, almost certainly. Exposing the engine description as JSON and adding the ability to distinguish app-shipped plugins via the nsIBrowserSearchService API seem like good first steps to me.
Plus, it was in the roadmap for 2011.

https://wiki.mozilla.org/Services/Sync/FxSync/Roadmap#Desktop
:ally will be looking at this once she gets back from vacation.
Assignee: nobody → ally
(In reply to Erwann MEST from comment #16)
> Plus, it was in the roadmap for 2011.
> 
> https://wiki.mozilla.org/Services/Sync/FxSync/Roadmap#Desktop

Standard disclaimer: roadmaps are subject to change :D

No plan survives contact with the enemy!
Ahah, you guys are crazy.

Yeah it'll be great. Thanks :gps, :ally, :rnewman
(In reply to Gregory Szorc [:gps] from comment #17)
> :ally will be looking at this once she gets back from vacation.

Thats sounds great! I dont like to test Chrome again.
This has been put on hold due to higher priorities. Unassigning myself.

See wiki for last known state of spec (to help whoever picks this up next)
Assignee: ally → nobody
please see :mconnor for future schedule or priority.
(In reply to Stop Kicking Cans down roadmaps from comment #26)

> How long will mozilla kick the can down the roadmap?

Firstly, patches are welcome. This is an open-source project[1]; if you consider this feature important, you are encouraged to work on it and submit patches for review and inclusion.

Secondly, we consider a broad array of conflicting engineering, product, and market requirements when prioritizing, resourcing, and ultimately delivering features and products; we have a responsibility to achieve Mozilla's mission, which — like any worthwhile human endeavor — involves making difficult decisions with conflicting inputs and limited resources.

In a world with infinite resources, this addition would already have been built. We don't live in that world. As Ally's thorough notes[2] make clear, this is not a trivial feature; it'll get done when we have time and resources to get it done right.

Thirdly, I encourage you to not hide behind a childish nom de plume, and also to respect Bugzilla Etiquette[3], which asks you to refrain from content-free "me too" and similar remarks. This project succeeds thanks to the contributions of tens of thousands of people who give their time to make the web a better place; please respect both the contribution and their time.


If you have any questions on this, please feel free to come and find me on irc.mozilla.org, #sync.


[1] <https://developer.mozilla.org/En/Introduction>
[2] <https://wiki.mozilla.org/User:Anaaktgeboren/SearchEngineSync>
[3] <https://bugzilla.mozilla.org/page.cgi?id=etiquette.html>
Duplicate of this bug: 823340
(In reply to axel.foster-eoagj6zt from comment #29)

> The OP used a nick.  Where was your "righteous" indignation then?

I was commenting on the childishness of the one-off complaint account, not the use of a nickname.


> > it'll get done when we have time and resources to get it done right.
> 
> obviously not.  you have such already.  see also: irrational, and causality.

I don't think you have enough information to make that assertion.


> Given the history of mozilla's willful blindness this is not an
> inappropriate question.

I disagree with your phrasing, but if you want a blunt answer: this bug will be "kicked down the roadmap" until:

• a contributor has time to build an implementation that meets the quality and maintenance standards of module peers, or
• a MoCo team deems it important enough to Mozilla's mission to assign a paid resource to be that contributor, or
• the bug is no longer relevant.

If you want to be the contributor, great; hop on IRC and we'll discuss the scope of the work. Otherwise, see Bugzilla Etiquette, item 1.2.

From my perspective, the question was not a serious request for information (though I answered it anyway), but either a childish outburst or an attempt to bully others into action. Neither is acceptable on Bugzilla.


> This is bugzilla where firefox USERS may submit bugs for developers to wend
> coding skills into patching.  Many users contribute corroboration, or argue
> for urgency or prioritization.   Suggesting "patches are welcome" as the
> response is facetious at best, Mr Pot.

I encourage you to read Bugzilla Etiquette, particularly points 1.1 and 1.2, and to more carefully read the second part of my response.

Use Bugzilla voting to express "me too", or "I want this feature". And refrain from implying that contributors have an obligation to build what you want them to build. The bug is filed, it's being tracked, and it will be fixed when it gets fixed.

If you wish to continue this conversation, please do so by emailing me directly, or taking it to IRC. This bug is not the place to discuss work prioritization, so please refrain from making any further comments that are not directly relevant to the implementation of this feature.
AFAIK the current Sync Implementation will not be improved/enhanced anymore.

See https://wiki.mozilla.org/Identity/AttachedServices for current (?) Plan(s) and Richards Comments in other Sync Enhancement Bug Reports directly and indirectly confirming above Guess.

Unsure if non-Implementation Talk (like this Comment and above) should go to mozilla.dev.apps.firefox (as Fx currently being the only Sync Consumer) or rather mozilla.dev.identity using the usual Ways?!?
(In reply to XtC4UaLL [:xtc4uall] from comment #32)
> AFAIK the current Sync Implementation will not be improved/enhanced anymore.

Correct. Addressing future user stories is under the PICL umbrella.

> Unsure if non-Implementation Talk (like this Comment and above) should go to
> mozilla.dev.apps.firefox (as Fx currently being the only Sync Consumer) or
> rather mozilla.dev.identity using the usual Ways?!?

dev.identity is the right forum.
> AFAIK the current Sync Implementation will not be improved/enhanced anymore.

That's one way to address user issues.


> https://wiki.mozilla.org/Identity/AttachedServices 

No use case for the user who wants to use sync but wants control over WHERE data lives, and HOW it is LOCALLY encrypted prior to upload?

That seems irresponsible.

> Addressing future user stories is under the PICL umbrella.
> dev.identity is the right forum.

Where are the respective interweb resources?  As google- and other interweb index engineers have pandered to the lowest common denominator far too long their tools are far more frustrating to use.
(In reply to zero-ads-are-acceptable+buggy from comment #35)

> > AFAIK the current Sync Implementation will not be improved/enhanced anymore.
> 
> That's one way to address user issues.

Indeed it is. It has the opportunity to discard some legacy baggage, which is important, and it more closely aligns identity attached services with the identity team, which makes a lot of sense.


> No use case for the user who wants to use sync but wants control over WHERE
> data lives, and HOW it is LOCALLY encrypted prior to upload?
> 
> That seems irresponsible.

On the contrary, the PICL team are actively exploring flexible security models to allow a sliding scale of recoverability, backup, and encryption strength. In that regard it will have *more* control than does Firefox Sync.


> Where are the respective interweb resources? 

https://groups.google.com/forum/#!forum/mozilla.dev.identity


> As google- and other interweb
> index engineers have pandered to the lowest common denominator far too long
> their tools are far more frustrating to use.

This is not the right place for whatever complaint you're making.

Please mind your manners, and carefully read Bugzilla Etiquette before you decide whether to comment again on this particular bug. Thanks!

https://bugzilla.mozilla.org/page.cgi?id=etiquette.html
+1 from me.
As a temporary fix, I did the following:
[as a Windows 7 user; should be similar for others]

Go to C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<random-name>\

You should see a folder called "searchplugins". Copy that folder into the similar profile folder in, say, another computer.
@chancephelps11 (comment 41). I've tried that, but it doesn't preserve assigned keywords, so you end up having to type all those in manually. Also, on linux, you have to be careful with permissions when copying that folder over, making sure you have permissions to it.

So we still need proper cloud syncing.
(In reply to nortexoid from comment #42)
> @chancephelps11 (comment 41). I've tried that, but it doesn't preserve
> assigned keywords, so you end up having to type all those in manually. Also,
> on linux, you have to be careful with permissions when copying that folder
> over, making sure you have permissions to it.

Hi, thank you for pointing that out, I hadn't noticed it. It seems the keywords can also be copied by restoring two more files from the main profile folder: "search.json" and "search-metadata.json". 

Of those, the latter stores the keywords, but I guess the former is needed as well. 

> 
> So we still need proper cloud syncing.

I agree, this just a quick-and-dirty fix, I hope this is resolved soon. Why not have Sync copy over the entire profile folder? Is there anything that must not be synced across devices?
> Is there anything that must not be synced across devices?

There are bandwith usage constraints (Mozilla's sync servers seem to be capped to 25MB storage per account - https://github.com/greasemonkey/greasemonkey/issues/1573) and files that can not be touched when they are in use.
Copying files is not an option for several reasons, not least of which are that different clients don't use the same storage backends, blindly copying files doesn't allow for conflict resolution, and disk storage formats are not designed for efficient incremental replication.
(In reply to sephie from comment #46)
> +1
> Say what you will about Opera (12), but it had search engine sync. ;)

Patches welcome!
Duplicate of this bug: 979288
Flags: firefox-backlog? → firefox-backlog+
This would be a very useful addition. Please also preserve the assigned search keywords and the user defined order of the search plugins when syncing.

Just on a sidenote: This feature request is now about 6 years old. The new user interface of Firefox 29 that almost everyone despises got prioritized above this.
Duplicate of this bug: 1023739
I just switched from Chrome to Firefox and so far this is the only drawback that I've had for switching (as others have said, Chrome has this feature). I use multiple computers and use around 20 search engines so not having this feature makes the transition to Firefox somewhat annoying, especially since I plan to keep adding to my list of search engines. I won't say that I'm going to switch back to Chrome due to Firefox lacking this feature, but it does make me want to keep using Chrome for whenever I want to make a search.
I found a solution for what Jorge writes above that works for me. Instead of entering search engines as search engines, a search can be bookmarked: right-click in the search field of a site and choose "enter a keyword for this search." This keyword works the same as the shortcuts for search engines in Chrome. The only difference I see so far is that I don't have an overview of all the keywords that I created (Chrome provides this under "manage search engines"), but the keyword you have assigned can be looked up from within the bookmarks manager. Clicking on a bookmark displays the properties of the bookmark in the bottom half of the window, including the keyword.
I might, for the ease of managing the shortcuts (or keywords as they are called here), put all "search engines" in a separate bookmark folder. 
Works for me.
Oh, and the good news is that the keywords, of course, now automatically will be synchronized as part of the bookmarks, without having to use add-ons.
It's not a solution.
I think that search engines synchronization should be included as a new Sync feature.
Now this feature is lost and it makes Sync functionality not complete.
I'm aware of that ff12345, whenever there is not an option to add a search engine through the search bar icon, I add it as a bookmark. The problem is that I already added most of the search engines through the search bar icon although I guess that for now the best is to convert them all to bookmarks. 

Also, instead of clicking the arrow to see the keyword for the bookmark, you can right click any of the column categories in the bookmark manager and check the option for "Keyword" so you can see all the keywords in a separate column.
As with all other [sync-engine-addition] bugs, this depends on a story for adding new sync engines.

That means Bug 608231, and also the declined engines work in Bug 978876.

New engines will only be possible in Sync 1.5, not in old Sync 1.1.

That implies some small UI/pref complexity, and there's a testing load, too.

I'm leaving this comment here lest someone think "huh, this is easy, let me write that sync engine". (It's actually not easy to write the engine, but this is even less easy than it looks!)
Depends on: 608231, 978876
Syncing search history is now Bug 1095138.
See Also: → 1095138
Summary: Sync search plugins and their history → Sync search plugins
The solution suggested by ff12345 seems to be the only working workaround for now. It's not perfect. The search engines added as bookmarks don't even have icons attached to them. But it does the job.
I think submitting patches is a good idea, if a person is capable of digging into the source, keeping the original around to diff, and 

I am just unsure of what exactly the config and software consists of the build environment. is that documented somewhere? are you folks using mingw-w64?

is there Sync API and documentation to look at? ahh, here they are: http://docs.services.mozilla.com/
it looks like http://docs.services.mozilla.com/storage/apis-1.5.html#api-instructions
has the 1.5 storage API and it uses a JSON object to put data in. I should suppose there is some firefox API for doing PUT, GET, POST.

finding the existing sync code should not be too difficult:
dir/s/b *sync*
(In reply to Jim Michaels from comment #64)

> I am just unsure of what exactly the config and software consists of the
> build environment. is that documented somewhere? are you folks using
> mingw-w64?

You probably want to start here:

https://developer.mozilla.org/docs/Introduction
Jim Michaels, maybe at first will be easier to implement this as Firefox Extension instead of patch? I don't a programmist, but I want help you with testing your addion, because I need this feature.
See Also: → 1054960
Priority: -- → P2
Duplicate of this bug: 1198305
Duplicate of this bug: 1221470
Blocks: syncmore
Flags: firefox-backlog+
Priority: P2 → --
Whiteboard: [sync-engine-addition] → [sync-engine-addition][fxsync]
Assignee: nobody → tchiovoloni
Status: NEW → ASSIGNED
Adding a comment so that future readers (including me) know that the planned approach and comments are in bug 1054960 comment 9 and 10.
Priority: -- → P1
That plan as written doesn't really take into account how our search and distribution partnerships work, especially across locales and territories.  That's a big problem and we should not proceed with work until we have a plan signed off by partner engineering.

For absolute clarity: we _must_ not sync the cached version of any plugins shipped with the app (in default or as a part of distributions).  This is not something we should risk.

Key requirements to keep in mind:

* All parameters for our partners must be preserved and used as configured in the build.
* Default and default-visible engines are/will be defined on a per-locale + per-territory basis.  (Right now that's mostly driven by server-side, but :mkaply is landing a set of changes to make that work out of the box as well.)
* Ordering will be less arbitrary in the new world.

I would strongly recommend that any solution follow these guidelines:

1) Mobile is tricky on a UX level and has different plugin sets. We should consider following the add-on sync model (sync mobile to mobile or desktop to desktop, but don't cross device type streams).

2) User-installed engines (i.e. those that don't show up in GetDefaultEngines) should have all parameters required for addEngineWithDetails added to a bundle.

3) Default engine setting should be synced as an engine name, mirroring the pseudo-pref.  If the default is used, we should sync nothing.  (If a user is syncing between different builds with different defaults, that's okay.)  (This one is super important for numerous reasons.)

4) Order should only be synced if the order is different from the default order.  If the only change is appending the user-installed engines synced in #2, we should sync that order and append to the default list on other devices.)

5) Conflict resolution should fail on the side of leaving engines present.

Thom, happy to discuss further if you have followup questions.
Thom, I CC'd you to an email to Javaun. Let's sync up with him to make sure this is done to spec so that legal and BD are pleased with the outcome.
Flags: needinfo?(tchiovoloni)
Yep, sounds great -- replied. I totally understand that doing the right thing here is super important, and definitley don't want to end up upsetting people with our implementation.
Flags: needinfo?(tchiovoloni)
Clearing Thom's assignment, pending reply from Javaun.
Assignee: tchiovoloni → nobody
Status: ASSIGNED → NEW
Priority: P1 → P3
Any updates to this that any Mozillians would like to share with the world? Was this blocked by partnership requirements?
(In reply to Mike Connor [:mconnor] from comment #74)
> 1) Mobile is tricky on a UX level and has different plugin sets. We should
> consider following the add-on sync model (sync mobile to mobile or desktop
> to desktop, but don't cross device type streams).

As there might be some movement on bug 648398 again, I'd like to point out for the record that
- bookmark keyword searches currently *are* synced between mobile and desktop
- there's a chance that keyword searches will often be things for which no ready-made search plugin is available that in a pinch could simply be redownloaded, in which case syncing them would be all the more valuable


I think that if
- Fennec would allow hiding search engines as well (it currently doesn't, maybe because you can't set search keywords either), and
- the hidden state of a search engine *wouldn't* be synced between Desktop and Mobile

I'd be quite happy, although maybe I'm overlooking some other problems.
(In reply to Daniel from comment #78)
> Any updates to this that any Mozillians would like to share with the world?
> Was this blocked by partnership requirements?

It was blocked due to that code undergoing refactors at the time, and nobody has found time to do it since. This is probably still the case, although it should be more doable now than it was in the past (but obviously it would still require someone finding the time to do it).

We never got far enough to get the sign-off from partner engineering mentioned in comment 74, and I don't fully remember the discussion around partner issues, but vague my understanding was that if this is implemented correctly (where it only synced changes made intentionally) it was probably going to be fine.

(In reply to Jan Henning [:JanH] from comment #79)
> - there's a chance that keyword searches will often be things for which no
> ready-made search plugin is available that in a pinch could simply be
> redownloaded, in which case syncing them would be all the more valuable

This is what I use for this, IMO it works well, but is quite undiscoverable; it's not clear that this is what "keyword" means in a bookmark, short of reading docs.
(In reply to Jan Henning [:JanH] from comment #79)
> I'd be quite happy, although maybe I'm overlooking some other problems.

Thinking some more, search keywords should of course be synced as well, but syncing the order of engines between mobile and desktop is another topic that might need some further thoughts. E.g. currently the default search engine is automatically the first engine in the settings on Fennec, while on desktop there's a separate setting for that. And speaking personally, I've ordered them by most used first on Fennec because of less screen space and touch screen usage, but simply alphabetically on Desktop.
Duplicate of this bug: 1407859
Duplicate of this bug: 1431317
Duplicate of this bug: 1451245
So many discussions here!
Any plans or plugins for this function yet or in the future?
Duplicate of this bug: 1174413
(In reply to maledong_forwork from comment #85)
> Any plans or plugins for this function yet or in the future?

Not at the moment, sorry. :-(
Component: Firefox Sync: Backend → Sync
Product: Cloud Services → Firefox
As workaround we can sync config file (via some file syncing service) with list of current search plugins, it is located at search.json.mozlz4 file in profile folder.
Duplicate of this bug: 1500681
Duplicate of this bug: sync-preferences
Duplicate of this bug: 1527185

This bug still on Firefox 65.0 (64-bit) for Ubuntu canonical-1.0

Duplicate of this bug: 1544125

Hi ! I'm not sure of the whole evolution of the keyword syncing feature throughout all these years.. I am personally trying to migrate as much as possible to Ubuntu, and setting up Firefox on it came with the challenge of syncing a-l-l my search keywords (it is such a valuable tool on my browsing!)
Does anyone know where is this issue at? Thank you for your reply

You need to log in before you can comment on or make changes to this bug.