Closed Bug 1432650 Opened 6 years ago Closed 6 years ago

api-maps.yandex.ru is always blocked by TP because of the yandex.ru entry

Categories

(Firefox :: Protections UI, enhancement, P2)

Product:
Firefox
Component:
Protections UI
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
platform-rel --- ?

People

(Reporter: francois, Assigned: groovecoder)

References

Details

(Whiteboard: tp-base [platform-rel-yandex]) 

api-maps.yandex.ru is in Disconnect's "Content" category:

https://github.com/mozilla-services/shavar-prod-lists/blob/e969a342103ef081438028d76a6d01b0a3ccd039/disconnect-blacklist.json#L7509

and so it should only be blocked when a user enables the "strict" list.

However, the bare domain, yandex.ru, is listed under the "Advertising" category:

https://github.com/mozilla-services/shavar-prod-lists/blob/e969a342103ef081438028d76a6d01b0a3ccd039/disconnect-blacklist.json#L5764

and has the effect of also matching URLs under the api-maps.yandex.ru domain, making the entry in Content ineffective.

We need to report this to Disconnect to see if they could use the fully-qualified hostnames instead of the top-level yandex.ru domain in the Advertising section.

Before doing that however, we should write a script to check whether or not there are other cases like that.
Whiteboard: tp-base
platform-rel: --- → ?
Whiteboard: tp-base → tp-base [platform-rel-yandex]
I've started the code to check whether or not there are other cases where an upper domain is already in a block-list.

https://github.com/mozilla-services/shavar-prod-lists/pull/33

Since it's part of verify.py it will automatically run on all PRs and fail the PR if a duplicate upper domain exists. I just need to debug its output.
Assignee: nobody → lcrouch
Status: NEW → ASSIGNED
Priority: -- → P2
Is there any progress on the issue? 
I see no activity for almost a month and we still have thousands of users facing the issue with maps.
Luke, am I right in thinking that your results on https://github.com/mozilla-services/shavar-prod-lists/pull/33 mean that yandex.ru is the only affected entry in the list?
Flags: needinfo?(lcrouch)
Yes, that's the only affected entry.
Flags: needinfo?(lcrouch)
Hi everyone!
Again no progress for 2 months and Yandex.Maps still unavailable for Mozilla users with tracking protection enabled. 
Am I right that we can recommend our users who need "incognito" mode to switch to other browsers?
We are still waiting on https://github.com/disconnectme/disconnect-tracking-protection/issues/39 from Disconnect.

FWIW, I haven't seen the end-user bug for this. I created a fresh Firefox profile, running Firefox Nightly 62.0a1 (2018-05-22), and I am able to use https://yandex.com/maps/ without any problems?
(In reply to Luke Crouch [:groovecoder] from comment #8)
> FWIW, I haven't seen the end-user bug for this. I created a fresh Firefox
> profile, running Firefox Nightly 62.0a1 (2018-05-22), and I am able to use
> https://yandex.com/maps/ without any problems?

It would have to be on a third-party domain. If you're doing it on Yandex.com, that's a related domain to yandex.ru (the domain where the blocked resource is hosted) and so we automatically allow it to go through.
Luke, you could try for example
https://mcdonalds.ru/restaurants/map

If you visit this URL with tracking protection enabled, map won't load. However, it present w/o tracking protection.

I even encountered it on our goverment site (gosuslugi.ru) and I had to use other browser to change my driving license —  I was blocked on "select police office to visit" step :-) 

So, I think this bug is actually showstopper for using tracking protection in Russia. Embedded yandex map is virtually everywhere.
Another broken site:
beru.ru (one of big shops).
Is there any way I could help or contribute to have this fixed? I could do some coding :-)
It's really most frustrating bug in Fx for me.
Not really. This needs an upstream fix by Disconnect: https://github.com/disconnectme/disconnect-tracking-protection/issues/39
Luke, I see that Mozilla actually have some patches on your level for Disconnect lists https://github.com/mozilla-services/shavar-prod-lists/blob/master/google_mapping.json
May be it could be done on that level for Yandex too?
The issue here is that Yandex *Content* elements (like api-maps.yandex.ru) are served from domains *under* Analytics domains (yandex.ru). (See https://github.com/disconnectme/disconnect-tracking-protection/issues/39#issuecomment-370338392)

If we made a yandex_mapping.json with "api-maps.yandex.ru" under Content, we would put "yandex.ru" under Analytics, which would result in the same problem.

So, we need Disconnect to enumerate and make the categorical decisions about all the yandex.ru domains, and/or yandex.ru paths. (See https://github.com/disconnectme/disconnect-tracking-protection/issues/39#issuecomment-371618771)
I don't understand. Could we use yandex_mapping.json to map:

    yandex.ru/cycounter
    yandex.ru/clck/click
    yandex.ru/clck/counter
    yandex.ru/set/s/rsya-tag-users/data
    yandex.ru/portal/set/any
to Analytics, 
all other yandex.ru to Content?

I understand that it better be done on Disconnect level, but according to Github thread I don't see them do it anytime soon.
Yes that would be possible. And yes - it needs to be done by Disconnect: they have full editorial control of the block-list.
Luke, sorry to repeat my initial question — if Mozilla patches this list on their level for Google, why it couldn't be done for Yandex?
We could map Yandex domains + paths to categories like we do with Google domains, but Disconnect must decide the domains & paths to include on the list in the first place. The above domains & paths may be comprehensive or they may not, so that's what we need Disconnect to decide.
You don't have bandwidth and/or expertise to decide if paths are correct in first place, do you? And obv. you can't trust random guy from Internet for it.... Have I understood problem correctly? 
Could we get someone from Yandex to give you the list or you couldn't trust them cause of intrest conflict? 

I'm pushing because disconnects is sitting on the bug for half-year and they seems just hope to get Yandex to move their analytics servers to another domains — it's best decision technically, but unlikely to happen over night, and probably will never happen.
That's correct. It's a tough spot. :(
Hi Luke! I've pinged Disconnect team once more and asked about the list of paths they assume to be analytics so you could make a patch. Anyway I'm concerned that tracking black list somehow transforms into whitelist in case of Yandex.  

Also I'd like to point that current settings block only Maps API and several similar widgets. Case is the following:
- all requests (even considered analytics) are allowed on Yandex' own domains;
- Maps API is blocked regardless it doesn't make any requests from analytics list.

As a result in yandex.ru* case none of "tracking" requests are blocked, only API resources loading. You may check it yourself and I hope it would be a reason to exclude yandex.ru* rule.
I've also noticed a Yandex map not showing up on my website in a private tab when tracking protection is off. This applied not just to a script at https://api-maps.yandex.ru/2.1/?lang=en&coordorder=latlong&load=package.full&mode=release&ns=&onload=_$_api_success&onerror=_$_api_error, but also to a static map image from static-maps.yandex.ru (e.g. https://static-maps.yandex.ru/1.x/?ll=45.01,53.217362&z=5&pt=45.1,53.19,org&size=650,450&l=map&lang=en_GB). The website shows it to users while the JS version gets initialised.
A breakthrough: we received and merged updated blocklist content from Disconnect that included changes to the yandex domains. https://github.com/mozilla-services/shavar-prod-lists/pull/42

It's been merged, and the updated lists are being served to Firefox clients now. (Clients check for list updates approx. every 30m)

I checked on a Fresh Firefox 63.0.1 and both yandex.com/maps and https://mcdonalds.ru/restaurants/map are now showing maps!

Marking this RESOLVED:FIXED!
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
I also confirm that Yandex maps are finally working on my website when Firefox tracking protection is enabled. This applies to JS maps as well as static maps (which I load first to make the website feel faster): https://www.emi-penza.com/contact

Many thanks for resolving this issue folks, much appreciated! 🎉
Feeling like it's my birthday =)
Great news! Thank you guys!
You need to log in before you can comment on or make changes to this bug.