>I'm highly skeptical of an approach that involves training users to rely on a black-box ML system.

google did it with youtube, if they do it to chrome i don't know if they can handle the developer frustration that will ensue(i'll put a nice red fullscreen browser incorrect banner on my website if users visit from chrome)