I think this has potential to nudge people in different directions, especially people who are looking for external input desperately. An AI which has knowledge about lot of topics and nuances can create a weight vector over appropriate pros and cons to push unsuspecting people in different directions.
Open source will keep good AI out there.. but I’m not looking forward to political arguments about which ai is actually lying propaganda and which is telling the truth…
Well, when you consider what it actually is (statistics and weights), it makes total sense that it can inform a decision. The decision is yours though, a machine cannot be held responsible.
LLMs are a stochastic (as opposed to a deterministic) system, which can make them better at tasks that by nature are difficult to express formally, but still require a degree of certainty ("how can I make this CV better").
I believe it's slightly more nuanced than a dice roll.