Trust isn't binary. Backdooring or exploitably bungling the deterministic parts of the CPU without getting caught is much harder than doing the same for the RNG part. Black box RNG's are notoriously hard to evaluate.