Well, it's certainly not the lowest level token - to the degree we can talk about tokens in our brains in the first place.

We can talk and think about such things as the shape of the letters, which it would take a Herculean effort for GPT-like models to learn, since it gets letters in a form where that information has been stripped away.

As pointed out by gerad above, we humans don't treat multi-digit numbers as one atomic token. We know 11 = 10 + 1. But if an AI model has no access to individual digits no matter how large the number is, then the task of doing simple maths is exponentially harder to learn.