Android applications can use their own private data directories just fine, you don't need any permission for that. Termux, for instance, can run entire Linux distros without ever being granted storage permissions.
They can also request the user for a specific folder to get read/write permissions for (rather than the entire shared storage space).
Just requesting full access to the entire shared storage is a lot easier to implement, but it's also not great for privacy and security. Google is trying to crack down on these permissions in the Play Store for this reason.
Several apps have been caught extracting location history data from pictures stored in the camera directories already, so it's not a theoretical threat. Unfortunately, app developers have been reluctant to implement the (much better) specific file picker introduced in Android 4.4, but now they're being forced to if the want Google Play access in the future.
The scoped storage API has some downsides, though, such as reduced performance. Not really a problem if all you want to do is upload a picture or save a document, but when you're running executables from there you may run into a wall. This can be circumvented by copying the files to the app's private data directory, but people may want to use the selected folder to copy output back to Android, and a copy action obviously doubles the space required at least temporarily.
That being said, I believe applications shouldn't refuse to work entirely without full access to every file on the shared storage space. They can ask the user what they want to do if the storage permission is denied rather than refuse to work.
Extra unfortunate is that the new file picker Google introduced a few months ago for media selection is absolutely terrible in my opinion (no directory support, just a flat list of images), but some other phone brands offer file pickers of their own.
I think both you and I agree that Android's implementation is awful. I don't think it is fair to blame developers because Google is being bad OS steward.
I don't think invidual developers are to blame necessarily, but Android developers at large have been ignoring privacy-friendly storage APIs for a literal decade at this point.
While I have my problems with the current state of things in Android, I think the new mechanism is better as a whole than the old mechanism where every flashlight app could upload your nudes to the cloud without you knowing. Had the Android developer landscape jumped ship to these APIs earlier, I'm sure the kinks would've been worked out years ago.
The unfortunate lesson Google seems to have learned is that giving developers the option to use safer, privacy-friendly APIs that put the user in control doesn't work, and forcefully disabling old APIs and forcing developers to update their apps to comply does. Hardly a surprise, to be honest, given Android's ecosystem is still full of shovelware and third-party "ad libraries" filled with adware and stalkerware (often making use of an app's legitimate permissions for nefarious means). I'd like Google to crack down on these, at least on their store, but then I'm sure we'd see weeks of HN threads about how Google broke the latest and greatest app by altering the API.
They can also request the user for a specific folder to get read/write permissions for (rather than the entire shared storage space).
Just requesting full access to the entire shared storage is a lot easier to implement, but it's also not great for privacy and security. Google is trying to crack down on these permissions in the Play Store for this reason.
Several apps have been caught extracting location history data from pictures stored in the camera directories already, so it's not a theoretical threat. Unfortunately, app developers have been reluctant to implement the (much better) specific file picker introduced in Android 4.4, but now they're being forced to if the want Google Play access in the future.
The scoped storage API has some downsides, though, such as reduced performance. Not really a problem if all you want to do is upload a picture or save a document, but when you're running executables from there you may run into a wall. This can be circumvented by copying the files to the app's private data directory, but people may want to use the selected folder to copy output back to Android, and a copy action obviously doubles the space required at least temporarily.
That being said, I believe applications shouldn't refuse to work entirely without full access to every file on the shared storage space. They can ask the user what they want to do if the storage permission is denied rather than refuse to work.
Extra unfortunate is that the new file picker Google introduced a few months ago for media selection is absolutely terrible in my opinion (no directory support, just a flat list of images), but some other phone brands offer file pickers of their own.