I'm not really sure how practical it is to sell an open-source desktop application; anyone can compile and run it for free and the 'sell support' model usually isn't viable for something as simple as a text editor.
You underestimate the difficulty of compiling from source for the layman. OSX and Windows do not come with compilers out of the box. The user would have to install them and the proper environment for them first. (cygwin, etc...) Then they'd have to get and install the dependencies properly. Finally they'd have to figure out the proper incantations to make it compile, which if you've never done it before can be extremely daunting. How on earth was I supposed to know what an LPATH is or pkgconfig? Not to mention the extremely verbose and scary looking errors from gcc.
That's assuming that it's a program written in C/C++. If it's another language then you might have another huge hurdle to go through. (Properly installing Go in my PATH took me a while). That's also assuming they don't rely on IDE-specific make files. A lot of the time OSX applications rely on xcode project files. The other day I was trying to compile an Android app before realizing that it was written in a language by Adobe which required the purchase of a full suite.
Open source can be really limiting if the developer doesn't make an effort to document the build process well and doesn't use open tools to create the product.
The problem is, only one person has to figure out how to build it, then if it's real free and open source software, they're legally allowed to distribute their own unofficial builds over the internet. The original XChat developer decided that it was too much effort to develop XChat for Windows, so he charged for it, which resulted in silverex's free X-Chat 2 builds and the HexChat fork.
I don't think users respond well to this either. You're essentially splitting your userbase into a group of people who know enough to build it themselves and receive it for free, and a group that have to pay, and I don't think people in the latter group appreciate being in that group.
Hmm I didn't think about that. I guess that's a genuine option for popular applications. Personally, for some unpopular applications I've tried to compile, I just gave up if the standard `./configure && make` didn't work.
As for your second point, I think it comes down to two things: if the user's time is more valuable than their money, and two, if they can get support for the paid product (at least that's how I've seen some products doing it), so it's not as if the paid is without it's benefits. Your Hexchat example shows how important it is to not alienate your users though.
How about releasing the source without any of the art assets or interface nibs? Full benefit of open source while still providing high value for money. (Sure, the geeks will go and create their own assets, but it won't be the same.)
Interesting idea, but I suspect that the first quality fork that provides a 'good-enough' version of those missing assets out of the box would start to steal the limelight from the main project, especially because the fork could pull in all the bug fixes and new features from upstream.
