I have built several twisted apps which are in use in production environments, including web and jabber/xmpp apps. I love twisted, and use it whenever possible...
...and I agree completely with the op.
Twisted is powerful, flexible, well architected, and reliable. It is also damn near impossible to get over the steep learning curve and to actually USE it.
Show me a competent Twisted developer, and I'll show you someone who had a gun jammed roughly in their mouth, and was FORCED to learn it at some point. We are all glad we know it _now_, but that learning process SUCKED.
Definitely agree on the learning curve. I'm using twisted for a distributed computing framework that is custom tailored for our requirements (looked into Hadoop, disco, etc. but decided against for valid technical and/or logistical reasons).
I go to a lot of pains so that clients of my framework never have to touch or even see twisted though it is under the hood. I think I spent about a week in the beginning really wrestling with Twisted and what was the appropriate way to transcribe my ideas into the "deferred" model. Once it clicked it was relatively smooth sailing (all in all, Twisted is a pretty sweet framework but not without its gotchas that you have to learn the hard way). A week spent learning a new framework isn't much time, but when clients want to get their app up and running ASAP it is an eternity and ultimately would prevent their adoption of my framework.
I have tried using twisted in pretty much the same way. Actually I tried it 3 times and eventually just gave up. With no schooling in the terminology and ideas it's just like a whole other language :D
Tornado does look nicer and I have to admit purely from the documentation. I read through it for about 10 minutes just now and I can for the first time actually see how it would work within the realms of our project.
With that said it might be a useful way to figure out what Twisted is all about too....
It's indeed the starting point to learn the fundamentals - but becomes almost hilarious after step 4 when they begin to introduce Components, Adapters, Perspective Broker and other things without losing many words about what all that stuff does, much less how to re-arrange it for custom needs.
The best investment in terms of mindshare the twisted project could make at this point would be to simply sit down and write documentation.
Grab sphinx (or another tool that makes doc-writing fun) and start out by writing a page about each Class that is being used in these examples. Why does it exist, what role does it have and don't forget the link to the API doc.
Then I'd suggest to make a high-level overview, perhaps even with a little diagram, that explains how everything plays together in the various application types. Hyperlinks to the previously mentioned detail-pages should then work wonders for anyone exploring the project.
Writing good documentation is hard, but you only have to do it once. The lack of documentation on the other hand forces every new user to wrap their heads around innocent lines like this:
I'm surely not the only one who put off learning about that whole Service metaphor and stuck with the simpler examples until I absolutely couldn't avoid it any longer. That's not the way you want users to approach your framework.
> The best investment in terms of mindshare the twisted project could make at this point would be to simply sit down and write documentation.
Making documentations (and contributions to it) a first-class contribution with its bugs & discussions & refactorings, and fun to write (the way the Django project handles it) would probably also go quite a way towards that.
Yup, agreed. Creating and endorsing good documentation is an artform in itself.
Currently the barrier of entry is too high and I guess too few people are deep enough in twisted to explain all the why's of even a simple baseline example properly.
It's kind of a deadlock situation because without that lowlevel insight it's also hard to get confident enough to contribute high-level docs.
> Currently the barrier of entry is too high and I guess too few people are deep enough
Thing is, through checking out Django I realized you don't really need to be "deep enough" to contribute to that stuff. The thousand-cut effects applies, and when you're trying to make heads or tails of the doc, you're bleeding from these thousand cuts so you know and understand the doc's pain points. If doc contribs are first-class, then right as you understand the issues and what the doc tries to say, you'll be steered towards opening a documentation bug (and maybe even contributing a patch) for that part.
Maybe it's just doing to be a bit of wording, maybe it'll be a link or two to other relevant parts of the doc; maybe it'll be things you'll have figured through your sweat and tears, maybe something you'll have learned by asking a question on IRC.
But even as a beginner, you'll have added a little stone and made the doc slightly better through your current understanding. And it's enough.
I barely contributed to Django myself, but my first ever contrib was simply pointing out that the `now` tag's format character `n` produced an RFC 2822 date, not an RFC 822 one as written in the doc at the time. It was almost nothing, but it still very slightly improved the documentation for the next guy reading it, and it actually made me feel involved in the project. Even though I didn't have anywhere near the knowledge of the codebase to contribute to the code itself.
edit: and re the thousand cuts effect, that's the great thing about making beginners contribute: "old timers" don't notice or remember these small issues (which pile up), because they fill in he blanks or missing parts through their knowledge.
What self-respecting programmer doesn't idle in the freenode channels of the software packages he or she is learning? You immerse yourself in the community and ask intelligent questions; if anything, just to see what the community is like before you commit your project to depend on something written by either douchebags or morons.
I did just that with the twisted community and I found them a good bunch.
You're on to something about community. A healthy OSS project has a healthy community. Maintaining a painless ramp-up with thoroughly organized documentation goes a long way towards nurturing an expansive, and inclusive community. By expansive and inclusive community, I do indeed mean that copy+paste jockeys can hack their way through it. If copypasters can get something done, then usually it means the defaults are usable, the common use-cases demonstrated, and the examples are up-to-date. When all of those conditions are met, it also means that the inertia to just trying something out is WAY less, and the interest:effort in just "screwing around" with it is now much better, which is the first step in getting some programmers (like me) to give something a try.
If I needed what twisted had to offer, I'd give it a try. But if I'm just screwing around (an exploratory phase that sometimes leads to much deeper commitment) then I'm going to go with whatever is easiest that seems to have sufficient depth.
Anyway, I have plenty of self-respect, thank you. I am sure there are others like me.
Not all of Twisted's documentation is bad. And if folks can't figure out the docs, Jean-Paul and Glyph are available to help in IRC and know everything about Twisted (since as far as I know they represent 90%+ of the current contribution and maintenance).
Besides, Tornado can afford to have (seemingly) more comprehensive documentation because it is far smaller in scope and support. Twisted has a large number of robust libraries for doing all sorts of things asynchronously, from web servers to ORMs to SSH and everything in between. It's like comparing an orange to a bushel of apples and it's silly. Twisted has never supplied a web framework, which is exactly what Tornado supplies.
The Twisted docs are lacking in many ways, I just don't understand how Tornado is objectively better because it doesn't currently have that problem -- they aren't the same thing!
"The Twisted docs are lacking in many ways, I just don't understand how Tornado is objectively better because it doesn't currently have that problem -- they aren't the same thing!"
Of course it's the same thing. If your documentation is lacking, don't complain when people point it out. It's awesome that Jean-Paul (who commented on the blog post) is available to help people...but if there was more documentation then the simple questions disappear and you are left answering the really tough ones. Easier said then done, obiously.
Upon reflection, this is really a comparison of Twisted.web with Tornado. The rest of Twisted got dragged into it, perhaps unfairly.
There seems to be a segment of the programming society that's just content with spewing whinage about not being able to get what they want, instead of using what they find and improving it to fit their needs.
I am no Python programmer, but i have looked at twisted and found it trivial to grok. I saw how they have wrapped the Unix system calls and put a layer of abstraction over it .. I didn't like it, and went back to using Common Lisp's iolib. I must have missed an opportunity to blog there :-(
I think you're taking a reductionist view, and I think you've missed the point the author was making. The author first expressed his understanding of the influences on individual behavior and then he made specific suggestions (with examples) of how twisted had failed to optimize for a specific outcome within that framework.
I consider this analysis and feedback, not whining.
You seem to suggest that the author is whining in lieu of being self-sufficient. I can't speak for the author, so I will just speak for myself. I wholeheartedly believe in the self-sufficiency that programming culture promotes. However, I also believe that the easier you make it for others to achieve self-sufficiency in your open-source library, the more successful it will be.
I'm happy that you found it trivial to drink deeply of twisted's essence. I also think that you are right that you have an opportunity to blog! I for one would love to read about why you think that CL's iolib is a better abstraction layer than twisted. You haven't missed the opportunity -- I'd upvote that over most of the stuff on the front page!
As usual, a simple "this versus that" comparison is only useful as an intellectual exercise.
Tornado is a web framework, in many ways closer to Django than Twisted. Beyond that the underlying approaches are different - Tornado does not use deferreds.
There are a number of great projects that use Twisted. Soon many new projects will use Tornado. But the frameworks' respective documentation does not tell the story.
Not really. It's the combination of an asynchronous HTTP server and an optional web framework. And work is being done [to add WSGI support](http://github.com/facebook/tornado/commit/8ca616088cfb26ff19...) so other frameworks can be plugged into the server.
It's obvious, documentation is always important. It's even more important for proprietary software, since you can't even study the source. But it seems that neither of this two projects has good and extensive docs/tutorials.
You could argue that if Twisted was better documented the FriedFeed guys wouldn't have started their own project. I don't buy it. There could have been many (technical and non-technical) reasons to do this anyway.
I (mostly) agree with a comment on the blog:
This is just pure NIH syndrome. It happened because it’s easier to write code than to read it, and it’s more fun to create than to fix.
I'm sorry, but http://twistedmatrix.com/trac/wiki/Documentation is the documentation, where I started and it shows nice examples, a high level introduction -- everything needed to dive in. Actually I learned a whole lot back in the days when I had to use this fine documentation.
Well, I think when friendfeed developers were writing tornado they didn't mean to opensource it or even talk about it in public, so they went for the name "Tornado".
I agree that documentation is really important in Open Source, and so is finding the documentation. Googling for such a generic names makes things hard. Considering all the hype Tornado got recently the homepage is not still listed in the first 5 pages of Google's results for "Tornado".
So I think it's a good advice to think twice (or thrice) before naming your next project.
I don't think the "documentation sucks" can be used to model very well the issue, there is another very important thing: is it simple to use, and the base case is obvious, or is it over engineered? I think in the Twisted / Tornado issue there is some of this elements too.
A lot of libs around here are, technically speaking, well coded, reliable, fast. But the API sucks. An example is the libpng, if you have some experience with it you'll understand. When an alternative with a clean API arise it's very hard to resist.
Every time I've thought about using Twisted for something, I go back to their documentation and remember why I wrote it myself with asyncore last time.
...and I agree completely with the op.
Twisted is powerful, flexible, well architected, and reliable. It is also damn near impossible to get over the steep learning curve and to actually USE it.
Show me a competent Twisted developer, and I'll show you someone who had a gun jammed roughly in their mouth, and was FORCED to learn it at some point. We are all glad we know it _now_, but that learning process SUCKED.