I just want to point out that anyone can help the Archive Team by running an "Archive Team Warrior". It's a virtual machine which scrapes things that need to be archived. Currently Posterous, Formspring, various URL shorteners and of course upcoming.org (in testing).
My Archive Warrior is still trying to get through all of posterous.com!
Internet Properties: Don't be a dick. If you're closing up shop, bundle the site up (public portion, not personal info), and ship it off to the Internet Archive.
I don't disagree with your suggestion, but when you're closing up shop, there's often a heck of a lot more pressing issues on your plate than wondering whether someone will be able to view your long-dead site after you've laid it to rest. Being a dick is a possible reason for not doing this, but there's a lot of others that don't require or imply malice.
I've been working on a replacement for Upcoming, called Daypace http://www.daypace.com but I'm still a good ways from getting a product out there. I really liked Upcoming before it went to hell with Yahoo and around 2010 got motivated to do something about it, but I wanted to also recommend events to people and avoid the spam that slowly took Upcoming down, and it's been a tough road getting that all to work and since I work at the most amazing non-profit in the world, I don't get a lot of time to put into it.
Regardless, thanks to the people who built Upcoming and sorry that Yahoo ruined it.
> But Yahoo's security makes scraping a challenge. Every time I've tried to back up pages, I can only grab a few files with curl or httrack before Yahoo starts serving blank responses. (...) If you have any idea how to scrape Upcoming's events
A way to bypass security is to distribute scraping over many, many clients.
For example, one could write a browser extension and have many people use it at once; this would have the added benefit of being able to scrape information generated by JavaScript (just do the scraping after the page is completely rendered).
If needed, I can help write such an extension for Chrome (I had written one for del.icio.us some time ago that did exactly that, and another one for IMDb).
(There would be a public server that would serve the ids of events yet to scrape, and that would receive the data that has been scraped).
But for this strategy to work, you need a big community of people willing to help (people who will download the extension and let it run).
ArchiveTeam already has a system for this. It's called "the warrior" and it's a little VirtualBox image that checks in with a centralized tracker to get jobs and upload rescued data. You can see the progress on the "upcoming" project here http://tracker.archiveteam.org/upcoming/
Still, it's much more complex than a simple webapp (so the audience willing to do it will be smaller). If there are ~12 million events, we need 10 000 people downloading 1200 pages...
(On the leaderboard, the number of "to do" items goes up instead of down: what does it mean?)
I don't know for sure, but I expect it's a standard thing from any spidering. Let's say you start at one page and find 10 links; your tracker will then have 10 items. If each of those pages has 10 links, then you can then have up to 100 items to download. And so on.
Over time you're going to start seeing duplicates (ie. things you've already archived/spidered), so the number will go down. But while you're still in the 'discovery' stage it's going to go up and up.
But here's what the OP said: "All of Upcoming's events and venues use autoincremented ids, making it dead simple to generate a list of URLs to scrape."
So if you go from 1 to {current_id} you get them all, and you have the list at the start.
Sometimes the "items" can get huge. Usually one user counts as one item, so some users run into multiple gigabytes and take days to finish. How would you handle that with a webapp?
I'm not sure about the details for this project, but sometimes it's not easy to get a dump of a list of the items you want all upfront. So the warrior process catalogs new items it encounters and updates the list on the tracker.
Ok, I get that now; but in this particular project every event id is known (see my other comment below).
A webapp would simply need to receive a list of ids and download one at a time (with a comfortable delay between each call so as to not trigger Yahoo's security).
While this was a good bit too "name-droppy" for my taste, it's an interesting story of decision-regret and a decent chronicle of a founder's view post acquisition. At the time, the decision to sell seemed right, and it would've been difficult to predict it would be so poorly managed by Yahoo! Along the way.
Indeed. I have to think that it's hardly like there are (or were at the time) no precedents. When you sell out^H^H^H your company, you give up ultimate control.
However, I've thought about this space a bit (the idea-space, not the solution-space), and I've always come to the conclusion it's too fragmented. Too many things to too many different players.
But if the field is wide open again, better to invite someone into the opportunity than dwell on the past.
All due respect to the work that went into Upcoming, but nothing else covers the same things? I can't remember why or when I stopped using Upcoming.org, but I was also using Eventful and Zvents at the same time, and I still use them, for the exact same thing.
Disclosure: 1st non-founding employee at Upcoming.
Neither Eventful nor Zvents plugs into a social network to tell you stuff your friends are going to, or thinking of going to. They are just sites that list events.
Facebook's events app is 100% social - it tells you what your friends are doing, and you can follow artists and venues. There are comments and such on events. It is also good for non-public, informal events, something Upcoming didn't do very well. But Facebook is useless for searching for events in your area, or for following a particular genre of music or type of event.
Upcoming was sort of in the middle of all of this. Which was part of the problem. It was hard to know what to focus on. I'm not sure the story is entirely that Yahoo was solely to blame for fucking it up. We also made some incorrect decisions. But unquestionably, Yahoo didn't give us the resources to succeed, and prioritized things that had nothing to do with success.
I'm sorry to hear about your loss. It always sucks to have something you worked so hard on (your baby) canned or messed around with for non-technical reasons.
Your elucidation of Upcoming's differences helps explain why I probably quit using it, though. While I can see the value in having your social network plugged into a facet of (if not your entire) calendar, I personally choose not to participate in third party social networks. If Upcoming required a Facebook (or just about any other social network) login, I would have stopped using it. And yes, I realize I'm in the minority :)
Thanks but I don't feel any "loss". It's just a website. Even if it had been a smashing success, by now it would either have evolved into something else or died a natural death.
It was a loss of time for me, maybe, but I was well compensated. The real losers were all the community members who built it into something.
I liked Upcoming a lot, and when I needed to learn Django I took a shot at making a similar clone, (though focused more on today and this weekend, something most sites don't feature).
It's a bit homely and needs some love but I'd like to upgrade and open-source it. Anyone interested? It's called kpasa and is at http://lax.kpasa.co/ .
[1] http://archiveteam.org/index.php?title=Main_Page
[2] https://archive.org/web/web.php