nerdstogether.org

Happy New Year! And a happy new name for The Application Formerly Known As Geek Event Aggregator (or geekeventfinder). Say hello to

nerdstogether.org

It's your one-stop resource for finding out, "How can I get together with some nerds in <insert place here>?"

nerdstogether.org comes with an exciting new feature: it actually works now. No, really. Go try it.

I achieved this by using some cutting-edge technology called "CGI scripting" and a "database". The problem of gathering the data has been pretty much solved for literally years, but I kept trying to find Web 2.0-ish solutions for getting the data to you: Oracle Application Express, Google Calendar, Google App Engine. All of them seemed great at first, but eventually frustrated me to tears.

The one nod to Web 2.0 is the very helpful use of Yahoo and Google web services for interpreting location information.

Anyway. I was so proud I actually bought a domain name for it. So go use and enjoy, and suggest missing events for it, and I'll see you at the VIC-20 club in Kalamazoo.

Geek Event Finder: now working

It's working! The Geek Event Finder on Google App Engine! Go play!

In some ways, the Google App Engine is a dream. Not thinking about the app server is wonderful. Deploying couldn't be easier.

I hate the GAE datastore with the passion of a thousand blazing suns, however. 90% of the work of this project has been trying to figure out workarounds and kludges for its bizzare limitations, like

• There is no mass delete. None. No truncate. No way to get rid of a large number of records at once.
• Bulk upload exists, but it always appends to the datastore - never replaces - which brings you right back to the "no mass delete" problem.
• No long-running operations - anything that would take more than a couple seconds dies - so you can't loop over all your records to do something (like delete them).
• Countless unexpected restrictions on queries. Can't filter on one property while sorting on another. Can't do inequality filters on more than one property. Can't filter on a string property if it is multiline (has \n's) or is longer than 500 characters (type Text). Can't use any function calls or arithmetic within a WHERE clause. Queries that fetch a large number of records die instead of completing (so I fetch in LIMIT 20 batches and assemble the results on the app side... crazy).

So if you look at the code and see some incredibly stupid stuff going on with data access - trust me, I tried fifteen different sane ways first. I am so not buying the buzz about this being "the future of databases". Fighting for hours to try to kludge your way to your data... that's the Bad Old Days, not the future.

My workaround for mass deletion was to write pages that would delete one record, then invoke it in a loop from my client computer. That has to run all night to clean out the datastore when new data is uploaded.

But anyway. I'm still very happy. It works!

Geek Event Aggregator code

I've told several people that I would post the code for the Geek Event Aggregator "as soon as I get it cleaned up".

Today, I swallowed my pride and admitted that I've really got to post something - that the much-needed code cleanup isn't going to happen for months. So, I created a page for it at Google Code: http://code.google.com/p/whatwhenwhere/

Not that you want to do that. It's ugly. Really. It will be cleaned up someday, but if you really want to make use of it now, there it is.

I decided to name the project "whatwhenwhere" because the code might well be used to aggregate events that are not at all geeky, and to host it at Google Code because, well, I'd never tried it.

Most of the stuff you might be conceivably interested in is in scan.py, ship.py, and events/model.py.

Geek Event Aggregator design notes

I finally created a central page for the Geek Event Aggregator - mostly it's design notes, but it's also a home less ephemeral than a bunch of blog posts:

http://catherine.devlin.googlepages.com/geekEventAggregator.html

Thanks to Google

... the Geek Event Aggregator is working again. Shortly after my post about the upload to gCal not working, I got an email from Ryan Boyd, of Google's Calendar team:

Someone brought the following post to my attention and I thought I'd follow up and see if I can be of any assistance... The problem that you are experiencing (500 error: Cannot access the calendar you requested) is due to the large size of (number of events in) the calendar.

Wow! Thanks to whoever passed my whining on to him... let me never speak ill of this social web stuff!

With that information, I was able to get it working again, so go ahead and check out that APL conference near your uncle in Kalamazoo.

Better yet, I was able to give my Lightning Talk on it, and it was very well received. The Lightning Talks are so popular that PyCon doesn't even schedule anything opposite them anymore, so basically the entire conference body of 593 people (!!!) was there; I've never spoken before so many in my life. I'm glad that didn't occur to me until afterward.

In the talk, I briefly outlined the architecture. At some point I'll post about it here. The basic principle is: HTML interpretation is hard, let's go datetutils.parser.parse()ing!

Yeah, PyCon! I just got back, and I've got so much to say; expect a flurry of blog posts as soon as I can get to it. I'm as happy as a clam. As several clams! As several unusually happy clams!

Geek Event Aggravator

I swear to you, the code that gathers data for the Geek Event Aggregator is running great.  It's harvesting over 5,000 events, with ever-improving completeness and accuracy.  

Unfortunately... the code that ships the harvested data up to Google Calendar - which worked perfectly at first - is now falling flat, with gCal returning 
"Error: Cannot access the calendar you requested." 
when I try to ship the new data up, or to clear out the old data - thus the redundant entries.  (No, it's not a simple password problem, I did check that...) 

I suspect that I triggered something at gCal that said, "Hey, there's way too much activity on this calendar, it must be some evil bot, lock it down."
  
I guess this is the problem with mashups.  It's great to springboard off the work of total strangers - but you're at the mercy of a black box.  Maybe I can buttonhole somebody from Google at PyCon next week (holy cow!  PyCon next week!) and ask for a contact somewhere in the bowels of the gCal team who might be willing to help me troubleshoot.  The sad thing is, this pretty much sinks my plan to do a Lightning Talk on it.

So anyway, my apologies if you've been trying to use the Aggregator and have found only stale and redundant information.  My box is overladen with fresh, hot event data; I just have to figure out a way to get it out to you.  I tried Swivel, but can't seem to get the data there, either.  And the old-fashioned solution - the app running in the Oracle Application Express sandbox - can only receive data by manual uploads, and then only in small chunks; I would have to do about fifty mouse clicks every time I wanted to refresh the Aggregator's data, and I'm sorry, but I'm not a clicker.

The ultimate solution, of course, is to build my own app on my own web server, or on a web host with permissive policies on installing software.  I don't have those resources right now, unfortunately.  (sigh)

[I apologize; Blogger has gone loony on the line breaks for this post, and I can't seem to fix it. Grumble. All my web resources are letting me down today.]

Geek Event Aggregator is back

Re-introducing the Geek Event Aggregator - the world's most comprehensive and least accurate compilation of technology events! The Aggregator regularly scans a huge variety of event announcement websites and gathers the information into a single location. Now, when the VIC-20 Users' Group schedules a meeting down the street from your in-laws in Pittsburgh, you'll know it.

(Yes, I've introduced this before, almost a year ago. But a LOT has changed, and the accuracy is much better now.)

The Aggregator uploads its data into a Google Calendar account - the most convenient way to access it is as a Google Calendar user (which I really do recommend.) Here's what you do.

1. Log into your Google Calendar account and choose "Manage Calendars".
   
   
2. Click "add calendar"
   
   
3. Search for "geek event aggregator", then select "Add Calendar".
   
   
4. Return to your calendar home. Oops, your calendar now looks dense with events. That's because you're seeing an unfiltered list of all events, everywhere. Better un-check "geekEventAggregator".
   
   
5. Now you can make real use of the Aggregator - search for the city, state, or region of interest. The Aggregator attaches several keywords to each event based on its location.
   
   
6. Don't take anything the Aggregator reports at face value! Double-check it all. Remember, it's just a poor computer trying to read webpages that were written for humans. It doesn't even attempt to find the time of day, it just declares that everything runs 9-5 Eastern time (who cares if it's in Johannesburg?) I expect to keep gradually improving the accuracy as time goes by.

If you don't have and don't want a Google Calendar account, you can also access it through a static HTML page, an ICS document, or an XML feed. In that case, you don't get Google's handy Search button, and you'll need to use your own patience/creativity to find events in the places you want.

The Geek Event Aggregator is written in Python (of course!) Behind the scenes, it uses TurboGears and PostgreSQL. (No, you don't see a TG-based web application; the web app generates an annotated version of each scanned webpage, which I can use to troubleshoot the scan; thus, the improved accuracy of this second-generation Aggregator.)

I expect to present a Lightning Talk on it at PyCon 2007 (you are going to PyCon, aren't you?)

The Geek Event Aggregator is ready!

The Geek Event Aggregator is more-or-less ready for prime time! It now collects many more events. Better yet, it is very easy to feed more event sources, so it is set for even more growth. In other words,

PLEASE FEED THE AGGREGATOR NEW SITE SUGGESTIONS!

Some design notes:

After fooling around with complex regular expressions, Beautiful Soup, etc., I found a quick-and-dirty way that works better. The Aggregator downloads a page's HTML, replaces all tags with carriage returns, breaks the remaining text into lines, and checks those lines for ones that appear to contain recognizable future dates. The Aggregator assumes that is an upcoming event date. (There are many reasons dates get put on websites, but future dates almost always refer to meetings or events.)

Wow, human beings have many, many, many ways to write dates. Fortunately, python's dateutil module can recognize most of them - I mainly have to modify that to avoid false hits (like interpreting '.' as 'today', or '2006' alone as 'Jan 1, 2006').

Some sites already aggregate events from several groups and places together; for those, the Aggregator uses a slightly different algorithm. It finds the dates as above, then finds the location and event name by their line-number position relative to the date. (A human being (me) needs to provide relative line-number positions for those values in advance; for instance, one site may always list location on the line immediately after the event date, and that fact is recorded in advance.)

For the multi-event sites, the Aggregator has a decent but kludgey algorithm to parse city and region, despite the great variety of ways to write a location. Part of that relies on a list of recognized city names. It can be used for the single-event sites, too; if a recognizable city name is in the site's Title, or in the text in the form "Blah Blah City Blah Blahware Group", the Aggregator can find it. But if your event is in Athens, GA, the Aggregator thinks it's in Greece.

HTML DB makes it very convenient to build a web interface to the data. Oracle is also very gracious to host a free sandbox for HTML DB projects (which is where the Aggregator lives right now.) Unfortunately, as far as I can see, HTML DB doesn't support RESTful interfaces, or serving up pages as XML. That's a pity, because this cries out to be a REST web service. Maybe eventually I'll buy/find a place to host the web app in TurboGears or something.

For now, I've given up on feeding upcoming.org. Upcoming mandates an actual street address, which is just too hard to find automatically. I'd still like to pull from it, although I'll have to check their legal requirements, and - dare I say it? - I don't know if it will really have many relevant events I don't already have.

I'm sorry the events are so U.S./Canada-centric. It only has a handful of events from elsewhere, and it doesn't break down regions within other countries. (What's wrong with going from St. Petersburg to Novosibirsk for a meeting, anyway? Isn't that what the Trans-Siberian Railway is for?) You can help fix this by suggesting new sites to scan, and volunteering to introduce regional granularity for other countries.

Actually, because I've been the only one to feed the Aggregator so far, the events are Ohio-centric. You folks in benighted backwaters like California and New York are just going to have to feed it your own favorite sites if you want to fix that.

Some of the many things that produce misses and false hits:

• Dates without years. Somebody puts on the website, "Our next meeting is Nov. 19." The Aggregator can't tell that they haven't updated the site since 2004.
• Years must be on the same line as the rest of the date. If you say,
  2005 events:
  Feb. 14
  Aug. 12
  ... the Aggregator doesn't see the "2005", and believes there are events on Feb. 14 and Aug 12 of this year.
• Frames. Well, you can't blame it; frames mess up everybody. But if you can dig into the HTML source and puzzle out the URL of the frame with the data, then that can be read. That's what I did for the OKCOUG webpage, for example.

Geek Event Aggregator's future

The strangest thing happened to me this morning. The clouds split open, and a beam of light shone down onto me. (Since I was sitting in my cubicle, this in itself was odd enough.) A Voice spoke from Heaven, and said,

"Catherine, remember that somewhat cheesy Geek Event Aggregator you put on HTML DB a while ago?"

"Yes, Lord?" I said. (That seemed like the obvious response. That the Almighty would speak with hyperlinks didn't seem too surprising.)

"Well, I have heard the cries of my people. I want you to rewrite your aggregator to both consume and provide events in iCalendar format. Also, interface it with upcoming.org - both to download and to upload. Behold, Python libraries for upcoming.org have already been written for you. And I'm thinking that Beautiful Soup might help you pluck event descriptions from the God-awful HTML jungles you find them in."

"It sounds wonderful, Lord," I said. "But when am I supposed to do this? I mean, you did just drop custody of our Godson into our lap. Time is not exactly abundant right now."

There was a long pause, then the Voice said, "Let me get back to you on that."