Crawler rewrite is still ongoing, goodbye Project Wonderful

I set it as a goal to get the new crawler done a couple of weeks ago. That didn't happen and I'm still not quite there yet. I'm far enough with it to do crawls for single comics with it and the hourly cycle code is almost done, it's mainly missing some logging I still want to add in there.

I think I found the reason why the crawler has been prone to database deadlocks lately. I had added caching for the users' unread comics stat shown in the sidebar (the "n1 new in n2" thing in there) and that's been updated by triggers. Which the crawler had been firing with its long held sessions. I think the deadlocks made the web site itself unresponsive at times as well. It'd be nice feature to save the few tens of milliseconds that computation takes when possible but I'll look into bringing it back some other way once the new crawler is in place.

It may take another couple of weeks for me to have the opportunity to put the new crawler in place, for real this time. There'll be nothing user visible with this change for now, though I'm hoping that I'll slowly work on new features based on it to make Piperka better catch updates. I just don't have the heart for another long slog with something that only takes me to where I was in the first place, even if the new version satisfies some odd aesthetic sense of mine.

In other news, Project Wonderful is shutting down. I've been using their ads nearly since they launched and I'm none too happy to see them go. I like their model a lot, it's fair and unintrusive and their ads have generally been relevant to Piperka's audience. I've liked them more for offering a place for comic authors to promote their art than for the ad revenue itself. I still would like to see Piperka generate enough income to support me but whatever that entails won't involve PW anymore.

I may or may not set up some new ads in July. Whatever I do, I care more about the user experince than whatever revenue they may or may not generate. If I had infinite time (I do not), I would set up some sort of self-hosted advertising system. Piperka will look quite empty without PW.

Ironically enough, PW sent me an email about receiving my 100000th bid on an ad box a few days after the shutdown announcement.