Database trouble, expect missing updates

I wish I had had a regular backup schedule. I'll be sure to set up one once I've cleaned up this mess.

I just overrode all the crawler configs from the database. Meaning that the crawler won't know how to parse the pages that it downloads. The last backup I had was from May. That leaves 400 or so comics with invalid configuration, and all the crawler fixes I have done after that are gone too. I expect that I can recover most of that data in a few days.

Please don't email me about broken comics quite yet. There are bound to be a number of stragglers but I'll try to fix as much as I can myself first.

This wasn't the blog post that I was hoping to make. Sorry about this.

Update, database mess cleaned

Everything's back to normal, as of Thursday evening. I got a bit lucky and still had all the updates to old comics' crawler settings available in an editor buffer, which had been open all year. Going through the new comics took some time but that's done now too. I fixed a few comics that had been missing updates even before I had explicitly broken their settings and found a few comics that had gone missing and should be removed from Piperka's end.

As far as database disasters go, this one was from the easier end. Had I wanted, I could have passed this one under the radar, but I didn't know that quite yet when it happened. On the positive side, I have regular database backups running now.

I should find ways to make maintaining the database easier for me, perhaps even to the point where I could hope to involve others in it. I'm doing a large part of the work by writing straight SQL, and that's error prone, laborous and I can't try to distribute that work if it's at that level. A large part of the update logic is humming along nicely automatically but it does get stuck from time to time and I'm watching that part only little. Though it's certainly the core part of what Piperka does, it's kind of invisible and I find trying to implement new, more visible, features more motivating. But I'll get to improve this part, yet.

Finding disappeared comics calls for another feature for Piperka: A list of removed comics. One that would retain all the data from the database and users' settings regarding that comic, but that would list them separately and aside from most of the logic that concerns active comics. I've been just removing dead comics so far, but that doesn't feel like the best solution. It's like they never existed when I do that.

That's it for now.