I've introduced a ticket system. There's a link to the ticket page on the comics' info pages now. I didn't yet apply any CSS on it so it looks basic and I know it. I've passed on on the idea of setting up a ticket system before but that was when I still had a lot fewer comics listed than currently, and I've finally all the pieces in place to set up a proper interface for crawler maintenance. Before this, I had pretty much expected to just do the manual update steps with hand written SQL and all even with a ticket and it would've mattered little whether I'd've read about an issue from an email or a ticket.
I only implemented ticket submissions as the first step. There's no way yet to view your submitted tickets. Nor is there any interface for me to view them either, other than by doing a database query. When I add one, removals of dead comics should be much easier to do. Other maintenance operations will still have to wait for later development goals to fall in place.
Sending me email about crawler issues has been a lamentably haphazard business. I'm afraid I'm not going to go through my old emails at this point. If you've sent me one about a comic that still hasn't been fixed, please go and open a ticket. I'm not yet promising more timely fixes than with the old way but at least it'll be much better organised from now on.
I'll be looking into ways to make the ticket system to bring the errors the crawler itself finds to my attention in a more actionable way. It really shouldn't be so that my users would need to ask me crawler issues this much when the crawler could work with me better in the first place.
I spotted an issue with parallel database accesses causing deadlocks with the web site. When that got triggered, the web server would freeze for some time and trying to access the site would just give a gateway timeout. Nasty stuff, I'm hoping that I've fixed it now or at least made the situation a lot less likely. It had to do with the unread comic counts (the numbers next to the check updates link) and how they got altered by using a redirect link. I don't know how many of you encountered that bug but the fix's been in since last Wednesday. The old site used a single database handle for all backend operations (other than AJAX endpoints) and it wouldn't even have run into a situation like this.
I've split the web site code and its authentication layer to separate projects. I'm hoping that the latter will find more users as it's always better for code health to have other people rely on it. I've switched over to using Gitlab for Piperka's code. It pleases me that they're open source, though I'm still somewhat on the fence about git itself. I still have a fondness for darcs even though I used git this time around. If you think you've found a bug or have an development idea then head to the issues page. I've opened a few myself already. Please don't use it to report about crawler bugs with singular comics. If anyone's interested in giving a ride to the code themselves then I'm ready to provide a copy of the database schema.
Time for a third update after the site transition. The biggest change you've likely noticed is that Piperka redirects you to use the HTTPS version of the site. There's no reason to not use it in this time and age, to better protect your settings. Setting that up wasn't quite that straightforward since there is one page on Piperka that needs to stay as HTTP. The iframe embedding that Reader does is not possible if the host page is HTTPS and the embedded page is HTTP and that makes it necessary to use HTTP instead. To make the reader page communicate with HTTPS with the rest of Piperka, I had to add something called CORS headers to the responses. It all added to complexity and I had to get it right or I would have risked blocking people from Piperka or even permanently redirecting their browsers to someone else's web page. But it all seems to work and I guess it's all right.
Another thing that I've done is to fix Piperka's side for the Android App. I'm hoping I did, I don't use it myself. The rest are smaller feature fixes, like fixing the link from another user's profile page to the map and correctly loading their selections to view. And there was an off by one error with the quick search function. I was actually told about it soon after the site transition but when I looked at it I saw another off by one error with the links to jump to a letter on the browse page and only ended up fixing that even though it was a separate issue. I only gave it a real fix when I got a second bug report about it, sorry about it.
Speaking of Piperka Map, I'm aware that it hasn't had an update for a long time. There's a daily data dump that makes the map calculating program segfault and I still haven't taken the time to figure out a fix for it. As the map compute is set up, doing it for a day requires the previous day's map, to align the comics to be as close to each other as possible, from day to day. I could have just jumped over the offending day but I'd rather fix it properly. Even though it's been that way for a long time now. At least there's good chance that it will be soon now, though it'll still take weeks to catch up, computing each missing daily map up to today.
There was a bug with using Piperka's bookmarking with a bookmarklet. I don't think there's currently any instructions for how to make one anywhere on Piperka. When I reintroduce it I'll make it so that you get a personal bookmarklet to use that makes making bookmarks a single stage process without that annoying CSRF confirmation check.
I've added a page about Supporting Piperka. I was more comfortable with setting it up so that the link to it on the side bar is visible only to logged in users. It might have been a better move if I wanted to maximize donations to simply offer a link to Patreon but I rather wanted to give an explanation on why I recommend Liberapay as an alternative. At the risk of making people read words. Don't get me wrong, it's okay if you use Patreon and I'm grateful if you do. It's okay if you never use either, as well.
Blog posts have been a weekly feature this month but there likely won't be one next week. I'm taking a mini vacation next weekend. Don't hold it against me if I still add some comics or fix crawler issues but I'll stay away from any feature development.
I'd say that the site transition went quite smoothly. The crawler had a hiccup where one faulty parser code failed silently with old library versions but killed the whole run with the new ones. Some redirects went nowhere but I caught on to that bug in short order. The Perl crawler code used a funny memcached key name for notifying about crawler progress and I needed to switch to another library with that. This is one of those situations where I was happy to receive just a couple of emails from my users. All in all, it all was comparatively boring. Boring is good.
Last week, I've been setting up various background jobs and testing the old site maintenance scripts with the new site and making some updates to them. One thing that didn't quite work was the submission handling code. The users with moderator rights weren't actually able to make edits. That bug was to be expected since I had written that part in a hurry and very late in the project when I thought I was about to get the code live.
There was a problem with users' timestamps with regards to comic additions. You may have noticed that Piperka was telling you about a couple of new comic entries for most of a day last Thursday and that notification didn't reset properly. I added a new timestamp column for that logic and now they seem to work properly again.
I found out that the script that I use for adding the comic's entry after the initial crawl that adds the archive page's to Piperka's index failed badly with comics with banners. The error looked very much like a stack trace, I didn't even know that Perl had those. I very quickly decided to not even try to fix that code and instead made the first fully new feature for the new code base: Adding comics to Piperka now works via the web page. In its current form it still is only usable by me since the initial crawling needs to be done with the old maintenance scripts on a shell account. Nonetheless, it's a step towards reducing site maintenance being tied solely to me.
Piperka now offers HTTPS connections. Thanks to Let's Encrypt setting it up was really easy. I could have done it with the old site already but I wanted to have one fewer moving part for the transition, despite the long delay. I still didn't set up a redirect from the HTTP side to HTTPS. I want to do it on a Sunday when there's less traffic but I was busy coding yesterday. Next week, then. Feel free to update your bookmarks to use "https://" already.
I'm afraid the unofficial Piperka Android App is currently inoperative. It's due to changes to Piperka's backend. I tried to retain compatibility but with other time constraints I didn't use all that much time on it or test it. I'm not its author and I can't as such quite offer support for it but I'd still rather keep it working. I may need a few days to find time to figure out what's going on with it.
I just figured out that Piperka's email sending was offline for last week. The SMTP process was supposed to be running and I did test it but apparently it was shut down at one point. The transition process was a whirlwind and it may have been due to some half finished action of mine. I'll keep an eye on it. Piperka uses email for password recoveries and for confirmations of newly added comic entries so no that large harm with that. I assume people would have told me if they tried password recovery and never received one.
I've talked much about OAuth2 and it didn't immediately work since I still needed to refresh the configuration on the providers' side. It should work now. I'll yet add at least Facebook as a login option.
The new site keeps a couple of Piperka server processes running and nginx acts as a proxy to forward traffic to one of them. Deploying changes involves zero downtime as I replace the secondary server process with an updated one and once it's done, just update the proxy port in nginx's config and reload it. I also have the option to instantly rollback to the last version if the new one turns out fail somehow. A sysadmin at work suggested this setup for me and it seems to work just fine, so far.
The point of no return was on last Friday. I had three weeks of overlap with the server contracts. I was a bit hasty with signing up with the new one but it's all well in the end. The contract period for the old server has expired and it's gone now, forever. Luckily it was soon clear that I had no need for that contingency plan. Whatever odd ends there still remain are well manageable. I was able to retain user sessions and nobody (as far as I know) was kicked out and needed to relogin. One issue that I suspect is that the account page won't accept password without logging in once, first. No need to log out even, it can be done with an incognito window. The account page's password check doesn't know about old style passwords.
I'll be giving a talk at work on an internal functional programming event day about Haskell and how to write a web site with it in a couple of days.
I'm considering that the next feature to implement is a ticketing system. I still need to act on them with the same old clunky scripts and raw SQL queries and the uptake may still not be all that great initially but I know that crawler bug reports very likely end up being just ignored if I take any more to my email. My goal is, naturally, to have much less to ticket about in the end. It's all too easy to spot stale comic entries currently.
At first, I was simply exhausted after getting the new code live but I'm slowly starting to see the new possibilities. I'm happy about the new backend code and it's much faster to do new feature development based on it.
I'll write a longer message later on, but this is just to let you know that the new site is now live. The old server is still in place in case there's still something catastrophically wrong but so far things seem to be going smoothly. My inbox is peacefully quiet so far. The crawler and other scheduled tasks are still offline but I'll enable them shortly. My own blog post will pass finely for a test for it.
Thanks to everyone who tested the server during the beta week. I know it still had some pretty obvious bugs when I announced it but I wanted to move forward with it already then.
I've had a pretty intense couple of weeks of coding. I thought it would be a week but just when I was finishing up at the end of it, I discovered that I still missed the code for processing comic info page edits and new comic submissions. I certainly was thinking of it earlier but it had slipped my mind since. Nor was account creation via the usual route of giving a user name and password working. So I had another week of pretty intense coding evenings.
The new server is live! Go poke at it at. I'll allow a week for testing but that'll become the new Piperka next weekend. It's running a fairly recent snapshot of the database and anything done on it will be reset on transition. It's not running the crawler yet and it's not getting any comic updates. Go create users and click on things and tell me when you run into issues. I'll rather fix them now instead of after it's in production. I trust that the code is in a good enough state to not blatantly leak any user data.
The backend code is all new. The first time you log in there's a longer delay as it generates a new cryptographic hash for you. When the transition occurs, old sessions should still work transparently. Or even now if you set the cookie manually. I expect to have the site down for less than half an hour when the transition happens.
The outward changes are all pretty minor. The updates page had display options on it but I moved all those over to the account page. I added more sort options for the updates page. As of this writing, the new server's not yet set up to send email and you can't ask it to reset your password. The comic info pages are still missing related and subscription history sections. I'll be using this week for fixing any bugs I find and for adding any still missing things.
I've been so long at this that I'm almost at a loss at this point. I've had variable amounts of time and energy to give to the project but that's just the way it is with a side project like Piperka. I know that some comics are missing updates as I've been giving my main attention to the rewrite. I'm not promising immediate improvement on that front as I have further development goals now, which include improving the crawler and its management interface. But I have more time to give to Piperka from now on and some of that will be on site maintenance. Be that it is still more laborous than what it needs to be.
Expect to see new things soon.