My last blog post was about describing comics that are updating. This time around, I'm talking about comics that don't update. Roughly speaking, more than half of the comics listed on Piperka don't see any updates. Thankfully there's much fewer of those that are due to an error on Piperka's end. Comics that have simply disappeared have been retired to the graveyard but there's still a lot left over. Comics that have been finished or abandoned but which are still online. I think both updating and non-updating comics have their place on Piperka but sometimes the trouble is telling them apart. I had already used the internal crawler update score to mark updating comics with a white corner on the listing pages but I'd like to go further with this.
I'm now classifying non-updating comics with a reason for their dormancy. I'm not even trying to make the crawler try to tell apart whether a comic has been completed or abandoned. Instead, it's a moderator action. It'll take a while until all of them have been looked at, of course.
My next plan is to make the comic listings have a bit more information on them besides just the comic name. I'll try out different layouts to see what seems sensible. Update or non-update status and perhaps the page and readers counts. I'll add the option for compact display. One goal with this change is to make it clearer to any new visitors to the site what Piperka is about: Updates and archives rather than just being one giant list of comic titles.
I made an update to the revert page. It's using the same listing code as the other listings now, with pagination and columns. The new listing also shows the time since the redirect was last used. Hopefully it's more usable now. The revert page used to show only the redirects done during the last 24 hours but that clean up got dropped by accident the last time I changed hosts a year and half ago and I decided to leave it as it is now.
Now that I got into touching the listing code again, I implemented another idea I've had for a while. As a shout-out to comic authors, I wouldn't be doing this without you guys. Support Comics is a user specific page which lists those comics they are subscribed to which have a Patreon or Ko-fi pages recorded for them.
Piperka's main page is kind of useless. I've never had any good idea about what to put there and it's only ever had a short introductory message along with a count of the amount of comics I've personally subscribed to. Yet it's what many well established users use as their landing page, either via search engines or directly, only to immediately click away from there to the updates page. Not anymore. I streamlined it and updates.html page is now gone and what was there is now shown on the main page for logged in users. Going to the old updates page will now redirect. Adjust your bookmarks if you will, even though I see that many of you still use a HTTP URL for accessing Piperka.
I've done a bit of log analysis to determine comics' weekly update schedules, whenever they have a stable enough one to tell. I haven't yet used the inferred schedules as an input for the hourly crawler run, it would be nice to have it ping on archives more often when there's an expectation of new content and let them rest at other times. There's no guarantee yet that the crawler would check on the comic when Piperka says that the update should be out. Once I get that part done I'd expect the accuracy of reported update schedules to increase as well.
As of this writing, out of 5326 active listed comics on Piperka 1313 had any updates during the 12 weeks of log history I'm using as data and 668 had schedules I could infer with my algorithm. I didn't use much time at all to try to optimize it so there likely is still room for making it more accurate. I encountered and reported a bug in GHC along the way.
The inferred update schedules are now a part of comics' info pages. If they have updates during the time period but the algorithm failed then there's a message about inconclusive update schedule. If there's no message about a schedule on it then it had no updates during that time.
A more prominent use for the inferred schedules is that they're displayed on the updates page for the comics a user is subscribed to but is up to date with. I suppose this is the most visible change to the part that people use the most on Piperka in quite a while. I'm not one to do any usability studies with the changes I do to my site. Most often I just come up with an idea and if I like it well enough for my own use then I go through with implementing it. Let me know if you don't like seeing Piperka's guesses about upcoming updates and would like to keep the updates page simpler then I can see about adding the option to not show them.
On a personal note, I'm on my annual vacation. Which means that I have a few weeks to give more attention to Piperka. I'm hoping to finish up with a feature or two.
I'm writing this one so that I might remember it better: Piperka's backend process runs as a systemd user service. The default mode for them is to shut down when the last user session closes. That means that everything is fine as long as I'm logged in but it shuts down down when the last ssh connection closes. The command to change services to persist is loginctl enable-linger which will make Piperka run even after I close the laptop lid. At least I had the fortune of having an SSH open from another computer until I did a reboot this morning to change the hostname. I had almost two hours of outage due to this today but I suppose I got lucky since I didn't cause any more of that. Sorry about this, this was a pure user error and I'll try to not make the same mistake the next time around. I should write some documentation about the process or even automate it somehow, this certainly wasn't the last server migration I'll be doing.
I'm not the first user of my new IP address. I don't mean to reflect badly on my new hosting provider but looks like the last holder of the address gathered some bad reputation. I was told that an anti-malware site was flagging Piperka after the migration. I contacted them and they told me that they had blacklisted my IP. I'm hoping that they'll clear the status soon if they haven't done it already.
Also, Google is rejecting all email from piperka.net at the moment, citing that my IP has sent bulk email before. I added DKIM, SPF and DMARC (email security and anti-phishing features) for my domain and I hope that that's enough to clear that one up. At least my IP isn't included in any of the public anti-spam blacklists.
Hopefully I'm not seeing any other issues like these. I'll just resolve them and it's not worth it to consider requesting a new IP over this or even to switch providers this soon. It's just frustrating that these aren't anything that I have direct control over.
Here's to hoping that I'll have a dreadfully uneventful and boring time with my new hosting and can concentrate on some feature development again.
Addendum: This one slipped my mind. I knocked out the crawler for more than half a day on Thursday. It happened at a bit inopportune time for me. Something in the new environment didn't agree with one of the parsers and that aborted the whole run. Sorry about that.
I moved Piperka to Hyperhost.ua today. I've had a few problems with my last hosting provider. The server I've been using has had a number of hang ups that I've had to resolve by issuing a remote reset on the control panel and their network has had more outages and slowdowns than what I'd like. Though as far as I can tell they've improved on that last respect lately.
I was hoping to stay on my previous server for a while longer but I decided to take the leap already now. I had no real need for the dedicated server and this time Piperka's on a VPS. I suppose it's okay. The hardware's newer and page renders are noticeably faster with it since that's mainly single core load. If you don't see it yet then you're likely still using the old DNS entry and have your traffic proxied through the old server. I had extravagant amounts of RAM and disk space on the old server but I was hardly using them.
My sysadmin friend at day job talked to me about streaming replication for my DB but I didn't listen to him this time around. As far as I'm concerned the 15 minutes of outage it took to dump, copy and restore the DB was quite acceptable. Everything seems to work now on the new server. It took a bit of manual work to set up everything on the new server and I hope to not have to think about this in a long while again. At least moving over was a lot less work than last time around.
If you're in need of any hosting for yourself then feel free to use the affiliate link that I gave above. Time will tell how this'll work out but it's looking good so far and they have received some very good reviews.
I'll keep the old server for the rest of the week but I certainly hope to not have to revert this change. I even went as far as to upgrade my PostgreSQL version along with the migration. I'll have to go with stale data if I must go back.
To top it off, I'm even paying less for the new hosting. Let's see if I can outgrow it.
If you see anything out of the ordinary then please let me know. I've tried to grab everything relevant from the old server and I'll have a second look tomorrow but it's still easier to fix any straggling issues while I still have it.
I've ported the Piperka client app to Android and it's been available on Google Play as of May 20th. I put the badge on the site template for a while but I'm moving that to just the new user landing page and the about page a while ago. I think you know that it exists by now. I marked the app as having ads but I didn't yet actually implement those. They'll be added in a later update, perhaps.
As of now, the uptake has been quite low. I'm seeing much more activity from the Sailfish version. There's still more I could do with the app, like using notifications to alert about updates, but I'm moving on to other things for now. If and when the app would gain some degree of popularity I would return my attention to it. I suspect that I'm just old school in that I don't have much of an intuition about what makes mobile devices' users tick. I don't use them much myself and as Piperka's design is mostly guided by what I'd like to use myself so I wasn't on a familiar territory when making the app. I hope it'll find some users at least, this kind of was what took most of my development time this spring.
I'm retiring the unofficial Piperka app that's been around for a few years now. Other people are welcome to write third party apps for Piperka (or to send pull requests to mine) but I had to poke a few holes to Piperka's authentication system that I never liked to make that particular app work and now that I have mine published I'm going to require other implementations to have that same base level. If there's something that it did that you'd like the new app to do then please let me know.
I'm going to focus on crawler improvements next. Much of that will be invisible and aimed at catching or even automatically fixing issues before anyone even submits a ticket but one feature that I plan to get out of this is to show comics' update schedules. I have enough data to tell if a comic updates on, say, Sundays or on Mondays and Fridays. I'll just have to look it up and display it.