Piperka blog

September updates

Looks like I got a bunch of new users just after I used a good part of my last blog post to complain about the lack thereof. I suppose someone went out to scout for them. I didn't quite mean it like that. I'll say that I'm grateful yet embarassed about this. I hope I'm doing something well enough with Piperka to keep you around.

I don't have any particular new feature to announce at this time so this is a grab bag of smaller things. 1st Sep update resulted in a considerable speed up with page loads. When the user is logged in, each one of them computed the total page count from the updates table for several comics potentially multiple times each time. It was pretty smart about using database indexes but it all still added up. Adding the expected updates table slowed the main page refresh further and I had a look at what I could do about it. I found a nice way to cache those results. I had tried to do a similar thing with the new backend from the start but that lead to some database deadlocks and didn't yield that much of a benefit after all so I rolled back it. Looks like I got it right this time. Faster page loads is a yay.

The mobile site version's been the default for mobile users for a while now. It's seen some slight adjustments since, thanks to everyone who gave me feedback on it. On a less cheerful note I noticed that I had broken logging in with the mobile app in June only on 5th Sep. I wouldn't want to find surprises like this but what's done is done. I had made a change and didn't test it properly. Nobody's sent me a message about it so I only noticed it myself later than what I'd like. But it works now again.

I've expanded the update frequency estimates to consider monthly schedules as well. As of this writing, 670 comics have a weekly schedule recognized, 46 have weekly update counts and 120 have monthly counts. I had a user contact me about this so I'll have a word about what they're about. I'm trying to figure out their current rates of updates, not what they've done over their whole run. If a comic has averaged to 3 pages each month over the years but has done 4 recently then that's what I show. If they, against all odds, start to update each monday at 5 AM then that's what I want to capture.

I've added some filtering options to the per user recommendations. I thought that newer comics are underrepresented in the results so I added the option to show only results among the latest 200 / 500 / 1000 / 2000 added comics. If you don't feel like sifting through all the newest additions then this may be of help. Even though someone still has to do that for them to show up as a recommendation at all.

I updated the crawler today to try one page backwards if the initial load of the last known page results an error or a redirect. Webtoons has been prone to getting stuck due changes to archive pages, I hope that this helps with it. It's a longer term goal to add more smarts to the crawler but this is something at least. It'd be cleaner to have all the compensatory actions and regular crawls separated and I've begun some work towards that but this latest change was added as a part of the regular crawls since it was simpler to do it that way at this point and this fixes some comics' updates now.

I've had some unnecessary personal drama going on lately and it's been a distraction. I'm hoping that I can set it aside soon enough and get to concentrate better on Piperka and other things I'd care more about.

Mon, 23 Sep 2019 16:19:31 UTC

Mobile site

I made a mobile version of the web site. It's basically just the same as the traditional version, but stretched out and enlarged, and with the side bar replaced by a top bar with a button to open a menu. The font size's larger with it. I didn't make it the default for anyone yet, even though I see from the web server logs that I have plenty of mobile users. I'm going to ask for feedback before I do so. I'll make sure that there's always the option to switch between the layouts seamlessly. There's a link on the side bar for switching to the mobile version if you're using a mobile device.

It took surprisingly little effort to make the alternate layout. I needed just a few conditionals to the backend code and a few more alternate cases to the templates and just a bit of JavaScript. Most of it is done with CSS. In retrospect I should have done this a lot earlier. Before going through the trouble of making an app, for certain.

Nobody's requested for a mobile layout yet. It's not something I'd see myself using, I like to stick to my desktop computer and laptop for my own web comic consumption. I know Google's penalizing Piperka in the search rankings since it's not mobile friendly, as they measure it. I hope I didn't end up implementing this just to placate Googlebot. Let me know what you think! The list size in the mobile layout is fixed at 18 elements and it's likely to only work sensibly in portrait mode. All the external links open in new tabs in mobile mode. I'll make changes to the layout yet, this isn't going to be the final version of it.

Another feature I implemented (based on a request) was to add personal blacklists. There's a button on the comics' info pages now to mark that you're not interested in that comic. It'll render them differently in the comic lists with a blacked out subscribe/unsubscribe button. It won't directly affect the results on the recommendations page but it'll filter out blacklisted comics. I hope it's useful for those cases where you've already decided that you don't care about a comic and end up running into it again and again, regardless. Rejected comics still aren't marked on the related comics list on info pages as those are currently statically generated and that'd need to change first before I could make user specific changes to their display code.

I have been making other changes to the listing code lately and it was easy to add this feature to it now. I'm not considering adding ratings for comics anytime soon but this is more of a feature for users to classify the comics for their own consideration.

I've been thinking about Piperka's user base. Piperka's peaked in 2015 and has been in a slow decline ever since. For much of that time frame, Piperka didn't see any new developments and the database was filling with cruft as I was busy reimplementing the backend. During the late 2018 I started to see some real benefit from having a new code base and got many comics' updates running again and cleaned up a ton of dead comics from the lists. Piperka's in a better shape than ever but I couldn't really tell the difference if I looked at my user stats. I suppose I would have bled more users if it was still stuck in the state it was in 2015. There's always going to be some rate of attrition but the rate it's getting new users at is quite anemic. Currently, Piperka gets something like a dozen new users each month and I'm lucky if a quarter of them sticks as active users.

I know Piperka's a niche site and it's going to remain as such. I reckon that my users are generally happy with what I'm doing. Some of them take the time to tell me so and I'm grateful to you for the encouragement. Still, what I'd really like to see as a validation for the effort I'm putting towards Piperka would be to see an inflection point in my user stats.

I suppose I could try advertising. At one level it's just about informing people about things they might care about. The people who never even hear about Piperka are never going to use it either. If I told you that My Life at War is a well drawn comic about a war (heh) between an ultra capitalist society where everything is about investments, property and debts to be paid and the other side is a hereditary nobility. That it isn't just about the war itself but also about how the different societies have a hard time even understanding each other's motives. That it's got mechas with diesel engines as well. If I wrote that on my development blog and you didn't even know about the comic and it would be something that's match your interest then you might start reading it. But who'd go and mention Piperka anywhere. Not many web comic sites even link back to Piperka.

I'd still like to think that marketing and advertising is needless and SEO is a three letter word, that by just running the best site I can people would naturally recognise it and come. That success would follow from having me just doing what I do best. And it's mostly worked for Piperka, so far. Up to a point. I suppose I should take a step from my lofty heights and get some dirt on my feet.

This is my blog, I'm entitled to ramble on it. I know I'm not entitled to gain hundreds of new users just due to putting effort towards something. For all they care, I could be digging a ditch and filling it again, every day. It takes effort even though nobody benefits from it.

Wed, 28 Aug 2019 18:46:26 UTC

Hiatus classifications

My last blog post was about describing comics that are updating. This time around, I'm talking about comics that don't update. Roughly speaking, more than half of the comics listed on Piperka don't see any updates. Thankfully there's much fewer of those that are due to an error on Piperka's end. Comics that have simply disappeared have been retired to the graveyard but there's still a lot left over. Comics that have been finished or abandoned but which are still online. I think both updating and non-updating comics have their place on Piperka but sometimes the trouble is telling them apart. I had already used the internal crawler update score to mark updating comics with a white corner on the listing pages but I'd like to go further with this.

I'm now classifying non-updating comics with a reason for their dormancy. I'm not even trying to make the crawler try to tell apart whether a comic has been completed or abandoned. Instead, it's a moderator action. It'll take a while until all of them have been looked at, of course.

My next plan is to make the comic listings have a bit more information on them besides just the comic name. I'll try out different layouts to see what seems sensible. Update or non-update status and perhaps the page and readers counts. I'll add the option for compact display. One goal with this change is to make it clearer to any new visitors to the site what Piperka is about: Updates and archives rather than just being one giant list of comic titles.

I made an update to the revert page. It's using the same listing code as the other listings now, with pagination and columns. The new listing also shows the time since the redirect was last used. Hopefully it's more usable now. The revert page used to show only the redirects done during the last 24 hours but that clean up got dropped by accident the last time I changed hosts a year and half ago and I decided to leave it as it is now.

Now that I got into touching the listing code again, I implemented another idea I've had for a while. As a shout-out to comic authors, I wouldn't be doing this without you guys. Support Comics is a user specific page which lists those comics they are subscribed to which have a Patreon or Ko-fi pages recorded for them.

Mon, 12 Aug 2019 16:36:24 UTC

Inferred schedules

Piperka's main page is kind of useless. I've never had any good idea about what to put there and it's only ever had a short introductory message along with a count of the amount of comics I've personally subscribed to. Yet it's what many well established users use as their landing page, either via search engines or directly, only to immediately click away from there to the updates page. Not anymore. I streamlined it and updates.html page is now gone and what was there is now shown on the main page for logged in users. Going to the old updates page will now redirect. Adjust your bookmarks if you will, even though I see that many of you still use a HTTP URL for accessing Piperka.

I've done a bit of log analysis to determine comics' weekly update schedules, whenever they have a stable enough one to tell. I haven't yet used the inferred schedules as an input for the hourly crawler run, it would be nice to have it ping on archives more often when there's an expectation of new content and let them rest at other times. There's no guarantee yet that the crawler would check on the comic when Piperka says that the update should be out. Once I get that part done I'd expect the accuracy of reported update schedules to increase as well.

As of this writing, out of 5326 active listed comics on Piperka 1313 had any updates during the 12 weeks of log history I'm using as data and 668 had schedules I could infer with my algorithm. I didn't use much time at all to try to optimize it so there likely is still room for making it more accurate. I encountered and reported a bug in GHC along the way.

The inferred update schedules are now a part of comics' info pages. If they have updates during the time period but the algorithm failed then there's a message about inconclusive update schedule. If there's no message about a schedule on it then it had no updates during that time.

A more prominent use for the inferred schedules is that they're displayed on the updates page for the comics a user is subscribed to but is up to date with. I suppose this is the most visible change to the part that people use the most on Piperka in quite a while. I'm not one to do any usability studies with the changes I do to my site. Most often I just come up with an idea and if I like it well enough for my own use then I go through with implementing it. Let me know if you don't like seeing Piperka's guesses about upcoming updates and would like to keep the updates page simpler then I can see about adding the option to not show them.

On a personal note, I'm on my annual vacation. Which means that I have a few weeks to give more attention to Piperka. I'm hoping to finish up with a feature or two.

Tue, 30 Jul 2019 15:18:51 UTC

Piperka migrated, part 2

I'm writing this one so that I might remember it better: Piperka's backend process runs as a systemd user service. The default mode for them is to shut down when the last user session closes. That means that everything is fine as long as I'm logged in but it shuts down down when the last ssh connection closes. The command to change services to persist is loginctl enable-linger which will make Piperka run even after I close the laptop lid. At least I had the fortune of having an SSH open from another computer until I did a reboot this morning to change the hostname. I had almost two hours of outage due to this today but I suppose I got lucky since I didn't cause any more of that. Sorry about this, this was a pure user error and I'll try to not make the same mistake the next time around. I should write some documentation about the process or even automate it somehow, this certainly wasn't the last server migration I'll be doing.

I'm not the first user of my new IP address. I don't mean to reflect badly on my new hosting provider but looks like the last holder of the address gathered some bad reputation. I was told that an anti-malware site was flagging Piperka after the migration. I contacted them and they told me that they had blacklisted my IP. I'm hoping that they'll clear the status soon if they haven't done it already.

Also, Google is rejecting all email from piperka.net at the moment, citing that my IP has sent bulk email before. I added DKIM, SPF and DMARC (email security and anti-phishing features) for my domain and I hope that that's enough to clear that one up. At least my IP isn't included in any of the public anti-spam blacklists.

Hopefully I'm not seeing any other issues like these. I'll just resolve them and it's not worth it to consider requesting a new IP over this or even to switch providers this soon. It's just frustrating that these aren't anything that I have direct control over.

Here's to hoping that I'll have a dreadfully uneventful and boring time with my new hosting and can concentrate on some feature development again.

Addendum: This one slipped my mind. I knocked out the crawler for more than half a day on Thursday. It happened at a bit inopportune time for me. Something in the new environment didn't agree with one of the parsers and that aborted the whole run. Sorry about that.

Fri, 21 Jun 2019 17:37:37 UTC