Piperka blog

OAuth2 logins

I've implemented OAuth2 logins for Piperka. If you've seen a "Sign in with Google" or somesuch link on a site, then Piperka has one now too.

I made it possible to have a Piperka account completely without a password, which made the changes needed a bit more intrusive than what it would have been without that. I kept the password protected section on account page as is, but now it is optionally OAuth2 protected.

The plan was to roll out an initial version with a choice of reddit or Google as identity providers, but the Google one didn't turn out to work on the production server. I suspect that it's because it tries to use IPv6 addresses there and they never worked after I set up bridging for a virtual server on the same host. I'll figure that out some other day. So the only option besides a plain old password, for now, is login with reddit.

The update for this change didn't go quite as smoothly as I'd liked to. Piperka was down for an hour or so, sorry for the inconvenience. Please test things and let me know if you see anything broken or unexpected.

I've been working on this on and off for several months now. Right now, I feel pretty burnt out regarding Piperka. I don't think I'll be pushing for any major changes until doing so would feel fun again. As you know, Piperka is a single man effort and I have a day job on top of that. I'd love to get others involved with the development and maintenance of Piperka, beyond asking me to fix crawlers, but I feel that getting that happen would require even more work than just doing whatever I do myself, whenever I feel like it.

I think I have griped about the exact same thing before on this blog. But you'll just have to excuse it. I have a month's worth of comics on the submit queue and a number of reports about broken crawlers. I'll work through those in the nearby future. After that, I'll do something fun, which most likely wouldn't involve Piperka.

submit to reddit
Sun, 08 Mar 2015 16:50:40 UTC

Whither Piperka

Let me tell you a story. There was a stone and a pot. And everyone was well fed by the stone soup. I may have skipped over a bit.

Let me talk about motivation, that is, mine, and let me talk about Piperka. I've been running it for over eight years now, which is a long time for doing anything. In many ways, it remains the same that it was from the start, the basic idea was there from the beginning and much of what I have been doing since has been auxiliary stuff or refinements. Besides the ever-so-present need for fixing crawler issues and adding a few thousand comics.

My motivation matters since I'm the only person behind Piperka and nothing much happens unless I do it. And it's not anything as simple as turning a profit, either. I had aspirations of commercial success when I started doing it, but that idea has been laid aside along the years. It's been a great way to practise my coding skills and accumulate proof of thereof. It has helped me get jobs. I can use it to explore techniques that I wouldn't have the opportunity for at my day job. I've been doing it for fun, and out of curiosity to see how people would react, and sometimes in anticipation of positive reviews.

It's been a fun ride, but right now I'm in a bit of an impasse. I've seen less progress last year than what I'd like. I changed jobs a year ago (to a much nicer one) and moved to a city where I can have a social life, again. Both of which have made me to neglect Piperka. I've had less time and when I've had it, I haven't necessarily felt like touching Piperka. Yet I'm not about to abandon Piperka, if only because I use it myself and I couldn't imagine reading web comics without, or with someone else's software. It's frustrating, seeing how Piperka could be better but feeling like I'm being the bottleneck, preventing that from happening.

I've tried. Build and they come, they say. I released Piperka's source code when it became feasible and I convinced myself to take the leap, inviting people to participate, and that's gone nowhere. One thing that I should have done already back then would be to arrange some other point of communication besides emailing me, personally. I've set up a public development mailing list, feel free to join to say what you'd like to see fixed or improved, or just follow along. A blog is fine for (sadly, infrequent) announcements but I hope that a mailing list would lend itself to a more open discussion.

So far, Piperka's story hasn't been that of the stone soup. It's been all me. I could have tried to commercialize it, but that's not the route that I wanted to try to take. I could have made look all serious and rename it as something less whimsical, as something that actually had something to do with web comics, but I didn't want to. Piperka's quirky and odd and so personal that there are days when I feel like taking it down just to avoid having others look at it, and I like it that way. I'm not sure how trying to get others to participate in running a web site would even work. I can release the source code but there will still be only one Piperka, which I control. That's something I can't even try to meaningfully change before the support would be there.

I'll say a few words about Piperka's crawler, comic updates and the database, to give a picture of what's involved in trying to share the burden of maintenance. It's about the plumbing, the hidden part which becomes visible only when it doesn't work properly. I take users' privacy seriously and that alone means that I'm not giving direct access to the database to anyone. I could, in principle, strip it of user data and make the crawler code contained in there available for people, in hopes that I would receive help in maintaining it. I'm not going to do it at this time. First off, I need to be convinced first that releasing the web site's source code was a good idea, even, before I release even more. I don't want to do it only to see nobody care. Maintaining it is unglorious janitorial work which even I find too much like work at times, and I don't expect people to jump in for that. Beyond that, there's the same problem I had with regard to the source code itself. If I made a dump of the data and someone improved on it, then there would be no easy way for someone to send me the fix, nor for me to accept the changes. It doesn't live in a version control system but in a database, and the web comics' archives it tries to reflect is quite an ephemeral bunch. Beyond that, it's not just a matter of fixing the comic archive index, but there are ways that the data is tied to users' settings. For example, when a page needs to be removed from Piperka's index, all the bookmarks need to be bumped down by one page to reflect the change. One future development plan I have is to allow users make annotations on a comic's archive page and that'd only make it more necessary to consider user data when updating the comic data.

What I'd really need to improve on Piperka's crawler would be a better user interface for the maintenance work. I do it all currently on a shell on the server, calling scripts to invoke the crawler or update the database directly via SQL queries. It works and updating a comic, on a comic per comic basis, doesn't usually take all that long. But it could be improved on, and all that effort does add up. I would have saved the time used for that many times over, had I used the time to build a better user interface for the work. There would be another benefit, too, if I had the interface: It would enable me to hand over parts of the maintenance to others.

Ultimately, this is a question of what kind of a site Piperka is. In a sense, its purpose is to stay out of the way. Nobody visits Piperka to see Piperka itself, but to get to read the web comics. But how does it do that, how does it present itself to people and who's working on it behind the scenes. I find myself unsatisfied with how the answers to these stand now, and I'd like to see that change. To put it simply, I want to make running Piperka feel like fun again.

submit to reddit
Mon, 06 Jan 2014 11:22:58 UTC

Piperka Map

Force-directed graph algorithms are fun. Imagine a collection of particles, each repelling each other much like electrical charges do. Add springs between some of the particles, and let them all bounce around until they attain a rest position.

Hence, Piperka Map. I've modelled user data as a force-directed graph and put it available on a map page. The bigger the circle, the more readers it has. Relative closeness means that the comics have common readers. I've deliberately de-emphasized the exact subscriber count from the map. Mouse wheel zooms and the map can be panned by dragging. Clicking a comic opens up controls, where you can open up the comic's info page or, if you are logged in, subscribe to a comic. The quick search dialog is available on the map.

I've added links to the map from comics' info pages and from users' profiles. There's the option of highlighting another user's comic picks. Only comics with readers get a place on the map.

The map viewer code is my own and I haven't paid all that much attention to cross browser compatibility. It's built using SVG, which can require more support from a browser than some other technique. It works best on Chrome, with some glitches on Firefox and I've no idea how IE copes though I'm certain nothing below version 10 would work. The map is a snapshot from today, but I'll enable daily updates later on. I may yet alter the algorithm in ways which would change its layout. This isn't something where there would be some intrinsic right way of doing this, but it's all determined by aesthetics. As for my graph layout program itself, I'm considering releasing it as an independent project.

I suppose Piperka's subscriber data is pretty challenging from a data mining perspective. Most people read xkcd and any algorithm would find that out first. I may yet write about the current related comics algorithm, the one that is still labelled as "experimental". It, too, involves graphs but in a more abstract sense and something alike Gaussian blur from image processing. It's my original work, which rarely is a good endorsement for something like this. I may replace it with something yet, but for now, it stays.

Piperka Map was inspired by Ruslan Enikeev's The Internet map. When I saw it, I got immediately the idea that I had similar data in my hands and wanted to do something similar. With the amount of data I had, I could well do it all with plain old CPU code. Kudos to AMD for my FX-8350 which dutifully crunched numbers during my numerous attempts at getting something sensible looking out. The first thing I did was to make my code run in parallel.

This all doesn't have much to do with catching comics' updates. Then again, Piperka isn't necessarily about that only. It was interesting so I wanted to do it.

submit to reddit
Sat, 26 Oct 2013 17:11:25 UTC

Getting social

I'm introducing a few social networking features to Piperka. Never fear, none of this will affect you unless you choose to use them. I've found myself checking out the comics' lists of readers, to see if can spot familiar names, and I thought that there's a case for making that kind of information more prominently available.

You can now follow other users. That means that comics that they read get hilited (currently, with a small "F" letter) in the listings, and there's a new profile privacy setting, protected, which lets other users see your comic picks once you give them the permission.

When you follow another user, they (and only they) can see a link back to you on their followers page, unless your profile is set as private. This means that with the private setting, you can only follow public profiles. Following is an asymmetrical relation, so being followed doesn't mean that you need to reciprocate. Permissions to view a profile are separate from the followee status. Having a permission but no followee status just means that the profile is visible but it won't be hilited.

Private profiles are as hidden as ever. Trying to see them makes Piperka neither confirm or deny that the user exists. Viewing a protected profile without a permission lets you ask to see their profile. I've made all the existing profiles either private or public and new users will get protected as their default setting.

I'm still going to work on how Piperka lets you access this information and on the presentation, but the basic ideas should be in place already. I'll see first whether anyone else cares to use this feature besides me.

I may do some data mining on the relationship data and show the results publicly. Nothing that would identify any single user. Basically the same thing I do with the comic subscription data, or would do if that part still wasn't pretty underdeveloped. I won't be giving anything sensitive to any third parties and I hope that you already expected that of me.

As an aside, singular they is weird.

submit to reddit
Mon, 15 Jul 2013 20:53:15 UTC

The story so far, 2013

It's been a while. I've purposefully not tried to stick to any update schedule, but I guess I could have written something earlier, already. So far, there have been two larger, yet not all that visible changes to Piperka this year.

But before I go into that, I should tell of an old bug. I redid the account and login handling code two years ago. After that, the new account creation code discarded the user's email instead of storing it. Please go check your account information page and check that you have an email address there. You'd need it for password recovery.

The rest of this post is going to get, unavoidably, more technical. In March, I introduced support for banner uploads. Prior to that, they were submitted as plain old text links. It worked, but it was a rather ungainly way of doing it. The comics have banners submitted more frequently now, so it seems that it solved a problem. I had to clean up quite a bit of old code to make that work, starting with how I was calling Mason. And since I'm using AJAX submits instead of plain old Web 1.0 submits, I had to use File API. I opted for simpler code and just tell older browsers that banner uploads are not supported.

The second, more recent, change is a bit fancier. I made the listing pages (top, browse and profile) navigation use AJAX calls and History API. The (un)subscribe buttons use AJAX now, too. It was remarkably easy to leave the old code path in place for browsers without working History API support. All I needed to do was to not add event hooks to the navigation links and let the old links work as is. By my rough testing, it looks like I made page load time drop by 25% and the page loads won't flash content anymore. I suppose I could further squeeze a pageful of comics to MTU, but that's a project for some other day.

Both changes introduced some bugs that some of you ran into and thankfully told me about. They have been fixed.

I solved a few annoying bits along the way and got to practise some HTML5 techniques. If I had to work on invisible parts of Piperka, I suppose that making the crawler errors easier to manage would have been better used time. But my whim took me here.

If you're interested in even more details than this, then I should remind you that Piperka's source code is available. Of which not all that many people are aware, even those who have contacted me about technical matters. I ought to add a link to a page about the source code to Piperka's templates.

submit to reddit
Mon, 27 May 2013 19:28:04 UTC