Magpie Analytics

my new analytics tool

04.04.2020 - 21:20
As mentioned in previous blogposts my sister has been working on her own website recently and I have been helping her with a few backend solutions. To get some insight on how many people were visiting her site, she wanted to use Google Analytics, but couldn't get it to work easily and she also had some privacy concerns. And while I am very certain that I could get it to run in a few minutes, anyone who knows me, knows that I like to create my own solutions. Years ago I had made my own analytics tool, which I have been using ever since. However, it has become quite outdated since then. It's not very detailed, isn't easy to use and is php based and therefore slowing down the page loading ever so slightly. Because I would have needed to redo 95% of it anyway, I started from scratch with a new and improved analytics tool. Unlike my game, finding a name was farily easy: The design of my sisters website resembles a half-timbered house and I like birds, so I just chose Magpie, a very smart and analytical bird common to rural areas.

Unlike the pure php based old analytics tool, Magpie only uses php on the backend and is completely JavaScript based on the frontend. This has a few advantages, it can be used on servers without php, is easier to use and most importantly it doesn't slow down page loading. While looking into the async keyword in script tags, I found the defer keyword which is even more suitable for my needs. A script tag with defer is loaded asynchronously like async, but the execution is delayed until the page loading is complete. Other than that there isn't really anything special about the JavaScript, it's just an ajax request to the server, which handles the rest.

Another improvement of Magpie is that it supports multiple users. In the old system there would have been no way to have my sister use it as well. Like most of my user based online tools, Magpie uses PingPoliNet for user authorization. Every user can add multiple websites in Magpie. Every website has its own unique token, which is passed in the ajax request. It doesn't need to be secret, because it is visible in the source code of the website anyway.



When somebody visits the website, Magpie creates a new unique visitor. Multiple hits from that visitor within a certain time period are just counted as normal hits. The same visitor is counted as a new unique visitor again after the time period has passed. At the end of the day a cronjob accumulates all daily hits/unique hits and creates a daily statistic. Magpie also offers IP anonymization, which may or may not be required or encouraged by different local laws. Germany requires IP anonymization when using Google Analytics, therefore I probably need to anonymize IPs in my own tool as well. For anonymization I use the same method that google uses: the last octet of IPv4 addresses and the last 80 bits of IPv6 addresses are set to 0 at the earliest possible moment. This may aggregate two unique visitors into one if one has 123.123.123.123 and the other has 123.123.123.178, because both of those would be anonymized to 123.123.123.0, but the chance of that happening is probably rather low.





A completely new feature to Magpie are subpages. Seeing how many unique visitors and hits a website gets is nice and all, but I wanted a little bit more information about what the visitor does. Therefore I added subpage analytics. Subpages can either be created manually, for example home, about or contact, but I didn't want to have to create a new subpage for every new blogpost I write by hand. So I added a convenient feature to automatically create subpages if Magpie doesn't find a match. With subpage analytics turned on, clicking on a visitor shows which subpages were visited and how often. Total subpage hits are also accumulated every day, but at the moment there are no daily subpage statistics, I may add those later.



When I first changed pingpoli.de over to Magpie it didn't work, because the pagespeed apache mod was messing up the JavaScript. Because the content of the JavaScript was quite short pagespeed automatically inlined it and the onload event for inline JavaScript tags does not happen and therefore Magpie was never executed. After a lot of searching I couldn't find a solution to turn off JavaScript inlining only for specific tags, but keep it on for the others, I don't think there is a pagespeed option to achieve this. But after a discussion and a bit of help from a friend we came to the conclusion that I could just turn off pagespeed all together, because it didn't really do much for my website anyway, it removed the whitespaces from my stylesheet, but that's about it.

---