Posts tagged with oops

I have always been sceptical of the new generic top level domains, I saw them as ICANN shamelessly cashing in on something it had the power to control. Because of this I have until now avoided them. However my current domain name is quite long, and I have for a long time wanted something shorter, but the good ones that may be applicable to me have all been taken.

But the time has come to admit that the new generic top level domains are here to stay, so I have swallowed my pride, and found that most of the good names are gone already anyway. But one was available that was suitable, so I have registered stewart.zone. I'm going to use it to set up a website that isn't hosted on my home connection. For this I'm going to use the hosted virtual servers I already have for my mail servers, but that is going to need me to set up a backup process for them, as they will no longer just be mail servers, so rather than trust my ability to reconfigure a new server from scratch I'm going to trust in my ability to back up the configurations in a sensible way, and save myself the trouble of having to manually rebuild their configurations if they go wrong.

Also this will give me an opportunity to build a website that isn't quite so ugly, and also isn't lumbered with some of the "features" of my current site that I haven't had the heart to do away with, but are a bit rubbish. Once this is done successfully I'll look at migrating my current site over to the new hosts, and the new design, and then I won't need to open up firewall rules on my router any more.

So I've had my mail servers set up and working for a month now, and there are a few things I haven't done. My old mail server is still set to send from a domain of craig-james-stewart.co.uk  by default, and it is no longer in the SPF record as a sender for that domain, so I have had to fix that so that I can continue to receive emails from it seamlessly. I've also had to alter the contact form on my website for the same reason. As well as these minor tweeks I have come to the realisation that I ignored time drift when setting up the mail servers, easily corrected by installing ntpd in it's default configuration on debian, apart from my rather strict iptables rules. So having fixed that, the only thing left to do, is configure certbot to auto-renew my ssl certificates, which is as simple as adding a couple of cron entries. So now I have two mail servers that will continue to work, with little maintenance effort. I still need to look at DKIM and DMARC, but those can wait.

So after my last blog post I decided that this one should be less rushed, and more practised and tested, which turns out to be a good thing. After my last blog post the hosted servers I have didn't work over IPv6, this is due to the hosting firm's use of SLAAC to configure the external IPv6 address and routing, and my use of iptables to block all traffic that wasn't otherwise allowed. Now I allowed icmp echo requests on IPv4 but those commands raised an error run I transposed them to IPv6 so I left them out. This led to SLAAC, which requires ICMP to work over IPv6 to not work. That has been rectified now. So onto apache, and SSL certs. Now one of the requirements I had for these servers was the ability to swap between them via DNS, and as I do not know how to configure postfix to use multiple SSL certs based upon the domain that is being connected too I decided the easiest way to do that would be to get a cert with a cname to that shared domain for each server. Using http authentication with lets encrypt you put a file on disk and they request that file from the domain they are validating. This would be a problem for the server that is not currently being pointed at for the shared domain.

I knew I relied heavily on the access that I get by carrying around a smart phone, I read my emails, send and receive text messages, look things up on the internet, etc. It is a very useful tool, but I didn't realise how much of an impact it has on my life until it died. So I immediately ordered a replacement, but that took two whole days to arrive. The fix for my old phone was outside my ability level, and I couldn't find anyone who could fix it in a hurry, so I elected to spend two days without it, or any other phone. And it has been two of the least stressful days I have had in a very long time. Disconnected from the world, I didn't need to worry about things I could do nothing about, or keep abreast of the latest goings on on twitter. I almost regret getting a new phone, the old one is off for repair, and will be back in two to four weeks. I can hardly imagine spending that much time without a smartphone. Before the old one broke it was because I thought I would become overly stressed, and fail to cope, but having spent two days with only having the internet when tethered to a desk I can't imagine how relaxed I may get without a smartphone. Maybe in future I shall turn the phone off for periods of time. Maybe too much connectivity is a bad thing?

So, I run My own server. It hosts this blog. It also, amongst other things, hosts my e-mail, and a local network share. (I know, I should use separate servers, but I do use containerisation to keep a modicum of separation)

To ensure the integrity of the data I use a software raid array, 4 disks in a raid 6 set, it's not a backup, and I should know better, but it has served me well enough. There have been a few disk failures, and I've not lost any data (at least none I care about enough to look at regularly enough to know it's gone) through any of them. One of those disk failures I put in a new disk, but it was slow, and had occasional read errors. Annoyingly these prevented me from installing the grub boot loader on that disk. But that's ok, there were three more disks, it's not a major problem. Critically however, it also prevented me installing grub on any other disks. And so begins our tale of fail. Since that disk has gone into the array there have been disk failures, I can't be certain of the number (disks don't fail in easy to identify patterns) but I can be certain that it is more than two. At least one more than two. Because the server I have doesn't support hot swapping drives, rebuilding the array requiires a restart of the system. Restating the system requires a working boot loader. The last of the disks with a working boot loader failed recently. This left me with a system that wouldn't boot, and installing grub wouldn't work with the slightly faulty disk in the system. I was left with a system that I couldn't repair without putting the integrity of my data at risk, or a long wait for the array to rebuild using a boot disk (knoppix as it happens. I highly recommend having a copy available to anyone who does any sort of computer support). I chose the latter. So it takes a long time to rebuild 3TBs of data onto a shiny new disk. And so my website, my blog, my emails too, have been offline for a long time. I have now replaced the slightly faulty drive, as well as the failed drive. The array is rebuilding (again) onto the newest drive. I have ordered enough disks to have a spare on hand. And I have learnt a lot about the grub-install command's modules flag. I have also now got the motivation to not only fix the technical debt that caused me to not have a server at home for three days, but also the technical debt that means I'm hosting a server at home, and not on a hosting service (I know what I'm doing this weekend).

So, I use Apache Roller as the application my blog runs on. A new version of this has come out (I was on 5.0 and 5.1 has been released) so I decided to upgrade.

This has resulted in the theme I was using breaking, badly, so I have had to move to the basic theme. I can't be bothered to tweak that right now, but I don't like it much either, so I am going to have too eventually. It also appears to have broken rss feeds, such that if you do follow my blog with an rss reader you get all my blog entries again (or at least it does in tiny tiny rss) so sorry about that.

Worse than all that however is the fact that I decided to take this opportunity to update to openjdk-7 (from openjdk-6) and tomcat 7 (from tomcat 6). The server this blog is running on used to be Debian 6, but was dist-upgraded to Debian 7 (which went terribly smoothly at the time) and the older versions of java and tomcat were left over from that. This process was far more laborious than it should have been, largely due to me having forgotten all the steps I had taken to get Roller working on tomcat 6 in the first place (the java upgrade was painless mind, so I did that bit right at least).

It's a good job I'm not getting paid for looking after this server, I'm apparently not doing a very good job of it.

So, my web server was a little out of date, running Debian Squeeze. The HeartBleed vulnerabilty was a bit of a wake up call to get up to date (despite the fact I wasn't vunerable to it because of the server being out of date), and I decided to do a Dist-upgrade.

This went reasonably smoothly, except my customised qmail install isn't allowing me to send email (or more specifically it is, but then generating an error) so I'll need to fix that (I am getting mail though, so not too urgent)

It also broke mysql, and I hadn't taken the time to take backups of the databases before hand! (BIG mistake)

Fortunately I was able to fix that without any data loss.

I then Discovered that it had broken my blog. The data appeared to be in the database, but the blog wouldn't load. I tried taking a backup and reinstalling the blog. This Did not help much. Although that then pointed me in the right direction. The update had removed the mysql connector that I had linked into the tomcat lib folder. So I fixed it by linking the new mysql connector that had been installed as part of the upgrade.

The Biggest fail here being the lack of backups, or contingency planning, particularly given that this is what I do for a living!

OOPS.