Posts tagged with fail

Debian have released the next stable version code named “BookWorm”, and so it is time to look at upgrading some of my servers. Now I have form here for things going smoothly. So I started with the server that I use to host a nextcloud instance. I mean what’s the worst that could happen? I’ve only set that server to hold my contacts, my calendar, to sync important files, to automatically backup any photos I take with my phone. It’s not like there is anything important I couldn’t afford to lose.

It’s been a couple of months since I posted anything, and honestly that’s more down to me being lazy and having nothing worth posting to talk about. But Qualys have updated their SSL Labs to make the use of TLS 1.1 and TLS 1.0 cap the score to a B. This has lead me to once again look at the TLS settings for the various servers that I run. Apache HTTPD is the easiest of server applications that I run to update for this so that’s where I decided to start. Rather than just updating to TLS version 1.2 only it would be nice to use TLS 1.3 as well. However I was mostly running Debian Stretch and out of the box the version of Apache HTTPD included in this release doesn’t support TLS 1.3. Fortunately for me Debian Buster has been out a while, and I have always found Debian upgrades to go smoothly so I decided to run a dist-upgrade, and update the TLS settings on Apache HTTPD.

So the SSL certificate that I used to secure my website (and other things) is no longer trusted by Chrome (as of version 57), and so I have been forced to upgrade to a Lets Encrypt SSL certificate. It's almost as if I could have predicted this state of affairs in advance. At least I can now rest assured that my SSL certs will be easy to keep up to date (I have set up what I believe to be the required automated steps to do just that, time will tell).

So recently I changed my mail server over to postfix, this not that long after I ranted about e-mail security with plusnet. It turns out this has led to an interesting problem. So SSL settings were set pretty strictly on the smtps port, so only strong TLSv1.2 ciphers were available. On the smtp port I was a little more permissive, as long as it was TLSv1.2 it would accept even very weak ciphers (well a weak cipher is better than no cipher at all, and I was accepting mail that didn't use the starttls command) and everything was good. Accept that it turns out emails coming from Plusnet's mail servers was failing, they would connect, try to starttls, not like any of the ciphers and fail, breaking the connection. Once they tried again they didn't remember that starttls didn't work so they tried again. Until the mail timed out, and was bounced. So I've had to make the setting even more permissive, as getting emails from people I know on plusnet (like my Father for example) is sort of important.

So, not very long ago I renewed the SSL certs for my website, I was happy with the changes that StartCom made to their free SSL certificate offering at the time. It turns out, however, that I should start looking at finding an alternative as StartCom are apparently being put on the naughty step. At least Let's Encrypt is up and running now. I'm also looking at changing my e-mail server, but more on that another time (maybe).

So, I run My own server. It hosts this blog. It also, amongst other things, hosts my e-mail, and a local network share. (I know, I should use separate servers, but I do use containerisation to keep a modicum of separation)

To ensure the integrity of the data I use a software raid array, 4 disks in a raid 6 set, it's not a backup, and I should know better, but it has served me well enough. There have been a few disk failures, and I've not lost any data (at least none I care about enough to look at regularly enough to know it's gone) through any of them. One of those disk failures I put in a new disk, but it was slow, and had occasional read errors. Annoyingly these prevented me from installing the grub boot loader on that disk. But that's ok, there were three more disks, it's not a major problem. Critically however, it also prevented me installing grub on any other disks. And so begins our tale of fail. Since that disk has gone into the array there have been disk failures, I can't be certain of the number (disks don't fail in easy to identify patterns) but I can be certain that it is more than two. At least one more than two. Because the server I have doesn't support hot swapping drives, rebuilding the array requiires a restart of the system. Restating the system requires a working boot loader. The last of the disks with a working boot loader failed recently. This left me with a system that wouldn't boot, and installing grub wouldn't work with the slightly faulty disk in the system. I was left with a system that I couldn't repair without putting the integrity of my data at risk, or a long wait for the array to rebuild using a boot disk (knoppix as it happens. I highly recommend having a copy available to anyone who does any sort of computer support). I chose the latter. So it takes a long time to rebuild 3TBs of data onto a shiny new disk. And so my website, my blog, my emails too, have been offline for a long time. I have now replaced the slightly faulty drive, as well as the failed drive. The array is rebuilding (again) onto the newest drive. I have ordered enough disks to have a spare on hand. And I have learnt a lot about the grub-install command's modules flag. I have also now got the motivation to not only fix the technical debt that caused me to not have a server at home for three days, but also the technical debt that means I'm hosting a server at home, and not on a hosting service (I know what I'm doing this weekend).

So, I use Apache Roller as the application my blog runs on. A new version of this has come out (I was on 5.0 and 5.1 has been released) so I decided to upgrade.

This has resulted in the theme I was using breaking, badly, so I have had to move to the basic theme. I can't be bothered to tweak that right now, but I don't like it much either, so I am going to have too eventually. It also appears to have broken rss feeds, such that if you do follow my blog with an rss reader you get all my blog entries again (or at least it does in tiny tiny rss) so sorry about that.

Worse than all that however is the fact that I decided to take this opportunity to update to openjdk-7 (from openjdk-6) and tomcat 7 (from tomcat 6). The server this blog is running on used to be Debian 6, but was dist-upgraded to Debian 7 (which went terribly smoothly at the time) and the older versions of java and tomcat were left over from that. This process was far more laborious than it should have been, largely due to me having forgotten all the steps I had taken to get Roller working on tomcat 6 in the first place (the java upgrade was painless mind, so I did that bit right at least).

It's a good job I'm not getting paid for looking after this server, I'm apparently not doing a very good job of it.

So, my web server was a little out of date, running Debian Squeeze. The HeartBleed vulnerabilty was a bit of a wake up call to get up to date (despite the fact I wasn't vunerable to it because of the server being out of date), and I decided to do a Dist-upgrade.

This went reasonably smoothly, except my customised qmail install isn't allowing me to send email (or more specifically it is, but then generating an error) so I'll need to fix that (I am getting mail though, so not too urgent)

It also broke mysql, and I hadn't taken the time to take backups of the databases before hand! (BIG mistake)

Fortunately I was able to fix that without any data loss.

I then Discovered that it had broken my blog. The data appeared to be in the database, but the blog wouldn't load. I tried taking a backup and reinstalling the blog. This Did not help much. Although that then pointed me in the right direction. The update had removed the mysql connector that I had linked into the tomcat lib folder. So I fixed it by linking the new mysql connector that had been installed as part of the upgrade.

The Biggest fail here being the lack of backups, or contingency planning, particularly given that this is what I do for a living!

OOPS.

So following on from my previous entry, where I decided I was going to build a cycle computer as an electronics project, to learn something new, I have purchased a RaspberryPi and some sensors to play with. (I purchased the stuff I thought I'd need from Pimoroni, a sheffield based company who give a proportion of their profits to the RaspberryPi foundation)

 So, I have started to have a look at what I can do.

 First things first, the RaspberryPi was bought with an SD card with Noobs pre-installed. I used this to install Rasbian (a port of my favourite Linux distribution optimised for the RaspberryPi). This was embarrassingly easy, or at least would have been if the small USB keyboard I was using wasn't faulty¹. Easy solution I'll get a new one this weekend (probably).

Next the sensors, these came with little header pins that needed soldering to the boards (if not soldering wires directly to them, which I am not doing as I am using a breadboard to try things out before fixing things in stone) and I have learnt, I either need a hotter soldering iron and a considerable amount of practise, or a seriously massive amount more practise at soldering.

So in conclusion I need to spend more time on thinking about this stuff, and I need to spend more money.

¹ This was not bought from pimoroni, but an old keyboard I had lying around.