Posts tagged with sysadmin - page 4

So recently I changed my mail server over to postfix, this not that long after I ranted about e-mail security with plusnet. It turns out this has led to an interesting problem. So SSL settings were set pretty strictly on the smtps port, so only strong TLSv1.2 ciphers were available. On the smtp port I was a little more permissive, as long as it was TLSv1.2 it would accept even very weak ciphers (well a weak cipher is better than no cipher at all, and I was accepting mail that didn't use the starttls command) and everything was good. Accept that it turns out emails coming from Plusnet's mail servers was failing, they would connect, try to starttls, not like any of the ciphers and fail, breaking the connection. Once they tried again they didn't remember that starttls didn't work so they tried again. Until the mail timed out, and was bounced. So I've had to make the setting even more permissive, as getting emails from people I know on plusnet (like my Father for example) is sort of important.

Not so long ago I suggested I may change my mail server software. I have recently done so, moving from a highly customised qmail installation to postfix. I have done so for a number of reasons, but that is not to say I dislike qmail.

What did I get out of qmail?

• Easy to configure, all the configuration was done using flat files, named for their purpose, there was no monolithic and confusing config file to search through
• Highly customisable, I had applied many patches, and made alterations to my specific installation that served my needs
• multi process mail system, this one mattered to me, and is why when I switched I switched to postfix, there is no single binary running as root, that does everything, each process runs with the privileges it needs.

So why did I want to change?

Well qmail, and specifically my installation, had become unwieldy to add new functionality to, I wanted to add greylisting, and there were many ways to do this, but they all required adding yet another patch, and out of laziness I had not committed all my changes to any sort of source control. I couldn't stomach manually going through another patch and seeing where it didn't apply cleanly and why, and fix it again. So I had a choice stomach the pain of another round of patching, rebuilding, and testing, and make things worse for myself, set up source control for my qmail set up, or move to something better supported in the community, and with more features.

Postfix suited my needs reasonably well, it is a multi-process mail system, using the idea of least privilege, it has a modular design allowing the addition of extra features much more quickly and easily. It is also better supported, and even has pakages within debian, my operating system of choice. Greylisting was added easily by simply installing another package (postgrey) and altering the config of postfix to use it. By setting up postfix to allow access over ssl on port 465 (as I had previously on qmail) it has also enabled opportunistic encryption for any mail servers sending email to me (something I had considered adding to qmail, but had decided wasn't worth the effort) and I have also been able to easily enable opportunistic encryption for when my server sends email out to other servers that support it.

So do I regret using qmail in the past? Not at all I learnt a great deal from using qmail, and I still prefer it as a basic mail system to postfix, it was just becoming too much hassle to support new features.

What mail server would I advise others to use? For the most part I would suggest google apps or office365 if you want your own domain, or any of a number of other paid for mail hosting solutions, very few poeple have the time and skills and patience to run their own mail server. It started as a learning exercise for me, and I like the control I have over my set up. If someone genuinely wanted to run their own mail server my advise would be to find out what suits their needs best, qmail is great if your needs are simple, and is relatively easy to learn if you have some basic knowledge of how networks and specifically email work, but everyone has different needs, and those needs can change over time, my certainly have.

So, not very long ago I renewed the SSL certs for my website, I was happy with the changes that StartCom made to their free SSL certificate offering at the time. It turns out, however, that I should start looking at finding an alternative as StartCom are apparently being put on the naughty step. At least Let's Encrypt is up and running now. I'm also looking at changing my e-mail server, but more on that another time (maybe).

So, about a year ago I renewed my SSL certificates, and I was using StartSSL as my certificate provider, because they were free, if a little awkward to use. One of the limitations they placed on the free certs is that they could only be valid for a year. At the time I was interested to see what would become of Let's Encrypt as it promised not only free certificates, but a much easier way to get, and manage those certificates. They went live in April this year. I have been considering setting up my cert through Let's Encrypt, and renewing my SSL certificate was the perfect opportunity to do so, however, I have not got myself into a possition to fully automate the renewal of all the places I use my SSL certificate, so while it is still a manual process, and I got the reminder from StartSSL I figured why not give them another go.

So, I run My own server. It hosts this blog. It also, amongst other things, hosts my e-mail, and a local network share. (I know, I should use separate servers, but I do use containerisation to keep a modicum of separation)

To ensure the integrity of the data I use a software raid array, 4 disks in a raid 6 set, it's not a backup, and I should know better, but it has served me well enough. There have been a few disk failures, and I've not lost any data (at least none I care about enough to look at regularly enough to know it's gone) through any of them. One of those disk failures I put in a new disk, but it was slow, and had occasional read errors. Annoyingly these prevented me from installing the grub boot loader on that disk. But that's ok, there were three more disks, it's not a major problem. Critically however, it also prevented me installing grub on any other disks. And so begins our tale of fail. Since that disk has gone into the array there have been disk failures, I can't be certain of the number (disks don't fail in easy to identify patterns) but I can be certain that it is more than two. At least one more than two. Because the server I have doesn't support hot swapping drives, rebuilding the array requiires a restart of the system. Restating the system requires a working boot loader. The last of the disks with a working boot loader failed recently. This left me with a system that wouldn't boot, and installing grub wouldn't work with the slightly faulty disk in the system. I was left with a system that I couldn't repair without putting the integrity of my data at risk, or a long wait for the array to rebuild using a boot disk (knoppix as it happens. I highly recommend having a copy available to anyone who does any sort of computer support). I chose the latter. So it takes a long time to rebuild 3TBs of data onto a shiny new disk. And so my website, my blog, my emails too, have been offline for a long time. I have now replaced the slightly faulty drive, as well as the failed drive. The array is rebuilding (again) onto the newest drive. I have ordered enough disks to have a spare on hand. And I have learnt a lot about the grub-install command's modules flag. I have also now got the motivation to not only fix the technical debt that caused me to not have a server at home for three days, but also the technical debt that means I'm hosting a server at home, and not on a hosting service (I know what I'm doing this weekend).

So, some time ago I had to admit that I needed TLSv1, well time marches on, and I started to look at SSL settings again (largely because my SSL certificate expired, and I needed to replace it, so why not review the SSL settings).

So, I use Apache Roller as the application my blog runs on. A new version of this has come out (I was on 5.0 and 5.1 has been released) so I decided to upgrade.

This has resulted in the theme I was using breaking, badly, so I have had to move to the basic theme. I can't be bothered to tweak that right now, but I don't like it much either, so I am going to have too eventually. It also appears to have broken rss feeds, such that if you do follow my blog with an rss reader you get all my blog entries again (or at least it does in tiny tiny rss) so sorry about that.

Worse than all that however is the fact that I decided to take this opportunity to update to openjdk-7 (from openjdk-6) and tomcat 7 (from tomcat 6). The server this blog is running on used to be Debian 6, but was dist-upgraded to Debian 7 (which went terribly smoothly at the time) and the older versions of java and tomcat were left over from that. This process was far more laborious than it should have been, largely due to me having forgotten all the steps I had taken to get Roller working on tomcat 6 in the first place (the java upgrade was painless mind, so I did that bit right at least).

It's a good job I'm not getting paid for looking after this server, I'm apparently not doing a very good job of it.

So following my work on fixing CVE-2014-4566 on my website, it turned out that I do indeed need to use lower versions of TLS than 1.2 a revelation that is a little embarrassing to admit. So I have been doing a little playing with the settings, and have tweaked the cipher suite to support TLSv1 TLSv1.1 and TLSv1.2 and only ciphers with forward secrecy.

So another day, another web security vulnerability. Once again a problem on the internet has prompted me to fix something on my home server, in this case the SSLv3 vulnerabilty that has been given the name "POODLE" (seriously who comes up with these names) and it has reminded me that the SSL settings on my server are woefully inadequate.

Given my site is just a personal site I figure there is no real reason to stay with SSLv3 as I don't much care about IE6 users. In fact, the stuff I use it for supports TLSv1.2 so I may as well stick to that, and the older protocols be damned. This does break a large number of older, and mobile clients. But that is their problem.

It's also a good time to play with different cipher suite orders. So I've removed all but those that support forward secrecy (again, this will break stuff, but not the stuff I use so I don't much care).

Obviously the choices I have made here are made in the absence of any pragmatic need to support legacy systems, but that is the beauty of having a personal site rather than a commercial one.

So, my web server was a little out of date, running Debian Squeeze. The HeartBleed vulnerabilty was a bit of a wake up call to get up to date (despite the fact I wasn't vunerable to it because of the server being out of date), and I decided to do a Dist-upgrade.

This went reasonably smoothly, except my customised qmail install isn't allowing me to send email (or more specifically it is, but then generating an error) so I'll need to fix that (I am getting mail though, so not too urgent)

It also broke mysql, and I hadn't taken the time to take backups of the databases before hand! (BIG mistake)

Fortunately I was able to fix that without any data loss.

I then Discovered that it had broken my blog. The data appeared to be in the database, but the blog wouldn't load. I tried taking a backup and reinstalling the blog. This Did not help much. Although that then pointed me in the right direction. The update had removed the mysql connector that I had linked into the tomcat lib folder. So I fixed it by linking the new mysql connector that had been installed as part of the upgrade.

The Biggest fail here being the lack of backups, or contingency planning, particularly given that this is what I do for a living!

OOPS.