Going Cloudy Part 6 - Monitoring and Load Balancing

The Monitoring Project

When using the traffic manager, it needs an endpoint on your service to hit in order to determine whether or not it is responding. This endpoint must be open (no authentication) and must be a path on your service. As you may have noticed, in the ServiceDefinition, I instructed my Site, Web Service and REST Services to only respond on port 80 or 443 to a specific host header, one which the traffic manager cannot provide as it will be accessing the instance directly, in other words via it’s cloudapp.net address.

The simple way to solve this would be to make one of the services also respond on an endpoint without a Host Header and give it an unauthenticated ActionResult somewehere that the Traffic Manager could access.
Now I’m no security expert, but I do my best, and I didn’t want any of my core services hosting an unauthenticated endpoint and I want to make sure that they are only accessible by their public urls. Therefore, I created a project that is just for monitoring the health of my services. Initially, this will just service the traffic manager. In the long term, it’s a convenient place to put any generic monitoring functionality.
In order to service the Traffic Manager, it just has a GET action on the Home controller which does a quick database connectivity check and returns a 200 if everything is okay. Before go live, this will be extended to check all the main services are responding.

Using the Azure Traffic Manager

As previously mentioned, I need to have service instances in both the EU and the US. I didn’t want users to have to decide which one they went to by going to eu.domain.com or us.domain.com, that’s just a bad user experience in my book.
The Azure Traffic Manager provides you with a load balancer that you can use between Cloud Services in the same or different regions. I am using it in the performance mode, which routes the user to the nearest (and presumably fastest) service to them.
The Traffic Manager uses the aforementioned Monitoring project to determine if the service is unavailable. It regularly hits the health endpoint and if it does not receive a 200 within 5 seconds, it considers the instance to be down. When the instance is determined to be down, on the next DNS refresh, the records will be updated to point to the next service on the list. There will still be some down time while all this happens, but it will most likely only be a couple of minutes.

Breathing new life in to old netbooks

You don't see many netbooks around these days, and for a very good reason. When they were new, their performance ranged from ok to rubbish and they were no good for any real computing. Looking back, they were more like the first stab at the sweet spot between a smart phone and a full-on laptop/desktop, a gap which has been much more successfully filled with tablets in recent years.

I have 2 netbooks, one is a Samsung NC10 and the other a Dell Mini 9. Both of these have an Intel Atom N270 @1.60 Ghz and 1GB of RAM.
The Dell ran XP (badly) and the Samsung has run XP (badly), Windows 7 (really badly), and Ubuntu (just about acceptable).

When Windows 8 was released, one of the things that I noticed the most was the very noticeable improvements in general performance. Some of this was clearly down to the removal of superfluous effects but that could not explain all of the improvement

Just after Windows 8 came out, I installed it on the Dell. Performance was much better than I expected, better than XP. The Dell's odd screen size meant some hacking to get the screen in to 1024x768, even then there is a small amount of blur. It works okay for some basic note taking and browsing, but I wouldn't stress it with anything further.

Recently, the other half graduated to using an ASUS MemoPad for her needs, leaving the Dell spare. I decided to install Windows 8 on this too and, surprisingly considering it has exactly the same processor and amount of ram as the Dell, it performs even better. Haven't put Office on yet so it could still go downhill, but so far I am very impressed.

So if you've got some old netbooks lying around that you had written off as being useless for anything, throw Windows 8 on them you may be surprised at how nippy they are as a result.

Making old machines immortal(-ish) with P2V

The time comes to every PC, when it's reached the end of it's life and it's time to be turned off once and for all...Except, when that PC has old software on it with a non-transferable license. In an ideal world this wouldn't happen, but sometimes it just can't be helped - either the software is no longer available to buy and transferring to an equivalent would cost a bomb, or you'd have to buy a new license for the most recent version which would also cost a bomb.

Fortunately, this is where virtualisation becomes really useful for a small business. While small businesses may not have workloads that warrant massive clusters of VMs spanning multiple centuply redundant clusters on fault tolerant blade servers, they will almost certainly at some point experience the imminent failure of that machine - the one machine in the entire company that simply must remain. It cannot be upgraded and it cannot be replaced, it must exist forever.

In short, P2V allows you to take an existing OS install on physical hardware and convert it to run as a virtual machine on a virtual host. It's easy to do, so easy in fact that I'm not going to tell you how to do it. Some detailed instructions for performing a P2V conversion can be found here
http://www.petri.co.il/virtual_convert_physical_machines_to_virtual_machines_with_vmware_converter.htm

Some tips when you're doing this:

1. Install the converter on the machine you want to convert - I've found the agent can sometimes be difficult to get to work, if you want to get things done quickly, just install on the machine and then uninstall from the converted VM when you're done.
2. Make sure you change the disks to Thin Provision - this will save you disk space on your virtual server. Space may not be an issue for companies with SANs, but if you're just using the drives in the server box, it doesn't hurt to save every gigabyte you can.
3. Make sure you get the number of CPUs right - I didn't do this on the first XP machine I converted, the physical hardware had 1 core and the virtualised version 2. I ended up in a tricky situation where I had to reactivate XP but couldn't because it couldn't connect to the activation service. Eventually, after numerous attempts, it connected and I could reactivate, but if I'd set the cores I may not have had to.

Virtualisation for me takes between 2 and 3 hours, after which I have a virtualised copy of the hardware machine. As a matter of course I go through a few steps here.

1. Create a snapshot before doing anything.
2. Uninstall VMWare converter.
3. A bit of general cleanup. Take out unnecessary items from services and startup, drop the visual effects (Classic mode for XP). This will make remoting a more pleasant experience as there won't be so many differing colors to transfer over the wire and render on the client side.
4. Almost forgot this one. On your physical PC, you probably have various software installed for the graphics card (ATI Catalyst Control Center, etc), network adapters, and any number of esoteric peripherals. The majority of these you will no longer need so remove them and save your new virtual machine some work.

Take a snapshot after each of these steps just in case you get anything wrong. 10 seconds to create a snapshot could save you 2-3 hours starting all over again.

Going Cloudy Part 5 - Scalability without breaking the bank

As detailed previously, the application consists of three major components:

• The web site, which predictably is used by everyone.
• The asmx web service which is used by about 80% of users but performs fairly low-resource actions.
• The REST service is hit only by external systems importing/retrieving data on a schedule, so it doesn’t need a lot of resources either.

 Going with the default cloud approach of separating everything would be prohibitively expensive. Currently a 2 instance cloud service is about £18 a month. Add on the monitoring project and this means 4 cloud services, giving me an £80 quid bill every month on servers whose utilisation would be pretty poor.

The smart thing to do is merge them all on to a single cloud service. It’s important that merging multiple sites/services on a single cloud service is only done where appropriate. Putting lots of services on the same instance when doing so will result in a shortage of resources defeats the point of moving to Cloud hosting. In this instance, they are all low-resource services which for the foreseeable future will happily sit on the same instance without problem. When traffic increases to the point where either of the services needs to be separated out to an instance of its own, this will be easy to do when required. The details of how to merge all of the services in to a single instance are detailed in my previous post.

 During initial deployment, I was getting some really appalling performance on all services, which I initially put down to the traffic manager. Using Remote Desktop to log on to one of the instances revealed that there was a process called VSPerf.exe running which was sapping all of the server’s CPU and a great deal of its RAM. There was one of these for each site initially (so 4) and then another 1 for each site was added whenever I did an upgrade deployment. This resulted in one instance where I had 16 VSPerfs running, it took me nearly 15 minutes to remote on to the server to kill them!

I could only find some anecdotal references to this problem on the internet which revealed no solutions other than to reboot after every deployment. Eventually I worked out that I had inadvertently turned on profiling at some point. I’d accidentally turned this on and didn’t need it, so I turned it off and the problem was solved. I’m sure the majority of users of profiling do not have this issue, but if you turn on profiling and then your performance falls through the floor, take a look at Task Manager and see if you are one of the unlucky few.

New Relic for Azure

Ever since my company moved our main management system to Azure, I've been slowly working on the ability to monitor the application better in order to find the bottle necks and improve the user experience. To date, I've added:

1. Timing on every request so I can query for the slowest pages.
2. Azure Diagnostics
3. StackExchange MiniProfiler to give me some insight in to every request, this has proved extremely useful in tracking down the exact cause of slow pages.

More recently, I've been planning to add Glimpse as well, as I like the insight it gives you in to the MVC ViewEngine and routing systems.

All these tools are great but each one adds an extra element of complexity to the system that is unavoidable, or at least it was until now.

Yesterday, I discovered New Relic. Their .net agent promises to hook in to your application with no code changes and it actually achieves it!
How?
Shamelessly lifted from the New Relic blog, this is how:

Code run by the CLR is considered ‘managed’ code, i.e., the CLR provides a managed environment in which memory object garbage collection and other services are ‘managed’ by the CLR. The Profiler API provides a mechanism for a profiler, such as the New Relic .NET agent, to inject code into whatever managed-code functions it desires. These injected bytes are in the form of MSIL, the .NET assembly language.

Personally, I think this is damn impressive, and as I already mentioned it allows the agent to hook in with no code changes. The agent can be installed on any server, but what impresses me is the simplicity with which this can be added to Azure Cloud Services.

The New Relic blog gives step by step instructions on how to add the agent using Nuget here, however this isn't all that is required. I deployed once and there was no data being reported, logging on to the server revealed that the agent had not been installed.
 I found that while the nuget package added a newrelic.cmd file to install the agent, it didn't add an entry to the cloud service's servicedefinition.csdef, so the script was never getting fired. After a few attempts, I found that the following entry works.

<Task commandLine="newrelic.cmd" executionContext="elevated" taskType="foreground" />

My initial attempt utilised a taskType of background, which meant that the task was processed asynchronously and everything was already initialised by the time the installation had completed - the agent had missed the boat to get it's hooks in.
I contacted New Relic support prior to working out the solution and they suggested that this would help them with a problem they were having with the nuget package (presumably, it was supposed to add it to the servicedefinition.csdef). The knowledge that I have possibly aided in making use of NR even smoother for other users is great, it feels good to contribute.

If you sign up through Azure, you get the Standard account free with a pro trial, all the details are on the aforementioned blog post, link below for those too lazy to scroll back up.
http://blog.newrelic.com/2012/08/21/x-ray-vision-into-your-azure-apps/

Note: I am not affiliated with New Relic in any way other than being a (free) customer and they have not paid me for this post (why would they, no one will read it!), I just think this is a really great tool.

Going Cloudy Part 4 - FTPServer and REST Service configuration

The FTP Server is a Windows Server 2012 Extra Small VM, running IIS for FTP and our in-house import/export agent.

The FTP server is used to exchange data between the application and external services. The reason for FTP is that the main external service we deal with only has that capability. We hope that external service can eventually switch to using JMS which can then be bridged to Service Bus, but for now this is all we have.

For the purposes of resilience and quick recovery after a failure, all files including the FTP folders and the executables for the agent are kept on a separate data disk and all of the configuration needed to get from brand new VM to all systems go is scripted with Powershell.

If the server ever dies, it’s just a case of firing up a new VM, adding the data disk, and running the Powershell script. I also intend to image the OS to give me the reimage option as well. The Powershell script is nothing special, but I intend to publish it for the sake of completeness.

You’ll notice from the diagram that the rest service is being directly accessed from rest.domain.com, bypassing the load balancer. You will also notice that only the EU instance is being used, the US instances are sitting there doing nothing.

There is a good reason for this.

The REST service can theoretically be accessed by any external system that is capable (as long as we grant it permission of course). Some of the imports carried out by the REST service are fairly destructive. If traffic was going through the load balancer, there is the possibility that the REST service in the US could be carrying out an import at the same time as the REST service in the EU, having some very scary consequences.

Now, in an ideal world, the REST service and the underlying data model would’ve been built to deal with this possibility. But it wasn’t, so we have to deal with that. Eventually, I intend for the REST service to simply be a receiver of files, which then puts the received data in to a queue to be picked up by a worker process, or worker processes, which WILL be designed to deal with multiple instances working at the same time without screwing everything up.

Having the entirety traffic go straight to the EU instance means that only one REST service will be taking request at any one time. It also means the US ones are doing nothing, but they use little to no resources if they are not being sent any work to do so I don’t think this is a major problem.

The only way around this would be to have a separate service definition set up for the US and other regions, which it seems to me is unnecessary duplication and extra work.

An alternative set up that I am considering is to have a second traffic manager configured for failover which is there just for the REST service. This would allow failover to the US instances if the EU ones ever became unavailable.

Azure downtime

Azure's storage system took a turn for the worse this weekend, reportedly because they forgot to renew the SSL certificate for the storage services.

Just in the comments section of this article, there are many comments chastising Microsoft for making such an amateur mistake (and rightly so). But there are also many who use this incident as a reason to write off cloud computing as a whole.

Letting a certificate expire is a massive cock-up, and as a customer I fully expect to see a report on the whys,hows, and what we'll do to stop it happening again from MS. However, let's not kid ourselves that cock-ups don't happen when we host and maintain these services ourselves. I work for a small company that couldn't afford to build and maintain equivalents of the Azure services ourselves. While it is frustrating and laughable that mistakes like this happen, at least when they happen on Azure, there is a whole team of highly intelligent well-paid developers and administrators working on solving the problem and preventing it from occurring again. In the on-site hosted premise, there's me. Now I am certain that if we were self-hosting, I'd be able to solve any problem that came my way although it would almost certainly take longer to fix purely due to available man hours, but I don't always have the time and resources to put in the necessary work to prevent it happening again, especially if it is something that is unlikely to recur.

If anything, incidents like this only serve to reinforce the message that cloud computing is not the silver bullet that it is often portrayed to be. It is not a suitable platform for everything nor is it devoid of fault or error. Just as you take the added expensive of the hardware, staffing, and management when hosting on-site, you also need to accept that things are to an extent out of your control when you move to cloud. Either way downtime will happen, and it will happen because someone made a stupid mistake. But at least if you're on Azure, you've got a team far more expensive than many companies could afford there to fix things when they go wrong, and also the resources to put processes and systems in place to prevent idiotic cock-ups like this from occurring again.