New Relic for Azure

Ever since my company moved our main management system to Azure, I've been slowly working on the ability to monitor the application better in order to find the bottle necks and improve the user experience. To date, I've added:

1. Timing on every request so I can query for the slowest pages.
2. Azure Diagnostics
3. StackExchange MiniProfiler to give me some insight in to every request, this has proved extremely useful in tracking down the exact cause of slow pages.

More recently, I've been planning to add Glimpse as well, as I like the insight it gives you in to the MVC ViewEngine and routing systems.

All these tools are great but each one adds an extra element of complexity to the system that is unavoidable, or at least it was until now.

Yesterday, I discovered New Relic. Their .net agent promises to hook in to your application with no code changes and it actually achieves it!
How?
Shamelessly lifted from the New Relic blog, this is how:

Code run by the CLR is considered ‘managed’ code, i.e., the CLR provides a managed environment in which memory object garbage collection and other services are ‘managed’ by the CLR. The Profiler API provides a mechanism for a profiler, such as the New Relic .NET agent, to inject code into whatever managed-code functions it desires. These injected bytes are in the form of MSIL, the .NET assembly language.

Personally, I think this is damn impressive, and as I already mentioned it allows the agent to hook in with no code changes. The agent can be installed on any server, but what impresses me is the simplicity with which this can be added to Azure Cloud Services.

The New Relic blog gives step by step instructions on how to add the agent using Nuget here, however this isn't all that is required. I deployed once and there was no data being reported, logging on to the server revealed that the agent had not been installed.
 I found that while the nuget package added a newrelic.cmd file to install the agent, it didn't add an entry to the cloud service's servicedefinition.csdef, so the script was never getting fired. After a few attempts, I found that the following entry works.

<Task commandLine="newrelic.cmd" executionContext="elevated" taskType="foreground" />

My initial attempt utilised a taskType of background, which meant that the task was processed asynchronously and everything was already initialised by the time the installation had completed - the agent had missed the boat to get it's hooks in.
I contacted New Relic support prior to working out the solution and they suggested that this would help them with a problem they were having with the nuget package (presumably, it was supposed to add it to the servicedefinition.csdef). The knowledge that I have possibly aided in making use of NR even smoother for other users is great, it feels good to contribute.

If you sign up through Azure, you get the Standard account free with a pro trial, all the details are on the aforementioned blog post, link below for those too lazy to scroll back up.
http://blog.newrelic.com/2012/08/21/x-ray-vision-into-your-azure-apps/

Note: I am not affiliated with New Relic in any way other than being a (free) customer and they have not paid me for this post (why would they, no one will read it!), I just think this is a really great tool.

Going Cloudy Part 4 - FTPServer and REST Service configuration

The FTP Server is a Windows Server 2012 Extra Small VM, running IIS for FTP and our in-house import/export agent.

The FTP server is used to exchange data between the application and external services. The reason for FTP is that the main external service we deal with only has that capability. We hope that external service can eventually switch to using JMS which can then be bridged to Service Bus, but for now this is all we have.

For the purposes of resilience and quick recovery after a failure, all files including the FTP folders and the executables for the agent are kept on a separate data disk and all of the configuration needed to get from brand new VM to all systems go is scripted with Powershell.

If the server ever dies, it’s just a case of firing up a new VM, adding the data disk, and running the Powershell script. I also intend to image the OS to give me the reimage option as well. The Powershell script is nothing special, but I intend to publish it for the sake of completeness.

You’ll notice from the diagram that the rest service is being directly accessed from rest.domain.com, bypassing the load balancer. You will also notice that only the EU instance is being used, the US instances are sitting there doing nothing.

There is a good reason for this.

The REST service can theoretically be accessed by any external system that is capable (as long as we grant it permission of course). Some of the imports carried out by the REST service are fairly destructive. If traffic was going through the load balancer, there is the possibility that the REST service in the US could be carrying out an import at the same time as the REST service in the EU, having some very scary consequences.

Now, in an ideal world, the REST service and the underlying data model would’ve been built to deal with this possibility. But it wasn’t, so we have to deal with that. Eventually, I intend for the REST service to simply be a receiver of files, which then puts the received data in to a queue to be picked up by a worker process, or worker processes, which WILL be designed to deal with multiple instances working at the same time without screwing everything up.

Having the entirety traffic go straight to the EU instance means that only one REST service will be taking request at any one time. It also means the US ones are doing nothing, but they use little to no resources if they are not being sent any work to do so I don’t think this is a major problem.

The only way around this would be to have a separate service definition set up for the US and other regions, which it seems to me is unnecessary duplication and extra work.

An alternative set up that I am considering is to have a second traffic manager configured for failover which is there just for the REST service. This would allow failover to the US instances if the EU ones ever became unavailable.

Azure downtime

Azure's storage system took a turn for the worse this weekend, reportedly because they forgot to renew the SSL certificate for the storage services.

Just in the comments section of this article, there are many comments chastising Microsoft for making such an amateur mistake (and rightly so). But there are also many who use this incident as a reason to write off cloud computing as a whole.

Letting a certificate expire is a massive cock-up, and as a customer I fully expect to see a report on the whys,hows, and what we'll do to stop it happening again from MS. However, let's not kid ourselves that cock-ups don't happen when we host and maintain these services ourselves. I work for a small company that couldn't afford to build and maintain equivalents of the Azure services ourselves. While it is frustrating and laughable that mistakes like this happen, at least when they happen on Azure, there is a whole team of highly intelligent well-paid developers and administrators working on solving the problem and preventing it from occurring again. In the on-site hosted premise, there's me. Now I am certain that if we were self-hosting, I'd be able to solve any problem that came my way although it would almost certainly take longer to fix purely due to available man hours, but I don't always have the time and resources to put in the necessary work to prevent it happening again, especially if it is something that is unlikely to recur.

If anything, incidents like this only serve to reinforce the message that cloud computing is not the silver bullet that it is often portrayed to be. It is not a suitable platform for everything nor is it devoid of fault or error. Just as you take the added expensive of the hardware, staffing, and management when hosting on-site, you also need to accept that things are to an extent out of your control when you move to cloud. Either way downtime will happen, and it will happen because someone made a stupid mistake. But at least if you're on Azure, you've got a team far more expensive than many companies could afford there to fix things when they go wrong, and also the resources to put processes and systems in place to prevent idiotic cock-ups like this from occurring again.

Going Cloudy Part 3 - Configuring your endpoints

In my previous post, I outlined the new architecture. You may have noticed how we no longer have urls containing relative paths. This is because all of the components will be hosted on the same web role and I don’t want to have to deal with complicated startup scripts to configure IIS, therefore I want to try and use the Azure-provided methods as much as possible. When deploying multiple sites to a single web role in Azure, there are essentially two choices for setting it up.

• Virtual directories
• Sites

Using virtual directories involves nesting all of the secondary services in the hierarchy of the main web role project (in this case the website). The up side of this is that I could retain my current url structure. The main down side of this is the inheritance of web.config settings. I initially prototyped using this set up and spent a lot of time overriding entries from the main web role config in the web.configs of the web and rest services to prevent missing assembly and conflicting configuration errors.

An example of this is the following snippet from the assemblies section in the web.config of the main website.

          
          <add assembly="System.Web.Helpers, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35, processorArchitecture=MSIL" />
          <add assembly="System.Web.Abstractions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Web.Routing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Web.WebPages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Data.Linq, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />

Looks harmless enough right? All of these assembly imports will get inherited down to the monitoring site’s child projects, the ASMX web service, and the Site and REST service. The latter 2 are also ASP.NET MVC so it’s not a problem. However, the asmx web service has none of the system.web assemblies in it, causing runtime errors post-deployment.

There were other items that caused me problems but this is the most obvious example. In the end, I decided the virtual directory set up was too brittle.

Using sites has the benefit of keeping all the individual applications on the server separate, so a failure or change in one won’t affect another. Because the applications are separate, we can’t use paths to distinguish which application we want when our request hits the server, so instead we use Host Headers.

The Host Header indicates the url that the user followed in order to get to us. By using subdomains for each service, we can tell from the host header which service the user is attempting to access.

The ServiceDefinition.csdef for the project looks like this.

  <WebRole name="domain.Monitoring" vmsize="ExtraSmall">
    <Sites>
      <Site name="Web">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint3" />
        </Bindings>
      </Site>
      <Site name="Site" physicalDirectory="../../../../../AzPublish/Site-Release">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint1" hostHeader="domain.com" />
          <Binding name="Endpoint2" endpointName="Endpoint2" hostHeader="domain.com" />
        </Bindings>
      </Site>
      <Site name="Gateway" physicalDirectory="../../../../../AzPublish/webservice-Release">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint1" hostHeader="webservice.domain.com" />
          <Binding name="Endpoint2" endpointName="Endpoint2" hostHeader="webservice.domain.com" />
        </Bindings>
      </Site>
      <Site name="RESTService" physicalDirectory="../../../../../AzPublish/Rest-Release">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint1" hostHeader="rest.domain.com" />
          <Binding name="Endpoint2" endpointName="Endpoint2" hostHeader="rest.domain.com" />
        </Bindings>
      </Site>
    </Sites>
    <Endpoints>
      <InputEndpoint name="Endpoint1" protocol="http" port="80" />
      <InputEndpoint name="Endpoint2" protocol="https" port="443" certificate="DomainCertificate" />
      <InputEndpoint name="Endpoint3" protocol="http" port="22202" />
    </Endpoints>

The Binding elements within each site define the endpoints that this site can be reached on and the hostHeader attribute maps this application to the url the user followed to get to us.

Notice how the domain.Monitoring project has no domain header. This means that this project will reply on endpoint3 (mapped to port 22202) for all traffic. Having it on a separate port means there is no chance of it ever grabbing traffic from the main projects.

You’ll notice that the Monitoring project is the main project in this deployment (I’ll go in to the monitoring project when I cover the traffic manager). The reason for this is that I had some initial trouble getting the host header set up to work with the Azure Traffic Manager and thought doing this may work. That later turned out to be incorrect but I see no harm in keeping the monitoring project as the main one here. Feel free to let me know if there are some caveats to doing this that I am not aware of.

Only the main project in the web role is compiled when publishing, so the ServiceDefinition needs to be pointed to a location containing the compiled project. In order to achieve this, I use file system deployment to publish the subprojects to the AzPublish folder. It’s not ideal and I intend to get all this scripted in Powershell before we go live. When I do that, I will of course publish the scripts on this blog. Also, if there is a better way, please use the comments.

Going Cloudy Part 2 - Out with the old, in with the new

In my previous post, I briefly went over the current situation and the rationale for making the move to Microsoft’s Azure service.

I also promised details on the current architecture and what I have determined to be the new architecture. I will also attempt to explain how I got from old to new and why I have made the decisions I have made.

The old architecture. 

Here’s a quick diagram:

As stated in my previous post, this server is a single VM running 2GB of RAM and a couple of Ghz CPU. With those resources, for the live environment we are running:

•  ASP.Net MVC Web site.
• ASMX Web Service.
• ASP.Net MVC-based REST service.
• SQL Server database.

This is duplicated in the staging site which is located on the same server. On top of this, there is also SQL Server itself, an FTP Server running through IIS, an in-house import/export agent which processes all incoming and outgoing data.

There are also a number of support tools, there is Linqpad, SSMS,Notepad++, off-site file backup, and a number of other minor services.

What this all means is that, whenever I RDP on to change anything, this creates a notable strain on the system.

Bear in mind that the current service doesn’t have many users, say a dozen or so. This will increase by 10x this year so, clearly, the current setup just will not do.

Key requirements

The new infrastructure has a number of key requirements. Other than the obvious speed, reliability,etc, there is also the following.

Speed must be the same regardless of where the user is.

This means website and web service hosted in both EU and US. This also means two databases which need to be kept in sync.

Needs to respond to rises and falls in demand.

This is what cloud does best. We can add and remove instances easily in a matter of minutes. There is also the possibility of using the Scaling Application Block to drop instances down to having a single server when it is night time in their region, halving the service costs during those times.

The new infrastructure

The new infrastructure is distributed, fairly fault tolerant and should be easy enough to scale - all those cloudy terms that get overused. Over the next few posts, I will go over my reasoning for laying it out this way and problems I have encountered while creating the infrastructure. I hope to do that in approximately this order.

• Changes to URL scheme.
• FTP server and REST service configuration.
• Instance configuration, the Monitoring project and the Azure Traffic Manager.
• SQL Azure and using Data Sync.

Going Cloudy Part 1 - The Beginning

If you believe the hype, "Cloud" is the future and everything will end up there eventually.
It's easy to think that "Cloud" is just another term for deploying your applications and services on virtual servers hosted in someone else's data center, mainstream media make this mistake all the time.

Making full use of the cloud, in my opinion anyway, is architecting and designing your system to take advantage of the availability and scalability that the cloud gives you.

Designing for the cloud is one thing, migrating an existing application to the cloud and taking advantage of all it has to offer, is another entirely. Recently, I have been tasked with planning and carrying out a migration to the cloud.

The application in question is currently hosted on a VM running with just 2Gb of RAM. This VM runs IIS, which hosts 3 individual web-based components of this application. It also hosts the database on SQL Server and a number of other additional tools and services. The user base is currently fairly small, limited to a dozen or so users located in the UK, China, USA, and South Africa.

The customer base is expected to expand massively this year, with the number of users from the US pushing over a hundred and a number of users coming on board from other countries around the world.

Clearly, the current server won't handle this, and Cloud is the way forward for this customer.

Amazon or Azure?

Amazon has a myriad of services which fill different niches and markets, but don't seem very joined up. In my opinion, while Amazon's cloud is much more fully featured and mature than Azure, it is a big bucket of disparate services trying to be everything to everyone.
Azure on the other hand, while being newer and not as feature complete as AWS, does what it does very well. All of Azure's services are aimed at allowing developers to build scalable and highly available services, the developer experience is first class and you can get started in minutes.
Microsoft finally recognizes that enticing developers is the key to success, and they have embraced other languages in Azure, creating SDKs for Node, PHP, Java, and Python. Azure is a great platform to develop for, and not just for .net developers.

So Azure it is.

Next time, we'll study the current architecture of the application and look at how it needs to change in order to make the move and meet the customer's requirements.

Disclaimer

I'm not a cloud expert so don't take what I am doing as best practice. This series is intended to help others learn from my experiences. If something can be done better, please use the comments to impart your wisdom.

My experience of Windows 8

I think it's fair to say that Windows 8 has been the most controversial and negatively received OS since Vista. Every mention of Windows 8 invokes cries about the Metro UI and how the OS is a schizophrenic nightmare. Occasionally, amongst these voices will be a quiet statement of "it's not that bad once you get used to it".

That little voice isn't far wrong.

That's not to say that after using Windows 8 for a while that I have grown to love the Metro UI and think it's the best thing since sliced bread. No, it's just that I've learned to ignore it to the extent that I barely notice it is there.
Worth noting that, despite it allegedly only being a code name, I will be using the term Metro repeatedly in this post, it's just easier to type than Windows 8 Store app.

Start screen

It wasn't until I had used Windows 8 for a few weeks that I realized how little I used the old Start Menu. Since the ability to pin applications to the taskbar was added in Windows 7 and the addition of a fantastic little application called Bins which allows me to have 4 apps in one 'slot' of the taskbar, everything I need is in the taskbar. This relegates the Start Menu to two things - Search and shutdown.

Search

One thing that Microsoft have gotten right in Windows 8 is the search experience, which is superior to that of previous Windows versions. I get a big full page list of search results which covers files, settings, and applications and also the ability to search things like eBay from the same screen (not a feature I see myself using, but a nice touch anyway).

Shutdown

Shutdown/Restart is the biggest thing that has frustrated me about Windows 8. There are a couple of ways to do it:
1. When using a mouse, hover the mouse over a mystical hotspot in the bottom right hand corner, click on Settings, click on power, and then select from shutdown/restart/update. This is such a massive pain in the ass when using a mouse that it is not even really an option.
2. Same as above but the charm bar is invoked using Win+C instead of the mystical hotspot.
3. Alt + F4 when on the Desktop - the preferred option and not a problem for a keyboard/mouse user, but absolutely impossible if you were using a tablet, which is supposed to be Metro's Raison d'être.
 My wife, who is fairly technically literate, could not figure out how to turn it off. I had to Google it, expecting that there would be some ridiculously easy method that I was completely overlooking. There was not. In order to make things easy for myself, I have added a shortcut to shutdown.exe on the Metro start screen so I have the options of one-click shutdown if I'm in Metro or Alt+F4 if I'm in classic desktop. Can't see your average Joe user coming up with this solution though and this will be one of the biggest transition headaches in my opinion.

Metro applications

The other biggest issue with Windows 8 is Metro apps. I work with Windows and C# on a daily basis and I must admit that I have found it really difficult trying to work with Windows' new paradigm. Local database usage is incredibly difficult, with portable libraries not allowing you to use some of the standard data libraries. Microsoft has tried really hard with their samples, but some of them fall short, such as the settings sample which shows how to use the settings flyout but does not show how to retrieve or save settings the "Metro" way.
 The expectation is very much that Metro apps will be consuming cloud data sources and so will not need local database access. But cloud does not work for everything, such as business applications or little apps I write to make my life easier. Microsoft have limited the usefulness of Metro apps by not doing the extra leg work to make the development experience equal or superior to writings a standard Windows app. For something so radical to be picked up, they need to make the community want to use it, not try to force them to do it a certain way and remove the tools they are used to using unnecessarily.

From my perspective, I have almost completely ignored the Metro apps as the vast majority are useless to me. I suspect that, unless major improvements are made to the support for standard .net libraries in Windows apps, that Metro apps will be relegated to time-killing games and media consumption, something that the Metro UI is perfect for.

Those are the biggest issues Windows 8 has which, for your average user, are absolutely massive hurdles to overcome and is the reason many businesses will be completely skipping Windows 8 and hoping that a future service pack or Windows 9 corrects some of Windows 8's mistakes.

But of the 3 main issues described above (Start screen, shutdown, Metro apps), 2 are largely ignorable and the other is easily worked around.

But it's not all bad...

Those not withstanding, Windows 8 has some plus points.

Upgrade experience

I upgraded from Windows 98 to ME, and regretted it for weeks after until I finally bit the bullet and did a complete reinstall. The upgrade experience from Windows 7 to 8 was as smooth as it could be. The upgrade adviser told me which applications would be incompatible ahead of time and uninstalled them for me as part of the upgrade, the upgrade itself was paid for without ever leaving the upgrade application.
Post-upgrade, everything worked and the only issue I have had is Linqpad not showing up as an installed application in the search screen which was easily solved by a quick re-install.

Faster than Windows 7

 Owing largely to the removal of superfluous visual effects but I suspect there has also been some serious optimization work done under the hood. The improved Task Manager is also a nice touch.

Hyper-V built in

 I haven't made massive use of this yet owing to having an almost-1 year old in the house and generally having less time to tinker than I used to, but when I have done some work requiring a VM, it has been more than adequate. My virtual server hasn't been fired up in almost a year now.

Great upgrade offer

I haven't paid for a Microsoft OS in a very long time, apart from as part of a new machine purchase. Despite the controversy over Windows 8, the cheap upgrade price of £15 and the possibility of being able to write applications and publish them through a store that would gain my applications exposure and possibly even make me a few quid, was enough for me to bite. The ecosystem possibility seems a little less likely now, but I still think the upgrade was worth it.