Automating repetitive tasks with Azure Functions.

Since the announcement of Azure Functions at Build 2016, I've been looking for an excuse to use them and I finally found it.

Whenever we release a new version of our software, the actual process for building and committing to source control is really simple and takes just a few minutes. Then I get to spend  at least 30 minutes on our internal job management system telling it about the new release and deprecating the old one. I then copy and paste the message from the release commit, enter it in to said system, then reformat it a bit and send it out to various internal users as release notes.

It's just wrong when doing admin after a release takes more time than the actual release, plus I hate doing boring repetitive takes that I know a computer could do for me.

Enter Azure Functions, below is a quick diagram of how I wanted everything to work.

Source control calls a function when we commit. This function determines if it is a release by checking for a specific string in the message, it then puts a message on a queue to another function which will then enter the release details straight in to the database of our internal system (which we host in azure). This system doesn't have an API so we'll do it the old fashioned way with raw SQL.

If this is successful, it then puts messages on to 2 more queues, one is picked up by another function which posts the message in to Slack, the other goes off to a pre-existing web job which will email the release notes out.

SETUP

I created a function app in the azure portal and hooked up a repository in my Bitbucket account to the function app.

I'm going to need to access a SQL database, so I put the connection string in my app settings same as for any other App Service app.

Let's see what the code looks like.

This first function simply takes a small json payload over a http post, this structure of this is as below.

{
    "Revision":"c1b49afddh7c",
 "Author" : "Author Name",
 "Created_At" : "2016-09-27T11:54:58+00:00",
 "Log" : "Updated version to cpm 1.9.40
    
    Added fluffy bunny controller",
 "Branch" : "develop",
    "Project":"cpm",
 
}

#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"

using System;
using System.Net;
using System.Threading.Tasks;
using Newtonsoft.Json;
using Microsoft.WindowsAzure.Storage.Table;

public static async Task<object> Run(HttpRequestMessage req, ICollector<CommitMessageTableEntity> commitLogTable, ICollector<<ommitMessage> releaseQueue, TraceWriter log)
{
    string jsonContent = await req.Content.ReadAsStringAsync();
    log.Info(jsonContent);
    var data = JsonConvert.DeserializeObject<CommitMessage>(jsonContent);
    
    log.Info(data.Log);
    
    var te = new CommitMessageTableEntity();
    te.Set(data);

    try
    {
        commitLogTable.Add(te);
    }
    catch (System.Exception ex)
    {
        log.Info("An error occurred: " + ex.Message);
        return req.CreateResponse(HttpStatusCode.OK, "An error occurred, see log for details.");
    }

    if(te.IsRelease) {
        log.Info("This is a release, adding to queue so it gets added to Radius.");
        releaseQueue.Add(data);
    }

    return req.CreateResponse(HttpStatusCode.OK, "Success");
}

We log this to an Azure Storage table, I've no use for this currently but it costs practically nothing and is an easy way to check if the function was called if I have any problems in the future.

Then if the commit was a release commit, as defined by checking for the commit message to begin in a specific way, we put a message on a storage queue for the next function to be triggered. Note that I'm just adding to ICollector in both cases, the are no explicit references to Tables or Queues which is one of the things I really like about functions.

Here is the function.json, this defines all my inputs and outputs, note how the output queue name shows up as an Icollector<string> parameter on my run method, this is the power of the WebJobs SDK at work.

{
  "bindings": [
    {
      "webHookType": "genericJson",
      "type": "httpTrigger",
      "direction": "in",
      "name": "req"
    },
    {
      "type": "http",
      "direction": "out",
      "name": "res"
    },
    {
      "type": "table",
      "name": "commitLogTable",
      "tableName": "commitlog",
      "connection": "SourceIntegrationSA",
      "direction": "out"
    },
    {
      "name": "releaseQueue",
      "queueName": "releasequeue",
      "connection": "SourceIntegrationSA",
      "type": "queue",
      "direction": "out"
    }
  ],
  "disabled": false
}

The next function is the one that does all the work. There is a large amount of SQL in this function. The key for me was to get this up and running quickly and using SQL does that. Once it has proved it's worth, I'll tidy it up a bit with all the time it saves me!

I haven't included the code here as it is very specific to me, all it does is use Dapper to insert a record for the new release and deprecate the previous one.

To import Dapper from Nuget, I added a project.json, see below.

{
  "frameworks": {
    "net46":{
      "dependencies": {
        "Dapper": "1.50.2"
      }
    }
   }
}

With this in place, whenever I redeploy my function, the Dapper Nuget will be downloaded for me to use.

Right at the end, once everything had been done, we pop a simple message on a queue to be picked up by another function that will send it to slack, and put the release notes from the commit message on another queue to be emailed out to internal staff members who will need to know.

The Slack function is really simple. It uses a Slack client class I found on the internet somewhere and just passes the message along.

UniqueIdentifier as a primary key, that will solve all of our problems!

No, no it won't.

In evolving a single-instance website to multi-instance one, one of the many problems I have faced is how to deal with database access when your website instances are on the opposite side of the world.
My solution to this was to use SQL Azure Data Sync, makes sense as my databases are already in SQL Azure anyway.
Facilitating this involved changing all of the integer primar keys of every table to a different type with a lower possibility of collisions when syncing between databases.

I thought a Guid in the .Net side and a UNIQUEIDENTIFIER in the database would be the perfect fit for this. I was very, very wrong.

While using uniqueidentifiers as PKs has virtually eliminated any possibility of a sync collision, there is a very undesirable side effect of very high index fragmentation, bringing my site crashing to it's knees as soon as the indexes reach a critical level of fragmentation.

As is customary, I decided to run some tests. Below are the results of inserting 10k records in to an empty table:

Ouch! 96% fragmentation vs 18% on 10k records. Now in reality I rarely insert 10k records at the same time, but certain operations involve hundreds and this level of fragmentation will occur over the course of time.

Regardless of what data type your PKs are, fragmentation will happen. But the massive downside of using uniqueidentifier is that this not only happens a lot faster, but also a simple defrag or rebuild indexes is not going to save you as the data is inherently, due to it's random nature, impossible to efficiently index.

My first idea was to use an identity column (in my case called clusterkey) for the clustered index and keep the PK as a Guid with the PK constraint being non-clustered. This would sort out the fragmentation problem. But unfortunately, SQL Azure Data Sync didn't like my clusterkey, I suspect because it is a non-PK identity column. Regardless of the reason, it's not viable, so I looked further.

A contact at Microsoft suggested using the SequentialId() function in SQL Azure (available as of the latest V12 release), but all of my Guids are generated in code, so this was too big a change for me. My colleague Dave came to the rescue by tracking down this article, which describes how to generate reasonably sequential guids in C# that should keep the clustered index happy.

I'll not repeat the contents of the article, which is a really informative read, but it seems to work. This is how my tests look now:

I'll take that, much better fragmentation and approximately the same cost for inserts.

Note: The insert time includes the time taken to generate the Guid/SequentialGuid in code. Also, don't take the fact that SequentialGuid is smaller here as an indication that it is consistently faster. During testing, the number was generally the same as Guids and ints, but was consistently faster as soon as I came to actually measuring it for this blog post!


As these kinds of distributed environments become more common, I suspect more and more people will hit similar issues so hopefully this will help someone avoid making the same mistakes I have.

Getting started with Azure Application Insights

Now that pricing information has been released for Application Insights, I decided to take the dive and deploy it on a few of the web applications I work on. The documentation is pretty good for your basic scenarios, but it glosses over what I think are probably common use cases.

• The instrumentation key (commonly referred to as the iKey) is located in an applicationinsights.config file, which is not very helpful if you want to change it when deploying to Live or Staging environments.
• If you use any monitoring system, such as Traffic Manager, Web Apps AlwaysOn, or any Web Testing application, this all gets included as "real traffic", which is fair enough as AppInsights has no way of knowing that it isn't real traffic. You may want to see it, but I personally do not so I wanted a way to filter it out.

There is a really great blog post by Victor Mushkatin which covers changing where AppInsights looks for the instrumentation key, adding your version number and your own tags to the telemetry so you can filter by version number or any tag you provide. I've added a siteIdentifier tag which contains an id for the particular instance of the application, identifying where in the world it is hosted.
My particular variation of his code adds the SiteIdentifier and the assembly version to the telemetry.

    public class AppInfoApplicationInsightsConfigInitializer : IContextInitializer
    {
        public void Initialize(TelemetryContext context)
        {
            context.Properties["SiteIdentifier"] = System.Configuration.ConfigurationManager.AppSettings["SiteIdentifier"];

            try
            {
                var verAtt = (AssemblyInformationalVersionAttribute)Assembly.GetExecutingAssembly().GetCustomAttributes(typeof(AssemblyInformationalVersionAttribute), false)[0];
                context.Component.Version = verAtt.InformationalVersion;
            }
            catch (Exception)
            {
                context.Component.Version = "Application Version Unknown";
            }
        }
    }

I also have a TelemetryInitializer that finds traffic from load balancers and web testers as explained at the top of this post, and reclassifies it as synthetic traffic, making it easier to exclude from charts and reports. I found that this traffic shows up as a different user every time, making my user count orders of magnitude higher than it should be.

    public class SyntheticSourceInitializer : ITelemetryInitializer
    {
        public void Initialize(Microsoft.ApplicationInsights.Channel.ITelemetry telemetry)
        {
            if (HttpContext.Current == null)
                return;

            //Set traffic manager check and web test request to synthetic.
            if (HttpContext.Current.Request.Url.ToString().EndsWith("/monitoring"))
            {
                telemetry.Context.Operation.SyntheticSource = "AzureAliveCheck";
            }

            //Set Azure Web Apps AlwaysOn pings to synthetic.
            if (HttpContext.Current.Request.UserAgent == "AlwaysOn")
            {
                telemetry.Context.Operation.SyntheticSource = "AzureAliveCheck";
            }
        }
    }

You've got full access to the current request so you can identify the traffic however you need to (referrer, headers, request url, etc)
I then tell AppInsights to use these classes with the below entries in Application_Start() in global.asax.cs

            //Configure application insights.
            TelemetryConfiguration.Active.InstrumentationKey = System.Configuration.ConfigurationManager.AppSettings["iKey"];

            TelemetryConfiguration.Active.ContextInitializers.Add(new AppInfoApplicationInsightsConfigInitializer());
            TelemetryConfiguration.Active.TelemetryInitializers.Add(new SyntheticSourceInitializer());

As I dig further in to Application Insights, if I find more examples of useful overrides for default behaviour, I will add additional blog posts detailing these.

Copying records between tables in different Azure accounts : The Next Generation

A few months back I posted a small piece of code for copying objects between Azure Table Storage Accounts. I have 2 bug-bears with the code I previously posted.

1.  It's all very custom, you need to give it the type that is in the table and add lambdas to make sure it doesn't try to select the wrong object. 
2. Due to a change in the Azure Storage Library, it no longer works. 

 I hadn't used this script in a while but now I have an impending need for something like it but better that will allow me to copy the contents of all the tables or a defined subset of tables in a given account, enter the DynamicTableEntity which allows you to get an entry from a table as a dynamic object.
To run the code, open up Linqpad and add a reference to the Windows Azure Storage library, best way to do this is using Nuget.

void Main()
{
  var srcClient = CreateClient("source account connection string");
  var destClient = CreateClient("destination account connection string");
  
  var mappings = new List<Tuple<string,string>>();
  
  //Manually setup mappings.
  //mappings.Add(new Tuple<string,string>("table1","table1copy"));
  //mappings.Add(new Tuple<string,string>("table2","table2copy"));
  
  //Copy all tables from the src account in to identically named tables in the destination account.
  var tables = srcClient.ListTables(null, new TableRequestOptions(){PayloadFormat = TablePayloadFormat.JsonNoMetadata});
  mappings = tables.Select (t => new Tuple<string,string>(t.Name,t.Name)).ToList();

  Copy(srcClient,destClient,mappings);
}

public void Copy(CloudTableClient src, CloudTableClient dest, List<Tuple<string,string>> mappings) {

  mappings.ForEach(x=>{
    var st = src.GetTableReference(x.Item1);
    var dt = dest.GetTableReference(x.Item2);
    dt.CreateIfNotExists();
    
    var query = new TableQuery<DynamicTableEntity>();

    foreach (var entity in st.ExecuteQuery(query))
    {
      dt.Execute(TableOperation.InsertOrReplace(entity));
    }
  });
}

public CloudTableClient CreateClient(string connString){
  var account = CloudStorageAccount.Parse(connString);
  return account.CreateCloudTableClient();
}

In the above code, we create CloudTableClients to represent the source and destination accounts, then we build a mapping of source and destination tables.
We can do this manually if we only want to copy some tables and/or we want the destination tables to have different names to the source tables.
Alternatively, we can get a list of all the tables from the source and use that to build a 1:1 map, this will have the result of copying all items in all tables from the source account to the destination.
The Copy method simply takes the clients and mapping and does some iteration to get the items from each table from the source account and save them to the destination account.

Note: The code above is horribly inefficient for copying large amounts of data as it inserts each request individually. In a follow up post, I'll make this more efficient by making use of the TableBatchOperation.

API Head-to-head: AWS S3 Vs Windows Azure Table Storage

Recently, I was experimenting with using S3 as a tertiary backup for my photos, an honour which eventually went to Azure as it was cheaper and I am more familiar with the Azure APIs as I use them in my day job.

I thought I’d take a deeper look at both APIs and see how they compare. I’ll go through some standard operations, comparing the amount of code required to perform the operation.

If you want a comparison of features, there are plenty of blog posts on the subject, just Bingle It

All the code in this test is being run in Linqpad, using the AWS SDK for .Net and Windows Azure Storage Nuget packages.

Create the client

Both Azure and S3 have the concept of a client, this represents the service itself and is where you provide credentials for accessing the service.

Azure

var account = Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse("connectionstring");
var client = account.CreateCloudBlobClient();

S3

var client = AWSClientFactory.CreateAmazonS3Client("accessKey", "secret",RegionEndpoint.EUWest1);

S3 wins on lines of code but I don’t like having to declare the datacenter the account is in. In my opinion, the application shouldn’t be aware of this. 1 point to Azure.

Creating a container

This is a folder, Azure refers to is a container, S3 calls it a bucket.

Azure

var container = client.GetContainerReference("test-container");
container.CreateIfNotExists();

S3

try
{         
 client.PutBucket(new PutBucketRequest { BucketName = "my-testing-bucket-123456", UseClientRegion = true});
}
catch (AmazonS3Exception ex)
{
 if(ex.ErrorCode != "BucketAlreadyOwnedByYou") {
  throw;
 }
}

S3 loses big time on simplicity here. To my knowledge, this is the only way to do a blind create of a container, that is creating it without knowing up front if it already exists. Azure makes this trivial with CreateIfNotExists. 2 points to Azure.

Uploading a file

Azure

var container = client.GetContainerReference("test-container");
var blob = container.GetBlockBlobReference("testfile");
blob.UploadFromFile(@"M:\testfile1.txt",FileMode.OpenOrCreate);

S3

var putObjectRequest = new PutObjectRequest {BucketName = "my-testing-bucket-123456", FilePath = @"M:\testfile.txt", Key = "testfile", GenerateMD5Digest = true, Timeout=-1};
var upload = client.PutObject(putObjectRequest);

They’re pretty much equal here, but the S3 code is more verbose. I like the idea of getting a reference to a blob while not knowing if it actually exists or not.

List Blobs

Azure

var container = client.GetContainerReference("test-container");
var blobs = container.ListBlobs(null, true, BlobListingDetails.Metadata);
blobs.OfType().Select (cbb => cbb.Name).Dump();

S3

var listRequest = new ListObjectsRequest(){ BucketName = "my-testing-bucket-123456"};
client.ListObjects(listRequest).S3Objects.Select (so => so.Key).Dump();

In terms of complexity, they’re pretty even here too. Azure has one more line but it’s not a difficult one. Notice that whereas with Azure, we get a reference to a container and then perform operations against that reference, with AWS all requests are individual so you end up having to explicitly tell the client for every operation what the bucket name is. Point to Azure.

Deleting a Blob

Azure

var dblob = container.GetBlockBlobReference("testfile");
dblob.Delete();

S3

var delRequest = new DeleteObjectRequest(){ BucketName = "my-testing-bucket-123456", Key="testfile"};
client.DeleteObject(delRequest);

Neither code is particularly complicated here, but I prefer Azure’s simplicity with the container and blob reference model so point Azure.

Delete a Container

Azure

var container = client.GetContainerReference("test-container");
container.Delete();

S3

var delBucket = new DeleteBucketRequest(){ BucketName = "my-testing-bucket-123456"};
client.DeleteBucket(delBucket);

Again, pretty equal. To micro-analyse the lines, you could say that for Azure, you’ve got one potentially reusable line, and one throw-away line. With S3, they’re both throw away. But in reality, unless you’re doing thousands of consecutive operations, it doesn’t really matter.

Conclusion

In terms of complexity, Azure’s and S3’s APIs are pretty much equal, but it’s easy to see where they each have their uses. Azure’s API is a much thicker abstraction over REST, whereas the S3 API is such a thin-veneer that you could imagine a home-grown API not turning out that differently (but most likely not as reliable).

In my mind, if you’re doing lots of operations against lots of different blobs and containers then S3’s API is more suitable as each operation is self-contained and there are no references to containers or blobs hanging around.

If you’re doing operations which share common elements, such as performing numerous operations on a blob or working with lots of blobs within a few containers, Azure’s API seems better suited as you create the references and then reuse them, reducing the amount of repeated code.

Bonus Section

If you could be bothered to read past my conclusion, congratulations on your determination! The comparative speed of Azure and AWS has been done to death, but I couldn’t resist getting my own stats.

These are ridiculously simple stats, essentially Stopwatch calls wrapped around the code in this post. The file I am uploading is only 6k. The simple reason for this is that everyone tests how these services handle lots of large objects, but no one seems to cover the probably more common scenario of users uploading very small files. The average size is probably higher than 6kb, but this is what I’ve got hanging around so this is what I’m using.

So here are my extremely simple and probably not at all reliable benchmarks.

Operation	S3	Azure
Create Container	573	279
Upload 6Kb file	99	55
List Blobs (1)	41	103
Delete Blob	55	45
Delete Container	221	38

All times are in milliseconds. I’ve got to admit; I was expecting a more even spread here. Azure is significantly faster creating and deleting containers and uploading the file. It is also faster at deleting a blob, but the difference is insignificant. S3 wins significantly listing blobs.

Not covered in this post: Both APIs also have the Begin/End style of async operations and Azure has the bonus of async operations based on the async/await pattern, I may do another post on that in the future.

TL;DR; Azure's API is in my opinion a better abstraction and it's faster for most operations.

Upgraded to Azure Storage Emulator 3.2, where have all my tables gone?

In an attempt to solve a 400 error accessing tables on the Azure Storage Emulator 3.0 today, I upgraded to 3.2 using the Web Platform Installer. This resulted in a kind of good news, bad news situation.

Good - The error stopped happening.
Bad - Where the f**k have all my tables gone!

I'll be buggered if I'm recreating and repopulating them all so I went hunting. I managed to find the emulator database is in C:\Users\<username>\. In that directory you'll find mdf files called WAStorageEmulatorDb**.mdf where ** is the version number. I had ones ending in 22, 30, 32. Each will be accompanied by a _log file.

I loaded them up in Linqpad and the schemas looked the same, so for a punt I just renamed the files ending in 32 to something else and the renamed the files ending in 30 to 32.

Start up the emulator and everything is present again. That saved me a few hours!

Copying records between tables in different Azure accounts

Today I had to quickly throw up a new instance of a customer's service in Hong Kong as they've got a big demo event coming up and want things to be as quick as possible. Now I haven't quite got things to a point where I can have multiple geographically distributed instances of the service all happily talking to each other and sharing data so this instance is it's own little island, a completely separate instance to the main one in the EU.

Deploying the new Cloud Service was easy.
Taking a backup of the EU database and deploying it to Hong Kong was also easy.

However, recently I've been making increasing use of Azure Table storage for trivial data storage scenarios where the data isn't relational and the data will want to be shared amongst multiple instances eventually without having to wait for a database sync. It was at this point that I realised, I have no way of copying data from one storage account to another.
Time to correct that!

public void Transfer<T>(Microsoft.WindowsAzure.Storage.CloudStorageAccount fromAcc, Microsoft.WindowsAzure.Storage.CloudStorageAccount toAcc, string table, Expression<Func<T,bool>> expr) where T: TableServiceEntity {
  
  var fromTC = fromAcc.CreateCloudTableClient();
  var fromT = fromTC.GetTableReference(table);
  
  var toTC = toAcc.CreateCloudTableClient();
  var toT = toTC.GetTableReference(table);
  toT.CreateIfNotExists();
  
  var fromContext = fromTC.GetTableServiceContext();
  var toContext = toTC.GetTableServiceContext();
    
  var fromData = fromContext.CreateQuery<T>(table).Where(expr);

  foreach (var item in fromData)
  {
    toContext.AttachTo(table,item);
    toContext.UpdateObject(item);
  }
  toContext.SaveChangesWithRetries(SaveChangesOptions.ReplaceOnUpdate);
}

This Transfer method takes in a from account and a to account and the name of the table.
The last parameter is an expression for the where clause. This is for scenarios where the same table contains multiple types of objects and you just want to query out the ones of a particular type for transfer using whatever clause is appropriate.

T must derive from TableServiceEntity and be the type of the object from which the record originated, or one that is similarly shaped.

The method is quite straight forward, it just fires up 2 table clients, gets a reference to the table specified by the table parameter, creates it on the receiving end if it doesn't exist (I think it's safe to assume that it already exists at the source end), queries out the data, then attaches it to the destination context and saves changes.
This upserts all of the data in to the source table.

Usage is simple:

var fromAccount = Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=accountname;AccountKey=accesskey");
var toAccount = Microsoft.WindowsAzure.Storage.CloudStorageAccount.Parse("DefaultEndpointsProtocol=https;AccountName=accountname;AccountKey=accesskey");

Transfer<MyProject.MyType1>(fromAccount, toAccount, "sharedtable", p=>p.PartitionKey == "Type1");
Transfer<MyProject.MyType2>(fromAccount, toAccount, "sharedtable", p=>p.PartitionKey == "Type2");
Transfer<MyProject.SomeOtherType>(fromAccount, toAccount, "someothertype", p=>p.PartitionKey != "");

Put all this together in Linqpad and you've got a simple way to transfer records between accounts on an ad hoc basis. As expected, it works with the Storage Emulator so you can use it to clone the contents of a production account down to your local dev machine and vice versa.

A simple WebCache Helper

As mentioned in most of my previous posts, the main project I work on is due to move to Azure in the future. Among the many gems of Azure is their caching infrastructure, which can either be hosted on a dedicated worker role or instructed to use spare memory on your web roles.

More information and pricing for Azure Caching can be found at http://www.windowsazure.com/en-us/services/caching/

I fully intend to make use of Azure’s in-built caching when we get there, but I can’t wait to start implementing some sort of caching and I don’t want to have to do a big find-replace in the code when we do get there, so I wrote a simple WebCacheHelper which provides easy access to caching anywhere in the application but is also easy to replace when I move to Azure.

The code is below.

    public static class 
        WebCacheHelper
    {
        public static T TryGetFromCache<T>(string cacheName, string itemKey) where T:class
        {
            if (HttpContext.Current == null || HttpContext.Current.Session == null) { return null; }
 
            var sessionResult = TryGetFromSessionCache<T>(cacheName, itemKey);
            if (sessionResult != null){return sessionResult;}
 
            var applicationResult = TryGetFromApplicationCache<T>(cacheName, itemKey);
            return applicationResult;
        }
 
        public static T TryGetFromCache<T>(string cacheName, string itemKey,CachingLevel cachingLevel) where T:class
        {
            if (HttpContext.Current == null || HttpContext.Current.Session == null) { return null; }
 
            switch (cachingLevel)
            {
                case CachingLevel.Session:
                    return TryGetFromSessionCache<T>(cacheName, itemKey);
                case CachingLevel.Application:
                    return TryGetFromApplicationCache<T>(cacheName, itemKey);
            }
            return null;
        }
 
        public static void AddToCache(string cacheName, string itemKey, object cacheItem, CachingLevel cachingLevel=CachingLevel.Application)
        {
            if (HttpContext.Current == null || HttpContext.Current.Session == null) { return; }
 
            switch (cachingLevel)
            {
                case CachingLevel.Session: AddToSessionCache(cacheName, itemKey, cacheItem);
                    break;
                case CachingLevel.Application: AddToApplicationCache(cacheName, itemKey, cacheItem);
                    break;
            }
        }
 
        public static void RemoveFromCache(string cacheName, string itemKey)
        {
            if (HttpContext.Current == null || HttpContext.Current.Session == null) { return; }
 
            RemoveFromApplicationCache(cacheName,itemKey);
            RemoveFromSessionCache(cacheName,itemKey);
        }
 
        public static void RemoveFromCache(string cacheName, string itemKey, CachingLevel cachingLevel)
        {
            if (HttpContext.Current == null || HttpContext.Current.Session == null) { return; }
 
            switch (cachingLevel)
            {
                case CachingLevel.Session: RemoveFromSessionCache(cacheName, itemKey);
                    break;
                case CachingLevel.Application: RemoveFromApplicationCache(cacheName, itemKey);
                    break;
            }
        }
 
        private static string GetCacheKey(string cacheName, string itemKey)
        {
            return cacheName + "-" + itemKey;
        }
 
        #region Session Cache
 
        private static void AddToSessionCache(string cacheName, string itemKey, object cacheItem)
        {
            HttpContext.Current.Session[GetCacheKey(cacheName, itemKey)] = cacheItem;
        }
 
        private static void RemoveFromSessionCache(string cacheName, string itemKey)
        {
            var result = HttpContext.Current.Session[GetCacheKey(cacheName, itemKey)];
            if (result != null)
            {
                HttpContext.Current.Session.Remove(GetCacheKey(cacheName, itemKey));
            }
        }
        
        public static T TryGetFromSessionCache<T>(string cacheName, string itemKey) where T:class
        {
            var result = HttpContext.Current.Session[GetCacheKey(cacheName, itemKey)];
            if (result == null)
            {
                return null;
            }
            return (T) result;
        }
 
        #endregion
 
        #region Application Cache
 
        private static void AddToApplicationCache(string cacheName, string itemKey, object cacheItem)
        {
            HttpContext.Current.Application[GetCacheKey(cacheName, itemKey)] = cacheItem;
        }
 
        private static void RemoveFromApplicationCache(string cacheName, string itemKey)
        {
            var result = HttpContext.Current.Application[GetCacheKey(cacheName, itemKey)];
            if (result != null)
            {
                HttpContext.Current.Application.Remove(GetCacheKey(cacheName, itemKey));
            }
        }
 
        public static T TryGetFromApplicationCache<T>(string cacheName, string itemKey) where T:class
        {
            var result = HttpContext.Current.Application[GetCacheKey(cacheName, itemKey)];
            if (result == null)
            {
                return null;
            }
            return (T) result;
        }
 
        #endregion
 
    }

As you can see there is nothing clever going on here, there is a single AddToCache method for adding data to the cache, with a default parameter for cachingLevel so the calling code can override the default application-level caching to cache at the session level.

 There are a couple of RemoveFromCache methods, one where the calling code can specify which cache to remove the item from, the other removes the data from whichever cache it resides in.
 There are also two TryGet methods for retrieving from a specified caching level or from any that has a matching key.

CachingLevel is just an enum with items for Session and Application

     public enum CachingLevel
    {
        Application,
        Session
    }

I also use the following static string class to save having magic strings peppered through the application

    public static class WebCacheKeys    {
        public const string Users = "Users";
    }

I can just add strings as required and change the backing value if I need to without having to make a large number of changes elsewhere. If I had IOC available, I probably wouldn't have made this static and would've abstracted the functionality behind an ICacheHelper interface. The way I look at it, this is the next best thing in terms of ability to make changes in the future. When I move to Azure, I'll post the Azure-centric version of this helper.

Going Cloudy Part 6 - Monitoring and Load Balancing

The Monitoring Project

When using the traffic manager, it needs an endpoint on your service to hit in order to determine whether or not it is responding. This endpoint must be open (no authentication) and must be a path on your service. As you may have noticed, in the ServiceDefinition, I instructed my Site, Web Service and REST Services to only respond on port 80 or 443 to a specific host header, one which the traffic manager cannot provide as it will be accessing the instance directly, in other words via it’s cloudapp.net address.

The simple way to solve this would be to make one of the services also respond on an endpoint without a Host Header and give it an unauthenticated ActionResult somewehere that the Traffic Manager could access.
Now I’m no security expert, but I do my best, and I didn’t want any of my core services hosting an unauthenticated endpoint and I want to make sure that they are only accessible by their public urls. Therefore, I created a project that is just for monitoring the health of my services. Initially, this will just service the traffic manager. In the long term, it’s a convenient place to put any generic monitoring functionality.
In order to service the Traffic Manager, it just has a GET action on the Home controller which does a quick database connectivity check and returns a 200 if everything is okay. Before go live, this will be extended to check all the main services are responding.

Using the Azure Traffic Manager

As previously mentioned, I need to have service instances in both the EU and the US. I didn’t want users to have to decide which one they went to by going to eu.domain.com or us.domain.com, that’s just a bad user experience in my book.
The Azure Traffic Manager provides you with a load balancer that you can use between Cloud Services in the same or different regions. I am using it in the performance mode, which routes the user to the nearest (and presumably fastest) service to them.
The Traffic Manager uses the aforementioned Monitoring project to determine if the service is unavailable. It regularly hits the health endpoint and if it does not receive a 200 within 5 seconds, it considers the instance to be down. When the instance is determined to be down, on the next DNS refresh, the records will be updated to point to the next service on the list. There will still be some down time while all this happens, but it will most likely only be a couple of minutes.

Going Cloudy Part 5 - Scalability without breaking the bank

As detailed previously, the application consists of three major components:

• The web site, which predictably is used by everyone.
• The asmx web service which is used by about 80% of users but performs fairly low-resource actions.
• The REST service is hit only by external systems importing/retrieving data on a schedule, so it doesn’t need a lot of resources either.

 Going with the default cloud approach of separating everything would be prohibitively expensive. Currently a 2 instance cloud service is about £18 a month. Add on the monitoring project and this means 4 cloud services, giving me an £80 quid bill every month on servers whose utilisation would be pretty poor.

The smart thing to do is merge them all on to a single cloud service. It’s important that merging multiple sites/services on a single cloud service is only done where appropriate. Putting lots of services on the same instance when doing so will result in a shortage of resources defeats the point of moving to Cloud hosting. In this instance, they are all low-resource services which for the foreseeable future will happily sit on the same instance without problem. When traffic increases to the point where either of the services needs to be separated out to an instance of its own, this will be easy to do when required. The details of how to merge all of the services in to a single instance are detailed in my previous post.

 During initial deployment, I was getting some really appalling performance on all services, which I initially put down to the traffic manager. Using Remote Desktop to log on to one of the instances revealed that there was a process called VSPerf.exe running which was sapping all of the server’s CPU and a great deal of its RAM. There was one of these for each site initially (so 4) and then another 1 for each site was added whenever I did an upgrade deployment. This resulted in one instance where I had 16 VSPerfs running, it took me nearly 15 minutes to remote on to the server to kill them!

I could only find some anecdotal references to this problem on the internet which revealed no solutions other than to reboot after every deployment. Eventually I worked out that I had inadvertently turned on profiling at some point. I’d accidentally turned this on and didn’t need it, so I turned it off and the problem was solved. I’m sure the majority of users of profiling do not have this issue, but if you turn on profiling and then your performance falls through the floor, take a look at Task Manager and see if you are one of the unlucky few.

New Relic for Azure

Ever since my company moved our main management system to Azure, I've been slowly working on the ability to monitor the application better in order to find the bottle necks and improve the user experience. To date, I've added:

1. Timing on every request so I can query for the slowest pages.
2. Azure Diagnostics
3. StackExchange MiniProfiler to give me some insight in to every request, this has proved extremely useful in tracking down the exact cause of slow pages.

More recently, I've been planning to add Glimpse as well, as I like the insight it gives you in to the MVC ViewEngine and routing systems.

All these tools are great but each one adds an extra element of complexity to the system that is unavoidable, or at least it was until now.

Yesterday, I discovered New Relic. Their .net agent promises to hook in to your application with no code changes and it actually achieves it!
How?
Shamelessly lifted from the New Relic blog, this is how:

Code run by the CLR is considered ‘managed’ code, i.e., the CLR provides a managed environment in which memory object garbage collection and other services are ‘managed’ by the CLR. The Profiler API provides a mechanism for a profiler, such as the New Relic .NET agent, to inject code into whatever managed-code functions it desires. These injected bytes are in the form of MSIL, the .NET assembly language.

Personally, I think this is damn impressive, and as I already mentioned it allows the agent to hook in with no code changes. The agent can be installed on any server, but what impresses me is the simplicity with which this can be added to Azure Cloud Services.

The New Relic blog gives step by step instructions on how to add the agent using Nuget here, however this isn't all that is required. I deployed once and there was no data being reported, logging on to the server revealed that the agent had not been installed.
 I found that while the nuget package added a newrelic.cmd file to install the agent, it didn't add an entry to the cloud service's servicedefinition.csdef, so the script was never getting fired. After a few attempts, I found that the following entry works.

<Task commandLine="newrelic.cmd" executionContext="elevated" taskType="foreground" />

My initial attempt utilised a taskType of background, which meant that the task was processed asynchronously and everything was already initialised by the time the installation had completed - the agent had missed the boat to get it's hooks in.
I contacted New Relic support prior to working out the solution and they suggested that this would help them with a problem they were having with the nuget package (presumably, it was supposed to add it to the servicedefinition.csdef). The knowledge that I have possibly aided in making use of NR even smoother for other users is great, it feels good to contribute.

If you sign up through Azure, you get the Standard account free with a pro trial, all the details are on the aforementioned blog post, link below for those too lazy to scroll back up.
http://blog.newrelic.com/2012/08/21/x-ray-vision-into-your-azure-apps/

Note: I am not affiliated with New Relic in any way other than being a (free) customer and they have not paid me for this post (why would they, no one will read it!), I just think this is a really great tool.

Going Cloudy Part 4 - FTPServer and REST Service configuration

The FTP Server is a Windows Server 2012 Extra Small VM, running IIS for FTP and our in-house import/export agent.

The FTP server is used to exchange data between the application and external services. The reason for FTP is that the main external service we deal with only has that capability. We hope that external service can eventually switch to using JMS which can then be bridged to Service Bus, but for now this is all we have.

For the purposes of resilience and quick recovery after a failure, all files including the FTP folders and the executables for the agent are kept on a separate data disk and all of the configuration needed to get from brand new VM to all systems go is scripted with Powershell.

If the server ever dies, it’s just a case of firing up a new VM, adding the data disk, and running the Powershell script. I also intend to image the OS to give me the reimage option as well. The Powershell script is nothing special, but I intend to publish it for the sake of completeness.

You’ll notice from the diagram that the rest service is being directly accessed from rest.domain.com, bypassing the load balancer. You will also notice that only the EU instance is being used, the US instances are sitting there doing nothing.

There is a good reason for this.

The REST service can theoretically be accessed by any external system that is capable (as long as we grant it permission of course). Some of the imports carried out by the REST service are fairly destructive. If traffic was going through the load balancer, there is the possibility that the REST service in the US could be carrying out an import at the same time as the REST service in the EU, having some very scary consequences.

Now, in an ideal world, the REST service and the underlying data model would’ve been built to deal with this possibility. But it wasn’t, so we have to deal with that. Eventually, I intend for the REST service to simply be a receiver of files, which then puts the received data in to a queue to be picked up by a worker process, or worker processes, which WILL be designed to deal with multiple instances working at the same time without screwing everything up.

Having the entirety traffic go straight to the EU instance means that only one REST service will be taking request at any one time. It also means the US ones are doing nothing, but they use little to no resources if they are not being sent any work to do so I don’t think this is a major problem.

The only way around this would be to have a separate service definition set up for the US and other regions, which it seems to me is unnecessary duplication and extra work.

An alternative set up that I am considering is to have a second traffic manager configured for failover which is there just for the REST service. This would allow failover to the US instances if the EU ones ever became unavailable.

Azure downtime

Azure's storage system took a turn for the worse this weekend, reportedly because they forgot to renew the SSL certificate for the storage services.

Just in the comments section of this article, there are many comments chastising Microsoft for making such an amateur mistake (and rightly so). But there are also many who use this incident as a reason to write off cloud computing as a whole.

Letting a certificate expire is a massive cock-up, and as a customer I fully expect to see a report on the whys,hows, and what we'll do to stop it happening again from MS. However, let's not kid ourselves that cock-ups don't happen when we host and maintain these services ourselves. I work for a small company that couldn't afford to build and maintain equivalents of the Azure services ourselves. While it is frustrating and laughable that mistakes like this happen, at least when they happen on Azure, there is a whole team of highly intelligent well-paid developers and administrators working on solving the problem and preventing it from occurring again. In the on-site hosted premise, there's me. Now I am certain that if we were self-hosting, I'd be able to solve any problem that came my way although it would almost certainly take longer to fix purely due to available man hours, but I don't always have the time and resources to put in the necessary work to prevent it happening again, especially if it is something that is unlikely to recur.

If anything, incidents like this only serve to reinforce the message that cloud computing is not the silver bullet that it is often portrayed to be. It is not a suitable platform for everything nor is it devoid of fault or error. Just as you take the added expensive of the hardware, staffing, and management when hosting on-site, you also need to accept that things are to an extent out of your control when you move to cloud. Either way downtime will happen, and it will happen because someone made a stupid mistake. But at least if you're on Azure, you've got a team far more expensive than many companies could afford there to fix things when they go wrong, and also the resources to put processes and systems in place to prevent idiotic cock-ups like this from occurring again.

Going Cloudy Part 3 - Configuring your endpoints

In my previous post, I outlined the new architecture. You may have noticed how we no longer have urls containing relative paths. This is because all of the components will be hosted on the same web role and I don’t want to have to deal with complicated startup scripts to configure IIS, therefore I want to try and use the Azure-provided methods as much as possible. When deploying multiple sites to a single web role in Azure, there are essentially two choices for setting it up.

• Virtual directories
• Sites

Using virtual directories involves nesting all of the secondary services in the hierarchy of the main web role project (in this case the website). The up side of this is that I could retain my current url structure. The main down side of this is the inheritance of web.config settings. I initially prototyped using this set up and spent a lot of time overriding entries from the main web role config in the web.configs of the web and rest services to prevent missing assembly and conflicting configuration errors.

An example of this is the following snippet from the assemblies section in the web.config of the main website.

          
          <add assembly="System.Web.Helpers, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Web.Mvc, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35, processorArchitecture=MSIL" />
          <add assembly="System.Web.Abstractions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Web.Routing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Web.WebPages, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
          <add assembly="System.Data.Linq, Version=4.0.0.0, Culture=neutral, PublicKeyToken=B77A5C561934E089" />

Looks harmless enough right? All of these assembly imports will get inherited down to the monitoring site’s child projects, the ASMX web service, and the Site and REST service. The latter 2 are also ASP.NET MVC so it’s not a problem. However, the asmx web service has none of the system.web assemblies in it, causing runtime errors post-deployment.

There were other items that caused me problems but this is the most obvious example. In the end, I decided the virtual directory set up was too brittle.

Using sites has the benefit of keeping all the individual applications on the server separate, so a failure or change in one won’t affect another. Because the applications are separate, we can’t use paths to distinguish which application we want when our request hits the server, so instead we use Host Headers.

The Host Header indicates the url that the user followed in order to get to us. By using subdomains for each service, we can tell from the host header which service the user is attempting to access.

The ServiceDefinition.csdef for the project looks like this.

  <WebRole name="domain.Monitoring" vmsize="ExtraSmall">
    <Sites>
      <Site name="Web">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint3" />
        </Bindings>
      </Site>
      <Site name="Site" physicalDirectory="../../../../../AzPublish/Site-Release">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint1" hostHeader="domain.com" />
          <Binding name="Endpoint2" endpointName="Endpoint2" hostHeader="domain.com" />
        </Bindings>
      </Site>
      <Site name="Gateway" physicalDirectory="../../../../../AzPublish/webservice-Release">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint1" hostHeader="webservice.domain.com" />
          <Binding name="Endpoint2" endpointName="Endpoint2" hostHeader="webservice.domain.com" />
        </Bindings>
      </Site>
      <Site name="RESTService" physicalDirectory="../../../../../AzPublish/Rest-Release">
        <Bindings>
          <Binding name="Endpoint1" endpointName="Endpoint1" hostHeader="rest.domain.com" />
          <Binding name="Endpoint2" endpointName="Endpoint2" hostHeader="rest.domain.com" />
        </Bindings>
      </Site>
    </Sites>
    <Endpoints>
      <InputEndpoint name="Endpoint1" protocol="http" port="80" />
      <InputEndpoint name="Endpoint2" protocol="https" port="443" certificate="DomainCertificate" />
      <InputEndpoint name="Endpoint3" protocol="http" port="22202" />
    </Endpoints>

The Binding elements within each site define the endpoints that this site can be reached on and the hostHeader attribute maps this application to the url the user followed to get to us.

Notice how the domain.Monitoring project has no domain header. This means that this project will reply on endpoint3 (mapped to port 22202) for all traffic. Having it on a separate port means there is no chance of it ever grabbing traffic from the main projects.

You’ll notice that the Monitoring project is the main project in this deployment (I’ll go in to the monitoring project when I cover the traffic manager). The reason for this is that I had some initial trouble getting the host header set up to work with the Azure Traffic Manager and thought doing this may work. That later turned out to be incorrect but I see no harm in keeping the monitoring project as the main one here. Feel free to let me know if there are some caveats to doing this that I am not aware of.

Only the main project in the web role is compiled when publishing, so the ServiceDefinition needs to be pointed to a location containing the compiled project. In order to achieve this, I use file system deployment to publish the subprojects to the AzPublish folder. It’s not ideal and I intend to get all this scripted in Powershell before we go live. When I do that, I will of course publish the scripts on this blog. Also, if there is a better way, please use the comments.

Going Cloudy Part 2 - Out with the old, in with the new

In my previous post, I briefly went over the current situation and the rationale for making the move to Microsoft’s Azure service.

I also promised details on the current architecture and what I have determined to be the new architecture. I will also attempt to explain how I got from old to new and why I have made the decisions I have made.

The old architecture. 

Here’s a quick diagram:

As stated in my previous post, this server is a single VM running 2GB of RAM and a couple of Ghz CPU. With those resources, for the live environment we are running:

•  ASP.Net MVC Web site.
• ASMX Web Service.
• ASP.Net MVC-based REST service.
• SQL Server database.

This is duplicated in the staging site which is located on the same server. On top of this, there is also SQL Server itself, an FTP Server running through IIS, an in-house import/export agent which processes all incoming and outgoing data.

There are also a number of support tools, there is Linqpad, SSMS,Notepad++, off-site file backup, and a number of other minor services.

What this all means is that, whenever I RDP on to change anything, this creates a notable strain on the system.

Bear in mind that the current service doesn’t have many users, say a dozen or so. This will increase by 10x this year so, clearly, the current setup just will not do.

Key requirements

The new infrastructure has a number of key requirements. Other than the obvious speed, reliability,etc, there is also the following.

Speed must be the same regardless of where the user is.

This means website and web service hosted in both EU and US. This also means two databases which need to be kept in sync.

Needs to respond to rises and falls in demand.

This is what cloud does best. We can add and remove instances easily in a matter of minutes. There is also the possibility of using the Scaling Application Block to drop instances down to having a single server when it is night time in their region, halving the service costs during those times.

The new infrastructure

The new infrastructure is distributed, fairly fault tolerant and should be easy enough to scale - all those cloudy terms that get overused. Over the next few posts, I will go over my reasoning for laying it out this way and problems I have encountered while creating the infrastructure. I hope to do that in approximately this order.

• Changes to URL scheme.
• FTP server and REST service configuration.
• Instance configuration, the Monitoring project and the Azure Traffic Manager.
• SQL Azure and using Data Sync.

Going Cloudy Part 1 - The Beginning

If you believe the hype, "Cloud" is the future and everything will end up there eventually.
It's easy to think that "Cloud" is just another term for deploying your applications and services on virtual servers hosted in someone else's data center, mainstream media make this mistake all the time.

Making full use of the cloud, in my opinion anyway, is architecting and designing your system to take advantage of the availability and scalability that the cloud gives you.

Designing for the cloud is one thing, migrating an existing application to the cloud and taking advantage of all it has to offer, is another entirely. Recently, I have been tasked with planning and carrying out a migration to the cloud.

The application in question is currently hosted on a VM running with just 2Gb of RAM. This VM runs IIS, which hosts 3 individual web-based components of this application. It also hosts the database on SQL Server and a number of other additional tools and services. The user base is currently fairly small, limited to a dozen or so users located in the UK, China, USA, and South Africa.

The customer base is expected to expand massively this year, with the number of users from the US pushing over a hundred and a number of users coming on board from other countries around the world.

Clearly, the current server won't handle this, and Cloud is the way forward for this customer.

Amazon or Azure?

Amazon has a myriad of services which fill different niches and markets, but don't seem very joined up. In my opinion, while Amazon's cloud is much more fully featured and mature than Azure, it is a big bucket of disparate services trying to be everything to everyone.
Azure on the other hand, while being newer and not as feature complete as AWS, does what it does very well. All of Azure's services are aimed at allowing developers to build scalable and highly available services, the developer experience is first class and you can get started in minutes.
Microsoft finally recognizes that enticing developers is the key to success, and they have embraced other languages in Azure, creating SDKs for Node, PHP, Java, and Python. Azure is a great platform to develop for, and not just for .net developers.

So Azure it is.

Next time, we'll study the current architecture of the application and look at how it needs to change in order to make the move and meet the customer's requirements.

Disclaimer

I'm not a cloud expert so don't take what I am doing as best practice. This series is intended to help others learn from my experiences. If something can be done better, please use the comments to impart your wisdom.

Speeding up Azure debugging your MVC application

When debugging an application that is intended for an Azure Web Role, you debug using the Compute and Storage emulators in order to give you a realistic Azure experience. These tools used to take a long time to initialize on Build, meaning debugging your app could take significantly longer than normal. Even though the tools have become a lot quicker, there is still an additional delay when debugging using them.

If, like me, you have just migrated a bog-standard web application to Azure and are not yet using any of the specific Azure features, essentially treating Azure as a web server and nothing else, there is something you can do to speed things up.

In your solution, simply change the start up project from the Azure Web Role to the actual MVC project that is going to be deployed in the Azure Web Role. This way you can debug in the normal way and then deploy to Azure when ready.

Note however, that if you start using any of the Azure features such as Blob, Table or Queue storage, then you just need to change the Start Up Project back to the Azure Web Role project