All posts by Bryan Avery

Git Workflow

Source control packages always have been a bit of a pain when you first start using them. You may know the principles of source control. Each environment has its own way and definitions and its own processes. Once you are used to using a source control system and the steps you follow it is easy. I call this the Source Control shuffle.

It’s always good to have a workflow that allows for code review and this process below will provide a method to do this by using branches and merging back into the master.

The branching, committing and merging workflow for projects using git source control.

Guiding Principals

  • master is always production ready and suitable for deployment
  • create a new branch for every feature or fix
  • on each branch, NEVER work on anything but the specific task related to it. If you see other work that needs doing during the course of development, submit a ticket for it, or check out a new branch.
  • rebase during feature development, explicit ( --no-ff ) merge when complete.

Feature Process (or the shuffle)

New features that are added to the product for a future release.

N.B. bug fixes that should go into the next feature release, rather than a hotfix, should also use this process.

  1. Pull down the latest changes from master.

    git checkout master
    git fetch origin
    git merge master
    
  2. Create a new branch for the work. Use the issue key followed by a descriptive name for the branch.
    git checkout -b PRJ-123-feature-name
  3. Work on the feature.
    1. Make regular commits
    2. Avoid clustering changes together.
    3. Push to remote as needed, to keep a backup of your work.
    4. If you need to work on another task, ensure all changes are committed before creating a new branch or switch to another one.
    5. Only one developer should work on a feature branch. If more than one developer needs to work on the task, each should branch off again (including the first developer to start work) from the shared branch. Merges should be made to the task branch as per the central workflow.
  4. Keep your branch up to date with the latest changes on master

    git fetch origin git rebase origin/master
  5. When ready for feedback and/or code review, push to remote and create a pull request.

    1. Push to remote:

      git push -u origin PRJ-123-feature-name

    2. Create a pull request on VSTS to merge the feature branch into master:\
  6. Code review should now occur. If changes are required, they can be committed and pushed to the same branch. The pull request will be automatically updated with any new commits.
    N.B. Any changes made from now on should occur using merges only. If master needs to be merged into the branch to update it, you should not use rebase, as this will cause problems with the pull request.
    Once the review is complete, perform a final rebase (see Feature Process above) to prepare to merge. Then push to remote again (see 5.a above).
  7. The pull request should now be completed on VSTS. This will create a merge commit. There should be no actual changes or conflicts in this commit if the previous procedures have been followed correctly.
  8. Delete the feature branch. This is an option on the remote during the pull request completion in VSTS. The branch will also need to be deleted locally by the developer:
    git branch -d PRJ-123-feature-name

Hotfix Process

Bug fixes that need to be applied to the currently released version (or to patch a previous version that is still supported) and require a new patch version release.

  1. Pull down the latest changes from master (see Feature Process 1. above).
  2. Check out the tag of the release you want to hotfix, branch from it and check out the new branch:

    git checkout v1.0.0 git branch PRJ-199-hotfix-name git checkout PRJ-199-hotfix-name
  3. Work on the bug fix, following the notes in Feature Process 3.
    N.B. It is important that you DO NOT follow step 4. of Feature Process, as changes from master should not be brought into your hotfix branch.
  4. When ready for feedback and/or code review, push to remote.
    N.B. DO NOT create a pull request to master.
  5. Follow the Release Process, using the hotfix branch instead of a release branch.
    N.B. hotfix/patch releases should contain just one bug fix.

Release Process

This process starts when preparing to make a release after all code tasks for that release are merged into master. Final work may still be necessary to complete the release.

  1. Create a new branch from, master either in git directly, or locally, having first pulled the latest changes from master. If you create the branch locally, the currently checked out branch will be used to branch from.
  2. The branch should be named release with the version number appended, e.g. release-1.1.0
    This marks the point at which no more features will be added to the release. Only adjustments and bug fixes necessary to put the release into production should be applied to this branch.
    However, work can continue on new features for other releases, by branching off, master as covered above.
  3. Any further work to prepare this release should be merged into this release branch, using the same process as Feature Process, but creating pull requests to and rebasing from the release branch, instead of master.
  4. Once the release is ready, update the assembly version in the release branch to match Jira and commit, then deploy the build via the deployment machine – this will usually involve checking out the new release branch, running the grunt script to grab the correct web configs and log4net configs and then deploying from Visual Studio.

  5. Note that the version number as reported in the health check will always have 4 digits even though we are only using 3.
  6. Create a tag on the release branch once you are happy that the release was successful

    git checkout release-1.1.0 git tag v1.1.0 git push --tags origin release-1.1.0
  7. Merge the release into master after the release has been tested and is finished: If there are any errors with the release, a decision will need to be made by the Development Manager whether to fix the release branch or to revert the release and do it again.
  8. Merging back into master should be done with a pull request on git.
  9. The release branch can now be deleted.

 

Symmetric encryption/decryption routine using AES

The following is a symmetric encryption/decryption routine using AES in GCM mode. This code operates in the application layer and is meant to receive user-specific and confidential information and encrypt it, after which it is stored in a separate database server. It also is called upon to decrypt encrypted information from the database.

The full description of the AES in GCM mode can be found in this document produced by David A. McGrew and John Viega The Galois/Counter Mode of Operation (GCM)

 

Database Layer

The DatabaseLayer is an abstraction for database access which means the calling application or library does not need to be tight-coupled to the database itself. A set of factory methods is available to call in order to create commands and parameters, all based on the .NET abstract base classes.

Currently supported engines are SQL server and MySQL, represented by “sql” and “mysql” in your web.config or app.config.

The DatabaseLayer does not use a provider pattern but the argument is passed in the factory to tell the library what database engine to use. An example is shown below. It is preferable to create one constant for the database type and share it across multiple places it is used.

The example below is lazy loaded and uses AppSettings and ConnectionStrings from the web.config or app.config file.

private static Database database;

internal static Database GetDb()
{
if ( database == null )
{
database = Database.GetDatabase(
ConfigurationManager.AppSettings["databasetype"],
ConfigurationManager.ConnectionStrings["myconnection"].ConnectionString);
}
return database;
}

DatabaseLayer solution

Security Links and References

To get a level of how many data breaches are happening Breach Level Index provides this information, along with many other statistics

http://breachlevelindex.com/

Need to check if you have an email account that has been compromised in a data breach when check out Have I Been Pwned to check if you have been compromised and also Signup to get email notifications

https://haveibeenpwned.com/

For a collection of Cryptography APIs in one library

https://www.bouncycastle.org

HSM hardware

https://www.thales-esecurity.com/products-and-services/products-and-services/hardware-security-moduleshttps://www.thales-esecurity.com/products-and-services/products-and-services/hardware-security-modules

Azure Resource Manager Templates Tips and Tricks for a Application Service

Over the past week, I spent some time automating Azure App Service infrastructure with Azure Resource Manager (ARM) templates. I discovered a few tips and tricks along the way that I’ll describe in detail below…

  • Take an environment name as a parameter
  • Create sticky slot settings
  • Variables aren’t just for strings
  • Change the time zone of your web app

Big thanks to David Ebbo’s talk at BUILD and his sample “WebAppManyFeatures” template; they are super handy resources for anyone who is working on writing their own ARM templates.

Environment Name Parameter

ARM templates are useful for creating the entire infrastructure for an application. Often this means creating infrastructure for each deployment environment (Dev, QA, Production, etc). Instead of passing in names for the service plans and web apps, I find it useful to pass in a name for the environment and use it as a suffix for all the names. This way we can easily use the same template to set up any environment.

{
    // ...
    "parameters": {
        "environmentName": {
            "type": "string"
        }
    },
    "variables": {
        "appServicePlanName": 
            "[concat('awesomeserviceplan-', parameters('environmentName'))]",
        "siteName": 
            "[concat('awesomesite-', parameters('environmentname'))]"
    }
    // ...
}

In the above example, if we pass in “dev” as the,environmentName we get “awesomeserviceplan-dev” and “awesomesite-dev” as our resource names.

Sticky Slot Settings

App settings and connection strings for a Web App slot can be marked as a “slot setting” (i.e., the setting is sticky to the slot). It’s not well documented how to specify sticky slot settings in an ARM template. It turns out we can do this using a config resource section called “slotconfignames” on the production site. Simply list the app setting and connection string keys that need to stick to the slot:

{
    "apiVersion": "2015-08-01",
    "name": "slotconfignames",
    "type": "config",
    "dependsOn": [
        "[resourceId('Microsoft.Web/Sites', variables('siteName'))]"
    ],
    "properties": {
        "connectionStringNames": [ "ConnString1" ],
        "appSettingNames": [ "AppSettingKey1", "AppSettingKey2" ]
    }
}

Here’s how it’ll look like in the portal:

Object Variables

Variables are not only useful for declaring text that is used in multiple places; they can be objects too! This is especially handy for settings that apply to multiple Web Apps in the template.

{
    // ...
    "variables": {
        "siteProperties": {
            "phpVersion": "5.5",
            "netFrameworkVersion": "v4.0",
            "use32BitWorkerProcess": false, /* 64-bit platform */
            "webSocketsEnabled": true,
            "alwaysOn": true,
            "requestTracingEnabled": true, /* Failed request tracing, aka 'freb' */
            "httpLoggingEnabled": true, /* IIS logs (aka Web server logging) */
            "logsDirectorySizeLimit": 40, /* 40 MB limit for IIS logs */
            "detailedErrorLoggingEnabled": true, /* Detailed error messages  */
            "remoteDebuggingEnabled": true,
            "remoteDebuggingVersion": "VS2013",
            "defaultDocuments": [
                "index.html",
                "hostingstart.html"
            ]
        }
    },
    // ...
}

And now we can use the siteProperties variable for the production site as well as its staging slot:

{
    "apiVersion": "2015-08-01",
    "name": "[variables('siteName')]",
    "type": "Microsoft.Web/sites",
    "location": "[parameters('siteLocation')]",
    "dependsOn": [
        "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]"
    ],
    "properties": {
        "serverFarmId": "[variables('appServicePlanName')]"
    },
    "resources": [
        {
            "apiVersion": "2015-08-01",
            "name": "web",
            "type": "config",
            "dependsOn": [
                "[resourceId('Microsoft.Web/Sites', variables('siteName'))]"
            ],
            "properties": "[variables('siteProperties')]"
        },
        {
            "apiVersion": "2015-08-01",
            "name": "Staging",
            "type": "slots",
            "location": "[parameters('siteLocation')]",
            "dependsOn": [
                "[resourceId('Microsoft.Web/Sites', variables('siteName'))]"
            ],
            "resources": [
                {
                    "apiVersion": "2015-08-01",
                    "name": "web",
                    "type": "config",
                    "dependsOn": [
                        "[resourceId('Microsoft.Web/Sites/Slots', variables('siteName'), 'Staging')]"
                    ],
                    "properties": "[variables('siteProperties')]"
                }
            ]
        }
    ]
}

Custom Time Zone

There’s a little-known setting in Azure App Service that allows you to set the time zone on a per-app basis. It is done by creating an app setting called WEBSITE_TIME_ZONE in the portal. It means we can also do this in an ARM template:

{
    "apiVersion": "2015-08-01",
    "name": "appsettings",
    "type": "config",
    "dependsOn": [
        "[resourceId('Microsoft.Web/Sites', variables('siteName'))]"
    ],
    "properties": {
        "WEBSITE_TIME_ZONE": "Pacific Standard Time"
    }
}

For more information on the time zone setting, check out this article.

Source code: WebAppManyFeatures.json

Allow access to Azure services, what does this actually mean?

I did some quick research on the SQL Server Firewall “Allow Access to Azure Services” option in Azure today.

And I sorry to say that my fears were right, that by setting this option does pose a significant security risk and leaves the SQL Server vulnerable.

Here is the extract of the article I found from Gaurav Hind at Microsoft

Access within Azure: This can be toggled by “Allow access to Azure services” Yes/No button on the portal (Firewall settings page). Please note, enabling this feature would allow any traffic from resources/services hosted in Azure (not just your Azure subscription) to access the database.

https://blogs.msdn.microsoft.com/azureedu/2016/04/11/what-should-i-know-when-setting-up-my-azure-sql-database-paas/

The big question now is how do you plug this gap in the firewall?  One possible solution is to build a virtual network within Azure or Filter network traffic with network security groups, this is beyond the scope of this article.

Comparing SecureStrings

After writing about SecureStrings, you soon come to realise that holding the secret information you are going to have to make some comparisons at some point. For example, if you need to check a password.

The important thing to remember is not to expose the password as text or hold it in memory for anyone or anything to sniff it out.

This code snippet does the job, but it what it does expose is a Timing Attack, but we will deal with that issue another day.

public static bool IsEqualTo(this SecureString ss1, SecureString ss2)
 {
 if (ss1 == null)
 throw new ArgumentNullException("s1");

if (ss2 == null)
 throw new ArgumentNullException("s2");

if (ss1.Length != ss2.Length)
 return false;

IntPtr bstr1 = IntPtr.Zero;
 IntPtr bstr2 = IntPtr.Zero;
 try
 {
 bstr1 = Marshal.SecureStringToBSTR(ss1);
 bstr2 = Marshal.SecureStringToBSTR(ss2);

int length1 = Marshal.ReadInt32(bstr1, -4);

for (int x = 0; x < length1; ++x)
 {
 byte b1 = Marshal.ReadByte(bstr1, x);
 byte b2 = Marshal.ReadByte(bstr2, x);
 if (b1 != b2) return false;
 }
 return true;
 }
 finally
 {
 if (bstr2 != IntPtr.Zero) Marshal.ZeroFreeBSTR(bstr2);
 if (bstr1 != IntPtr.Zero) Marshal.ZeroFreeBSTR(bstr1);
 }
 }

SecureString

SecureString

I’ve come across a class in the System.Security namespace that is often overlooked. The class is SecureString. In this post, I will go over what SecureString is and why it is needed.

Have you ever come across these scenarios before?

  • A password appears in a log file accidentally.
  • A password is being shown at somewhere – once a GUI did show a command line of application that was being run, and the command line consisted of the password.
  • Using memory profiler to profile software with your colleague. Colleague sees your password in memory.
  • Using RedGate software that could capture the “value” of local variables in case of exceptions, amazingly useful. Though, I can imagine that it will log “string passwords” accidentally.
  • A crash dump that includes string password.

Do you know how to avoid all these problems? SecureString. It makes sure you don’t make silly mistakes as such. How does it avoid it? By making sure that password is encrypted in unmanaged memory and the real value can be only accessed when you are 90% sure what you’re doing.

In the sense, SecureString works pretty easily:

  1. Everything is encrypted
  2. User calls AppendChar
  3. Decrypt everything in UNMANAGED MEMORY and add the character
  4. Encrypt everything again in UNMANAGED MEMORY.

What if the user has access to your computer? Would a virus be able to get access to all the SecureStrings? Yes. All you need to do is hook yourself into RtlEncryptMemory then decrypt the memory, and you will get the location of the unencrypted memory address, and read it out. Voila! In fact, you could make a virus that will continuously scan for usage of SecureString and log all the activities with it. I am not saying it will be an easy task, but it can be done. As you can see, the “powerfulness” of SecureString is completely gone once there’s a user/virus on your system.

You have few points in your post. Sure, if you use some of the UI controls that hold “string password” internally, using actual SecureString is not that useful.

The bottom line is; if you have sensitive data(passwords, credit cards, ..), use SecureString. This is what C# Framework is following. For example, NetworkCredential class stores password as SecureString. If you look this, you can see over ~80 different usages in .NET framework of SecureString.

There are many cases when you have to convert SecureString to string because some API expects it.

The usual problem is either:

  • The API is GENERIC. It does not know that there’s a sensitive data.
  • The API knows that it’s dealing with sensitive data and uses “string” – that’s just bad design.

You raised a good point: what happens when you convert SecureString to a string? That can only happen because of the first point. Eg the API does not know that it’s sensitive data. I have personally not seen that happening. Getting string out of SecureString is not that simple.

It’s not simple for a simple reason; it was never intended to let the user convert SecureString to string, as you stated: GC will kick in. If you see yourself doing that, you need to step back and ask yourself: Why am I even doing this, or do I need this, why?

You can always extend the SecureString class with an extension method, such as ToEncryptedString(__SERVER__PUBLIC_KEY), which gives you a string instance of SecureString that is encrypted using server’s public key. The only server can then decrypt it. Problem solved, GC will never see the “original” string, as you never expose it in managed memory. This is precisely what is being done in PSRemotingCryptoHelper (EncryptSecureStringCore(SecureString secureString)).

And as something very almost-related: Mono SecureString does not encrypt at all. The implementation has been commented out because ..wait for it. “It somehow causes nunit test breakage”, which brings to my last point:

Not supported everywhere is SecureString. If the platform/architecture does not support SecureString, you’ll get an exception. The recommended list of deployment platforms is documentation.

Using SecureString for Sensitive Data

The standard System.String class has never been a very secure solution for storing sensitive strings such as passwords or credit card numbers. Using a string for this purposes has numerous problems including it’s not pinned, so the garbage collector can move it around and will leave several copies in memory, it’s not encrypted, so anyone who can read your processor’s memory will be able to see the values of the string easily. Also, if your process gets swapped out to disk, the unencrypted contents of the string will be sitting in your swap file. And it’s not mutable, so whenever you need to modify it there will be an old version and the new version both in memory.

Since it’s not mutable, there’s no effective way to clear it out when you’re done using it. Secure strings are held in encrypted memory by the CLR using the Data Protection API, or DPAPI, and they’re only unencrypted when they are accessed.  This limits the amount of time that your string is in plain text for an attacker to see.

The garbage collector will not move the encrypted string around in memory, so you never have to worry about multiple copies of your string sitting in your address space, unless you make copies of those.

SecureString also implements IDisposable, and when it’s disposed of or finalised if you forget to dispose of it, the memory that was used to hold your encrypted string will be zeroed out.

They also provide a feature that lets you lock them down as read only preventing other code from modifying your string.

You can create a secure string with a pointer to a character array and a length of that array. When constructed this way, the secure string will make a copy of your array, allowing you to zero out your insecure copy.

A secure string can also be constructed without an existing character array, and the data can be copied in one character at a time.

One important thing to note though is that SecureString shouldn’t be used as a blanket replacement for System.String, it should only be used in places where you need to store sensitive information that you do not want to spread around in memory for longer than is required.

To add data or modify data in your string, standard operations are provided. For instance, you’ll find an AppendChar, InsertAt, RemoveAt, and SetAt methods. MakeReadOnly and IsReadOnly allows you to lock down the secure string. Clear, Dispose, and the finaliser takes care of removing any trace of the safe string from memory.

The main idea with SecureString is that you would never store a password or other textual secret in plain text. Unfortunately, SecureString was introduced into the framework only after plenty of APIs were built and shipped using passwords stored in a string, such as the System.Net.NetworkCredential. So any application that must use these APIs had no option but to convert secure strings to strings.

However, the SecureString class itself doesn’t provide any method to get back a plain string with its content, precisely to discourage this type of usage. What a developer has to do is use functions from the System.Runtime.InteropServices.Marshal to get a native buffer with the plain string, marshal a value into managed string, and then very importantly free the native buffer. The best implementation to get a string out of a secure string is to use a try/finally block to free the native buffer.

Here is a sample application showing how you can use SecureString, please remember in production do not use the SecureString to String as this just does not make sense and would make it insecure.

SecureString

Here is a short method to convert a string to a SecureString

public static SecureString ToSecureString(this string source)
{
 if (string.IsNullOrWhiteSpace(source))
 return null;
 else
 {
 SecureString result = new SecureString();
 foreach (char c in source.ToCharArray())
 result.AppendChar(c);
 return result;
 }
}

Raising Multiple Exceptions with AggregateException

There are occasions where you are aware of many exceptions within your code that you want to raise together in one go. Perhaps your system makes a service call to a middleware orchestration component that potentially returns many exceptions detailed in its response, or another scenario might be a batch processing task dealing with many items in one process that require you to collate all exceptions until the end and then throw them together.

Let us look at the batch scenario in more detail. In this situation, if you raised the first exception that you found it would exit the method without processing the remaining items. Alternatively, you could store the exception information in a variable of some sort and once all the elements are processed using the information to construct an exception and throw it. While this approach works, there are some drawbacks. There is the extra effort required to create a viable storage container to hold the exception information, and this may mean modifying existing code not to throw an exception but instead to log the details in this new ‘exception detail helper class’. This solution also lacks the additional benefits you get with creating an exception then, for example, the numerous intrinsic properties that exist within Exception objects that provide valuable additional context information to support the message within the exception. Even when all the relevant information has been collated into a single exception class, then you are still left with one exception holding all that information when you may need to handle the exceptions individually and pass them off to existing error handling frameworks which rely on a type deriving from Exception.

Luckily included in .Net Framework 4.0 is the simple but very useful AggregateException class which lives in the System namespace (within mscorlib.dll). It was created to use the Task Parallel Library, and its use within that library is described on MSDN here. Don’t think that is it’s only used though, as it can be put to good use within your code in situations like those described above where you need to throw many exceptions, so let’s see what it offers.

The AggregateException class is an exception type, inheriting from System.Exception, that acts a wrapper for a collection of child exceptions. Within your code, you can create instances of any exception based type and add them to the AggregateException’s collection. The idea is a simple one, but the AggregateException’s beauty comes in the implementation of this simplicity. As it is a regular exception class, it can handle in the usual way by existing code but also as a unique exception collection by the particular system that cares about all the exceptions nested within its bowels.

The class accepts the child exceptions on one of its seven constructors and then exposes them through its InnerExceptions property. Unfortunately, this is a read-only collection, and so it is not possible to add inner exceptions to the AggregateException after it has instantiated (which would have been nice) and so you will need to store your exceptions in a collection until you’re ready to create the Aggregate and throw it:

// create a collection container to hold exceptions 
List<Exception> exceptions = new List<Exception>();
// do some stuff here ........
// we have an exception with an innerexception, so add it to the list
 
exceptions.Add(new TimeoutException("It timed out", new ArgumentException("ID missing"))); 

// do more stuff ..... 
// Another exception, add to list exceptions.Add(new NotImplementedException("Somethings not implemented")); 

// all done, now create the AggregateException and throw it 
AggregateException aggEx = new AggregateException(exceptions); 
throw aggEx;

The method you use to store the exceptions is up to you as long as you have them all ready at the time you create the AggregateException class. Seven constructors are allowing you to pass combinations of nothing, a string message, collections or arrays of inner exceptions.

Once created you interact with the class as you would any other exception type:

try 
{    // do task } 
catch (AggregateException ex) 
{    // handle it  }

The key as it means that you can make use of existing code and patterns for handling exceptions within your (or third parties) codebase.

In addition to the general Exception members, the class exposes a few custom ones. The common InnerException property is there for compatibility, and this appears to return the first exception added to the AggregateException class via the constructor, so in the example above it would be the TimeoutException instance. All of the child exceptions expose via the InnerExceptions read-only collection property (as shown below).

The Flatten() method is another custom property that might prove useful if you find the need to nest Exceptions as inner exceptions within several AggregateExceptions. The method will iterate the InnerExceptions collection, and if it finds AggregateExceptions nested as InnerExceptions, it will promote their child exceptions to the parent level. As you can see in this example:

AggregateException aggExInner = 
 new AggregateException("inner AggEx", new TimeoutException());
AggregateException aggExOuter1 = 
 new AggregateException("outer 1 AggEx", aggExInner);
AggregateException aggExOuter2 = 
 new AggregateException("outer 2 AggEx", new ArgumentException());
AggregateException aggExMaster =
 new AggregateException(aggExOuter1, aggExOuter2);

If we create this structure above of AggregrateExceptions with inner exceptions of TimeoutException and ArgumentException then the InnerExceptions property of the parent AggregateException (i.e. aggExMaster) shows, as expected, two objects, both being of type AggregrateException and both containing child exceptions of their own:

But if we call Flatten()…

AggregateException aggExFlatterX = aggExMaster.Flatten();

…we get a new ArgumentException instance returned that contains still two objects, but this time the AggregrateException objects have gone, and we have the two child exceptions of TimeoutException and ArgumentException:

A useful feature to discard the AggregateException containers (which are effectively just packaging) and expose the real meat, i.e. the real exceptions that have been thrown and needs to be addressed.

If you’re wondering how the ToString() is implemented then the aggExMaster object in the examples above (without flattening) produces this:

System.AggregateException: One or more errors occurred. ---> System.AggregateException
: outer 1 AggEx ---> System.AggregateException: inner AggEx ---> 
System.TimeoutException: The operation has timed out. --- End of inner exception 
stack trace --- --- End of inner exception stack trace --- --- End of inner exception 
stack trace ------> (Inner Exception #0) System.AggregateException: outer 1 AggEx ---> 
System.AggregateException: inner AggEx ---> System.TimeoutException: The operation
 has timed out. --- End of inner exception stack trace --- --- End of inner 
exception stack trace ------> (Inner Exception #0) System.AggregateException: inner
AggEx ---> System.TimeoutException: The operation has timed out. --- End of inner 
exception stack trace ------> (Inner Exception #0) System.TimeoutException: The 
operation has timed out.<---<---<------> (Inner Exception #1) System.AggregateException
: outer 2 AggEx --- System.ArgumentException: Value does not fall within the expected
 range. --- End of inner exception stack trace ------> (Inner Exception #0) 
System.ArgumentException: Value does not fall within the expected range.

As you can see the data has been formatted in a neat and convenient way for readability, with separators between the inner exceptions.

In summary, this is a very useful class to be aware of and have in your arsenal whether you are dealing with the Parallel Tasks Library or you just need to manage multiple exceptions. I like simple and neat solutions, and to me, this is a good example of that philosophy.

Orginal Article