High-performance JSON API

Raw System.Text.Json

.NET Core 3.0, has shipped with the new System.Text.Json APIs, which provides built-in support for JSON, including reader/writer, read-only DOM, and serializer/deserializer. With the performance, the main improvement to the methods, check and see the typical speedups of up to 2x over Json.NET.

Doing micro-benchmarks to compare System.Text.Json with Json.NET yields the following output:

ScenarioSpeedMemory
Deserialization2x fasterParity or lower
Serialization1.5x fasterParity or lower
Document (read-only)3-5x faster~Allocation free for sizes < 1 MB
Reader2-3x faster~Allocation free (until you materialize values)
Writer1.3-1.6x faster~Allocation free

ASP.NET Core 3.0 includes support for System.Text.Json, which is enabled by default.

Using System.Text.Json directly

For all the samples, make sure you import the following two namespaces:

using System.Text.Json;
using System.Text.Json.Serialization;

Using the serializer

The System.Text.Json the serializer can read and write JSON asynchronously and is optimized for UTF-8 text, making it ideal for REST API and back-end applications.

class WeatherForecast
{
public DateTimeOffset Date { get; set; }
public int TemperatureC { get; set; }
public string Summary { get; set; }
}
string Serialize(WeatherForecast value)
{
return JsonSerializer.ToString(value);
}

By default, a minified JSON is produced. To produce something that is human-readable, you can pass in an instance of JsonSerializerOptions to the serializer. This is also the way you configure other settings, such as handling of comments, trailing commas, and naming policies.

string SerializePrettyPrint(WeatherForecast value)
{
var options = new JsonSerializerOptions
{
WriteIndented = true
};
return JsonSerializer.ToString(value, options);
}

Deserialization works similarly:

 // {
 //      "Date": "2013-01-20T00:00:00Z",
 //      "TemperatureC": 42,
 //      "Summary": "Typical summer in Seattle. Not.",
 // }
 WeatherForecast Deserialize(string json)
 {
     var options = new JsonSerializerOptions
     {
         AllowTrailingCommas = true
     };
     return JsonSerializer.Parse(json, options);
}

The support asynchronous serialization and deserialization:

async Task SerializeAsync(WeatherForecast value, Stream stream)
{
    await JsonSerializer.WriteAsync(value, stream);
}

You can also use custom attributes to control serialization behaviour, for example, ignoring properties and specifying the name of the property in the JSON:

class WeatherForecast
 {
     public DateTimeOffset Date { get; set; }
// Always in Celsius.
 [JsonPropertyName("temp")]
 public int TemperatureC { get; set; }
 public string Summary { get; set; }
// Don't serialize this property.
 [JsonIgnore]
 public bool IsHot => TemperatureC >= 30;
}

Original article can be found here: https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis/

Async Streams

The new feature Async Streams in C# has removed the scalar result limitation and allows the async method to returns multiple results.

This change makes the async pattern more flexible so that you can retrieve data in lazy asynchronous sequence from a database or you can download data from an asynchronous sequence that returns the data in chunks when they become available.  

Here is an example

await foreach (var issue in runPagedQueryAsync(client, PagedIssueQuery, "docs"))
                 Console.WriteLine(issue);

This makes full use of the service you are calling with the speed and response it was designed for.

Here is a fully working example that calls the GitHub API

Cross-Site Request Forgery (CSRF)

Description

It is possible to trick a user into executing potentially dangerous actions against the target site due to a lack of Cross- Site-Request-Forgery (CSRF) protections. CSRF attacks are a class of confused deputy attacks that exploit the behaviour of browsers always sending authorization cookies in requests. The target site has no secure way of verifying the request was initiated from a link on a trusted domain.

Recommendations

Protect all forms by generating a random unique nonce value each time the page is requested. This nonce value should be included in the form as a parameter, usually as a hidden input field. These nonce values should be linked to the user’s session ID and validated on submission by comparing the generated nonce value for the form with the one that was submitted by the user. When possible, use built-in CSRF protections offered by various vetted libraries or frameworks instead of developing a custom implementation.

If your deployment is to a stateless environment then you will need to ensure that the state on the server is stored in a central reusable location.

Useful links:

Prevent Cross-Site Request Forgery (XSRF/CSRF) attacks in ASP.NET Core

 

Improper Certificate Validation

Description

The SSL Certificate Details provide information about the certificate associated with the HTTPS server. This information describes important attributes of the certificate and should be reviewed for the correctness of the contact information, whether it is self-signed, and how soon it will expire. The Supported Ciphers provides information about each available encryption scheme available for each of the possible SSL/TLS protocols.

Recommendations

The expiration date for certificates should be monitored. New certificates should be available before the expiration date in order to maintain continuity of service and avoid downtime or displaying certificate expiration errors in users’ web browsers.

Server-Side Request Forgery (SSRF)

Description

“Server Side Request Forgery” (a.k.a. SSRF) is a class of web-application vulnerability in which an attacker can cause a website to access unintended server-side resources, including the unauthorized reading, writing, or execution of server resources.

Web-applications designed to pass URLs (references to server accessible content) or portions thereof, from the client browser to the application server using HTTP request parameters or HTTP request headers, are at risk for Server Side Request Forgery exploitation unless they protect against it.

Server-side Request Forgery (SSRF) is related to another type of vulnerability called Open Redirects in the sense that they both rely on untrusted input to reference other web-resources.

Strategies for avoiding and/or fixing Server-Side Request Forgery include:

  • Design around it: Unless there is a reason why URL information must be passed, avoid the problem entirely by implementing an alternative design.
  • Validation: When a URL value is received by the application, it must be white-list validated against the domain of possible legitimate values and rejected if it is not a member.
  • Indirect References: In some cases, it may be possible to pass a (cryptographically strong) random value that represents the target URL and maintain a token: URL mapping on the server.  Since URLs are never passed and the tokens are (practically) un-guessable, the vulnerability is eliminated.

If URI is hard-coded, then the attacker cannot influence where the request is going, so it would look to be a false positive. However, although some scanning software is known for false positives, you need to be careful, as most scanning software are correct in their analysis. You need to check the whole source-to-sink trace that scanning software provides? If it is reporting only that one line as the source and sink, then yes it is a false positive.

Useful links:

https://www.owasp.org/index.php/Server_Side_Request_Forgery

 

Improper Neutralization of CRLF Sequences in HTTP Headers

Description

A function call contains an HTTP response splitting flaw. Writing untrusted input into an HTTP header allows an attacker to manipulate the HTTP response rendered by the browser, leading to cache poisoning and cross-site scripting attacks.

Recommendations

Remove unexpected carriage returns and line feeds from untrusted data used to construct an HTTP response. Always validate untrusted input to ensure that it conforms to the expected format, using centralized data validation routines when possible.

A simple string extension can resolve this issue:

public static string CRLFRemoval(this string token)

{

    return token.Replace("%0d", string.Empty, StringComparison.InvariantCultureIgnoreCase)

                .Replace("%0D", string.Empty, StringComparison.InvariantCultureIgnoreCase)

                .Replace("\r", string.Empty)

                .Replace("\n", string.Empty)

                .Replace("%0a", string.Empty, StringComparison.InvariantCultureIgnoreCase)

                .Replace("%0A", string.Empty, StringComparison.InvariantCultureIgnoreCase);

}

Links for reference:

https://en.wikipedia.org/wiki/HTTP_response_splitting

https://securiteam.com/securityreviews/5WP0E2KFGK

https://www.owasp.org/index.php?title=Testing_for_HTTP_Splitting/Smuggling_(OTG-INPVAL-017)

ASP.NET Core – MongoDB Repository Pattern & Unit Of Work

The implementation of generic repositories and Unit Of Work are complex tasks in MongoDB. The lack of mechanisms in the component hinders its adoption. See how to implement this article.

Unit of Work

Unit Of Work is a standard used to group one or more operations (usually database operations) into a single transaction or “unit of work”, so that all operations are approved or disapproved as one.

Repository Pattern

Repositories are classes or components that encapsulate the logic needed to access the database (repository). Centralize common data access functionality. Providing better maintenance and decoupling of the infrastructure from the domain layer.

Benefits

Code decoupling and re-usability.

Creating a Repository Pattern with Generics in .NET allows you to implement CRUD operations with very little effort and code.

Together with the Unit Of Work, in case of any error in the flow of the operation, it avoids any modification in the bank.

And this control is in the application rather than opening transactions in the bank and avoiding locks in the tables.

Where’s MongoDB?

MongoDB does not have transaction control in the .NET driver. With MongoDB’s popularity increasing, more and more systems are being implemented with it. Soon this article will address how to implement one.

Implementation

For this implementation, you will need the MongoDB.Driver and ServiceStack.Core components.

To install the components, open the Package Manager (View> Other Windows> Package Manager Console) and enter the commands:

Install-Package ServiceStack.Core
Install-Package MongoDB.Driver

Generic Repository

To implement the Repository Pattern, an abstract generics class will be used with CRUD operations.

Interface

public interface IRepository<TEntity> : IDisposable where TEntity : class
{
Task Add(TEntity obj);
Task<TEntity> GetById(Guid id);
Task<IEnumerable<TEntity>> GetAll();
Task Update(TEntity obj);
Task Remove(Guid id);
}

Implementation

public abstract class BaseRepository < TEntity >: IRepository < TEntity > where TEntity : class
{
protected readonly IMongoContext _context ;
protected readonly IMongoCollection < TEntity > DbSet ;

protected BaseRepository ( IMongoContext context )
{
_context = context ;
DbSet = _context . GetCollection < TEntity > ( typeof ( TEntity ). Name );
}

public virtual Task Add ( TEntity obj )
{
return _context . AddCommand ( async () => await DbSet . InsertOneAsync ( obj ));
}

public virtual async Task < TEntity > GetById ( Guid id )
{
var data = await DbSet . FindAsync ( Builders < TEntity >. Filter . Eq ( " _id " , id ));
return data . FirstOrDefault ();
}

public virtual async Task < IEnumerable < TEntity >> GetAll ()
{
var all = await DbSet . FindAsync ( Builders < TEntity >. Filter . Empty );
return all . ToList ();
}

public virtual Task Update ( TEntity obj )
{
return _context . AddCommand ( async () =>
{
await DbSet . ReplaceOneAsync ( Builders < TEntity >. Filter . Eq ( " _id " , obj . GetId ()) obj );
});
}

public virtual Task Remove ( Guid id ) => _context . AddCommand (() => DbSet . DeleteOneAsync ( Builders < TEntity >. Filter . Eq ( " _id " , id )));

public void Dispose ()
{
GC . SuppressFinalize ( this );
}
}

Each Model will have its own implementation. For example a Product class:

public interface IProductRepository : IRepository < Product >
{
}

// Implementation
public class ProductRepository : BaseRepository < Product >, IProductRepository
{
public ProductRepository ( IMongoContext context ): base ( context )
{
}
}

Unit of Work

Unit Of Work will be responsible for performing the transactions that the Repositories have made. For this work to be done, a Mongo Context must be created. This Context will be the connection between the Repository and UoW.

Mongo Context

public class MongoContext : IMongoContext
{
private IMongoDatabase Database { get; set; }
private readonly List<Func<Task>> _commands;
public MongoContext(IConfiguration configuration)
{
// Set Guid to CSharp style (with dash -)
BsonDefaults.GuidRepresentation = GuidRepresentation.CSharpLegacy;

// Every command will be stored and it'll be processed at SaveChanges
_commands = new List<Func<Task>>();

RegisterConventions();

// Configure mongo (You can inject the config, just to simplify)
var mongoClient = new MongoClient(configuration.GetSection("MongoSettings").GetSection("Connection").Value);

Database = mongoClient.GetDatabase(configuration.GetSection("MongoSettings").GetSection("DatabaseName").Value);
}

private void RegisterConventions()
{
var pack = new ConventionPack
{
new IgnoreExtraElementsConvention(true),
new IgnoreIfDefaultConvention(true)
};
ConventionRegistry.Register("My Solution Conventions", pack, t => true);
}

public int SaveChanges()
{
var qtd = _commands.Count;
foreach (var command in _commands)
{
command();
}

_commands.Clear();
return qtd;
}

public IMongoCollection<T> GetCollection<T>(string name)
{
return Database.GetCollection<T>(name);
}

public void Dispose()
{
GC.SuppressFinalize(this);
}

public Task AddCommand(Func<Task> func)
{
_commands.Add(func);
return Task.CompletedTask;
}
}

UoW implementation

public interface IUnitOfWork : IDisposable
{
bool Commit ();
}

public class UnitOfWork : IUnitOfWork
{
private readonly IMongoContext _context ;

public UnitOfWork ( IMongoContext context )
{
_context = context ;
}

public bool Commit ()
{
return _context . SaveChanges () > 0 ;
}

public void Dispose ()
{
_context . Dispose ();
}
}

Configuring Startup.cs

To finalize the configuration, open the project’s Startup.cs and add the DI settings.

public void ConfigureServices ( IServiceCollection services )
{
services . AddMvc (). SetCompatibilityVersion ( CompatibilityVersion . Version_2_2 );
services . AddScoped < IMongoContext , MongoContext > ();
services . AddScoped < IUnitOfWork , UnitOfWork > ();
services . AddScoped < IProductRepository , ProductRepository > ();
}

The project code is available on Bruno Brito’s GitHub

I have then the post by Bruno Brito and translated it in to English, mainly because the article was simple and what I was working on

DONE – when done means DONE

When is something done?  What does DONE mean?

In the teams of software development, here is a definition that can apply to this process:

Definition of Done

The meaning of done is an agreed upon list of the activities necessary to get product increment to a production ready state.

  • Development completed
  • Passed code and peer review (Pull Request in BitBucket)
  • All Technical documentation is created and stored in the source repository
  • Sanity check to confirm technical debt is identified and logged in the tech debt register identified and logged/recorded?
  • Testing has been completed and no issues logged against acceptance tests.
  • Deployed and tested against the acceptance criteria in a valid environment
  • All non-functional requirements have been met
  • Regression test packs updated
  • Update tickets, including documentation, annotation and outcomes
  • Product owner/Business Designer has signed off the work.
  • Release to Production

These are the necessary steps that apply to the development process where every you work

How to export all tables to csv by one export job

If you just want to export all the tables in a SQL database in to separate CSV files, then this is a quick and easy way of doing it.

  1. Execute below query which generates BCP commands

    SELECT ‘bcp ‘ + st.NAME + ‘ out c:\Target\’ + st.NAME + ‘.csv -c -r -d ‘ + DB_NAME() + ‘ -U user@??????.database.windows.net -S tcp:?????.database.windows.net -P ?????? FROM sys.tables st
  2. Paste the result set into text file. Make batch file and schedule it. (It can also be run in CMD manually)