I’ve been using an Event Bus for some time now in the designs of a system, for reason of performance and scalability. There is also an issue with the message size. But it would be best if you were using the Event Bus as a messaging system, not a data transport mechanism.

There is a solution to this, and it is the claim-check pattern.

Split a large message into a claim check and a payload. Send the claim check to the messaging platform and store the payload to an external service. This pattern allows large messages to be processed while protecting the message bus and the client from being overwhelmed or slowed down. This pattern also helps to reduce costs, as storage is usually cheaper than resource units used by the messaging platform.

Context and problem

A messaging-based architecture at some point must be able to send, receive, and manipulate large messages. Such messages may contain anything, including images (for example, MRI scans), sound files (for example, call-centre calls), text documents, or any kind of binary data of arbitrary size.

Sending such large messages to the message bus directly is not recommended, because they require more resources and bandwidth to be consumed. Large messages can also slow down the entire solution because messaging platforms are usually fine-tuned to handle huge quantities of small messages. Also, most messaging platforms have limits on message size, so you may need to work around these limits for large messages.

Solution

Store the entire message payload into an external service, such as a database. Get the reference to the stored payload, and send just that reference to the message bus. The reference acts like a claim check used to retrieve a piece of luggage, hence the name of the pattern. Clients interested in processing that specific message can use the obtained reference to retrieve the payload if needed.

Issues and considerations

Consider the following points when deciding how to implement this pattern:

• Consider deleting the message data after consuming it, if you don’t need to archive the messages. Although blob storage is relatively cheap, it costs some money in the long run, especially if there is a lot of data. Deleting the message can be done synchronously by the application that receives and processes the message, or asynchronously by a separate dedicated process. The asynchronous approach removes old data with no impact on the throughput and message processing performance of the receiving application.
• Storing and retrieving the message causes some additional overhead and latency. You may want to implement logic in the sending application to use this pattern only when the message size exceeds the data limit of the message bus. The pattern would be skipped for smaller messages. This approach would result in a conditional claim-check pattern.

When to use this pattern

This pattern should be used whenever a message cannot fit the supported message limit of the chosen message bus technology. For example, Event Hubs currently has a limit of 256 KB (Basic Tier), while Event Grid supports only 64-KB messages.

The pattern can also be used if the payload should be accessed only by services that are authorized to see it. By offloading the payload to an external resource, stricter authentication and authorization rules can be put in place, to ensure that security is enforced when sensitive data is stored in the payload.

Working Example

A good example of this pattern is provided by Microsoft

The example is using Blob Store to store data, but any service that supports Event Grid integration can be used too. A client just needs to drop the payload to be shared into the designated Azure Blob Store and Event Grid will automatically generate a Claim Check message and send it to one of the supported message bus. In this sample, the message bus is created using Azure Storage Queues.

This allows a client application to poll the queue, get the message and then use the stored reference data to download the payload directly from Azure Blob Storage. The same message, without the need to go through a message bus, can also be directly consumed via Azure Function, leveraging, in this case, the serverless nature of both Azure Event Grid and Azure Functions.

https://github.com/mspnp/cloud-design-patterns/tree/master/claim-check/code-samples/sample-1

There are other samples available, but this is the easiest to follow.

For more information which out this:

Claim-Check Pattern by Microsoft using Azure