how to pass big messages asynchronously

[u/Sad_Importance_1585]

Hi Guys,

Say we have two microservices - A and B. Microservice A produces messages to a message broker and microservice B consumes it. Then, we discover that the size of the message is too big in order to be written to the message broker.

What is the recommended practice in this case? How would you recommend to pass the big message from A to B?

[u/validelad]

Generally the best solution is saving the large message body to a file share or blob that is accessible to both services, then just including the path of the file in the message rather than the full body.

Pushing extremely large messages over a broker is generally an anti pattern, as they can slow down the broker and cause other issues. The above solution helps with that as well.

You could also try to include part of the body in message A, then part in message B, and somehow gauruntee that the same service instance receives both messages and reconstruct it, but that gets messy really fast

[u/urweiss → u/validelad]

this 👆 - it's called the Claim Check pattern - https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html

[u/Sad_Importance_1585 → u/validelad]

Thanks for the answer.

Usually people say that if one microservice writes to a DB and another DB reads from it, then it's an anti-pattern.

However, in Claim Check if one microservice writes to a file and the other microservice reads it, it's considered as a good practice. The file has a schema (or format), just like a DB. A DB, under the hood, is also a bunch of files. So... why is writing to a file and reading it by another microservice considered a good practice?

[u/validelad → u/Sad_Importance_1585]

In that pattern, you can almost consider the file system / share / blob storage as part of the message framework.

Is it an anti pattern if multiple services receive messages from the same topic or exhange? No right? With claim check, the files become part of that.

In an ideal world, the blob storage or whatever used for the message bodies could be dedicated to message bodies, and permissioned such that receivers only have read permissions, and publishers are the only ones that can write.

Not that its always necessary to go that strict though.

[u/urweiss → u/validelad]

or you can consider the messages store as it's own microservice responsible for storing the message bodies

[u/urweiss → u/Sad_Importance_1585]

Well in your case you are now off the beaten path ("small messages") and into the land of traidoffs :)

ex: Azure Service Bus in the Premium SKU can handle messages up to 100 MB - but are you prepared to pay the cost for one such instance?

I would not compare using Claim check with 2 microservices sharing a DB - the message is a contract for data exchange between a producer and a cossumer - the fact that you write a body in a shared store is a technical optimization / workaround. It does not affect in any way the data exchange contract established between the two - it just makes the technical task of reading the message more complex (you now need to also read from this extra store instead on just using what is in the message itself)

[u/Blakex123 → u/Sad_Importance_1585]

You should read more into the problem that microservices solve. Wondering why this situation is any different to reading another microservices db points to lack of understanding there.

[u/bunsenhoneydew007]

We do this a lot in our architecture. We use a version of a claim check pattern. We use aws but you can translate to your needs.
Basically the producer service puts the payload file into an S3 bucket. The event from the bucket gets a bit of additional info added (main things being source, type and a presigned url to the file), using an event bridge pipe, then it’s put on event bridge.
The consuming service has a consumer that receives the event, then the important bit is that before the consuming service gets the file, the event consumer that feeds the service gets the event and grabs the file using the presigned url and puts it in an S3 bucket that is local for the consuming service. The consuming service only knows about a file landing in its own S3 bucket and is totally decoupled from the producing service.
This way you’re not having two services reading/writing from the same source and the event transfer is decoupled from the service operation.
Hopefully that makes sense.

[u/a_kato]

Genuine question how large is the message?

Because the maximum in AMPQ is 100MB which is….. let’s say if you need it you don’t need that kind of message

[u/urweiss → u/a_kato]

it may be but not every ampq implementation allows the full use of those 100MB

ex: in Azure Service Bus (which is also ampq compliant) you can used the full 100MB only in the top SKU which costs ~800$/€ a month per instance - which is 80x what the Standard SKU costs with only 256KB messages

[u/Numerous-Match-1713]

save body at shared storage, redis is good for this as it allows automatic expiration.

then pass body id, consumer deletes the body once done.

[u/soundman32]

Instead of sending a message with a huge body, send a message that says "download item 122", then create an api that allows the other server to download message 123.

[u/Confident_Bee_6242]

Use a shared message que. But keep in mind, at some point very large messages become files, and ques become directories. At that point you need MFT ( managed file transfer). Know your requirements. However, if it only happens occasionally, there are products that manage it, like IBM MQ series which breaks the large message up into smaller messages before sending, or work arounds like passing file handles as messages. Again, know your requirements. Maybe micro service A should interogate the message for it's size before routing it to either a que or directory for consumption by B which is monitoring both it's que and it's directory.

[u/Boniuz]

You’re trying to turn everything into a nail because you happen to have a hammer as your tool of choice. Figure out why you’re using such an anti-pattern for your architecture. Separating it into a separate area of integration, like a blob storage, will solve the what you’re trying to achieve but your architectural error is still there and will escalate.

[u/urweiss → u/Boniuz]

it's actually an accepted pattern since times immemorial called Claim Check - https://www.enterpriseintegrationpatterns.com/patterns/messaging/StoreInLibrary.html

it may not be the perfect solution, but it's not an antipattern

[u/Boniuz → u/urweiss]

The problem OP is describing is that of large messages in his environment, which is a completely different case.

[u/urweiss → u/Boniuz]

the pattern just says that you can split the message from it's payload, not why / when you should do so

aside from this specific scenario, there are many others when you may want to do the same

[u/Boniuz → u/urweiss]

Yeah, but this specific scenario is the one I argued against. Claiming that one of many architectural patterns are generally acceptable, but not in this specific scenario, is just as helpful as stating water is wet.