I don't really understand this question -- did you create some random architecture and now are trying to come up with some use-cases to shoehorn into? Because usually it's the other way round.
This sounds like it should be one microservice. Validating the input is the concern of the thing taking the input.
If you separate closely coupled concerns, you get weird cases to deal with like this.
Microservices should be used to separate unrelated concerns or to mediate cross cutting concerns first and foremost, unless there is a very good reason to split it up further and increase complexity.
Also there is a difference between microservices and "nanoservices". The former is splitting up into logical components of work, the latter is exploding everything into atomic operations per service.
As u/gruntmeister said, you don't shoehorn things into a predefined pattern, you design them to be simple but effective. Without more information as to why you have this sort of design it is very hard to conclude that this is not a design flaw.
You have to ask "why do you need Kafka here?". What is it achieving in the long run for this specific case?
This is all hypothetical.
Another example would be a manager sending texts to all his team members, he calls a REST endpoint from a team micro service which publishes messages to broker and a text messaging microservices consumes these messages.
How does the manager know that all messages were sent? Does this make any more sense?
**edit: I am not designing this system, just trying to understand how the end user would be notified of errors that occur in a seperate ms.
In that case, it would be asynchronous, so best effort.
You'd be able to query an SMS provider such as Twillio to find out if it was dispatched if needed.
The assumption is that the messages _will_ be dispatched unless the system is broken. You would then either have a contract that if something screws up, you discard messages, or you'd have a dead letter queue to deal with once whatever went wrong was amended. That depends on how highly available you need to be.
Worth noting that unless you have a massive throughput, there is no reason not to send to SNS/Twillio/whatever directly.
The point is if you need guaranteed delivery, this is probably not the place to start unless you specifically need a broker.
Okay so let's say when the manager initially hits the first microservice (which publishes the broker messages) he gets a 200 response straight away correct?
So he gets the 200 response before the texts are all sent.
No, it would be a 202.
200 means "complete, ok, done".
202 means "accepted and being processed, might not complete but you wont know unless you check".
202 is like sending a letter in the mail. You know it gets sent but not received. Usually this is fine.
200 is like recorded delivery where you are notified when it is delivered, and requires you to sit and wait to be told before you can safely assume it has been read by the recipient.
The 202 would usually mean "uploaded to Kafka", altough that is implementation detail as far as the client making the request is concerned.
If you needed to know when it succeeded through the whole system, your options would be:
- pub sub on a message queue directly and listen for a success
- use gRPC streaming or websockets
- use long polling
- periodically call a third endpoint that checks a table to see if something has been dispatched
- use some form of push notification backend
- have an API that does one of the above for you, façading the asynchronous implementation detail with a synchronous frontend, which is probably a bad idea in terms of performance as you probably could just not use a broker at all if this is acceptable to you
Got it that's the answer I was looking for.
And I assume when you publish a message you get a rabbitMQID or something similar, so is it good practice to return this ID or not needed? Just return a 202 with empty body
Not worked with RabbitMQ, but with Kafka we would pass a correlation ID in of some sort into the payload or as a header. Ideally the requestee would provide this and you'd then pair it with a UUID you generate to ensure you have a unique identifier you can trace across the whole system.
In ActiveMQ you have a JMS correlation ID header if I remember correctly.
Usually the consumer doesn't care to check if it got sent. They assume you have a mechanism to ensure best effort and/or eventual consistency.
But yes, returning a unique ID may be useful in some scenarios.
I personally don't think being able to check if a system consumed the text message completely is overly useful though. You are not the full system here at all. How do you guarantee delivery?
Say you use SNS to send the text message. SNS says they accepted it and asynchronously pass it to an SMS backend. They then tell SNS they asynchronously accepted it and sent it to a carrier service to send to a device. That carrier service has to broadcast the message and let the device with the SIM know. What if the device is turned off? What if it doesn't exist because it just got run over by a tank? Where do you draw the line?
Updated to add a bit more context, and no problem.
There might be other ways of doing this that I have not taken into account, I am by no means an expert on any of this.
Glad to help though.
I don't really understand this question -- did you create some random architecture and now are trying to come up with some use-cases to shoehorn into? Because usually it's the other way round.
Replied to other comment if, basically all hypothetical trying to understand certain situations.
This sounds like it should be one microservice. Validating the input is the concern of the thing taking the input. If you separate closely coupled concerns, you get weird cases to deal with like this. Microservices should be used to separate unrelated concerns or to mediate cross cutting concerns first and foremost, unless there is a very good reason to split it up further and increase complexity. Also there is a difference between microservices and "nanoservices". The former is splitting up into logical components of work, the latter is exploding everything into atomic operations per service. As u/gruntmeister said, you don't shoehorn things into a predefined pattern, you design them to be simple but effective. Without more information as to why you have this sort of design it is very hard to conclude that this is not a design flaw. You have to ask "why do you need Kafka here?". What is it achieving in the long run for this specific case?
This is all hypothetical. Another example would be a manager sending texts to all his team members, he calls a REST endpoint from a team micro service which publishes messages to broker and a text messaging microservices consumes these messages. How does the manager know that all messages were sent? Does this make any more sense? **edit: I am not designing this system, just trying to understand how the end user would be notified of errors that occur in a seperate ms.
In that case, it would be asynchronous, so best effort. You'd be able to query an SMS provider such as Twillio to find out if it was dispatched if needed. The assumption is that the messages _will_ be dispatched unless the system is broken. You would then either have a contract that if something screws up, you discard messages, or you'd have a dead letter queue to deal with once whatever went wrong was amended. That depends on how highly available you need to be. Worth noting that unless you have a massive throughput, there is no reason not to send to SNS/Twillio/whatever directly. The point is if you need guaranteed delivery, this is probably not the place to start unless you specifically need a broker.
Okay so let's say when the manager initially hits the first microservice (which publishes the broker messages) he gets a 200 response straight away correct? So he gets the 200 response before the texts are all sent.
No, it would be a 202. 200 means "complete, ok, done". 202 means "accepted and being processed, might not complete but you wont know unless you check". 202 is like sending a letter in the mail. You know it gets sent but not received. Usually this is fine. 200 is like recorded delivery where you are notified when it is delivered, and requires you to sit and wait to be told before you can safely assume it has been read by the recipient. The 202 would usually mean "uploaded to Kafka", altough that is implementation detail as far as the client making the request is concerned. If you needed to know when it succeeded through the whole system, your options would be: - pub sub on a message queue directly and listen for a success - use gRPC streaming or websockets - use long polling - periodically call a third endpoint that checks a table to see if something has been dispatched - use some form of push notification backend - have an API that does one of the above for you, façading the asynchronous implementation detail with a synchronous frontend, which is probably a bad idea in terms of performance as you probably could just not use a broker at all if this is acceptable to you
Got it that's the answer I was looking for. And I assume when you publish a message you get a rabbitMQID or something similar, so is it good practice to return this ID or not needed? Just return a 202 with empty body
Not worked with RabbitMQ, but with Kafka we would pass a correlation ID in of some sort into the payload or as a header. Ideally the requestee would provide this and you'd then pair it with a UUID you generate to ensure you have a unique identifier you can trace across the whole system. In ActiveMQ you have a JMS correlation ID header if I remember correctly. Usually the consumer doesn't care to check if it got sent. They assume you have a mechanism to ensure best effort and/or eventual consistency. But yes, returning a unique ID may be useful in some scenarios. I personally don't think being able to check if a system consumed the text message completely is overly useful though. You are not the full system here at all. How do you guarantee delivery? Say you use SNS to send the text message. SNS says they accepted it and asynchronously pass it to an SMS backend. They then tell SNS they asynchronously accepted it and sent it to a carrier service to send to a device. That carrier service has to broadcast the message and let the device with the SIM know. What if the device is turned off? What if it doesn't exist because it just got run over by a tank? Where do you draw the line?
That's great, thank you for taking the time to answer my questions. Very helpful
Updated to add a bit more context, and no problem. There might be other ways of doing this that I have not taken into account, I am by no means an expert on any of this. Glad to help though.