Schema-oriented message destination AUTHOR ====== Ricky Ho (rickyphyllis@hotmail.com) 08/28/2001 CONTEXT ======= When a message producer send a message to a destination (queue or topic), he wants to make sure that he is sending messages to a destination where the consumers can understand. Similarly, when a message consumer receive a message from a destination, he wants to make sure that he can understand all messages he take from the destination. PROBLEMS ======== In JMS, the producer and consumer are tightly coupled with the message structure. However, there is no relationship between the destination and the message structure. Therefore, the consumer application and producer application needs to agree upfront (at design time) what message structure will be produced or consumed. (Problem 1): There is NO standard way for a new producer app or consumer app to learn what message structure they should be used for a destination. They will try to look for existing design documents to find if they mention anything about the message structure. Nevertheless, the only reliable way is to look at source code from existing producer and consumer app. This process is error-prone. (Problem 2): If the consumer consumes a message from a queue with an unexpected structure. At best, it detects the message is not understandable and drops it, in this case the message is lost. At worst, the message with wrong structure will crash the consumer app. Well-written consumer applications will take the burden to code the message structure validation logic, resulting in undesirable complexity. Ideally, the message structure should have been verified before passing to the consumer application. (Problem 3): Whenever a change is needed for a message structure (usually initiated from one of the message producer), nightmare start. First, we need to trace all the destination where the new message (with the changed message structure) will go to. Second, for each of the destination, we need to trace all the consumer applications and make sure that they are changed apropriately to deal with the new message structure. This tracing can be very difficult when the consumer application dynamically subscribing and receiving from destinations. If we miss the upgrade for one consuming app, we are introducing problem 2. SOLUTION ======== A destination should only accept one type of message with a particular message structure. The metadata associated with each destination should include the definition of this ONE message structure that it accepts. Whenever a message is sent to a destination, the JMS container should verifies the message with the specified structure. Whenever a change is made to a message structure, a new version of destination (by destination naming convention)is created, at the same time all existing consumer applications will received a callback method "onSchemaChange()". Now this new destination will only accept message with the new (changed) message structure. Old producer app without aware of the message structure change can keep sending message with old structure to the old destination which will be consumed by old consumer apps. Because, existing consumer apps will never receive the new message type until they explicitly change to the new destination, they won't break. This mechanism requires the producer and consumer applications to explicitly move to a new destination and accomodate the message structure changes at the same time. CONSEQUENCE =========== By enforcing the association between a destination with a message type. We can standardize the mechanism how the producer app and consumer app discover the correct message schema. By introducing metadata of the destination, a producer app and consumer app can examine that at run time, and hence reducing design-time assumptions. The destination versioning helps to prevent incompatibilities problems when schema changes. 4 replies in this thread Reply Schema-oriented message destination Posted By: Gal Binyamini on August 29, 2001 in response to this message. Some notes: 1. When a schema changes, how do you suppose the clients will start producing new valid versions? New schemas usually mean different data. In order to make clients that can adapt to new schemas on-the-fly you need a lot of technology, and this pattern alone won't help you at that (I belive some work is done in this direction in JXTA). 2. Having a callback to a client is *extremely* undersireable in message-based application. It makes strong coupling between consumers and producers, and should be avoided whenever possible. I think this alone rules out this particular strategy. 3. You have made no proposal as for the schema language to use, and I don't think any one can propose such language. Therefore I don't see a way for the JMS provider to do the validation for you. Besides, the fact is the JMS provider doesn't do the validation, so the current proposal is not a "design pattern" - it can't work with existing specs. As I see it, this pattern comes down to: define the structure of the message in some language and before you start processing a message make sure it conforms to the definition. This is generally what people do today, and I think it's more of a "good practice" than a "design pattern". The parts about making this definition formal and available are again, IMHO, just good practice. Note that in some areas such as XML messaging defining the input schema using a DTD/XML Schema is possible. Most such systems will support this notion. Gal 0 replies in this thread Reply Schema-oriented message destination Posted By: Ricky Ho on August 29, 2001 in response to this message. Of course we verify the message structure using schema validation (in case of XML message) but there is no association between a JMS destination with an XML schema. This pattern suggest to associate the schema definition with the destination so that the validation can be done outside the consumer application (possibly by the JMS container). My response to your 3 notes 1) The pattern doesn't advocate application (producer or consumer) needs to be adapt to new schema on the fly. Although the metadata of the destination allows this to happen by enabling introspection-style programming, it is not recommended because the code will become less readable (similar to using Java reflection API instead of regular method calls). The pattern actually enable the producer app and consumer app to upgrade their code (design-time, not run-time) in a controllable way so that new producer would never get a chance to talk to an old consumer. 2) Callback function is used extensively in JMS. Whenever a subscriber subscribe to a topic, it register a callback function "onMessage()" which will be called whether a message is delivered to that topic. Same for MessageDrivenBean, "callback" almost becomes a dominant case. Would you mind to elaborate more detail about the tight coupling scenario you mentioned. 3) For XML message, the Schema language will be XML Schema. For Map message, the Schema language will specify the list of key names. For Object message, the Schema language will specify the class name of the Java Object. Best regards, Ricky 2 replies in this thread Reply Schema-oriented message destination Posted By: Giedrius Trumpickas on August 30, 2001 in response to this message. I don't think that this design suggestion can be treated as pattern. It just design suggestion for MOM API. In simple terms it sounds: messages version support in MOM and message routing to consumers depending on version. 0 replies in this thread Reply Schema-oriented message destination Posted By: Gal Binyamini on August 30, 2001 in response to this message. As you said, the association of schema definition with a destination is important at design time, not run time. I don't see a reason for letting the clients know about new schemas as they arrive unless they can adapt to them. If you want a seperate service of notifying clients as new versions of the service get released, you can use a callback (with a topic) as you described. I don't see how this is specifically relevant to this pattern. If choosing a schema language was that simple, JMS would have incorporated it. However, JMS is designed to be used in a variety of different ways, and validation may mean different things to differet people. When I validate a Map, I usually need to check the actual mapped values, not only that *some* value exists for them. I see no special reason why JMS should support your specific schemas rather than others. If you need these particular types of schemas, you can extend JMS (for instance, by providing a common base class for listeners). If JMS will be extended to support XML messaging in specific (which isn't obvious), it may support XML Schemas. Anyway, my point is not that what's described here is not a good design approach. It's simply not a design pattern, IMHO. Regards Gal 0 replies in this thread Reply Schema-oriented message destination Posted By: Polina Alber on September 12, 2001 in response to this message. Introduction of "schema-oriented message destination" comes against loosely coupled system approach. In addition a number of destinations can be limited per system. I think the easiest way to enforce message delivery to appropriate consumer is to use a message selector option and custom message properties settings ( like: proprietary message format type). 1 replies in this thread Reply Schema-oriented message destination Posted By: Ricky Ho on September 13, 2001 in response to this message. The unfortunate fact is, in current JMS, the producer and consumer code are "tightly coupled" with the message structure. Message selector doesn't help in this case because there is nothing preventing the producer code putting the right message properties but the wrong message type and crash the consumer code. The solution is some schema checking needs to be done before the message get dispatched to the consumer. Therefore, we need to associate some schema information with the destination (queue/topic) to enable this kind of checking. Ricky 0 replies in this thread Reply Schema-oriented message destination Posted By: Peter Donald on September 16, 2001 in response to this message. I am not sure this is strictly needed as part of the API or even the infrastructure. Names in JMS roughly corespond to ports in TCP/IP world. The TCP/IP world has several "schemas" and generally they are associated with a port (though not necessarily). For instance the HTTP "schema" is associated with port 80, 8080 and a few others ontop of SSL. If there was well known and published schemas then I think they could have well known and published names. However as I don't know of any organization that does such standardization shouldn't it just be a company internal policy? 1 replies in this thread Reply Schema-oriented message destination Posted By: Ricky Ho on September 16, 2001 in response to this message. Peter, There are some interesting difference ... 1) Port 80 is well-known, but not any JMS destination. 2) HTTP parameters has a well-published encoding standard, but JMS doesn't. In fact, HTTP suffer from a similar problem, but to a lesser degree, because ... a) HTTP has a different programming model. Typically the Servlet will extract only the parameters it expect, so whatever it doesn't expect will simply be ignored. For those it expects, the Servlet code will check if they are "null" or any application specific test. Of course in JMS, such checking responsibility can be put into the receiver code also, but having another layer to enforce this is cleaner, because the checking is declared externally, rather than buried into the Java code. b) HTTP is a sync protocol, so whatever the sender immediately aware of any problem that has occured and deal with it immediately. However, exception handling in an async protocol is very clumsy. The default exception handling is just abort the receiving side transaction and keep re-dispatching the wrong message. Usually the solution requires some more complex async notification and sometime manual intervention. It will be much better if the checking can be done by the message broker when the sender sends the message. (the interaction between the sender and message broker is usually sync). Best regards, Ricky 0 replies in this thread