end2end

End-To-End Arguments in System Design

J.H. Saltzer, D.P. Reed and D. D. Clark, MIT, 1984

Summary. The paper states a design principle called the end-to-end argument and provides several examples of where the principle can be used. Regarding an applications functionality, the end-to-end argument states that:

The function in question can completely and correctly be implemented only with the knowledge and help of the application standing at the endpoints of the communication system. Therefore, providing that questioned function as a feature of the [end application] is not possible.
Sometimes an incomplete version of the function provided by the communication system may be useful as a performance enhancement.

The idea is that often the designers of lower level layers try to ``guess'' .what the higher level layers (and applications) will need and attempt to ``help'' by providing that functionality in the layer. Often, however, some necessary piece is missing from the lower-level functionality so the application ends up re-implementing the functionality, anyway. The resulting redundancy takes away from performance.

More Detail

Performance aspects. Sometimes functionality that cannot be implemented correctly at the lower level should be implemented at least in part as a optimization. For example, in the case of file-transfer, the end-to-end argument says that we should implement error checks at the application level since errors can occur at any level from the app down to the network. However, if delivery reliability were not checked for, a lossy connection would require the entire application to be re-transferred perhaps several times before successful. However, it is not necessary to implement great reliability since app must re-implement. Finally, implementing error recovery at low-levels can sometimes be costly: 1) apps that do not require will have anyway 2) lower levels don't have as much info so can't do the job as efficiently.

Identifying the ends. It is not always easy to identify the ends. In other words, the level at which error detection is placed can vary from system to system. For example, in a real-time voice transmission (phone call), the end would be the humans as doing guaranteed error recovery at the network level would most likely induce unacceptable delays. In a non-real time voice transmission, however, error recovery may be desirable. In airline reservation systems, the human will check that the reservation was confirmed through system crashes, etc.. End-to-end examples: reliable tape subsystem (humans, machines unreliable) and RISC chips (no use trying to guess the user's needs).

Examples

Much of the paper gave examples of where/when the end-to-end arg applies.

Open Operating System: In an operating system, functions shouldn't be permanent parts of lower level module. Rather, they should be replaceable. This way, those apps that need to can implement their own functionality.

Careful file transfer. When doing file transfer, errors can come up in any level. The error checking at the app level can be done by computing a checksum at each end and comparing.

Delivery guarantees. An app might really want to know if host acted on message rather than that the message arrived. So, can have the end app send a reply message stating what action was taken. Could try to avoid this by having the host guarantee to act but the host might refuse to act. This also won't work for two-phase commit transactions as all involved nodes must communicate before knowing to commit.

Secure transmission of Data. May be necessary at network level to ensure that misbehaving app doesn't send private data out without encrypting, but the encryption algorithm for this level is much sophisticated than that required by app: no user keys, all hosts can use same key & change frequently.

data transmission layer must be trusted to manage keys.
data vulnerable before/after reaching communication layer.
since network layer managing keys, still need authentication at app level.

Duplicate message suppression. App may accidently send dups that network .layer can't see so may as well do this at end. Another case: after crash, sender might send second message out but that ``cash''may have just been a delayed ack rather than a real crash. The resent message will be a duplicate that the network layer can't detect.

In-order delivery. Although many protocols guarantee in-order delivery, it cannot detect out of order messages in a distributed application environment.

Transaction management. MIT built a storage system called SWALLOW that sent with each message an object id to read/write, read/write mode and the version of the object. SWALLOW sends a write request reponse saying that the data written safely. Because SWALLOW can detect duplicates using the object ID and versions, no duplicate suppression is necessary. Also, no acknowledgement is needed at the network level since SWALLOW sends a high-level acknowledgement.