Chunking
-
Standard scheme: this is used by most browsers. It consists in specifying a Content-Length header in the HTTP headers. This allows the receiver to know how much data is coming and when to stop reading.
The problem with this approach is that the length needs to be pre-determined. The data cannot be streamed as generated as the length needs to be calculated upfront.
Thus, if chunking is turned off, we need to buffer the data in a byte buffer (or temp file if it is too large) so that the Content-Length can be calculated.
-
Chunked scheme: with this mode, the data is sent to the receiver in chunks. Each chunk is preceded by a hexadecimal chunk size. When a chunk size is 0, the receiver knows that all the data has been received.
This mode allows better streaming, as we just need to buffer a small amount, up to 8K by default. When the buffer fills, the chunk is written out.
Some parameters in Configuration properties allow you to specify the details of the chunking.
- Many proxy servers don't understand it, especially older proxy servers. Many proxy servers want the Content-Length up front so they can allocate a buffer to store the request before passing it onto the real server.
- Some of the older Web Services stacks also have problems with Chunking - specifically, older versions of .NET.