When handling HTTP responses, especially with streaming data, properly verify content integrity and length to prevent data corruption or unexpected termination. Key considerations:
- Content-Length may not match actual data size when responses use compression (gzip, br, deflate). In such cases:
- Check for Content-Encoding header to determine if response is compressed
- Use actual byte counts rather than Content-Length for progress tracking
- Consider sending Accept-Encoding: identity for critical operations requiring precise progress tracking
- Implement proper stream handling based on environment:
- For Node.js, use Transform streams rather than ‘data’ event listeners to avoid backpressure issues:
```javascript
// GOOD: Use piping with Transform streams
var contentLengthCheckerStream = new ContentLengthCheckerStream(
parseInt(headers[‘content-length’], 10)
);
responseStream.pipe(contentLengthCheckerStream).pipe(stream);
// AVOID: Direct data event listeners can cause backpressure issues in older Node.js
responseStream.on(‘data’, function(chunk) {
receivedLen += chunk.length;
// potential for data loss in Node.js <= 0.10.x
});
```
- Handle content integrity verification for both Node.js and browser environments:
- In Node.js, implement pipe-through verification streams
- In browsers, buffer response data for integrity checks
- Verify checksums/hashes when provided in response headers
- Emit appropriate errors with descriptive messages when content length or integrity checks fail
Implementing these practices ensures data is received completely and correctly, preventing subtle bugs in network communication.
