Add support for new message format

[xstevens]

Kafka uses a new messaging format in the most recent versions. This is denoted by the magic byte being set to 1 and is followed by some message attributes which let you know if compression was used, and if so what type. See the Kafka design guide for more details.

FYI: The ruby client looks like they've updated for this already.

[sthomas-github]

This might be along the same lines. When I run a consumer against our kafka installation (0.7.2), I get:

Traceback (most recent call last):
File "readkafka.py", line 10, in
for message in consumer.loop():
File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 46, in loop
messages = self.consume()
File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 38, in consume
self.send_consume_request()
File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 68, in send_consume_request
self.write(self.encode_request())
File "/usr/local/lib/python2.7/dist-packages/kafka/consumer.py", line 64, in encode_request
return struct.pack('>HH%dsiQi' % length, self.request_type, length, self.topic, self.partition, self.offset, self.max_size)
struct.error: integer out of range for 'Q' format code

I'd like to use python for kafka interrogation, but we will have to fall back to Java unless I can get around issues like this, so any help would be appreciated.

Thanks,

Steve