deflate error when using git

[gemorin]

I am rebasing a linux-stable tree with git and getting deflate failure with the current git master.

$ git pull --rebase
remote: Counting objects: 256, done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 256 (delta 218), reused 254 (delta 217)
Receiving objects: 100% (256/256), 44.38 KiB | 0 bytes/s, done.
fatal: unable to deflate appended object (-5)
fatal: index-pack failed

The annoying thing is that when running this under gdb, it does not fail so there is definitely something fishy going on. Using vanilla zlib works just fine. It's fairly easy to reproduce for me.

I am going to try debugging some of this but wanted to get it out there. Hopefully you guys have suggestions on what I can try.

[jtkukunas]

Hi gemorin,

Thanks for reporting this. Looks like the -5 corresponds to a Z_BUF_ERROR. Is your tree publically available that I can investigate this further?

Git uses two configuration options to control the deflate compression level, core.loosecompression and pack.compression. The loosecompression defaults to level 1 and pack.compression defaults to level 6. You might be able to temporarily avoid this problem while I work to fix it by changing the git compression levels. If this works, let me know what option you had to change, and the working value. That'll provide some additional insights.

Thanks and sorry for the inconvenience.

[gemorin]

Unfortunately no, my tree is not public. I am trying to see if I can make it easy to reproduce by using a public tree.

I tried to change the parameters as you suggested. It was a bit of random walk

• Set pack.compression to 1, no change
  -Then set core.compression to 1, no change
• Then set core.compression to 5 and it worked

Then I started from a fresh clone and I verified that setting core.compression to 5 was enough

[jtkukunas]

Okay. Thanks.

The common denominator there seems to be level 1. Could you verify that it doesn't work when set to 1 and does when set to 2-9?

Thanks.

[gemorin]

Yes, core.compression 2-9 works just fine. core.compression 1 fails

[xnox]

That's interesting. I have similar errors reproducible with public git repositories. Cloning linux master repository fails.

[jtkukunas]

Interesting. This seems to be caused by deflate_quick returning finish_started a bit too early. This happens when the output buffer is exhausted before all of the input is loaded. Then deflate() sees the user submitting more input after we've entered the FINISH_STATE, and returns an error.

I've pushed a new branch called "issue10." Could you check whether this branch, specifically commit fa19b0b, fixes the issue?

Thanks.

[gemorin]

I've just tried fa19b0b but while index-pack does not fail anymore, the resulting buffer seems to be corrupted:

error: inflate: data stream error (incorrect data check)
error: failed to read delta base object a210766279d380c550c8302725b9ab344a512417 at offset 4617837 from .git/objects/pack/pack-ecb1f703ea9732b91e3a423176aab38fc69514b1.pack
error: inflate: data stream error (incorrect data check)
error: failed to read delta base object b901371ca361a1e6eafa411f5b144f6e8866aa9c at offset 4836578 from .git/objects/pack/pack-ecb1f703ea9732b91e3a423176aab38fc69514b1.pack
error: inflate: data stream error (incorrect data check)
error: failed to read delta base object 6650df70bb35aa48320ed25d5573703be789ea18 at offset 5517189 from .git/objects/pack/pack-ecb1f703ea9732b91e3a423176aab38fc69514b1.pack
error: inflate: data stream error (incorrect data check)

[xnox]

@gemorin the original bug is about deflate, which seems to work correctly. The error you are seeing is a new one about inflate, no? If yes, then it's a new/different bug to investigate. For my use case, I no longer see deflate errors from git and thus the "deflate error when using git" is fixed. Should we open a new issue about inflate error?

[gemorin]

Hmm, I would assume that the errors are connected, don't you think? If deflate corrupts the file, inflate will fail.

[jtkukunas]

I'm going to assume that these are related until we determine otherwise.

gemorin,

I can't seem to reproduce this. Would you be able to put together a reproducible test case?

Thanks.

[gemorin]

I am trying but so far I can only reproduce it with a specific tree which I can't make publicly available

[jtkukunas]

Okay. Could you provide some system information (OS/kernel, compiler version, git version) to help me narrow my search?

[gemorin]

Sure, it's a debian 7.6 with a 3.14 kernel, gcc 4.7, git 2.0.1

CPU: Intel(R) Core(TM) i7-3820 CPU @ 3.60GHz