Question about the details of ig.py

[GuPeijian]

Hello, nice work! I have a question about the details of ig.py in line140-line150:

if "llava" in model_name or "gemma" in model_name:
    original_ffn_activations = [
        original_ffn_activations[layer].detach()[:, 0, :]
        for layer in original_ffn_activations
    ]
if "Qwen" in model_name:
    original_ffn_activations = [
        original_ffn_activations[layer].detach()[:, 0] # (bs, ffn_hidden_size)
        for layer in original_ffn_activations
    ]

I wonder why you use the ffn_activations in postion 0 of the whole sequence. I think these tokens are all the same such as the bos token for all input texts and have no further information considering the causal setting of the model.