Inquiry About DMN Reproduction: Gap Between Paper Results (Table 3) and GitHub Logs

[1108zyzy]

Thanks for your great work on Dual Memory Networks. I am trying to reproduce the results in Table 3 of your paper using the code from your GitHub repository.

However, I noticed some differences between the numbers in the provided log files and the paper. For example, on DTD, the log file camera_ready_dmn_tf_searched_vit_0.txt reports 54.61% accuracy, while the paper reports 55.85%. A similar ~1-2% gap is also observed on EuroSAT.

I would like to confirm if there are any differences in the experimental setup that I might have missed. Specifically, I am wondering about:

1. Whether the reported results in the paper are averaged over multiple random seeds runs?
2. Any specific data preprocessing steps (e.g., multi-scale testing, multi-crop, test-time augmentation) that might not be in the default code?
3. Whether you used any ensemble of different models or hyperparameters for the final numbers?
4. Any other details such as beta, selection_p, or memory_size that were tuned per dataset?

Thank you for your time and help!

[YBZh]

Thanks for your comments.

1. Not yet. However, we conducted experiments on ImageNet and observed very small fluctuations with different seeds, as presented in the paper.

2. No specific data preprocessing steps are used.

3. No ensemble used.

4. We do not tune the hyperparameters per dataset.

Regarding the difference you noticed on the DTD dataset, this might be due to the following reason: In our initial experiments, we used the APE method (https://arxiv.org/pdf/2304.01195) to select important channels, which you can see in the code (e.g., num_important_channel, lambda_ape). However, in the released code, we disabled this method (by setting num_important_channel=0). Using the APE method might bring about an additional 1% improvement. Even without this technique, our method still significantly outperforms competitors and is essentially comparable to the reported results (less than 1% difference). If you want to fully reproduce the paper results or achieve enhanced results, you can try activating num_important_channel or even search for num_important_channel per dataset like APE.

[1108zyzy]

First of all, thank you for your great work on the paper and for making the code publicly available. It has been very helpful for my research.

I am writing to you regarding an issue I encountered while trying to reproduce the results on the DTD dataset. I noticed a slight performance gap compared to the numbers reported in the paper. After reading your previous response (in the GitHub issues), I learned that this might be due to the disabled APE module in the released code.

As you suggested, I have tried to activate the APE module by setting num_important_channel to a non-zero value (e.g., 512...) and also adjusted lambda_ape. However, the performance remains largely unchanged from the baseline without APE.

Do you have any recommended hyperparameter combinations (like num_important_channel and lambda_ape)

Thank you very much for your time and support. I greatly appreciate your help in resolving this matter.

[1108zyzy]

Dear author,

Thank you for your work . In my experiments, the discrepancy I raised earlier mainly appears in the zero-shot setting. I now realize APE’s core gains rely on the few-shot cache (support), so pure zero-shot will inevitably reduce to the standard CLIP text–image score.

Besides APE, what other reasons could cause a 1–2 point drop in zero-shot accuracy?