A Dataset and Framework for Learning State-invariant Object Representations

ObjectsWithStateChange (OWSC) Dataset

We introduce a new dataset of 406 household objects from 21 categories, undergoing diverse state changes in addition to other transformations such as pose and viewpoint changes. The goal of introducing this dataset is to facilitate research in learning object representations that are invariant to state changes while also staying invariant to transformations induced by changes in viewpoint, pose, illumination, etc., for fine-grained recognition and retrieval. Details of the dataset are provided in our paper.

The OWSC dataset comprises 13837 images of 406 household objects from 21 categories, captured using smartphone cameras under various state changes from arbitrary viewpoints. The dataset is divided into two splits:

OWSC-SI for state invariance:

This split is to evaluate invariance across different states and other transformations. It comprises 11328 images of 331 objects, randomly partitioned into:

• Test Set of 3,428 images (≈ 10 per object).
• Train Set of 7,900 images (≈ 24 per object) with no overlap between the two partitions.

OWSC-GN for generalization to novel objects:

This split is to test generalization to novel objects. It comprises 2509 images of 75 unseen objects (not present in OWSC-SI), randomly partitioned into:

• Gallery Set: Images of one state per object
• Probe Set: Images from the remaining states per object, with no overlap between the two partitions.

This split is to evaluate the recognition and retrieval of novel objects amidst state variations and other transformations by correctly matching the probe images of the objects with their gallery images.

The datasets are organized such that the images of each object identity are stored in a separate subfolder with an integer ID indicating the object-identity. The mapping of object-identities to categories (also indicated by an integer ID) are also provided as a split_o2c.npy file.

The dataset can be downloaded from Google Drive.

Evaluation

Requirements and Setup

Please clone this repo and install the dependencies using:

conda env create -f environment_owsc.yml

Downloading datasets

Download the datasets (OWSC, ObjectPI [1], ModelNet-40 [2], and FG3D [3]) from Google Drive

Please unzip the data.zip file using and place the datasets in a folder named data

unzip data.zip

We have organized these datasets such that the multi-view images of each object identity are stored in a separate subfolder with an integer ID indicating the object-identity. The train and test splits for the above-mentioned datasets can be downloaded from the link provided above. The mapping of object-identities to categories is also provided as train_o2c.npy and test_o2c.npy files.

Downloading our trained models

Download the model weights from Google Drive and place them in a folder named model_weights

Benchmarking different methods using our dataset:

In the paper, eight invariant recognition and retrieval tasks are proposed. These tasks are category and object-based where either a single or multiple images are used during inference.

To evaluate different methods on these tasks, please run the following commands:

OWSC-SI split

For Our method (using curriculum learning):

python evaluate_OWSC_SI.py ours model_weights/OWSC/PiRO2024/Ours_OWSCSI_CURRICULUM_nH1_nL2_2.pth 1 2

For PiRO method:

python evaluate_OWSC_SI.py piro model_weights/OWSC/PiRO2024/PiRO_OWSCSI_RAND_CATG_nH1_nL1.pth 1 1

OWSC-GN split

For Our method (using curriculum learning):

python evaluate_OWSC_GN.py ours model_weights/OWSC/PiRO2024/Ours_OWSCGN_CURRICULUM_nH1_nL2.pth 1 2

For PiRO method:

python evaluate_OWSC_GN.py piro model_weights/OWSC/PiRO2024/PiRO_OWSCGN_RAND_CATG_nH1_nL1.pth 1 1

For PI-CNN, PI-Proxy, and PI-TC methods:

python evaluate_OWSC_SI/GN.py picnn model_weights/OWSC/PIE2019/PICNN_1.0_1.0_1.0_1_150.pth 1 1
python evaluate_OWSC_SI/GN.py piprx model_weights/OWSC/PIE2019/PIPRX_1.0_1.0_1.0_1_150.pth 1 1
python evaluate_OWSC_SI/GN.py pitc model_weights/OWSC/PIE2019/PITC_1.0_0.2_1.0_1_150.pth 1 1

Ablation for our Curriculum Learning Approach:

For this ablation, we compare performance of the same dual-encoder model with same number of self-attention layer and heads trained using different object pair sampling strategies:

• Randomly sampling object pairs from the same category (models named as RAND_CATG)
• Mining object pairs based on our Curriculum Learning approach (models named as CURRICULUM)

For Random Sampling from Same Category using PiRO's dual-encoder architecture (with nHeads = 1, nLayers = 1)

python evaluate_curriculum.py OWSC model_weights/OWSC/PiRO2024/PiRO_OWSCSI_RAND_CATG_nH1_nL1.pth 1 1

For Curriculum Learning using the same architecture (with nHeads = 1, nLayers = 1)

python evaluate_curriculum.py OWSC model_weights/OWSC/PiRO2024/PiRO_OWSCSI_CURRICULUM_nH1_nL1.pth 1 1

Similarly, for comparing performance with random sampling from same category (RAND_CATG) and Curriculum Learning (CURRICULUM) on pose-invariant tasks using the other multi-view datasets, please run the following commands:

For ObjectPI (OOWL):

python evaluate_curriculum.py OOWL model_weights/ObjectPI/RAND_CATG_OOWL_nH1_nL1.pth 1 1
python evaluate_curriculum.py OOWL model_weights/ObjectPI/CURRICULUM_OOWL_nH1_nL1.pth 1 1

For ModelNet-40:

python evaluate_curriculum.py MNet40 model_weights/ModelNet40/RAND_CATG_MNet40_nH1_nL1.pth 1 1
python evaluate_curriculum.py MNet40 model_weights/ModelNet40/CURRICULUM_MNet40_nH1_nL1.pth 1 1 

For FG3D:

python evaluate_curriculum.py FG3D model_weights/FG3D/RAND_CATG_FG3D_nH1_nL1.pth 1 1
python evaluate_curriculum.py FG3D model_weights/FG3D/CURRICULUM_FG3D_nH1_nL1.pth 1 1 

Ablation for Architecture:

For this ablation, we compare performance of models with different number of self-attention layers trained using the same curriculum learning approach.

For model with self-attention nHeads = 1 and nLayers = 1 trained using curriculum learning on our OWSC dataset:

python evaluate_OWSC_SI.py ours model_weights/OWSC/PiRO2024/PiRO_OWSCSI_CURRICULUM_nH1_nL1.pth 1 1

For model with self-attention nHeads = 1 and nLayers = 2 trained using curriculum learning on our OWSC dataset:

python evaluate_OWSC_SI.py ours model_weights/OWSC/PiRO2024/Ours_OWSCSI_CURRICULUM_nH1_nL2.pth 1 2

References

[1] Chih-Hui Ho, Pedro Morgado, Amir Persekian, and Nuno Vasconcelos. PIEs: Pose invariant embeddings. In Computer Vision and Pattern Recognition (CVPR), 2019.
[2] Zhirong Wu, S. Song, A. Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and J. Xiao. 3D shapenets: A deep representation for volumetric shapes. In Computer Vision and Pattern Recognition (CVPR), pages 1912–1920, Los Alamitos, CA, USA, 2015.
[3] Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Fine-grained 3D shape classification with hierarchical part-view attentions. IEEE Transactions on Image Processing, 2021.
[4] Rohan Sarkar, Avinash Kak; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 17077-17085

Citation

If you use the dataset or curriculum learning approach in your work, please cite our paper:

@InProceedings{Sarkar_2026_WACV,
    author    = {Sarkar, Rohan and Kak, Avinash},
    title     = {A Dataset and Framework for Learning State-invariant Object Representations},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {March},
    year      = {2026},
    pages     = {4715-4723}
}