Question about the encode function in the huggingface DeepSeek-DeepEncoder

[hhlim333]

Can you provide the encode function for the following demo that you provided in huggingface?

import torch
from deepencoder import build_sam_vit_b, build_clip_l, MlpProjector
from easydict import EasyDict as adict

Load models

sam = build_sam_vit_b(checkpoint=None)
sam.load_state_dict(torch.load('sam_encoder.pth'))

clip = build_clip_l()
clip.load_state_dict(torch.load('clip_encoder.pth'))

projector_cfg = adict({'projector_type': 'linear', 'input_dim': 2048, 'n_embed': 1280})
projector = MlpProjector(projector_cfg)
projector.load_state_dict(torch.load('projector.pth'))

Run encoder

vision_tokens = encode(image) # [1, 256, 1280]

Thank you for your help.