Add new model EfficientVit-SAM to transformers#47275
Conversation
…all variants - Update get_config in convert_efficientvitsam_to_hf.py to use exact string matching instead of substring checks, resolving an issue where the 'xl0' model was incorrectly matched to 'l0' configuration. - Update test_l0_hf/test.py to loop through and validate all 5 model variations (l0, l1, l2, xl0, xl1) against their original checkpoints. - Verify 100% key and shape mapping equivalence across all 5 variations.
|
Thank you for your contribution 🤗! CI Security Gate — automatic approval blockedThis PR was not automatically approved for CI because the security gate failed. Possible reasons:
See the workflow run for the exact violations. A maintainer can review and manually approve CI if a finding is a false positive. |
|
cc @molbap |
molbap
left a comment
molbap
left a comment
There was a problem hiding this comment.
Hello @abhishekkapoorx , I started reviewing but it seems the current implementation is still a draft/not aligned with our library's standards, using modular transformers to the max, attention interface, and so on: I recommend running this experimental self-review skill on your end and then coming back, it'll help us a lot!
| # ============================================================================= | ||
| # 2. Helpers and Basic Layers | ||
| # ============================================================================= | ||
|
|
||
|
|
||
| def val2list(x: list | tuple | Any, repeat_time: int = 1) -> list: | ||
| if isinstance(x, (list, tuple)): | ||
| return list(x) | ||
| return [x for _ in range(repeat_time)] | ||
|
|
||
|
|
||
| def val2tuple(x: list | tuple | Any, min_len: int = 1, idx_repeat: int = -1) -> tuple: | ||
| x = val2list(x) | ||
| if len(x) > 0: | ||
| x[idx_repeat:idx_repeat] = [x[idx_repeat] for _ in range(min_len - len(x))] | ||
| return tuple(x) | ||
|
|
||
|
|
||
| def get_same_padding(kernel_size: int | tuple[int, ...]) -> int | tuple[int, ...]: | ||
| if isinstance(kernel_size, tuple): | ||
| return tuple([get_same_padding(ks) for ks in kernel_size]) | ||
| else: | ||
| assert kernel_size % 2 > 0, "kernel size should be odd number" | ||
| return kernel_size // 2 | ||
|
|
||
|
|
||
| class LayerNorm2d(nn.LayerNorm): | ||
| def forward(self, x: torch.Tensor) -> torch.Tensor: | ||
| out = x - torch.mean(x, dim=1, keepdim=True) | ||
| out = out / torch.sqrt(torch.square(out).mean(dim=1, keepdim=True) + self.eps) | ||
| if self.elementwise_affine: | ||
| out = out * self.weight.view(1, -1, 1, 1) + self.bias.view(1, -1, 1, 1) | ||
| return out | ||
|
|
||
|
|
||
| def build_norm(name: str, num_features: int, **kwargs) -> nn.Module: | ||
| if name == "bn2d": | ||
| return nn.BatchNorm2d(num_features, **kwargs) | ||
| elif name == "ln": | ||
| return nn.LayerNorm(num_features, **kwargs) | ||
| elif name == "ln2d": | ||
| return LayerNorm2d(num_features, **kwargs) | ||
| else: | ||
| return nn.Identity() | ||
|
|
||
|
|
||
| def build_act(name: str | None) -> nn.Module: | ||
| if name == "relu": | ||
| return nn.ReLU() | ||
| elif name == "relu6": | ||
| return nn.ReLU6() | ||
| elif name == "hswish": | ||
| return nn.Hardswish() | ||
| elif name == "silu": | ||
| return nn.SiLU() | ||
| elif name == "gelu": | ||
| return nn.GELU(approximate="tanh") | ||
| else: | ||
| return nn.Identity() |
There was a problem hiding this comment.
we have existing layers and helpers + activation function mapping in the library, we should not need this
| self.norm = build_norm(norm, num_features=out_channels) if norm else None | ||
| self.act = build_act(act_func) if act_func else None |
There was a problem hiding this comment.
typically, it's better to derive it from config here, and pass config to the Module init.
| def relu_quadratic_att(self, qkv: torch.Tensor) -> torch.Tensor: | ||
| B, _, H, W = list(qkv.size()) | ||
| device_type = qkv.device.type | ||
| autocast_context = ( | ||
| torch.autocast(device_type=device_type, enabled=False) | ||
| if device_type in {"cuda", "cpu"} | ||
| else contextlib.nullcontext() | ||
| ) | ||
|
|
||
| with autocast_context: | ||
| if qkv.dtype in [torch.float16, torch.bfloat16]: | ||
| qkv = qkv.float() | ||
|
|
||
| qkv = torch.reshape( | ||
| qkv, | ||
| ( | ||
| B, | ||
| -1, | ||
| 3 * self.dim, | ||
| H * W, | ||
| ), | ||
| ) | ||
| q, k, v = ( | ||
| qkv[:, :, 0 : self.dim], | ||
| qkv[:, :, self.dim : 2 * self.dim], | ||
| qkv[:, :, 2 * self.dim :], | ||
| ) | ||
|
|
||
| q = self.kernel_func(q) | ||
| k = self.kernel_func(k) | ||
|
|
||
| att_map = torch.matmul(k.transpose(-1, -2), q) # b h n n | ||
| att_map = att_map / (torch.sum(att_map, dim=2, keepdim=True) + self.eps) # b h n n | ||
| out = torch.matmul(v, att_map) # b h d n | ||
|
|
||
| out = torch.reshape(out, (B, -1, H, W)) | ||
| return out |
There was a problem hiding this comment.
This should be made to be aligned with current attention implementation, defining an eager_attention_forward on the side, and so on.
Thanks for the review! I will revert back after making the necessary changes. |
…SamVisionConfig`. - move attention to `eager_attention_forward` in generated modeling file
… norm and act_func
- refactor config for EfficientViTSamConfig
…odel in EfficientViTSamConfig
- remove eager attention for LiteMLA block (its diff from transformer attention) - use get_activation for getting activation
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, efficientvitsam |
CI recapDashboard: View test results in Grafana |
4 participants
What does this PR do?
Add new model
EfficientVit-SAMto transformersFixes #45175
Code Agent Policy
The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. These often are low-quality, or fix extremely minor issues that occur rarely or never in practice.
As a result, we're instituting a rule that first-time contributors should not use code agents to submit PRs or issues.
We'd also ask autonomous "OpenClaw"-like agents not to open any PRs or issues.
Issues/PRs from first-time contributors that violate this rule will probably just be closed without review, and we
might block you, especially if you open more than one or appear to be deliberately ignoring this. We especially do not
want new contributors to jump in on random issues to contribute an agent-written fix. This creates lots of noise
for reviewers and other users and will almost certainly get you blocked.
For more information, please read
CONTRIBUTING.md.Before submitting
Pull Request checks?
to it if that's the case.
Feature request: Add EfficientViT-SAM (efficientvitsam) to Transformers #45175
Who can review?
@yonigozlan @Cyrilvallez can help to review and make it to production.