Model issue

[zjd2024]

(uncomp) jdzhou@MANLP-yliu:/exper/uncomp$ bash ./scripts/scripts_longBench/eval.sh --max_capacity_prompts 512 --attn_implementation eager --source_path ./results/ --model_path /home/jdzhou/exper/uncomp/models/tinyllama --eval_batch_size 1 --method uncomp --name ./output_tinyllama_1b --gpu_id 0 --fp16 1 --seed 43 --logger_pattern info --port 1236
Current path: /home/jdzhou/exper/uncomp
Output Catalogue /home/jdzhou/exper/uncomp/./output_tinyllama_1b/
2025-11-28 22:08:46,983 - [Process 0/1] - INFO - args.seed is 43
2025-11-28 22:08:47,012 - [Process 0/1] - INFO - model_path is /home/jdzhou/exper/uncomp/models/tinyllama
2025-11-28 22:08:47,012 - [Process 0/1] - INFO - uncomp
2025-11-28 22:08:48,427 - [Process 0/1] - INFO - Working on max_capacity_prompts 512 dataset narrativeqa - 0/16
2025-11-28 22:08:48,427 - [Process 0/1] - INFO - Loading data...
2025-11-28 22:08:48,460 - [Process 0/1] - INFO - Max Length is 36418
2025-11-28 22:08:48,461 - [Process 0/1] - INFO - Finish loading model and tokenizer
0%| | 0/200 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/jdzhou/exper/uncomp/run_longbench.py", line 1329, in
main(args,manager)
File "/home/jdzhou/exper/uncomp/run_longbench.py", line 291, in main
output = model.generate(
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/transformers/generation/utils.py", line 1527, in generate
result = self._greedy_search(
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/transformers/generation/utils.py", line 2411, in _greedy_search
outputs = self(
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1196, in forward
outputs = self.model(
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1016, in forward
layer_outputs = decoder_layer(
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jdzhou/exper/uncomp/uncomp/llama_model.py", line 71, in LlamaDecoderLayer_forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/jdzhou/exper/uncomp/uncomp/llama_model.py", line 958, in llama_attn_forward_Uncomp
key_states_compress, value_states_compress = self.kv_cluster.update_kv(key_states, query_states, value_states, attn_weights[:,:,-self.kv_cluster.window_size:,:],attn_weights)
File "/home/jdzhou/exper/uncomp/uncomp/uncomp_utils.py", line 316, in update_kv
self.head_indices1 = self.head_indices1.to(indices1.device)
AttributeError: 'int' object has no attribute 'to'
Traceback (most recent call last):
File "/home/jdzhou/miniconda3/envs/uncomp/bin/accelerate", line 7, in
sys.exit(main())
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command
simple_launcher(args)
File "/home/jdzhou/miniconda3/envs/uncomp/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/jdzhou/miniconda3/envs/uncomp/bin/python3.10', 'run_longbench.py', '--method', 'uncomp', '--seed', '43', '--model_path', '/home/jdzhou/exper/uncomp/models/tinyllama', '--max_capacity_prompts', '512', '--attn_implementation', 'eager', '--save_dir', './results/results_long_bench', '--use_cache', 'True', '--method_name', 'uncomp', '--fphalf', '1', '--pattern', 'info', '--eval_batch_size', '1']' returned non-zero exit status 1.
(uncomp) jdzhou@MANLP-yliu:/exper/uncomp$

Are there any models that are compatible with 1B–2B?
If they are not compatible, a lot of errors will occur.

[menik1126]

If you need to support more models such as TinyLlama, note that its architecture is different from Llama, so you’ll need to modify the class files accordingly to make them compatible. @zjd2024

[menik1126]

We’ve now fixed this issue, and tinyLLaMA is supported. @zjd2024