text:
RuntimeError: The size of tensor a (0) must match the size of tensor b (5632) at non-singleton dimension 1

dataset: pixel200k
scripts: HLLM-Pixel.sh
batchsize:2

Hi. Could you provide more logs and information about your env? It's good on our system.

Hi. Could you provide more logs and information about your env? It's good on our system.
env：torch2.0.1-cuda11.7 device:v100-32g(same error in A100-80g)
The complete log is as follows：
run_yaml:  ../TORCHRUN run.py --config_file overall/LLM_deepspeed.yaml HLLM/HLLM.yaml --MAX_ITEM_LIST_LENGTH 10 --epochs 5 --optim_args.learning_rate 1e-4 --checkpoint_dir checkpoint_dir --loss nce --MAX_TEXT_LENGTH 256 --dataset Pixel200K --text_path /sfs/users/zcj3/llm4rec/HLLM/information --item_pretrain_dir /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4 --user_pretrain_dir /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4 --train_batch_size 1 --eval_batch_size 4 --gradient_checkpointing True --stage 3
++ date +%FT%T

start_time=2024-10-30T16:40:41
[[ '' == '' ]]
[[ '' == '' ]]
nnodes=1
node_rank=0
master_port=12345
nproc_per_node=1
torchrun --master_port=12345 --node_rank=0 --nproc_per_node=1 --nnodes=1 run.py --config_file overall/LLM_deepspeed.yaml HLLM/HLLM.yaml --MAX_ITEM_LIST_LENGTH 10 --epochs 5 --optim_args.learning_rate 1e-4 --checkpoint_dir checkpoint_dir --loss nce --MAX_TEXT_LENGTH 256 --dataset Pixel200K --text_path /sfs/users/zcj3/llm4rec/HLLM/information --item_pretrain_dir /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4 --user_pretrain_dir /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4 --train_batch_size 1 --eval_batch_size 4 --gradient_checkpointing True --stage 3
[2024-10-30 16:40:49,282] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
local_rank:  0
Unable to convert the string 'checkpoint_dir' to None / Bool / Float / Int, retaining the original string.
Unable to convert the string 'nce' to None / Bool / Float / Int, retaining the original string.
Unable to convert the string 'Pixel200K' to None / Bool / Float / Int, retaining the original string.
Unable to convert the string '/sfs/users/zcj3/llm4rec/HLLM/information' to None / Bool / Float / Int, retaining the original string.
Unable to convert the string '/sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4' to None / Bool / Float / Int, retaining the original string.
Unable to convert the string '/sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4' to None / Bool / Float / Int, retaining the original string.
30 Oct 16:40    INFO  Update text_path to /sfs/users/zcj3/llm4rec/HLLM/information/Pixel200K.csv
30 Oct 16:40    INFO  Loading <class 'REC.data.dataload.Data'> from scratch with self.data_split = None.
30 Oct 16:41    INFO  Interaction feature loaded successfully from [../dataset/Pixel200K.csv].
30 Oct 16:41    INFO  self.user_num = 200001 self.item_num = 96283
30 Oct 16:41    INFO  self.inter_feat['item_id'].isna().any() = False self.inter_feat['user_id'].isna().any() = False
30 Oct 16:41    INFO  build Pixel200K dataload
30 Oct 16:41    INFO  Use random sample True for mask id
30 Oct 16:41    INFO  Text path: /sfs/users/zcj3/llm4rec/HLLM/information/Pixel200K.csv
30 Oct 16:41    INFO  Text keys: ['title', 'tag', 'description']
30 Oct 16:41    INFO  Item prompt: Compress the following sentence into embedding:
30 Oct 16:41    INFO  Text Item num: 96281
30 Oct 16:41    INFO  [Training]: train_batch_size = [1]
30 Oct 16:41    INFO  [Evaluation]: eval_batch_size = [4]
len(train_loader) = 303525
30 Oct 16:41    INFO  create item llm
30 Oct 16:41    INFO  ******* create LLM /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4 *******
30 Oct 16:41    INFO  hf_config: LlamaConfig {
"_name_or_path": "/sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 5632,
"max_position_embeddings": 2048,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 22,
"num_key_value_heads": 4,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 10000.0,
"tie_word_embeddings": false,
"torch_dtype": "float32",
"transformers_version": "4.39.3",
"use_cache": false,
"vocab_size": 32003
}

30 Oct 16:41    INFO  xxxxx starting loading checkpoint
30 Oct 16:41    INFO  Using flash attention False for llama
30 Oct 16:41    INFO  Init True for llama
You are using an old version of the checkpointing format that is deprecated (We will also silently ignore gradient_checkpointing_kwargs in case you passed it).Please update to the new format on your modeling file. To use the new format, you need to completely remove the definition of the method _set_gradient_checkpointing in your model.
30 Oct 16:41    INFO  create user llm
30 Oct 16:41    INFO  ******* create LLM /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4 *******
30 Oct 16:41    INFO  hf_config: LlamaConfig {
"_name_or_path": "/sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4",
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 2048,
"initializer_range": 0.02,
"intermediate_size": 5632,
"max_position_embeddings": 2048,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 22,
"num_key_value_heads": 4,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 10000.0,
"tie_word_embeddings": false,
"torch_dtype": "float32",
"transformers_version": "4.39.3",
"use_cache": false,
"vocab_size": 32003
}
30 Oct 16:41    INFO  xxxxx starting loading checkpoint
30 Oct 16:41    INFO  Using flash attention False for llama
30 Oct 16:41    INFO  Init True for llama
You are using an old version of the checkpointing format that is deprecated (We will also silently ignore gradient_checkpointing_kwargs in case you passed it).Please update to the new format on your modeling file. To use the new format, you need to completely remove the definition of the method _set_gradient_checkpointing in your model.
30 Oct 16:41    INFO  nce thres setting to 0.99
30 Oct 16:41    INFO  item_emb_tokens torch.Size([1, 1, 2048]) True
30 Oct 16:41    INFO  logit_scale torch.Size([]) True
30 Oct 16:41    INFO  item_llm.model.embed_tokens.weight torch.Size([32003, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.0.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.1.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.2.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.3.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.4.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.5.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.6.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.7.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.8.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.9.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.10.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.11.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.12.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.13.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.14.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.15.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.16.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.17.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.18.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.19.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.20.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.layers.21.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.model.norm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  item_llm.lm_head.weight torch.Size([32003, 2048]) True
30 Oct 16:41    INFO  user_llm.model.embed_tokens.weight torch.Size([32003, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.0.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.1.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.2.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.3.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.4.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.5.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.6.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.7.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.8.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.9.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.10.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.11.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.12.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.13.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.14.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.15.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.16.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.17.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.18.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.19.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.20.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.self_attn.q_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.self_attn.k_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.self_attn.v_proj.weight torch.Size([256, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.self_attn.o_proj.weight torch.Size([2048, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.mlp.gate_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.mlp.up_proj.weight torch.Size([5632, 2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.mlp.down_proj.weight torch.Size([2048, 5632]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.input_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.layers.21.post_attention_layernorm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.model.norm.weight torch.Size([2048]) True
30 Oct 16:41    INFO  user_llm.lm_head.weight torch.Size([32003, 2048]) True
30 Oct 16:41    INFO
World_Size = 1
30 Oct 16:41    INFO
General Hyper Parameters:
seed = 2020
state = INFO
use_text = True
reproducibility = True
checkpoint_dir = checkpoint_dir
show_progress = True
log_wandb = False
data_path = ../dataset/
strategy = deepspeed
precision = bf16-mixed
model = HLLM
Training Hyper Parameters:
epochs = 5
train_batch_size = 1
optim_args = {'learning_rate': 0.0001, 'weight_decay': 0.01}
eval_step = 1
stopping_step = 5
Evaluation Hyper Parameters:
eval_batch_size = 4
topk = [5, 10, 50, 200]
metrics = ['Recall', 'NDCG']
valid_metric = NDCG@200
metric_decimal_place = 7
eval_type = EvaluatorType.RANKING
valid_metric_bigger = True
Dataset Hyper Parameters:
MAX_ITEM_LIST_LENGTH = 10
MAX_TEXT_LENGTH = 256
text_keys = ['title', 'tag', 'description']
item_prompt = Compress the following sentence into embedding:
Other Hyper Parameters:
wandb_project = REC
text_path = /sfs/users/zcj3/llm4rec/HLLM/information/Pixel200K.csv
item_emb_token_n = 1
loss = nce
scheduler_args = {'type': 'cosine', 'warmup': 0.1}
stage = 3
item_pretrain_dir = /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4
item_llm_init = True
user_pretrain_dir = /sfs/users/zcj3/llm4rec/HLLM/pretrain_dir/TinyLlama-1.1B-Chat-v0.4
user_llm_init = True
use_ft_flash_attn = False
MODEL_INPUT_TYPE = InputType.SEQ
device = cuda:0
gradient_checkpointing = True
30 Oct 16:41    INFO  Pixel200K
The number of users: 200001
Average actions of users: 19.82828
The number of items: 96283
Average actions of items: 41.187927130720176
The number of inters: 3965656
The sparsity of the dataset: 99.9794063532928%
30 Oct 16:41    INFO  HLLM(
(item_llm): LlamaForCausalLM(
(model): LlamaModel(
(embed_tokens): Embedding(32003, 2048)
(layers): ModuleList(
(0-21): 22 x LlamaDecoderLayer(
(self_attn): LlamaAttention(
(q_proj): Linear(in_features=2048, out_features=2048, bias=False)
(k_proj): Linear(in_features=2048, out_features=256, bias=False)
(v_proj): Linear(in_features=2048, out_features=256, bias=False)
(o_proj): Linear(in_features=2048, out_features=2048, bias=False)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): LlamaMLP(
(gate_proj): Linear(in_features=2048, out_features=5632, bias=False)
(up_proj): Linear(in_features=2048, out_features=5632, bias=False)
(down_proj): Linear(in_features=5632, out_features=2048, bias=False)
(act_fn): SiLUActivation()
)
(input_layernorm): LlamaRMSNorm()
(post_attention_layernorm): LlamaRMSNorm()
)
)
(norm): LlamaRMSNorm()
)
(lm_head): Linear(in_features=2048, out_features=32003, bias=False)
)
(user_llm): LlamaForCausalLM(
(model): LlamaModel(
(embed_tokens): Embedding(32003, 2048)
(layers): ModuleList(
(0-21): 22 x LlamaDecoderLayer(
(self_attn): LlamaAttention(
(q_proj): Linear(in_features=2048, out_features=2048, bias=False)
(k_proj): Linear(in_features=2048, out_features=256, bias=False)
(v_proj): Linear(in_features=2048, out_features=256, bias=False)
(o_proj): Linear(in_features=2048, out_features=2048, bias=False)
(rotary_emb): LlamaRotaryEmbedding()
)
(mlp): LlamaMLP(
(gate_proj): Linear(in_features=2048, out_features=5632, bias=False)
(up_proj): Linear(in_features=2048, out_features=5632, bias=False)
(down_proj): Linear(in_features=5632, out_features=2048, bias=False)
(act_fn): SiLUActivation()
)
(input_layernorm): LlamaRMSNorm()
(post_attention_layernorm): LlamaRMSNorm()
)
)
(norm): LlamaRMSNorm()
)
(lm_head): Linear(in_features=2048, out_features=32003, bias=False)
)
)
Trainable parameters: 2200123393.0
30 Oct 16:41    INFO  Use consine scheduler with 151762.5 warmup 1517625 total steps
world_size: 1 local_world_size: 1
30 Oct 16:41    INFO  Use deepspeed strategy
initializing deepspeed distributed: GLOBAL_RANK: 0, MEMBER: 1/1
Enabling DeepSpeed BF16. Model parameters and inputs will be cast to bfloat16.
30 Oct 16:41    INFO  Added key: store_based_barrier_key:2 to store for rank: 0
30 Oct 16:41    INFO  Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 1 nodes.
Parameter Offload: Total persistent parameters: 186369 in 92 params
Train [  0/  5]:   0%|                                                                                | 0/303525 [00:00<?, ?it/s]/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
[debug] data:
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
{'pos_item_ids': tensor([[24071,   613,  2968, 16345, 37844, 12581, 31803,  2716,  1362, 27250,
6276]]), 'neg_item_ids': tensor([[20766, 81222, 81198, 22876, 87871, 60427, 57998, 62150, 48319, 54961,
71171]]), 'pos_input_ids': tensor([    1,   422,  2139,  ..., 20556,  5648,     0]), 'pos_cu_input_lens': tensor([ 81, 122,  77,  46,  42, 257,  97, 102,  68, 102,  70]), 'pos_position_ids': tensor([176, 177, 178,  ..., 254, 255, 256]), 'neg_input_ids': tensor([    1,   422,  2139,   278,  1494, 10541,   964, 23655, 29901,  3611,
29901,  1128,  1784,  2305,   505,  1063,  5796,  1312,   491,   278,
4103, 24784, 29973,   390,  3631, 14000,   505, 12456,   985, 17135,
29892,  6460,   734,   264, 18150, 26277,   278,  6460,   322,  5360,
278,  8261, 29973,   450,  8760,   338,  2089, 29991,  4039, 29901,
9586,  3305,  1383,  1242,  8216, 29901,  1094,   278,   937,   970,
2669,  1842,   653,  1824, 12789,  4746,   373, 12397,  5849,   297,
7551, 29892,  2020,   756,   376, 10095,   314,  5676, 19263, 29908,
4953,   278,  2181,   328,   280,   310, 12670, 20157, 29916,  5810,
29899, 28990, 29973,  1128,  1784,  7934,  8760, 29879,   526,   727,
5742,   278, 20407, 29973,     0,     1,   422,  2139,   278,  1494,
10541,   964, 23655, 29901,  3611, 29901,   518,  4495,   655, 29962,
19565,  2701,   674,  2867,   701,   565,   896,  1708,   411,  1269,
916, 29892,   825,  1048,  8743,   411,  7535, 29973,  3563,  1111,
12504,  4039, 29901, 16740, 29899,  9106, 12482,  8216, 29901,   450,
8448,  6811,  1111, 12504, 29908,  1106,   322,  4459,  2289,  7423,
363,   592,   373, 20147, 29901,  1732,   597,   705,   747, 29877,
29889,   510, 29914,     0,     1,   422,  2139,   278,  1494, 10541,
964, 23655, 29901,  3611, 29901,  1932,   278,  6940, 18930,   671,
315, 11960,   968,   304, 21334, 29883,   873,  7213,   895,  2718,
435,   713, 29892,  2718,   435,   713,  7339,  8908, 29892,  1999,
1657,   368, 29901,   508, 29915, 29873,  2274, 21023, 29991,  4039,
29901,   383,   273,  6760,   362,  8216, 29901,   450,  3564,     0,
1,   422,  2139,   278,  1494, 10541,   964, 23655, 29901,  3611,
29901, 24300,  5320, 10013,  2305, 13590,  2201, 23443,   400,   351,
29901,  3423,  4011,  8216, 29901,   450,  2793,   310,   278,  4863,
338, 24837, 28709,  1288,  8287, 14332,   304,  6505,   278,  4863,
1434,   366,   748,  1016, 29915, 29873,  9566,   304,  5967,  1023,
1302,  1144,  3855, 29893, 29939,     0,     1,   422,  2139,   278,
1494, 10541,   964, 23655, 29901,  3611, 29901,   518,  1451,  8233,
2896, 21706, 29962,  8905, 22467,  7339, 29885,  1228, 29915, 29879,
25616, 18988,   310,   306, 28593,   411,  2654, 11315,   338, 16232,
29991,  4039, 29901,   315,  6146, 26549,   583, 23478,  8216, 29901,
14711,     0,     1,   422,  2139,   278,  1494, 10541,   964, 23655,
29901,  3611, 29901,  4971,   453, 15472, 29901,   678, 17347,  4942,
453, 24403, 29901, 24620,  3321, 12453,  8216, 29901,   319,  1629,
310,   521, 17347,  6694,  8988,   714,   491, 25458, 29991,     0,
1,   422,  2139,   278,  1494, 10541,   964, 23655, 29901,  3611,
29901,   450,  1556,  9150, 17524,  8090,   310,   599,   931,   313,
27215,   467,  4039, 29901, 16740, 29899,  9106, 12482,     0,     1,
422,  2139,   278,  1494, 10541,   964, 23655, 29901,  3611, 29901,
518,  6156, 10403, 29962,  1724,   887, 20340,   292,  1152, 29915,
29879, 15586, 29884,   940, 29872, 10134,   324, 29915, 29879, 21256,
2909,  5735,  7408,  1722, 29991,  4039, 29901, 10782,  6125,  8216,
29901, 14711,     0,     1,   422,  2139,   278,  1494, 10541,   964,
23655, 29901,  3611, 29901, 29871, 31478,  6147,  1227,   847,   450,
8141,   936, 10079,   847,   341,  1446, 29884, 17029,  1920,   895,
29875, 31472, 29903,  1742,   322, 17784, 29883,   708,  6212,   547,
382, 12907, 28804,   265,   583,   310,   278,   382, 11303,   567,
287,  8168,   309,   523, 12630,   349, 29963, 29871, 31478, 29924,
4741, 10013, 10079,  1632,   283,   415,   351, 29901, 10343,  8216,
29901, 10564,   317,  1742,   669,  3925,   272, 29883,   708,  6212,
547,   382, 12907, 28804,   265,   583,   310,   278,   382, 11303,
567,   287,  8168,   309,   523, 12630,   349, 29963,     0,     1,
422,  2139,   278,  1494, 10541,   964, 23655, 29901,  3611, 29901,
518, 29903,  5098, 29962,   350, 29931, 11375, 29925,  1177, 29968,
29915, 29879,   830,  2467,   304, 24274,   292,   350,  6259, 29933,
19453, 27313,   383,  6247, 29968, 13315,   313, 29949,  3554,   806,
5564, 29897,   313,  8263,   271, 29889,  3725, 29963,  3919, 29923,
1430, 29897,  4039, 29901, 18189,  8565,  1558,  1080,  8216, 29901,
612, 12015,  7466, 29923,   317,  9851,  9362, 12788,   349,   682,
512, 29895, 29987,  1160, 29936,  1160, 29936, 24274,  7997, 14320,
29915, 29879, 19848,   746,   540, 23233,   383,   557,  1952,   313,
2541,  7623, 29899,   262, 29899, 12095, 29897,     0,     1,   422,
2139,   278,  1494, 10541,   964, 23655, 29901,  3611, 29901, 26647,
29899,  2780,   287,   365,  2878,   405,   404,  1601,  2475,   767,
1179,   304, 15220,   345,  1156,   321,  1218,  6808,   814,   322,
25088, 28396, 24745,   373,   967,  1250,  4039, 29901, 20929,  3729,
23584,  8216, 29901,  2216,   304,  3585,   278,  1652, 13163,  1302,
1144,  3013, 13271,   278,  9881, 29899,  2780,   287,   365,  2878,
405,   404,  2598,  2475,  8745,   304, 15220,   345,  1156,   321,
1218,  6808,   814,   322,  7689,   449,   292, 28396, 24745,   373,
967,  1250,     0]), 'neg_cu_input_lens': tensor([105,  69,  56,  56,  46,  38,  29,  44,  96,  99,  85]), 'neg_position_ids': tensor([152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165,
166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179,
180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193,
194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221,
222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235,
236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249,
250, 251, 252, 253, 254, 255, 256, 188, 189, 190, 191, 192, 193, 194,
195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208,
209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222,
223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236,
237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250,
251, 252, 253, 254, 255, 256, 201, 202, 203, 204, 205, 206, 207, 208,
209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222,
223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236,
237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250,
251, 252, 253, 254, 255, 256, 201, 202, 203, 204, 205, 206, 207, 208,
209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222,
223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236,
237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250,
251, 252, 253, 254, 255, 256, 211, 212, 213, 214, 215, 216, 217, 218,
219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232,
233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246,
247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 219, 220, 221, 222,
223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236,
237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250,
251, 252, 253, 254, 255, 256, 228, 229, 230, 231, 232, 233, 234, 235,
236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249,
250, 251, 252, 253, 254, 255, 256, 213, 214, 215, 216, 217, 218, 219,
220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247,
248, 249, 250, 251, 252, 253, 254, 255, 256, 161, 162, 163, 164, 165,
166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179,
180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193,
194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207,
208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221,
222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235,
236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249,
250, 251, 252, 253, 254, 255, 256, 158, 159, 160, 161, 162, 163, 164,
165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178,
179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192,
193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206,
207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220,
221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234,
235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248,
249, 250, 251, 252, 253, 254, 255, 256, 172, 173, 174, 175, 176, 177,
178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191,
192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205,
206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219,
220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233,
234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247,
248, 249, 250, 251, 252, 253, 254, 255, 256]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]), 'time_ids': tensor([[[2021,    2,    3,    3,   32,    7],
[2021,    2,    9,   13,   22,   59],
[2021,    2,   10,   12,   53,   38],
[2021,    2,   11,   14,   20,    7],
[2021,    2,   12,    5,   31,   13],
[2021,    2,   18,   15,    0,   45],
[2021,    2,   21,    4,   46,    8],
[2021,    2,   24,    3,    5,   54],
[2021,    2,   24,    9,   46,   47],
[2021,    2,   24,   15,   12,    4],
[2021,    2,   26,    7,    0,   43]]])}
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:41: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
storage = elem.storage().new_shared(numel)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [11], which does not match the required output shape [1, 11]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [10], which does not match the required output shape [1, 10]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
/workspace/llm4rec/HLLM/code/REC/data/dataset/collate_fn.py:43: UserWarning: An output with one or more elements was resized since it had shape [66], which does not match the required output shape [1, 11, 6]. This behavior is deprecated, and in a future PyTorch release outputs will not be resized unless they have zero elements. You can explicitly reuse an out tensor t by resizing it, inplace, to zero elements with t.resize_(0). (Triggered internally at ../aten/src/ATen/native/Resize.cpp:26.)
return torch.stack(batch, 0, out=out)
[debug] loss:
2.453125
[debug] total loss:
2.453125
Traceback (most recent call last):
File "/workspace/llm4rec/HLLM/code/run.py", line 140, in
run_loop(local_rank=local_rank, config_file=config_file, extra_args=extra_args)
File "/workspace/llm4rec/HLLM/code/run.py", line 110, in run_loop
best_valid_score, best_valid_result = trainer.fit(
File "/workspace/llm4rec/HLLM/code/REC/trainer/trainer.py", line 346, in fit
train_loss = self._train_epoch(train_data, epoch_idx, show_progress=show_progress)
File "/workspace/llm4rec/HLLM/code/REC/trainer/trainer.py", line 201, in _train_epoch
self.lite.backward(losses)
File "/opt/venv/lib/python3.10/site-packages/lightning/fabric/fabric.py", line 446, in backward
self._strategy.backward(tensor, module, *args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/lightning/fabric/strategies/strategy.py", line 188, in backward
self.precision.backward(tensor, module, *args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/lightning/fabric/plugins/precision/deepspeed.py", line 91, in backward
model.backward(tensor, *args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1976, in backward
self.optimizer.backward(loss, retain_graph=retain_graph)
File "/opt/venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
ret_val = func(*args, **kwargs)
File "/opt/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/stage3.py", line 2213, in backward
self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
File "/opt/venv/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
scaled_loss.backward(retain_graph=retain_graph)
File "/opt/venv/lib/python3.10/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/opt/venv/lib/python3.10/site-packages/torch/autograd/init.py", line 200, in backward
Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: The size of tensor a (0) must match the size of tensor b (5632) at non-singleton dimension 1

Could you please try to use the same pkgs as we show in requirement.txt ? There would be some bugs with gradient_checkpoint + deepspeed zero3 on some different versions.

Could you please try to use the same pkgs as we shown in requirement.txt ? There would be some bugs with gradient_checkpoint + deepspeed zero3 on some different versions.

Can you provide a complete Dockerfile? When I installed the requirement, various package versions did not match.

Can you provide a complete Dockerfile? When I installed the requirement, various package versions did not match.
We are sorry for using an internal docker image, open source a new dockerfile will take a considerable amount of time.

authors: cijiezheng ssyzeChen

repo: bytedance/HLLM


text:
Add indicator of maintainence mode

There should be an easily identifiable marker to show which alerts are in maintenance mode when the filter is disabled.

@thailandmatt Do you have a preference on how this should look?

kind of like labels in github - just a little bubble was what I was envisioning.

For visual's sake, this is what Matt is talking about:

source: gharchive/issue

authors: bwinterton thailandmatt

repo: byuoitav/smee

license_type: permissive

license_source: github-api


text:
Problem: normalize_unnamed_job throws exception w/ argument

I just discovered a bug in v0.5.0 where Quantum.Normalizer.normalize_unnamed_job is not properly handling jobs that take arguments. Reproduction:
iex(2)> ["* * * * * Kernel.node"] |> Enum.map(&Quantum.Normalizer.normalize/1)
[nil: %Quantum.Job{args: [], name: nil, nodes: [:nonode@nohost],
  schedule: "* * * * *", state: :active, task: {"Kernel", "node"}}]

iex(3)> ["* * * * * Kernel.node(self)"] |> Enum.map(&Quantum.Normalizer.normalize/1)
** (MatchError) no match of right hand side value: []
    (quantum) lib/quantum/normalizer.ex:68: Quantum.Normalizer.normalize_unnamed_job/1
    (quantum) lib/quantum/normalizer.ex:50: Quantum.Normalizer.normalize_job/1
    (quantum) lib/quantum/normalizer.ex:6: Quantum.Normalizer.normalize/1
     (elixir) lib/enum.ex:1043: anonymous fn/3 in Enum.map/2
     (elixir) lib/enum.ex:1387: Enum."-reduce/3-lists^foldl/2-0-"/3
     (elixir) lib/enum.ex:1043: Enum.map/2

Sorry for the convoluted execution job, I had to find a STDLIB method that optionally took an argument to make this easily reproducible, hence node.

Edit: In looking through the usage examples there is appears to be few cases of argument passing except the use of args w/ named jobs.
It'll require a bit of a refactor as I dont see an easy way to enable argument passing here (fix regex, pass opts to normalize_task, etc etc) so maybe it should just be made clear its a limitation within the readme?  Not sure how you would like to proceed.

@theoretick I guess you mean v1.5.0 :)
Actually usage examples describe task definition w/o args:
"* * * * *": {MyApp.MyModule, :my_method}

and
"* * * * * MyApp.MyModule.my_method"

Otherwise you should be able to send any parameter (e.g. functions):
"* * * * * MyApp.MyModule.my_method(fn(a, b) -> a + b end)"

but it's wrong way to define args. For this case you should use named jobs. Or it is possible to implement support for the following:
"* * * * *": {MyApp.MyModule, :my_method, [1, 2, 3]}

Anyway I think you just can change your config from
config :quantum, cron: [
  "* * * * * Kernel.node(self)"
]

to
config :quantum, cron: [
  some_job_name: [
    schedule: "* * * * *",
    task: &Kernel.node/1,
    args: [self]
  ]
]

Or for realtime job creation (from your example):
j = [
  schedule: "* * * * *",
  task: &Kernel.node/1,
  args: [self]
]
[{nil, j}] |> Enum.map(&Quantum.Normalizer.normalize/1)

@theoretick Thanks for reporting!
Our documentation should be improved on this point. PRs are welcome.
And if someone wants to implement support for @alPacino's idea "* * * * *": {MyApp.MyModule, :my_method, [1, 2, 3]}, PRs are welcome, too :-)

Ha, yes v1.5.0.  I ended up doing as you described replacing my config w/ "* * * * * MyApp.MyModule.my_method(fn(a, b) -> a + b end)" to get it working, but I wouldn't think this behavior is expected and seems limited.
It wasn't clear to me if the usage examples lacked args because that was the nature of the code or because it was a limitation of those methods of job declaration.
If named jobs is the suggested method of argument passing, perhaps that should be noted in the readme?  But as a user, I'd expect to be able to pass methods outside of either a fn closure or named job.

Fixed in https://github.com/c-rack/quantum-elixir/pull/47

authors: alPacino c-rack theoretick

repo: c-rack/quantum-elixir

license_source: bigquery


text:
🛑 www.ssc.sk is down

In 77d00c9, www.ssc.sk (https://www.ssc.sk/sk/Aktualne.ssc) was down:

HTTP code: 403
Response time: 814 ms

Resolved: www.ssc.sk is back up in 4919b38.

license: MIT


text:
🛑 cms2.alejtech.eu is down

In ecf0ae8, cms2.alejtech.eu (http://cms2.alejtech.eu/UserSystem/login/login.alej) was down:

HTTP code: 403
Response time: 735 ms

Resolved: cms2.alejtech.eu is back up in 9f63f12.


text:
🛑 www.garancnyfond.sk is down

In c34a7dc, www.garancnyfond.sk (https://www.garancnyfond.sk) was down:

HTTP code: 0
Response time: 0 ms

Resolved: www.garancnyfond.sk is back up in 7885c11.


text:
Fixed & Enhanced the Footer section

Previously, Footer section had large gap between the columns. Now it's fixed and compatible for all devices.
@c4coderandcreator

bro you don't understand the consequences of merging code with conflicts !! please resolve it , there's no other option

source: gharchive/pull-request


text:
PT sample data pages not rendering properly

the ju jitsu data not rendering in PT but IS rendering properly in Spanish and English

So weird - but this turned out to be related to a translation error where a typing mistake made %(nodes) into %(node) and then that turned into a Key that python couldn't find so it was throwing a KeyError. Same thing happened with %(conector) which should stay %(connector) even in the translated text

license: mit


text:
Seems DrawNode::setLineWidth(>) is not working correct (tested on Windows 10)

Windows 10 / VS2019
Steps to Reproduce:

Start: 28:Node: Draw
Both circle having the same width, but it should be different 1 and 10 (on v4 it has the same behavior).

More info:
After some tests I see the test have to be rewrite (each DrawNode has it own width, changing it is not working => only the latest width will be used on the renderer).
I am right? How is the behavior on android?
I will rewrite the test.

Yeah, you are right, I also realized that, I think you can rewrite the test, and guess cocos2d-x v3 also will have the same behavior

The test was written for the reason: Issue11942Test::Issue11942Test()
What was the Issue11942?

The issue should be the official cocos2d-x repo

https://github.com/cocos2d/cocos2d-x/issues/11942

so the behavior maybe what they are want.


text:
fix: removed outdated regex

Motivation
Relates to https://github.com/caas-team/sparrow/issues/80

Changes
Removed outdated regex tag.

For additional information look at the commits.
TODO

[x] I've assigned this PR to myself
[x] I've labeled this PR correctly

Please also take a look at these images in the sparrow package and delete them manually if needed.

Please also take a look at these images in the sparrow package and delete them manually if needed.

@y-eight @NiklasTreml @puffitos Can one of you remove these images manually?

Please also take a look at these images in the sparrow package and delete them manually if needed.

@lvlcn-t @JTaeuber images pruned

authors: JTaeuber lvlcn-t puffitos

repo: caas-team/sparrow


text:
Feat multiplier and gestalt checks now validate whether a unit is in the party or a party member's pet

Resolves #242.

@ArcaneTrixter Reminder to close #242 as well, if this resolves it.

authors: Amariithynar ArcaneTrixter

repo: cabarius/ToyBox


text:
op not found page

https://cables.gl/op/75c726d6-7cc5-48e0-b4ec-7b6af6802fd9
https://dev.cables.gl/op/75c726d6-7cc5-48e0-b4ec-7b6af6802fd9

this page should check and let the user know that the op exists on dev.cables.gl
maybe just a client side api request to check if that exsits on dev ?

add some endpoint that checks dev via http request "in the back" so that we can add more environments eventually

/api/op/:opIdentifier/find

repo: cables-gl/cables


text:
🛑 CBTB Coffee House Shop is down

In 63a71ff, CBTB Coffee House Shop (https://shop.cbtb.coffee/) was down:

HTTP code: 403
Response time: 2022 ms

Resolved: CBTB Coffee House Shop is back up in f2e4f77.

repo: cachetech/service-status


text:
🛑 CBTB Coffee House Shop is down

In 28c330a, CBTB Coffee House Shop (https://shop.cbtb.coffee/) was down:

HTTP code: 403
Response time: 1193 ms

Resolved: CBTB Coffee House Shop is back up in a8216af.


text:
🛑 CBTB Coffee House Shop is down

In 6e48bcc, CBTB Coffee House Shop (https://shop.cbtb.coffee/) was down:

HTTP code: 403
Response time: 1023 ms

Resolved: CBTB Coffee House Shop is back up in a87d78b.


text:
🛑 CBTB Coffee House Shop is down

In 983856a, CBTB Coffee House Shop (https://shop.cbtb.coffee/) was down:

HTTP code: 403
Response time: 840 ms

Resolved: CBTB Coffee House Shop is back up in a591a5e after 764 days, 5 hours, 10 minutes.


text:
🛑 CBTB Coffee House Shop is down

In 66abe9c, CBTB Coffee House Shop (https://shop.cbtb.coffee/) was down:

HTTP code: 403
Response time: 1188 ms

Resolved: CBTB Coffee House Shop is back up in c32bac3.


text:
🛑 CBTB Coffee House Shop is down

In ce89a9f, CBTB Coffee House Shop (https://shop.cbtb.coffee/) was down:

HTTP code: 403
Response time: 1104 ms

Resolved: CBTB Coffee House Shop is back up in 676995e after 5 minutes.


text:
🛑 CBTB Coffee House Shop is down

In d03b9fa, CBTB Coffee House Shop (https://shop.cbtb.coffee/) was down:

HTTP code: 403
Response time: 1124 ms

Resolved: CBTB Coffee House Shop is back up in e860a29 after 15 minutes.


text:
Add upstream proxy support

1. What does this change do, exactly?
This is built on top of #22. Separate PR from my branch, since I can't push into jim3ma:upstream
I've added HTTPS proxy and tests, so now it looks like something I could merge.
2. Please link to the relevant issues.
https://github.com/caddyserver/forwardproxy/issues/17
3. Which documentation changes (if any) need to be made because of this PR?
I've added full documentation for this feature to the README.md
4. Checklist

[x] I have written tests and verified that they fail without my change
[ ] I made pull request as minimal and simple as possible. If change is not small or additional dependencies are required, I opened an issue to propose and discuss the design first
[ ] I have squashed any insignificant commits. -- didn't squash @jim3ma commits, to avoid diverging of git histories and wiping authorship of their commits.
[x] This change has comments for package types, values, functions, and non-obvious lines of code

I decided to add more tests to increase coverage and sure enough found out that out previous approach was wrong. Will fix and rebase the whole thing.

repo: caddyserver/forwardproxy


text:
TTSN and TNI don't update when manually adding flights from previous days

When adding flights from previous days, the Total Time Since New (TTSN) and the Time to Next Inspection (TNI) are not updated to indicate the new times.

Knowing how this should work is a dilemma. It seems obvious at first that if I add a flight last week, the TTSN for that day and every day since should be incremented by the duration of that flight. But what happens if the user has already manually adjusted the TTSN of the more recent days to be correct? Then incrementing would make them wrong. Let's take a real example:
I fly a cross country in the towplane to gliding site X. I don't have an iPad that day, so I just write the times in the paper Journey log. The next day, site X staff adjust the TTSN for that aircraft to match the journey logs and start flying. After a few weeks, someone says, "shouldn't that cross country be in the iPad?" so they enter it in. Now in this case all the timesheets they've made in the weeks from when they manually adjust the TTSN to when they enter the missing flight are correct and shouldn't be tampered with.
We could solve this problem by not letting you manually adjust the TTSN, but that would make things less flexible and potentially make other errors harder to fix.

Yes, something we need to explore further for sure. We should aim to keep as much flexibility as possible. Maybe even just popping up a reminder that they need to double check the TTSN and TNI after they add flights manually?

Yeah. There is a feature that allows you to set the initial TTSN of one entry to the final TTSN of the prior one so you can manually go through and adjust them if that’s the right thing to do. In addition, mismatches of final and initial TTSN are highlighted in red in the journey log entries list.

authors: kirvanp markmacpherson

repo: cadets-ca/ets


text:
Android Oreo Issue --Notification is not receiving in background

Hi,
Notification is not receiving in background on Oreo android devices.
If anyone has some relevant solution, please let me know.
Version: 4.1.1
Tested SDK: 7.4.0/7.4.1
Thanks & Regards,
Hemant Solanki

https://github.com/caffeinalab/ti.goosh/issues/114#issuecomment-436813556

@furthurr : Thanks for your great help.

We've just released a new version of Ti.Goosh that should solve this problem. You can find it here.
I'm closing this issue, since it's a duplicate of #114.

authors: Hemantsolanki103 Jei furthurr


text:
Deny permissions

There should be an option to deny a permission to a user. That way, a user that has a Role with X permissions, could have a specific permission(s) denied from that X without the need to fork the Role.
Best regards,
António Fernandes

@antoniopaisfernandes
It's like
@cannot('access.dashboard')
    Access Dennied for {{user}}
@endcannot

?

authors: antoniopaisfernandes showcheap

repo: caffeinated/shinobi