Merge branch 'main' of hf.co:tangledgroup/tangled-alpha-0.9-core

README.md

@@ -76,6 +76,10 @@ Total number of tokens in the optimized dataset '../core-data-6-32769-65537-6553
 
 i=7, min_len=65537, max_len=131073, block_size=131073, chunk_size=16384125, len(dataset)=634, len(dataset) * block_size=83100282
 Total number of tokens in the optimized dataset '../core-data-7-65537-131073-131073-125' is 83100282
+
+real    292m54.341s
+user    2118m1.154s
+sys     12m2.746s
 ```
 
 ```bash
@@ -83,6 +87,45 @@ CUDA_VISIBLE_DEVICES=0 CUDA_LAUNCH_BLOCKING=0 PYTORCH_CUDA_ALLOC_CONF=expandable
 ```
 
 ```
+Seed set to 23
+Time to instantiate model: 0.44 seconds.
+Total parameters: 234,914,304
+Verifying settings ...
+Measured TFLOPs: 55520.94
+Epoch 1 | iter 64 step 1 | loss train: 11.977, val: n/a | iter time: 490.27 ms (step) remaining time: 6 days, 22:47:04
+Epoch 1 | iter 128 step 2 | loss train: 11.970, val: n/a | iter time: 351.11 ms (step) remaining time: 4 days, 16:53:01
+Epoch 1 | iter 192 step 3 | loss train: 11.971, val: n/a | iter time: 353.74 ms (step) remaining time: 3 days, 23:43:23
+Epoch 1 | iter 256 step 4 | loss train: 11.974, val: n/a | iter time: 355.03 ms (step) remaining time: 3 days, 14:41:57
+Epoch 1 | iter 320 step 5 | loss train: 11.964, val: n/a | iter time: 357.36 ms (step) remaining time: 3 days, 9:21:54
+Epoch 1 | iter 384 step 6 | loss train: 11.957, val: n/a | iter time: 362.27 ms (step) remaining time: 3 days, 5:53:20
+Epoch 1 | iter 448 step 7 | loss train: 11.948, val: n/a | iter time: 359.89 ms (step) remaining time: 3 days, 3:26:34
+Epoch 1 | iter 512 step 8 | loss train: 11.938, val: n/a | iter time: 363.84 ms (step) remaining time: 3 days, 1:37:54
+Epoch 1 | iter 576 step 9 | loss train: 11.920, val: n/a | iter time: 362.75 ms (step) remaining time: 3 days, 0:13:59
+Epoch 1 | iter 640 step 10 | loss train: 11.900, val: n/a | iter time: 363.46 ms (step) remaining time: 2 days, 23:07:06
+# ...
+Epoch 1 | iter 643264 step 10051 | loss train: 2.834, val: 2.669 | iter time: 360.50 ms (step) remaining time: 0:03:59
+Epoch 2 | iter 643328 step 10052 | loss train: 2.837, val: 2.669 | iter time: 359.53 ms (step) remaining time: 0:03:37
+Epoch 2 | iter 643392 step 10053 | loss train: 2.768, val: 2.669 | iter time: 362.83 ms (step) remaining time: 0:03:15
+Epoch 2 | iter 643456 step 10054 | loss train: 2.695, val: 2.669 | iter time: 363.85 ms (step) remaining time: 0:02:53
+Epoch 2 | iter 643520 step 10055 | loss train: 2.768, val: 2.669 | iter time: 365.40 ms (step) remaining time: 0:02:30
+Epoch 2 | iter 643584 step 10056 | loss train: 2.710, val: 2.669 | iter time: 364.72 ms (step) remaining time: 0:02:08
+Epoch 2 | iter 643648 step 10057 | loss train: 2.749, val: 2.669 | iter time: 365.00 ms (step) remaining time: 0:01:46
+Epoch 2 | iter 643712 step 10058 | loss train: 2.748, val: 2.669 | iter time: 363.42 ms (step) remaining time: 0:01:24
+Epoch 2 | iter 643776 step 10059 | loss train: 2.710, val: 2.669 | iter time: 364.49 ms (step) remaining time: 0:01:02
+Epoch 2 | iter 643840 step 10060 | loss train: 2.738, val: 2.669 | iter time: 364.43 ms (step) remaining time: 0:00:39
+Epoch 2 | iter 643904 step 10061 | loss train: 2.734, val: 2.669 | iter time: 364.94 ms (step) remaining time: 0:00:17
+Validating ...
+Final evaluation | val loss: 2.669 | val ppl: 14.422
+Saving checkpoint to '../out/pretrain-core-0/final/lit_model.pth'
+----------------------------------------
+| Performance
+| - Total tokens  : 5,275,279,360
+| - Training Time : 223314.37 s
+| - Tok/sec       : 5541.09 tok/s
+| ----------------------------------------
+| Memory Usage
+| - Memory Used   : 22.33 GB
+----------------------------------------
 ```
 
 Backup `wandb`:
@@ -91,6 +134,12 @@ Backup `wandb`:
 mv wandb wandb-pretrain-core-0
 ```
 
+Copy config:
+
+```bash
+cp ../config-0.json ../out/pretrain-core-0/final
+```
+
 Chat with model:
 
 ```bash

config.json → config-0.json

@@ -14,13 +14,13 @@
   "max_position_embeddings": 131072,
   "mlp_bias": false,
   "model_type": "llama",
-  "num_attention_heads": 4,
+  "num_attention_heads": 8,
   "num_hidden_layers": 32,
-  "num_key_value_heads": 4,
+  "num_key_value_heads": 8,
   "pretraining_tp": 1,
   "rms_norm_eps": 1e-05,
   "rope_scaling": null,
-  "rope_theta": 10000.0,
+  "rope_theta": 4300.0,
   "tie_word_embeddings": true,
   "torch_dtype": "bfloat16",
   "transformers_version": "4.45.0.dev0",