schonsense/70B_llama3_1_Genre_ties

genre_ties

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the TIES merge method using meta-llama/Llama-3.1-70B as a base.

Models Merged

The following models were included in the merge:

• schonsense/70B_ero_horror
• schonsense/70B_llama3_1_Base_IKM
• D:\mergekit_My_YAMLS\llama_3_1_ero_nearstock
• schonsense/70B_llama3_1_Base_GW
• schonsense/70B_llama3_1_Base_SunVorGhast

Configuration

The following YAML configuration was used to produce this model:

merge_method: ties

models:
   
  - model: schonsense/70B_llama3_1_Base_IKM
    parameters:
      density: 1
      weight: 1

  - model: schonsense/70B_llama3_1_Base_SunVorGhast
    parameters:
      density: 1
      weight:
      - filter: self_attn
        value: [0, 0.1, 0.5, 0.9, 0.4, 0.2, 0]
      - filter: mlp
        value: [0, 0.9, 0.8, 0.4, 0.1, 0.01, 0]
      - filter: embed_tokens
        value: 0.8
      - filter: lm_head
        value: 0.6
      - value: 0.1


  - model: schonsense/70B_llama3_1_Base_GW
    parameters:
      density: 1
      weight:
      - filter: self_attn
        value: [0, 0.1, 0.5, 0.9, 0.4, 0.2, 0]
      - filter: mlp
        value: [0, 0.9, 0.8, 0.4, 0.1, 0.01, 0]
      - filter: embed_tokens
        value: 0.8
      - filter: lm_head
        value: 0.6
      - value: 0.1


  - model: "D:\\mergekit\\_My_YAMLS\\llama_3_1_ero_nearstock"
    parameters:
      density: 1
      weight:
      - filter: self_attn
        value: [0, 0.1, 0.5, 0.9, 0.4, 0.2, 0]
      - filter: mlp
        value: [0, 0.9, 0.8, 0.4, 0.1, 0.01, 0]
      - filter: embed_tokens
        value: 0.8
      - filter: lm_head
        value: 0.6
      - value: 0.1

  - model: schonsense/70B_ero_horror
    parameters:
      density: 1
      weight: [0, 0.5, 0]

  - model: meta-llama/Llama-3.1-70B

base_model: meta-llama/Llama-3.1-70B
parameters:
  normalize: true
  int8_mask: true
  lambda: 1.1

dtype: float32
out_dtype: bfloat16

tokenizer:
  source: union
  pad_to_multiple_of: 8