text:
[LEADERBOARD UPDATE] DanskGPT-chat-v2.1

I believe that should be all datasets for da, no, sv. Somehow it scored negative for some of the norwegian benchmarks.
{"dataset": "angry-tweets", "task": "sentiment-classification", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.5184235804314162, "macro_f1": 0.6682555518604737}, {"mcc": 0.6345769712834416, "macro_f1": 0.7476473474655149}, {"mcc": 0.5561641594346012, "macro_f1": 0.6974840010519857}, {"mcc": 0.537915335723865, "macro_f1": 0.6904345654345655}, {"mcc": 0.48739583711175405, "macro_f1": 0.647520975787618}, {"mcc": 0.5642581574270951, "macro_f1": 0.7067332309267793}, {"mcc": 0.5849207506619116, "macro_f1": 0.7148941236490539}, {"mcc": 0.5924582679961513, "macro_f1": 0.7238829771186478}, {"mcc": 0.5098156324102479, "macro_f1": 0.6577043977089223}, {"mcc": 0.5578826012873085, "macro_f1": 0.7027417027417027}]}, "total": {"test_mcc": 55.43811293767791, "test_mcc_se": 2.690915415249613, "test_macro_f1": 69.57298873745265, "test_macro_f1_se": 1.9138897568342794}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "dansk", "task": "named-entity-recognition", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5419734904270986, "micro_f1": 0.43227091633466136}, {"micro_f1_no_misc": 0.503225806451613, "micro_f1": 0.4498977505112475}, {"micro_f1_no_misc": 0.5031446540880503, "micro_f1": 0.36160714285714285}, {"micro_f1_no_misc": 0.48933143669985774, "micro_f1": 0.3859275053304905}, {"micro_f1_no_misc": 0.46441947565543074, "micro_f1": 0.37812789620018533}, {"micro_f1_no_misc": 0.4371069182389937, "micro_f1": 0.3612334801762115}, {"micro_f1_no_misc": 0.527086383601757, "micro_f1": 0.37307692307692303}, {"micro_f1_no_misc": 0.5590230664857531, "micro_f1": 0.46153846153846156}, {"micro_f1_no_misc": 0.5234708392603129, "micro_f1": 0.36950146627565983}, {"micro_f1_no_misc": 0.48286604361370716, "micro_f1": 0.3733333333333333}]}, "total": {"test_micro_f1_no_misc": 50.316481145225744, "test_micro_f1_no_misc_se": 2.270658779043772, "test_micro_f1": 39.46514875634317, "test_micro_f1_se": 2.360289644215353}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "scala-da", "task": "linguistic-acceptability", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.242535625036333, "macro_f1": 0.5618703177473521}, {"mcc": 0.27801921874276636, "macro_f1": 0.6061538461538462}, {"mcc": 0.3431193197164929, "macro_f1": 0.612488174077578}, {"mcc": 0.22307395758779516, "macro_f1": 0.5766692889366962}, {"mcc": 0.36831064649905937, "macro_f1": 0.6780960559406244}, {"mcc": 0.3189650372515433, "macro_f1": 0.6379511775195228}, {"mcc": 0.26278380719013816, "macro_f1": 0.5466734125980268}, {"mcc": 0.3473566391288959, "macro_f1": 0.6547701185473248}, {"mcc": 0.23102761521458104, "macro_f1": 0.5925642984466514}, {"mcc": 0.40713222588774883, "macro_f1": 0.6929261492576023}]}, "total": {"test_mcc": 30.22324092255354, "test_mcc_se": 3.941749096409534, "test_macro_f1": 61.60162839225225, "test_macro_f1_se": 3.0434494669515764}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "scandiqa-da", "task": "question-answering", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 51.19047619047619, "f1": 65.96357346357347}, {"em": 53.75, "f1": 64.15774258421317}, {"em": 59.036144578313255, "f1": 67.5347544022243}, {"em": 50.0, "f1": 61.49834292691437}, {"em": 56.774193548387096, "f1": 66.33077316948285}, {"em": 50.588235294117645, "f1": 61.43043884220356}, {"em": 60.122699386503065, "f1": 68.27928717499272}, {"em": 52.17391304347826, "f1": 61.17618061717442}, {"em": 56.32911392405063, "f1": 64.1430580671087}, {"em": 56.96969696969697, "f1": 67.73700373700376}]}, "total": {"test_em": 54.69344729350231, "test_em_se": 2.252931557437419, "test_f1": 64.82511549848913, "test_f1_se": 1.7053400440301674}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "nordjylland-news", "task": "summarization", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.6725106660742313, "rouge_l": 0.2075343468823907}, {"bertscore": 0.6717358950991184, "rouge_l": 0.2061237345344845}, {"bertscore": 0.6617896785028279, "rouge_l": 0.19948939192807538}, {"bertscore": 0.6768606544937938, "rouge_l": 0.2154658706749286}, {"bertscore": 0.6501816539093852, "rouge_l": 0.19741787300146646}, {"bertscore": 0.673794281668961, "rouge_l": 0.21073368525734845}, {"bertscore": 0.6693948113825172, "rouge_l": 0.19984827073078543}, {"bertscore": 0.6765127489343286, "rouge_l": 0.2022917237334853}, {"bertscore": 0.6692628439050168, "rouge_l": 0.19448978984456267}, {"bertscore": 0.673038718290627, "rouge_l": 0.2065912712375019}]}, "total": {"test_bertscore": 66.95081952260807, "test_bertscore_se": 0.49803492902172003, "test_rouge_l": 20.399859578250293, "test_rouge_l_se": 0.39997361880934734}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "danske-talemaader", "task": "knowledge", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.845948293862559, "accuracy": 0.8828125}, {"mcc": 0.7769235704383787, "accuracy": 0.83203125}, {"mcc": 0.7944328539576512, "accuracy": 0.84375}, {"mcc": 0.7865711878642629, "accuracy": 0.8359375}, {"mcc": 0.810291560769786, "accuracy": 0.859375}, {"mcc": 0.8176784333961894, "accuracy": 0.86328125}, {"mcc": 0.7636876524768063, "accuracy": 0.82421875}, {"mcc": 0.8358571996268044, "accuracy": 0.87890625}, {"mcc": 0.7762022519463603, "accuracy": 0.83203125}, {"mcc": 0.8248024706608752, "accuracy": 0.8671875}]}, "total": {"test_mcc": 80.32395474999673, "test_mcc_se": 1.7229194885457089, "test_accuracy": 85.1953125, "test_accuracy_se": 1.3035625972228844}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "danish-citizen-tests", "task": "knowledge", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.624875087309511, "accuracy": 0.75}, {"mcc": 0.7199575821095028, "accuracy": 0.8125}, {"mcc": 0.7028872607481234, "accuracy": 0.7890625}, {"mcc": 0.8370328106283629, "accuracy": 0.890625}, {"mcc": 0.7170951146610519, "accuracy": 0.8125}, {"mcc": 0.734466707394504, "accuracy": 0.8203125}, {"mcc": 0.8105226915786738, "accuracy": 0.875}, {"mcc": 0.7042024993516115, "accuracy": 0.796875}, {"mcc": 0.7489873180522583, "accuracy": 0.8359375}, {"mcc": 0.7138944795271756, "accuracy": 0.8046875}]}, "total": {"test_mcc": 73.13921551360775, "test_mcc_se": 3.6539764675388495, "test_accuracy": 81.875, "test_accuracy_se": 2.5295253040617895}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "hellaswag-da", "task": "common-sense-reasoning", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.61328125, "mcc": 0.4927889745228705}, {"accuracy": 0.609375, "mcc": 0.4754202242513178}, {"accuracy": 0.609375, "mcc": 0.484296862345627}, {"accuracy": 0.5859375, "mcc": 0.4577940215733695}, {"accuracy": 0.640625, "mcc": 0.5203993355979124}, {"accuracy": 0.62109375, "mcc": 0.49675504775869844}, {"accuracy": 0.53125, "mcc": 0.37592886265479897}, {"accuracy": 0.57421875, "mcc": 0.4319241006922182}, {"accuracy": 0.49609375, "mcc": 0.33314638763080434}, {"accuracy": 0.6171875, "mcc": 0.4907135935277734}]}, "total": {"test_accuracy": 58.984375, "test_accuracy_se": 2.7793113171376542, "test_mcc": 45.591674105553906, "test_mcc_se": 3.674485330227178}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "speed", "task": "speed", "dataset_languages": ["ab", "aa", "af", "sq", "am", "ar", "an", "hy", "as", "av", "ae", "ay", "az", "bm", "ba", "eu", "be", "bn", "bi", "bs", "br", "bg", "my", "ca", "ch", "ce", "ny", "zh", "cu", "cv", "kw", "co", "cr", "hr", "cs", "da", "dv", "nl", "dz", "en", "eo", "et", "ee", "fo", "fj", "fi", "fr", "fy", "ff", "gd", "gl", "lg", "ka", "de", "el", "kl", "gn", "gu", "ht", "ha", "he", "hz", "hi", "ho", "hu", "is", "io", "ig", "id", "ia", "ie", "iu", "ik", "ga", "it", "ja", "kn", "kr", "ks", "kk", "km", "ki", "rw", "ky", "kv", "kg", "ko", "kj", "ku", "lo", "la", "lv", "li", "ln", "lt", "lu", "lb", "mk", "mg", "ms", "ml", "mt", "gv", "mi", "mr", "mh", "mn", "na", "nv", "nd", "nr", "ng", "ne", "no", "nb", "nn", "ii", "oc", "oj", "or", "om", "os", "pi", "ps", "fa", "pl", "pt", "pa", "qu", "ro", "rm", "rn", "ru", "se", "sm", "sg", "sa", "sc", "sr", "sn", "sd", "si", "sk", "sl", "so", "st", "es", "su", "sw", "ss", "sv", "tl", "ty", "tg", "ta", "tt", "te", "th", "bo", "ti", "to", "ts", "tn", "tr", "tk", "tw", "ug", "uk", "ur", "uz", "ve", "vi", "vo", "wa", "cy", "wo", "xh", "yi", "yo", "za", "zu"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"test_speed": 1989.9599999999998, "test_speed_short": 275.0}, {"test_speed": 3570.84, "test_speed_short": 478.2}, {"test_speed": 4528.799999999999, "test_speed_short": 916.5600000000001}, {"test_speed": 5868.02, "test_speed_short": 1065.02}, {"test_speed": 6373.2, "test_speed_short": 1265.6000000000001}, {"test_speed": 6666.6, "test_speed_short": 1616.16}, {"test_speed": 7577.68, "test_speed_short": 1806.9099999999999}, {"test_speed": 7999.76, "test_speed_short": 1993.64}, {"test_speed": 7860.16, "test_speed_short": 2174.53}, {"test_speed": 8289.38, "test_speed_short": 2359.5}]}, "total": {"test_speed": 6072.4400000000005, "test_speed_se": 1304.76761109485, "test_speed_short": 1395.112, "test_speed_short_se": 441.39118878476864}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "swerec", "task": "sentiment-classification", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.69095866370862, "macro_f1": 0.7168455609050719}, {"mcc": 0.7361269468754462, "macro_f1": 0.6989835425190116}, {"mcc": 0.7254067496599732, "macro_f1": 0.6862918069584736}, {"mcc": 0.7621414251312847, "macro_f1": 0.7837403823328167}, {"mcc": 0.7073030251576995, "macro_f1": 0.6699038179534806}, {"mcc": 0.7116848385508566, "macro_f1": 0.7213776276276276}, {"mcc": 0.6711478263200392, "macro_f1": 0.6466924976742648}, {"mcc": 0.7654263808983799, "macro_f1": 0.7649551839824297}, {"mcc": 0.7441698122531218, "macro_f1": 0.7389492753623189}, {"mcc": 0.6853714040477759, "macro_f1": 0.7077615833456955}]}, "total": {"test_mcc": 71.99737072603197, "test_mcc_se": 2.0001358930410476, "test_macro_f1": 71.35501278661192, "test_macro_f1_se": 2.5845960587530614}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "norec", "task": "sentiment-classification", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.5123638337725782, "macro_f1": 0.6876623438026219}, {"mcc": 0.6185275565832171, "macro_f1": 0.7342516781839882}, {"mcc": 0.49402241622129955, "macro_f1": 0.6394654509698757}, {"mcc": 0.6236648042734055, "macro_f1": 0.7587692844803198}, {"mcc": 0.639372236904214, "macro_f1": 0.7618797311852425}, {"mcc": 0.5229781259861302, "macro_f1": 0.6851363784477966}, {"mcc": 0.5284645821648246, "macro_f1": 0.6837441390286919}, {"mcc": 0.5847167838247488, "macro_f1": 0.7376748452189865}, {"mcc": 0.48381014678561474, "macro_f1": 0.6367258702164362}, {"mcc": 0.5559668394434493, "macro_f1": 0.7069517245456023}]}, "total": {"test_mcc": 55.638873259594824, "test_mcc_se": 3.522408892038271, "test_macro_f1": 70.3226144607956, "test_macro_f1_se": 2.7814812396301085}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "sb10k", "task": "sentiment-classification", "dataset_languages": ["de"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.5538703035003155, "macro_f1": 0.6981968465089246}, {"mcc": 0.5017594472385768, "macro_f1": 0.6628540671466057}, {"mcc": 0.5474218013267408, "macro_f1": 0.6935274657597706}, {"mcc": 0.5141117283086541, "macro_f1": 0.6691257517688919}, {"mcc": 0.5407736930212017, "macro_f1": 0.6879537153275391}, {"mcc": 0.5413282941970623, "macro_f1": 0.689798828622358}, {"mcc": 0.5553685444480709, "macro_f1": 0.6964868542975586}, {"mcc": 0.6057952110542372, "macro_f1": 0.7305588228099641}, {"mcc": 0.561464635499108, "macro_f1": 0.7057086463515337}, {"mcc": 0.6473286855856728, "macro_f1": 0.7647492922218198}]}, "total": {"test_mcc": 55.69222344179641, "test_mcc_se": 2.6185656248035625, "test_macro_f1": 69.98960290814968, "test_macro_f1_se": 1.8244848560290465}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "suc3", "task": "named-entity-recognition", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5685071574642128, "micro_f1": 0.29327902240325865}, {"micro_f1_no_misc": 0.6373165618448637, "micro_f1": 0.5457685664939551}, {"micro_f1_no_misc": 0.49242424242424243, "micro_f1": 0.3952451708766716}, {"micro_f1_no_misc": 0.5657894736842105, "micro_f1": 0.41348973607038125}, {"micro_f1_no_misc": 0.6521739130434782, "micro_f1": 0.5764705882352941}, {"micro_f1_no_misc": 0.525179856115108, "micro_f1": 0.43515850144092216}, {"micro_f1_no_misc": 0.5146443514644352, "micro_f1": 0.39938080495356043}, {"micro_f1_no_misc": 0.45614035087719296, "micro_f1": 0.38386648122392214}, {"micro_f1_no_misc": 0.535645472061657, "micro_f1": 0.45997088791848617}, {"micro_f1_no_misc": 0.5133928571428572, "micro_f1": 0.3936}]}, "total": {"test_micro_f1_no_misc": 54.61214236122258, "test_micro_f1_no_misc_se": 3.811900624300252, "test_micro_f1": 42.96229759616452, "test_micro_f1_se": 5.075157351350615}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "scala-sv", "task": "linguistic-acceptability", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.19445958948234918, "macro_f1": 0.5925642984466514}, {"mcc": 0.18324142829300047, "macro_f1": 0.556923076923077}, {"mcc": 0.19069255717628847, "macro_f1": 0.5915915915915916}, {"mcc": 0.15949410976371273, "macro_f1": 0.5403511559545563}, {"mcc": 0.13788300768252423, "macro_f1": 0.5686274509803921}, {"mcc": 0.10953287859164815, "macro_f1": 0.5501858158013404}, {"mcc": 0.22604350622091474, "macro_f1": 0.5717686418323361}, {"mcc": 0.13118111565056956, "macro_f1": 0.47442039024553084}, {"mcc": 0.16532411231568353, "macro_f1": 0.5641021635554168}, {"mcc": 0.050429861013149516, "macro_f1": 0.5156102175969726}]}, "total": {"test_mcc": 15.482821661898406, "test_mcc_se": 3.1103586100609903, "test_macro_f1": 55.26144802927865, "test_macro_f1_se": 2.2183480818114587}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "scandiqa-sv", "task": "question-answering", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 57.73809523809524, "f1": 66.0851714423143}, {"em": 58.125, "f1": 67.15530303030303}, {"em": 61.44578313253012, "f1": 67.14624732697023}, {"em": 56.54761904761905, "f1": 64.71253711274719}, {"em": 56.774193548387096, "f1": 64.04356933389192}, {"em": 54.705882352941174, "f1": 63.77765794374792}, {"em": 56.441717791411044, "f1": 63.50038509547712}, {"em": 54.037267080745345, "f1": 64.46910780122325}, {"em": 52.53164556962025, "f1": 63.107164614982935}, {"em": 53.333333333333336, "f1": 61.54794700249245}]}, "total": {"test_em": 56.16805370946827, "test_em_se": 1.6355892372243956, "test_f1": 64.55450907041504, "test_f1_se": 1.111941584235325}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "swedn", "task": "summarization", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.6623412126209587, "rouge_l": 0.17948454407729558}, {"bertscore": 0.6599227430997416, "rouge_l": 0.18302905814325443}, {"bertscore": 0.6627107572276145, "rouge_l": 0.17499728151514926}, {"bertscore": 0.6633944308850914, "rouge_l": 0.1916089717003644}, {"bertscore": 0.6505317895207554, "rouge_l": 0.1647751980400869}, {"bertscore": 0.6566043742932379, "rouge_l": 0.17679153100593195}, {"bertscore": 0.6612215854693204, "rouge_l": 0.18809099047776226}, {"bertscore": 0.6557645436841995, "rouge_l": 0.1761696877720468}, {"bertscore": 0.6572468352969736, "rouge_l": 0.1728377007983859}, {"bertscore": 0.6563620776869357, "rouge_l": 0.1725852011441733}]}, "total": {"test_bertscore": 65.86100349784829, "test_bertscore_se": 0.25003180436800815, "test_rouge_l": 17.803701646744507, "test_rouge_l_se": 0.4885628760584288}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "mmlu-sv", "task": "knowledge", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.2979829315592265, "accuracy": 0.46484375}, {"mcc": 0.30219011126313344, "accuracy": 0.48046875}, {"mcc": 0.2842043875802984, "accuracy": 0.46484375}, {"mcc": 0.27954251743401765, "accuracy": 0.453125}, {"mcc": 0.27975966786229495, "accuracy": 0.46875}, {"mcc": 0.29063540898681967, "accuracy": 0.4609375}, {"mcc": 0.2743039399430729, "accuracy": 0.453125}, {"mcc": 0.31799334285149655, "accuracy": 0.484375}, {"mcc": 0.2990218406832985, "accuracy": 0.46875}, {"mcc": 0.23279176978767205, "accuracy": 0.41796875}]}, "total": {"test_mcc": 28.58425917951331, "test_mcc_se": 1.4121480054266322, "test_accuracy": 46.171875, "test_accuracy_se": 1.1401844659174427}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "hellaswag-sv", "task": "common-sense-reasoning", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.5, "mcc": 0.35109659520283093}, {"accuracy": 0.5390625, "mcc": 0.38924592859732543}, {"accuracy": 0.53515625, "mcc": 0.39492877738111176}, {"accuracy": 0.61328125, "mcc": 0.49342896445070344}, {"accuracy": 0.4296875, "mcc": 0.24935198001581968}, {"accuracy": 0.5, "mcc": 0.3462325188112799}, {"accuracy": 0.53515625, "mcc": 0.40125872987962813}, {"accuracy": 0.47265625, "mcc": 0.32075799645282244}, {"accuracy": 0.5078125, "mcc": 0.35378486829076256}, {"accuracy": 0.56640625, "mcc": 0.4339232819270325}]}, "total": {"test_accuracy": 51.9921875, "test_accuracy_se": 3.1349100244736112, "test_mcc": 37.34009641009317, "test_mcc_se": 4.097815289356593}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "norne-nb", "task": "named-entity-recognition", "dataset_languages": ["nb"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5974025974025975, "micro_f1": 0.5748898678414097}, {"micro_f1_no_misc": 0.5919732441471571, "micro_f1": 0.49736842105263157}, {"micro_f1_no_misc": 0.5392022008253096, "micro_f1": 0.5010482180293501}, {"micro_f1_no_misc": 0.5502063273727648, "micro_f1": 0.48654244306418215}, {"micro_f1_no_misc": 0.6932668329177057, "micro_f1": 0.6774891774891775}, {"micro_f1_no_misc": 0.537037037037037, "micro_f1": 0.504950495049505}, {"micro_f1_no_misc": 0.5633001422475107, "micro_f1": 0.5365853658536585}, {"micro_f1_no_misc": 0.6053882725832013, "micro_f1": 0.5272277227722773}, {"micro_f1_no_misc": 0.7223065250379362, "micro_f1": 0.5754310344827587}, {"micro_f1_no_misc": 0.6616161616161617, "micro_f1": 0.5434995112414468}]}, "total": {"test_micro_f1_no_misc": 60.61699341187381, "test_micro_f1_no_misc_se": 4.062276410086982, "test_micro_f1": 54.25032256876398, "test_micro_f1_se": 3.513523284964431}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "norne-nn", "task": "named-entity-recognition", "dataset_languages": ["nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.624505928853755, "micro_f1": 0.5005086469989828}, {"micro_f1_no_misc": 0.5992217898832685, "micro_f1": 0.4440400363967243}, {"micro_f1_no_misc": 0.7063953488372092, "micro_f1": 0.5619834710743802}, {"micro_f1_no_misc": 0.6064356435643565, "micro_f1": 0.5113077679449362}, {"micro_f1_no_misc": 0.6796785304247991, "micro_f1": 0.4724279835390946}, {"micro_f1_no_misc": 0.6630872483221476, "micro_f1": 0.5527426160337553}, {"micro_f1_no_misc": 0.6374695863746959, "micro_f1": 0.4903581267217631}, {"micro_f1_no_misc": 0.5997425997425998, "micro_f1": 0.6002265005662515}, {"micro_f1_no_misc": 0.6380832282471626, "micro_f1": 0.5813221406086043}, {"micro_f1_no_misc": 0.6389610389610388, "micro_f1": 0.46389891696750907}]}, "total": {"test_micro_f1_no_misc": 63.935809432110325, "test_micro_f1_no_misc_se": 2.1838284208450554, "test_micro_f1": 51.78816206852, "test_micro_f1_se": 3.299234005147171}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "scala-nb", "task": "linguistic-acceptability", "dataset_languages": ["nb"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.0, "macro_f1": 0.33506493506493507}, {"mcc": 0.0, "macro_f1": 0.3435897435897436}, {"mcc": 0.0, "macro_f1": 0.3469387755102041}, {"mcc": 0.0, "macro_f1": 0.32808398950131235}, {"mcc": 0.0, "macro_f1": 0.33678756476683935}, {"mcc": 0.0, "macro_f1": 0.3333333333333333}, {"mcc": -0.06117151949384656, "macro_f1": 0.33678756476683935}, {"mcc": 0.0, "macro_f1": 0.33678756476683935}, {"mcc": -0.04611093636816612, "macro_f1": 0.32917683202313186}, {"mcc": -0.0636086697675111, "macro_f1": 0.32808398950131235}]}, "total": {"test_mcc": -1.708911256295238, "test_mcc_se": 1.7277990679865893, "test_macro_f1": 33.54634292824491, "test_macro_f1_se": 0.389876486882204}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "scala-nn", "task": "linguistic-acceptability", "dataset_languages": ["nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.0, "macro_f1": 0.33506493506493507}, {"mcc": 0.0, "macro_f1": 0.2967032967032967}, {"mcc": 0.0, "macro_f1": 0.3298429319371728}, {"mcc": -0.1047193895811623, "macro_f1": 0.33678756476683935}, {"mcc": 0.0, "macro_f1": 0.3298429319371728}, {"mcc": 0.0, "macro_f1": 0.33506493506493507}, {"mcc": 0.0027743440830113124, "macro_f1": 0.3473147462784768}, {"mcc": 0.0, "macro_f1": 0.3452685421994885}, {"mcc": 0.0, "macro_f1": 0.33678756476683935}, {"mcc": 0.0, "macro_f1": 0.3435897435897436}]}, "total": {"test_mcc": -1.0194504549815098, "test_mcc_se": 2.0592511480456666, "test_macro_f1": 33.362671923088996, "test_macro_f1_se": 0.8858024363228787}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "norquad", "task": "question-answering", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 50.90909090909091, "f1": 75.34228861076286}, {"em": 40.880503144654085, "f1": 69.63786340956872}, {"em": 41.566265060240966, "f1": 69.06001579302945}, {"em": 44.785276073619634, "f1": 73.6690181676046}, {"em": 63.81578947368421, "f1": 83.96266001772563}, {"em": 52.69461077844311, "f1": 75.8999389098787}, {"em": 48.148148148148145, "f1": 75.42818410249167}, {"em": 48.10126582278481, "f1": 74.0422604692764}, {"em": 46.45161290322581, "f1": 73.81725191981056}, {"em": 48.484848484848484, "f1": 72.64292761635951}]}, "total": {"test_em": 48.58374107987402, "test_em_se": 4.0396570237355345, "test_f1": 74.35024090165082, "test_f1_se": 2.537905108209652}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "no-sammendrag", "task": "summarization", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.6353648062795401, "rouge_l": 0.1351466297391169}, {"bertscore": 0.6300802999176085, "rouge_l": 0.1333336653535952}, {"bertscore": 0.6331870991270989, "rouge_l": 0.132078847695249}, {"bertscore": 0.6276011622976512, "rouge_l": 0.10901477811879332}, {"bertscore": 0.6435821207705885, "rouge_l": 0.14886354830393322}, {"bertscore": 0.6403231008443981, "rouge_l": 0.12914402363423663}, {"bertscore": 0.6294122280087322, "rouge_l": 0.1286917677593034}, {"bertscore": 0.6377897658385336, "rouge_l": 0.13818278734244666}, {"bertscore": 0.637333108112216, "rouge_l": 0.14196654246906998}, {"bertscore": 0.6341048267204314, "rouge_l": 0.13332118649699987}]}, "total": {"test_bertscore": 63.487785179167986, "test_bertscore_se": 0.3129566622612293, "test_rouge_l": 13.297437769127441, "test_rouge_l_se": 0.6445513599381921}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "mmlu-no", "task": "knowledge", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.3563139029828732, "accuracy": 0.50390625}, {"mcc": 0.2528934203448545, "accuracy": 0.43359375}, {"mcc": 0.29720914116733815, "accuracy": 0.47265625}, {"mcc": 0.22918061196985817, "accuracy": 0.41796875}, {"mcc": 0.30005523389604605, "accuracy": 0.47265625}, {"mcc": 0.3768653101057093, "accuracy": 0.52734375}, {"mcc": 0.2383056903441827, "accuracy": 0.41796875}, {"mcc": 0.3158667410686986, "accuracy": 0.48046875}, {"mcc": 0.27423783412506847, "accuracy": 0.4453125}, {"mcc": 0.29922553645335226, "accuracy": 0.4609375}]}, "total": {"test_mcc": 29.401534224579816, "test_mcc_se": 2.971468031791036, "test_accuracy": 46.328125, "test_accuracy_se": 2.2195790215683044}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}
{"dataset": "hellaswag-no", "task": "common-sense-reasoning", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.5625, "mcc": 0.42105828530643863}, {"accuracy": 0.578125, "mcc": 0.4358491750179072}, {"accuracy": 0.53125, "mcc": 0.37786324531150445}, {"accuracy": 0.578125, "mcc": 0.4438196167788708}, {"accuracy": 0.47265625, "mcc": 0.3049754039182944}, {"accuracy": 0.55859375, "mcc": 0.4176493345310223}, {"accuracy": 0.5703125, "mcc": 0.43371789710839936}, {"accuracy": 0.56640625, "mcc": 0.42773173686642846}, {"accuracy": 0.58203125, "mcc": 0.44385966679745836}, {"accuracy": 0.53125, "mcc": 0.37593764350045056}]}, "total": {"test_accuracy": 55.3125, "test_accuracy_se": 2.077088557332113, "test_mcc": 40.824620051367745, "test_mcc_se": 2.7102387617418664}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": true, "scandeval_version": "11.0.0"}

Thanks! Should it be registered as mhenrichsen/danskgpt-chat-v2.1, even though that model is private on the Hub, or more like danskgpt-chat-v2.1, to emphasise that it's not available?

@mhenrichsen  I see that you've evaluated it only on the validation split. Can you evaluate it on the full test set?

Here you go :)
{"dataset": "angry-tweets", "task": "sentiment-classification", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.5470151068743645, "macro_f1": 0.7047250329278935}, {"mcc": 0.5509690251753908, "macro_f1": 0.7022794811767601}, {"mcc": 0.5395830864902568, "macro_f1": 0.6863279486150139}, {"mcc": 0.5601647856420113, "macro_f1": 0.7062123686954361}, {"mcc": 0.5262341779021951, "macro_f1": 0.6681850368248451}, {"mcc": 0.5694551924277325, "macro_f1": 0.7122101309862009}, {"mcc": 0.516581263560336, "macro_f1": 0.6736354224164008}, {"mcc": 0.5597581131546965, "macro_f1": 0.7081448226150671}, {"mcc": 0.5467387241736854, "macro_f1": 0.6893435885109872}, {"mcc": 0.5525322086485214, "macro_f1": 0.6988609918653976}]}, "total": {"test_mcc": 54.69031684049192, "test_mcc_se": 0.9904368322842158, "test_macro_f1": 69.49924824634003, "test_macro_f1_se": 0.9331129988021167}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "dansk", "task": "named-entity-recognition", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5400089405453732, "micro_f1": 0.3580246913580247}, {"micro_f1_no_misc": 0.5028773793714032, "micro_f1": 0.39281045751633986}, {"micro_f1_no_misc": 0.47290209790209786, "micro_f1": 0.3587360594795539}, {"micro_f1_no_misc": 0.505229283990346, "micro_f1": 0.37739181630850754}, {"micro_f1_no_misc": 0.5161290322580645, "micro_f1": 0.344225352112676}, {"micro_f1_no_misc": 0.5201923076923076, "micro_f1": 0.3684377982818963}, {"micro_f1_no_misc": 0.47560975609756095, "micro_f1": 0.299298519095869}, {"micro_f1_no_misc": 0.543313708999159, "micro_f1": 0.4024024024024024}, {"micro_f1_no_misc": 0.4918478260869565, "micro_f1": 0.3354763296317943}, {"micro_f1_no_misc": 0.5400516795865633, "micro_f1": 0.3462149285336157}]}, "total": {"test_micro_f1_no_misc": 51.08162012529831, "test_micro_f1_no_misc_se": 1.601688418285135, "test_micro_f1": 35.83018354720679, "test_micro_f1_se": 1.8420954603454487}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-da", "task": "linguistic-acceptability", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.2707853552154174, "macro_f1": 0.5865546953255694}, {"mcc": 0.30352782788839133, "macro_f1": 0.6185297484185476}, {"mcc": 0.3310526009952438, "macro_f1": 0.646833382999043}, {"mcc": 0.30295938829684965, "macro_f1": 0.6079233247658888}, {"mcc": 0.2968410572843576, "macro_f1": 0.6348737679195147}, {"mcc": 0.3256648126595716, "macro_f1": 0.6526086890051914}, {"mcc": 0.28083741862334227, "macro_f1": 0.5591316706916011}, {"mcc": 0.31816687902908636, "macro_f1": 0.6311555240817968}, {"mcc": 0.3138504791931172, "macro_f1": 0.6328910780964487}, {"mcc": 0.35152957070308083, "macro_f1": 0.6442164870041693}]}, "total": {"test_mcc": 30.952153898884582, "test_mcc_se": 1.4819869752353665, "test_macro_f1": 62.1471836830777, "test_macro_f1_se": 1.8283456553613204}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scandiqa-da", "task": "question-answering", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 55.30596436870643, "f1": 64.2806621799194}, {"em": 56.58914728682171, "f1": 64.25636592975852}, {"em": 57.95981452859351, "f1": 65.04863101503464}, {"em": 55.06230529595015, "f1": 63.85892781492223}, {"em": 57.76061776061776, "f1": 65.07433931803675}, {"em": 54.58750963762529, "f1": 62.82309817061955}, {"em": 56.79574791192103, "f1": 64.13631643565981}, {"em": 57.87432117920869, "f1": 64.72082710660143}, {"em": 57.333333333333336, "f1": 64.50587203389968}, {"em": 56.2888198757764, "f1": 64.4702692443147}]}, "total": {"test_em": 56.55575811785543, "test_em_se": 0.7608640090317561, "test_f1": 64.31753092487665, "test_f1_se": 0.40328720275943436}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "nordjylland-news", "task": "summarization", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.668609851389192, "rouge_l": 0.2039914021393196}, {"bertscore": 0.6746063534519635, "rouge_l": 0.21926533456511507}, {"bertscore": 0.664634121421841, "rouge_l": 0.20472851618203852}, {"bertscore": 0.6726909662247635, "rouge_l": 0.21167626165322267}, {"bertscore": 0.6668700768059352, "rouge_l": 0.23086788686031476}, {"bertscore": 0.6725016877899179, "rouge_l": 0.2130333954792637}, {"bertscore": 0.6683111188176554, "rouge_l": 0.20418486333637614}, {"bertscore": 0.6710484639770584, "rouge_l": 0.20629890099620568}, {"bertscore": 0.6630088706006063, "rouge_l": 0.19357379913847286}, {"bertscore": 0.6676352790091187, "rouge_l": 0.20195556646682122}]}, "total": {"test_bertscore": 66.89916789488052, "test_bertscore_se": 0.22959075664787143, "test_rouge_l": 20.895759268171503, "test_rouge_l_se": 0.6429274915522064}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "danske-talemaader", "task": "knowledge", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.773350766605174, "accuracy": 0.826171875}, {"mcc": 0.7915544946103718, "accuracy": 0.83984375}, {"mcc": 0.7728212208829291, "accuracy": 0.8271484375}, {"mcc": 0.7902566489409734, "accuracy": 0.83984375}, {"mcc": 0.8261098835829227, "accuracy": 0.8671875}, {"mcc": 0.7981074441251266, "accuracy": 0.84765625}, {"mcc": 0.7972037226725296, "accuracy": 0.8447265625}, {"mcc": 0.7802465571650927, "accuracy": 0.833984375}, {"mcc": 0.7887728080473971, "accuracy": 0.837890625}, {"mcc": 0.820831763126859, "accuracy": 0.8642578125}]}, "total": {"test_mcc": 79.39255309759375, "test_mcc_se": 1.1105654030739633, "test_accuracy": 84.287109375, "test_accuracy_se": 0.8576338599946044}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "danish-citizen-tests", "task": "knowledge", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.7027328105687023, "accuracy": 0.794921875}, {"mcc": 0.7585038183877986, "accuracy": 0.837890625}, {"mcc": 0.752970274864937, "accuracy": 0.83203125}, {"mcc": 0.735234481490622, "accuracy": 0.822265625}, {"mcc": 0.695602885466217, "accuracy": 0.794921875}, {"mcc": 0.737212069532255, "accuracy": 0.826171875}, {"mcc": 0.7016433025242871, "accuracy": 0.796875}, {"mcc": 0.7611053735916435, "accuracy": 0.83984375}, {"mcc": 0.7020157251736459, "accuracy": 0.796875}, {"mcc": 0.7750934968107404, "accuracy": 0.849609375}]}, "total": {"test_mcc": 73.22114238410849, "test_mcc_se": 1.8346295360263907, "test_accuracy": 81.9140625, "test_accuracy_se": 1.3226588905680146}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "hellaswag-da", "task": "common-sense-reasoning", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.61865234375, "mcc": 0.49565878624255083}, {"accuracy": 0.6279296875, "mcc": 0.5053010308414206}, {"accuracy": 0.60986328125, "mcc": 0.4805284991610356}, {"accuracy": 0.5849609375, "mcc": 0.45646051830773443}, {"accuracy": 0.62353515625, "mcc": 0.5006686175994569}, {"accuracy": 0.609375, "mcc": 0.4815019384075108}, {"accuracy": 0.59765625, "mcc": 0.465792646961778}, {"accuracy": 0.59619140625, "mcc": 0.4684679853431774}, {"accuracy": 0.5888671875, "mcc": 0.4603454233870459}, {"accuracy": 0.6259765625, "mcc": 0.5015713200705473}]}, "total": {"test_accuracy": 60.830078125, "test_accuracy_se": 0.9740846080825533, "test_mcc": 48.16296766322258, "test_mcc_se": 1.1370770405809758}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "speed", "task": "speed", "dataset_languages": ["ab", "aa", "af", "sq", "am", "ar", "an", "hy", "as", "av", "ae", "ay", "az", "bm", "ba", "eu", "be", "bn", "bi", "bs", "br", "bg", "my", "ca", "ch", "ce", "ny", "zh", "cu", "cv", "kw", "co", "cr", "hr", "cs", "da", "dv", "nl", "dz", "en", "eo", "et", "ee", "fo", "fj", "fi", "fr", "fy", "ff", "gd", "gl", "lg", "ka", "de", "el", "kl", "gn", "gu", "ht", "ha", "he", "hz", "hi", "ho", "hu", "is", "io", "ig", "id", "ia", "ie", "iu", "ik", "ga", "it", "ja", "kn", "kr", "ks", "kk", "km", "ki", "rw", "ky", "kv", "kg", "ko", "kj", "ku", "lo", "la", "lv", "li", "ln", "lt", "lu", "lb", "mk", "mg", "ms", "ml", "mt", "gv", "mi", "mr", "mh", "mn", "na", "nv", "nd", "nr", "ng", "ne", "no", "nb", "nn", "ii", "oc", "oj", "or", "om", "os", "pi", "ps", "fa", "pl", "pt", "pa", "qu", "ro", "rm", "rn", "ru", "se", "sm", "sg", "sa", "sc", "sr", "sn", "sd", "si", "sk", "sl", "so", "st", "es", "su", "sw", "ss", "sv", "tl", "ty", "tg", "ta", "tt", "te", "th", "bo", "ti", "to", "ts", "tn", "tr", "tk", "tw", "ug", "uk", "ur", "uz", "ve", "vi", "vo", "wa", "cy", "wo", "xh", "yi", "yo", "za", "zu"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"test_speed": 1854.72, "test_speed_short": 267.96}, {"test_speed": 3206.84, "test_speed_short": 466.2}, {"test_speed": 3995.68, "test_speed_short": 875.52}, {"test_speed": 4984.74, "test_speed_short": 1022.25}, {"test_speed": 5365.24, "test_speed_short": 1184.96}, {"test_speed": 5560.92, "test_speed_short": 1519.22}, {"test_speed": 6225.2, "test_speed_short": 1679.0900000000001}, {"test_speed": 6411.360000000001, "test_speed_short": 1840.0}, {"test_speed": 6455.400000000001, "test_speed_short": 2025.0500000000002}, {"test_speed": 6792.06, "test_speed_short": 2182.4}]}, "total": {"test_speed": 5085.215999999999, "test_speed_se": 997.596031123359, "test_speed_short": 1306.2649999999999, "test_speed_short_se": 403.70464961454957}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "swerec", "task": "sentiment-classification", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.7894648234072962, "macro_f1": 0.7759912640124949}, {"mcc": 0.7654572606337021, "macro_f1": 0.7489558358137534}, {"mcc": 0.7557188131326368, "macro_f1": 0.7273013447731036}, {"mcc": 0.7849187448096291, "macro_f1": 0.7672188326965398}, {"mcc": 0.7377184536121024, "macro_f1": 0.7257700761977937}, {"mcc": 0.7672900567039164, "macro_f1": 0.7419858487432142}, {"mcc": 0.7459985691267025, "macro_f1": 0.7426501838866844}, {"mcc": 0.752393291149162, "macro_f1": 0.7446800657874899}, {"mcc": 0.7659286725951657, "macro_f1": 0.7520686041410336}, {"mcc": 0.7331393879951842, "macro_f1": 0.7178062782876803}]}, "total": {"test_mcc": 75.98028073165499, "test_mcc_se": 1.1497547453351087, "test_macro_f1": 74.44428334339787, "test_macro_f1_se": 1.1228753910099514}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "suc3", "task": "named-entity-recognition", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5180309101316544, "micro_f1": 0.27184466019417475}, {"micro_f1_no_misc": 0.591520861372813, "micro_f1": 0.4599114608952288}, {"micro_f1_no_misc": 0.5550108932461875, "micro_f1": 0.48843069873997713}, {"micro_f1_no_misc": 0.5770277626565052, "micro_f1": 0.41251885369532426}, {"micro_f1_no_misc": 0.5928733889310083, "micro_f1": 0.4817221770917953}, {"micro_f1_no_misc": 0.5074626865671642, "micro_f1": 0.4134768036690774}, {"micro_f1_no_misc": 0.5370509922130118, "micro_f1": 0.4095221048863433}, {"micro_f1_no_misc": 0.45767260307392044, "micro_f1": 0.39913317572892043}, {"micro_f1_no_misc": 0.6058242329693188, "micro_f1": 0.48494453248811414}, {"micro_f1_no_misc": 0.494269340974212, "micro_f1": 0.39412181303116145}]}, "total": {"test_micro_f1_no_misc": 54.367436721357954, "test_micro_f1_no_misc_se": 3.0408651819481616, "test_micro_f1": 42.15626280420117, "test_micro_f1_se": 4.00307853371975}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-sv", "task": "linguistic-acceptability", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.20800892290815678, "macro_f1": 0.5562440473677197}, {"mcc": 0.23625733991911496, "macro_f1": 0.5889964840330992}, {"mcc": 0.16454921084867935, "macro_f1": 0.5749857508220679}, {"mcc": 0.17364684117215298, "macro_f1": 0.5449568899111477}, {"mcc": 0.1864211090170882, "macro_f1": 0.5932104130526792}, {"mcc": 0.1632541662149232, "macro_f1": 0.5722908024895561}, {"mcc": 0.11713757558186298, "macro_f1": 0.4953290993987531}, {"mcc": 0.16533046320669464, "macro_f1": 0.5109076332781571}, {"mcc": 0.19727737785051067, "macro_f1": 0.5796387520525452}, {"mcc": 0.18574379658164988, "macro_f1": 0.584247756636554}]}, "total": {"test_mcc": 17.976268033008335, "test_mcc_se": 1.9652139226795657, "test_macro_f1": 56.00807629042279, "test_macro_f1_se": 2.0789679982826867}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scandiqa-sv", "task": "question-answering", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 55.92563903950426, "f1": 64.09381252256908}, {"em": 54.34108527131783, "f1": 62.69805293505699}, {"em": 56.723338485316845, "f1": 64.11175029378353}, {"em": 55.99688473520249, "f1": 64.81609608870099}, {"em": 54.51737451737452, "f1": 65.0532325939225}, {"em": 54.81881264456438, "f1": 65.34450828039142}, {"em": 56.264236902050115, "f1": 63.72365366312695}, {"em": 54.38324282389449, "f1": 64.97338625326209}, {"em": 54.98039215686274, "f1": 64.96405821392156}, {"em": 52.71739130434783, "f1": 62.59441094878467}]}, "total": {"test_em": 55.06683978804356, "test_em_se": 0.7352570166629493, "test_f1": 64.23729617935199, "test_f1_se": 0.6081691684127206}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "swedn", "task": "summarization", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.642759829206625, "rouge_l": 0.1336418787814596}, {"bertscore": 0.6430029694747645, "rouge_l": 0.1493357532185356}, {"bertscore": 0.645746031368617, "rouge_l": 0.1442643167422919}, {"bertscore": 0.6460787011601496, "rouge_l": 0.15676738302228888}, {"bertscore": 0.6429613325744867, "rouge_l": 0.1470871844596599}, {"bertscore": 0.6439343006932177, "rouge_l": 0.13539146116418171}, {"bertscore": 0.646564692739048, "rouge_l": 0.15507257658782064}, {"bertscore": 0.6439566720218863, "rouge_l": 0.1399306961547083}, {"bertscore": 0.6440661973756505, "rouge_l": 0.14531053589482085}, {"bertscore": 0.643348487035837, "rouge_l": 0.1348610192225992}]}, "total": {"test_bertscore": 64.42419213650282, "test_bertscore_se": 0.08620035004355621, "test_rouge_l": 14.416628052483665, "test_rouge_l_se": 0.5088446252579157}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "mmlu-sv", "task": "knowledge", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.2889733153027504, "accuracy": 0.46435546875}, {"mcc": 0.32703854250703696, "accuracy": 0.49267578125}, {"mcc": 0.33975385098511274, "accuracy": 0.49853515625}, {"mcc": 0.3260286360570039, "accuracy": 0.4921875}, {"mcc": 0.32906737139693976, "accuracy": 0.49462890625}, {"mcc": 0.3298178943443183, "accuracy": 0.494140625}, {"mcc": 0.3391359815944487, "accuracy": 0.50048828125}, {"mcc": 0.33475919167627105, "accuracy": 0.49853515625}, {"mcc": 0.33679315607231053, "accuracy": 0.4970703125}, {"mcc": 0.32974629467668537, "accuracy": 0.49560546875}]}, "total": {"test_mcc": 32.811142346128776, "test_mcc_se": 0.9053869606271412, "test_accuracy": 49.2822265625, "test_accuracy_se": 0.6420040053060785}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "hellaswag-sv", "task": "common-sense-reasoning", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.5087890625, "mcc": 0.34689433379792856}, {"accuracy": 0.51611328125, "mcc": 0.35866931184646894}, {"accuracy": 0.5263671875, "mcc": 0.3709954992377235}, {"accuracy": 0.5078125, "mcc": 0.3491125435659484}, {"accuracy": 0.50634765625, "mcc": 0.34537491454028013}, {"accuracy": 0.5009765625, "mcc": 0.33878096639546923}, {"accuracy": 0.53125, "mcc": 0.3778417360269989}, {"accuracy": 0.50244140625, "mcc": 0.3402985955117182}, {"accuracy": 0.5556640625, "mcc": 0.4089659594179417}, {"accuracy": 0.54052734375, "mcc": 0.3873901934546367}]}, "total": {"test_accuracy": 51.962890625, "test_accuracy_se": 1.1325364022372872, "test_mcc": 36.243240537951145, "test_mcc_se": 1.4421217508758581}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norec", "task": "sentiment-classification", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.5199216796788708, "macro_f1": 0.6926228224676874}, {"mcc": 0.5136515685609524, "macro_f1": 0.6811989402838887}, {"mcc": 0.5331141312259325, "macro_f1": 0.6972400832870266}, {"mcc": 0.5495746909203776, "macro_f1": 0.6998746867167919}, {"mcc": 0.5057952405371908, "macro_f1": 0.6764520265536144}, {"mcc": 0.5396848239175875, "macro_f1": 0.6976697446977274}, {"mcc": 0.5779230343642895, "macro_f1": 0.717798197848134}, {"mcc": 0.5245640391293556, "macro_f1": 0.7001559720399547}, {"mcc": 0.5138402163681871, "macro_f1": 0.6658200199404446}, {"mcc": 0.5633210627670239, "macro_f1": 0.7200431676290447}]}, "total": {"test_mcc": 53.41390487469768, "test_mcc_se": 1.4561652097071172, "test_macro_f1": 69.48875661464314, "test_macro_f1_se": 1.05365118000823}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norne-nb", "task": "named-entity-recognition", "dataset_languages": ["nb"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5982575024201355, "micro_f1": 0.5229615745079662}, {"micro_f1_no_misc": 0.6260289210233593, "micro_f1": 0.5463749774001085}, {"micro_f1_no_misc": 0.6208092485549133, "micro_f1": 0.4622100954979536}, {"micro_f1_no_misc": 0.6138350437096161, "micro_f1": 0.5003598675687346}, {"micro_f1_no_misc": 0.6342674707343029, "micro_f1": 0.5738731856378916}, {"micro_f1_no_misc": 0.6170171715222843, "micro_f1": 0.506601607347876}, {"micro_f1_no_misc": 0.6370428627605191, "micro_f1": 0.5823820071562448}, {"micro_f1_no_misc": 0.6398996235884568, "micro_f1": 0.49569142388182197}, {"micro_f1_no_misc": 0.6091541868804049, "micro_f1": 0.4823259372609028}, {"micro_f1_no_misc": 0.6465344733490995, "micro_f1": 0.4916677776296493}]}, "total": {"test_micro_f1_no_misc": 62.42846504543091, "test_micro_f1_no_misc_se": 0.9433340339075545, "test_micro_f1": 51.64448453889149, "test_micro_f1_se": 2.4523713715188853}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norne-nn", "task": "named-entity-recognition", "dataset_languages": ["nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.6083701939435181, "micro_f1": 0.4985934360348292}, {"micro_f1_no_misc": 0.6128075379514919, "micro_f1": 0.470603110380579}, {"micro_f1_no_misc": 0.6072714182865371, "micro_f1": 0.5005007869509228}, {"micro_f1_no_misc": 0.6186868686868686, "micro_f1": 0.5260008548226243}, {"micro_f1_no_misc": 0.5981112277019937, "micro_f1": 0.41155155374986274}, {"micro_f1_no_misc": 0.6204895738893925, "micro_f1": 0.5106620209059233}, {"micro_f1_no_misc": 0.5871023215821152, "micro_f1": 0.48978479978207573}, {"micro_f1_no_misc": 0.6097431355181576, "micro_f1": 0.5722239406124342}, {"micro_f1_no_misc": 0.5884571859675595, "micro_f1": 0.491734921816828}, {"micro_f1_no_misc": 0.6171254855598717, "micro_f1": 0.41967358720994785}]}, "total": {"test_micro_f1_no_misc": 60.68164949087505, "test_micro_f1_no_misc_se": 0.7399436092330949, "test_micro_f1": 48.91329012266027, "test_micro_f1_se": 2.931975609435959}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-nb", "task": "linguistic-acceptability", "dataset_languages": ["nb"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": -0.04287591753068066, "macro_f1": 0.3393548387096774}, {"mcc": 0.0, "macro_f1": 0.3298429319371728}, {"mcc": 0.0, "macro_f1": 0.3249835201054713}, {"mcc": 0.0223412017732118, "macro_f1": 0.3368070888162737}, {"mcc": -0.02953780100896495, "macro_f1": 0.33419941456972146}, {"mcc": 0.0, "macro_f1": 0.3354964308890331}, {"mcc": 0.0, "macro_f1": 0.3393548387096774}, {"mcc": -0.037855095812506526, "macro_f1": 0.335280753002272}, {"mcc": -0.026956049530778597, "macro_f1": 0.3507947466769319}, {"mcc": -0.0009506322153399019, "macro_f1": 0.3302041709658256}]}, "total": {"test_mcc": -1.1583429432505885, "test_mcc_se": 1.3113520583052751, "test_macro_f1": 33.563187343820566, "test_macro_f1_se": 0.43217276144460004}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-nn", "task": "linguistic-acceptability", "dataset_languages": ["nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.0, "macro_f1": 0.3427471116816431}, {"mcc": 0.0, "macro_f1": 0.33093760209082}, {"mcc": 0.0, "macro_f1": 0.33289902280130296}, {"mcc": -0.011894517461593933, "macro_f1": 0.34314096185417237}, {"mcc": 0.03222654806539256, "macro_f1": 0.34222288482075}, {"mcc": 0.0, "macro_f1": 0.3431686978832585}, {"mcc": 0.009629037033826615, "macro_f1": 0.3467582721314064}, {"mcc": 0.0, "macro_f1": 0.34401024983984624}, {"mcc": 0.0, "macro_f1": 0.34401024983984624}, {"mcc": 0.0, "macro_f1": 0.33764553686934023}]}, "total": {"test_mcc": 0.29961067637625244, "test_mcc_se": 0.7105977657800933, "test_macro_f1": 34.07540589812386, "test_macro_f1_se": 0.3218228465206786}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norquad", "task": "question-answering", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 54.59057071960298, "f1": 79.36951428853958}, {"em": 43.02422723475355, "f1": 72.59505991085315}, {"em": 46.03174603174603, "f1": 74.91474302744389}, {"em": 44.334160463192724, "f1": 74.57191484736167}, {"em": 53.3278553820871, "f1": 78.74243671306567}, {"em": 53.54785478547855, "f1": 79.73160568808481}, {"em": 53.25638911788953, "f1": 79.63118245797781}, {"em": 51.4571190674438, "f1": 77.40938609661133}, {"em": 44.778613199665834, "f1": 74.44491768825759}, {"em": 47.15242881072027, "f1": 76.30986403907234}]}, "total": {"test_em": 49.15009648125803, "test_em_se": 2.7881405920218296, "test_f1": 76.77206247572677, "test_f1_se": 1.591448668867276}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "no-sammendrag", "task": "summarization", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.6378745920374058, "rouge_l": 0.13519110877785326}, {"bertscore": 0.6354336176300421, "rouge_l": 0.1313200771817863}, {"bertscore": 0.6381137947028037, "rouge_l": 0.13495672279710427}, {"bertscore": 0.6272821271995781, "rouge_l": 0.10965880225960993}, {"bertscore": 0.6399186215421651, "rouge_l": 0.1390233906055227}, {"bertscore": 0.6391916212742217, "rouge_l": 0.13663830296139362}, {"bertscore": 0.6349123492109356, "rouge_l": 0.13233740974921115}, {"bertscore": 0.6412511183298193, "rouge_l": 0.13743312777585692}, {"bertscore": 0.6388069704989903, "rouge_l": 0.13561420005745967}, {"bertscore": 0.6324046073568752, "rouge_l": 0.1268492208308276}]}, "total": {"test_bertscore": 63.651894197828376, "test_bertscore_se": 0.25863791732394154, "test_rouge_l": 13.190223629966253, "test_rouge_l_se": 0.5301625009511327}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "hellaswag-no", "task": "common-sense-reasoning", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.5556640625, "mcc": 0.4122086372085589}, {"accuracy": 0.53857421875, "mcc": 0.3890224904225414}, {"accuracy": 0.5791015625, "mcc": 0.43980946095646395}, {"accuracy": 0.57421875, "mcc": 0.43406367212481545}, {"accuracy": 0.47607421875, "mcc": 0.30757349479748075}, {"accuracy": 0.5185546875, "mcc": 0.35960215174016225}, {"accuracy": 0.53564453125, "mcc": 0.38194778103284993}, {"accuracy": 0.5517578125, "mcc": 0.40416942271026934}, {"accuracy": 0.595703125, "mcc": 0.46359424711689784}, {"accuracy": 0.56201171875, "mcc": 0.4209720858399295}]}, "total": {"test_accuracy": 54.873046875, "test_accuracy_se": 2.114622816396836, "test_mcc": 40.129634439499696, "test_mcc_se": 2.771846782237344}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}

@mhenrichsen Seems like you're missing the MMLU-no evaluation?

@saattrupdan should be correct now :)
{"dataset": "angry-tweets", "task": "sentiment-classification", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.5470151068743645, "macro_f1": 0.7047250329278935}, {"mcc": 0.5509690251753908, "macro_f1": 0.7022794811767601}, {"mcc": 0.5395830864902568, "macro_f1": 0.6863279486150139}, {"mcc": 0.5601647856420113, "macro_f1": 0.7062123686954361}, {"mcc": 0.5262341779021951, "macro_f1": 0.6681850368248451}, {"mcc": 0.5694551924277325, "macro_f1": 0.7122101309862009}, {"mcc": 0.516581263560336, "macro_f1": 0.6736354224164008}, {"mcc": 0.5597581131546965, "macro_f1": 0.7081448226150671}, {"mcc": 0.5467387241736854, "macro_f1": 0.6893435885109872}, {"mcc": 0.5525322086485214, "macro_f1": 0.6988609918653976}]}, "total": {"test_mcc": 54.69031684049192, "test_mcc_se": 0.9904368322842158, "test_macro_f1": 69.49924824634003, "test_macro_f1_se": 0.9331129988021167}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "dansk", "task": "named-entity-recognition", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5400089405453732, "micro_f1": 0.3580246913580247}, {"micro_f1_no_misc": 0.5028773793714032, "micro_f1": 0.39281045751633986}, {"micro_f1_no_misc": 0.47290209790209786, "micro_f1": 0.3587360594795539}, {"micro_f1_no_misc": 0.505229283990346, "micro_f1": 0.37739181630850754}, {"micro_f1_no_misc": 0.5161290322580645, "micro_f1": 0.344225352112676}, {"micro_f1_no_misc": 0.5201923076923076, "micro_f1": 0.3684377982818963}, {"micro_f1_no_misc": 0.47560975609756095, "micro_f1": 0.299298519095869}, {"micro_f1_no_misc": 0.543313708999159, "micro_f1": 0.4024024024024024}, {"micro_f1_no_misc": 0.4918478260869565, "micro_f1": 0.3354763296317943}, {"micro_f1_no_misc": 0.5400516795865633, "micro_f1": 0.3462149285336157}]}, "total": {"test_micro_f1_no_misc": 51.08162012529831, "test_micro_f1_no_misc_se": 1.601688418285135, "test_micro_f1": 35.83018354720679, "test_micro_f1_se": 1.8420954603454487}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-da", "task": "linguistic-acceptability", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.2707853552154174, "macro_f1": 0.5865546953255694}, {"mcc": 0.30352782788839133, "macro_f1": 0.6185297484185476}, {"mcc": 0.3310526009952438, "macro_f1": 0.646833382999043}, {"mcc": 0.30295938829684965, "macro_f1": 0.6079233247658888}, {"mcc": 0.2968410572843576, "macro_f1": 0.6348737679195147}, {"mcc": 0.3256648126595716, "macro_f1": 0.6526086890051914}, {"mcc": 0.28083741862334227, "macro_f1": 0.5591316706916011}, {"mcc": 0.31816687902908636, "macro_f1": 0.6311555240817968}, {"mcc": 0.3138504791931172, "macro_f1": 0.6328910780964487}, {"mcc": 0.35152957070308083, "macro_f1": 0.6442164870041693}]}, "total": {"test_mcc": 30.952153898884582, "test_mcc_se": 1.4819869752353665, "test_macro_f1": 62.1471836830777, "test_macro_f1_se": 1.8283456553613204}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scandiqa-da", "task": "question-answering", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 55.30596436870643, "f1": 64.2806621799194}, {"em": 56.58914728682171, "f1": 64.25636592975852}, {"em": 57.95981452859351, "f1": 65.04863101503464}, {"em": 55.06230529595015, "f1": 63.85892781492223}, {"em": 57.76061776061776, "f1": 65.07433931803675}, {"em": 54.58750963762529, "f1": 62.82309817061955}, {"em": 56.79574791192103, "f1": 64.13631643565981}, {"em": 57.87432117920869, "f1": 64.72082710660143}, {"em": 57.333333333333336, "f1": 64.50587203389968}, {"em": 56.2888198757764, "f1": 64.4702692443147}]}, "total": {"test_em": 56.55575811785543, "test_em_se": 0.7608640090317561, "test_f1": 64.31753092487665, "test_f1_se": 0.40328720275943436}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "nordjylland-news", "task": "summarization", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.668609851389192, "rouge_l": 0.2039914021393196}, {"bertscore": 0.6746063534519635, "rouge_l": 0.21926533456511507}, {"bertscore": 0.664634121421841, "rouge_l": 0.20472851618203852}, {"bertscore": 0.6726909662247635, "rouge_l": 0.21167626165322267}, {"bertscore": 0.6668700768059352, "rouge_l": 0.23086788686031476}, {"bertscore": 0.6725016877899179, "rouge_l": 0.2130333954792637}, {"bertscore": 0.6683111188176554, "rouge_l": 0.20418486333637614}, {"bertscore": 0.6710484639770584, "rouge_l": 0.20629890099620568}, {"bertscore": 0.6630088706006063, "rouge_l": 0.19357379913847286}, {"bertscore": 0.6676352790091187, "rouge_l": 0.20195556646682122}]}, "total": {"test_bertscore": 66.89916789488052, "test_bertscore_se": 0.22959075664787143, "test_rouge_l": 20.895759268171503, "test_rouge_l_se": 0.6429274915522064}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "danske-talemaader", "task": "knowledge", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.773350766605174, "accuracy": 0.826171875}, {"mcc": 0.7915544946103718, "accuracy": 0.83984375}, {"mcc": 0.7728212208829291, "accuracy": 0.8271484375}, {"mcc": 0.7902566489409734, "accuracy": 0.83984375}, {"mcc": 0.8261098835829227, "accuracy": 0.8671875}, {"mcc": 0.7981074441251266, "accuracy": 0.84765625}, {"mcc": 0.7972037226725296, "accuracy": 0.8447265625}, {"mcc": 0.7802465571650927, "accuracy": 0.833984375}, {"mcc": 0.7887728080473971, "accuracy": 0.837890625}, {"mcc": 0.820831763126859, "accuracy": 0.8642578125}]}, "total": {"test_mcc": 79.39255309759375, "test_mcc_se": 1.1105654030739633, "test_accuracy": 84.287109375, "test_accuracy_se": 0.8576338599946044}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "danish-citizen-tests", "task": "knowledge", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.7027328105687023, "accuracy": 0.794921875}, {"mcc": 0.7585038183877986, "accuracy": 0.837890625}, {"mcc": 0.752970274864937, "accuracy": 0.83203125}, {"mcc": 0.735234481490622, "accuracy": 0.822265625}, {"mcc": 0.695602885466217, "accuracy": 0.794921875}, {"mcc": 0.737212069532255, "accuracy": 0.826171875}, {"mcc": 0.7016433025242871, "accuracy": 0.796875}, {"mcc": 0.7611053735916435, "accuracy": 0.83984375}, {"mcc": 0.7020157251736459, "accuracy": 0.796875}, {"mcc": 0.7750934968107404, "accuracy": 0.849609375}]}, "total": {"test_mcc": 73.22114238410849, "test_mcc_se": 1.8346295360263907, "test_accuracy": 81.9140625, "test_accuracy_se": 1.3226588905680146}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "hellaswag-da", "task": "common-sense-reasoning", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.61865234375, "mcc": 0.49565878624255083}, {"accuracy": 0.6279296875, "mcc": 0.5053010308414206}, {"accuracy": 0.60986328125, "mcc": 0.4805284991610356}, {"accuracy": 0.5849609375, "mcc": 0.45646051830773443}, {"accuracy": 0.62353515625, "mcc": 0.5006686175994569}, {"accuracy": 0.609375, "mcc": 0.4815019384075108}, {"accuracy": 0.59765625, "mcc": 0.465792646961778}, {"accuracy": 0.59619140625, "mcc": 0.4684679853431774}, {"accuracy": 0.5888671875, "mcc": 0.4603454233870459}, {"accuracy": 0.6259765625, "mcc": 0.5015713200705473}]}, "total": {"test_accuracy": 60.830078125, "test_accuracy_se": 0.9740846080825533, "test_mcc": 48.16296766322258, "test_mcc_se": 1.1370770405809758}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "speed", "task": "speed", "dataset_languages": ["ab", "aa", "af", "sq", "am", "ar", "an", "hy", "as", "av", "ae", "ay", "az", "bm", "ba", "eu", "be", "bn", "bi", "bs", "br", "bg", "my", "ca", "ch", "ce", "ny", "zh", "cu", "cv", "kw", "co", "cr", "hr", "cs", "da", "dv", "nl", "dz", "en", "eo", "et", "ee", "fo", "fj", "fi", "fr", "fy", "ff", "gd", "gl", "lg", "ka", "de", "el", "kl", "gn", "gu", "ht", "ha", "he", "hz", "hi", "ho", "hu", "is", "io", "ig", "id", "ia", "ie", "iu", "ik", "ga", "it", "ja", "kn", "kr", "ks", "kk", "km", "ki", "rw", "ky", "kv", "kg", "ko", "kj", "ku", "lo", "la", "lv", "li", "ln", "lt", "lu", "lb", "mk", "mg", "ms", "ml", "mt", "gv", "mi", "mr", "mh", "mn", "na", "nv", "nd", "nr", "ng", "ne", "no", "nb", "nn", "ii", "oc", "oj", "or", "om", "os", "pi", "ps", "fa", "pl", "pt", "pa", "qu", "ro", "rm", "rn", "ru", "se", "sm", "sg", "sa", "sc", "sr", "sn", "sd", "si", "sk", "sl", "so", "st", "es", "su", "sw", "ss", "sv", "tl", "ty", "tg", "ta", "tt", "te", "th", "bo", "ti", "to", "ts", "tn", "tr", "tk", "tw", "ug", "uk", "ur", "uz", "ve", "vi", "vo", "wa", "cy", "wo", "xh", "yi", "yo", "za", "zu"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"test_speed": 1854.72, "test_speed_short": 267.96}, {"test_speed": 3206.84, "test_speed_short": 466.2}, {"test_speed": 3995.68, "test_speed_short": 875.52}, {"test_speed": 4984.74, "test_speed_short": 1022.25}, {"test_speed": 5365.24, "test_speed_short": 1184.96}, {"test_speed": 5560.92, "test_speed_short": 1519.22}, {"test_speed": 6225.2, "test_speed_short": 1679.0900000000001}, {"test_speed": 6411.360000000001, "test_speed_short": 1840.0}, {"test_speed": 6455.400000000001, "test_speed_short": 2025.0500000000002}, {"test_speed": 6792.06, "test_speed_short": 2182.4}]}, "total": {"test_speed": 5085.215999999999, "test_speed_se": 997.596031123359, "test_speed_short": 1306.2649999999999, "test_speed_short_se": 403.70464961454957}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "swerec", "task": "sentiment-classification", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.7894648234072962, "macro_f1": 0.7759912640124949}, {"mcc": 0.7654572606337021, "macro_f1": 0.7489558358137534}, {"mcc": 0.7557188131326368, "macro_f1": 0.7273013447731036}, {"mcc": 0.7849187448096291, "macro_f1": 0.7672188326965398}, {"mcc": 0.7377184536121024, "macro_f1": 0.7257700761977937}, {"mcc": 0.7672900567039164, "macro_f1": 0.7419858487432142}, {"mcc": 0.7459985691267025, "macro_f1": 0.7426501838866844}, {"mcc": 0.752393291149162, "macro_f1": 0.7446800657874899}, {"mcc": 0.7659286725951657, "macro_f1": 0.7520686041410336}, {"mcc": 0.7331393879951842, "macro_f1": 0.7178062782876803}]}, "total": {"test_mcc": 75.98028073165499, "test_mcc_se": 1.1497547453351087, "test_macro_f1": 74.44428334339787, "test_macro_f1_se": 1.1228753910099514}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "suc3", "task": "named-entity-recognition", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5180309101316544, "micro_f1": 0.27184466019417475}, {"micro_f1_no_misc": 0.591520861372813, "micro_f1": 0.4599114608952288}, {"micro_f1_no_misc": 0.5550108932461875, "micro_f1": 0.48843069873997713}, {"micro_f1_no_misc": 0.5770277626565052, "micro_f1": 0.41251885369532426}, {"micro_f1_no_misc": 0.5928733889310083, "micro_f1": 0.4817221770917953}, {"micro_f1_no_misc": 0.5074626865671642, "micro_f1": 0.4134768036690774}, {"micro_f1_no_misc": 0.5370509922130118, "micro_f1": 0.4095221048863433}, {"micro_f1_no_misc": 0.45767260307392044, "micro_f1": 0.39913317572892043}, {"micro_f1_no_misc": 0.6058242329693188, "micro_f1": 0.48494453248811414}, {"micro_f1_no_misc": 0.494269340974212, "micro_f1": 0.39412181303116145}]}, "total": {"test_micro_f1_no_misc": 54.367436721357954, "test_micro_f1_no_misc_se": 3.0408651819481616, "test_micro_f1": 42.15626280420117, "test_micro_f1_se": 4.00307853371975}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-sv", "task": "linguistic-acceptability", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.20800892290815678, "macro_f1": 0.5562440473677197}, {"mcc": 0.23625733991911496, "macro_f1": 0.5889964840330992}, {"mcc": 0.16454921084867935, "macro_f1": 0.5749857508220679}, {"mcc": 0.17364684117215298, "macro_f1": 0.5449568899111477}, {"mcc": 0.1864211090170882, "macro_f1": 0.5932104130526792}, {"mcc": 0.1632541662149232, "macro_f1": 0.5722908024895561}, {"mcc": 0.11713757558186298, "macro_f1": 0.4953290993987531}, {"mcc": 0.16533046320669464, "macro_f1": 0.5109076332781571}, {"mcc": 0.19727737785051067, "macro_f1": 0.5796387520525452}, {"mcc": 0.18574379658164988, "macro_f1": 0.584247756636554}]}, "total": {"test_mcc": 17.976268033008335, "test_mcc_se": 1.9652139226795657, "test_macro_f1": 56.00807629042279, "test_macro_f1_se": 2.0789679982826867}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scandiqa-sv", "task": "question-answering", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 55.92563903950426, "f1": 64.09381252256908}, {"em": 54.34108527131783, "f1": 62.69805293505699}, {"em": 56.723338485316845, "f1": 64.11175029378353}, {"em": 55.99688473520249, "f1": 64.81609608870099}, {"em": 54.51737451737452, "f1": 65.0532325939225}, {"em": 54.81881264456438, "f1": 65.34450828039142}, {"em": 56.264236902050115, "f1": 63.72365366312695}, {"em": 54.38324282389449, "f1": 64.97338625326209}, {"em": 54.98039215686274, "f1": 64.96405821392156}, {"em": 52.71739130434783, "f1": 62.59441094878467}]}, "total": {"test_em": 55.06683978804356, "test_em_se": 0.7352570166629493, "test_f1": 64.23729617935199, "test_f1_se": 0.6081691684127206}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "swedn", "task": "summarization", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.642759829206625, "rouge_l": 0.1336418787814596}, {"bertscore": 0.6430029694747645, "rouge_l": 0.1493357532185356}, {"bertscore": 0.645746031368617, "rouge_l": 0.1442643167422919}, {"bertscore": 0.6460787011601496, "rouge_l": 0.15676738302228888}, {"bertscore": 0.6429613325744867, "rouge_l": 0.1470871844596599}, {"bertscore": 0.6439343006932177, "rouge_l": 0.13539146116418171}, {"bertscore": 0.646564692739048, "rouge_l": 0.15507257658782064}, {"bertscore": 0.6439566720218863, "rouge_l": 0.1399306961547083}, {"bertscore": 0.6440661973756505, "rouge_l": 0.14531053589482085}, {"bertscore": 0.643348487035837, "rouge_l": 0.1348610192225992}]}, "total": {"test_bertscore": 64.42419213650282, "test_bertscore_se": 0.08620035004355621, "test_rouge_l": 14.416628052483665, "test_rouge_l_se": 0.5088446252579157}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "mmlu-sv", "task": "knowledge", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.2889733153027504, "accuracy": 0.46435546875}, {"mcc": 0.32703854250703696, "accuracy": 0.49267578125}, {"mcc": 0.33975385098511274, "accuracy": 0.49853515625}, {"mcc": 0.3260286360570039, "accuracy": 0.4921875}, {"mcc": 0.32906737139693976, "accuracy": 0.49462890625}, {"mcc": 0.3298178943443183, "accuracy": 0.494140625}, {"mcc": 0.3391359815944487, "accuracy": 0.50048828125}, {"mcc": 0.33475919167627105, "accuracy": 0.49853515625}, {"mcc": 0.33679315607231053, "accuracy": 0.4970703125}, {"mcc": 0.32974629467668537, "accuracy": 0.49560546875}]}, "total": {"test_mcc": 32.811142346128776, "test_mcc_se": 0.9053869606271412, "test_accuracy": 49.2822265625, "test_accuracy_se": 0.6420040053060785}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "hellaswag-sv", "task": "common-sense-reasoning", "dataset_languages": ["sv"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.5087890625, "mcc": 0.34689433379792856}, {"accuracy": 0.51611328125, "mcc": 0.35866931184646894}, {"accuracy": 0.5263671875, "mcc": 0.3709954992377235}, {"accuracy": 0.5078125, "mcc": 0.3491125435659484}, {"accuracy": 0.50634765625, "mcc": 0.34537491454028013}, {"accuracy": 0.5009765625, "mcc": 0.33878096639546923}, {"accuracy": 0.53125, "mcc": 0.3778417360269989}, {"accuracy": 0.50244140625, "mcc": 0.3402985955117182}, {"accuracy": 0.5556640625, "mcc": 0.4089659594179417}, {"accuracy": 0.54052734375, "mcc": 0.3873901934546367}]}, "total": {"test_accuracy": 51.962890625, "test_accuracy_se": 1.1325364022372872, "test_mcc": 36.243240537951145, "test_mcc_se": 1.4421217508758581}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norec", "task": "sentiment-classification", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.5199216796788708, "macro_f1": 0.6926228224676874}, {"mcc": 0.5136515685609524, "macro_f1": 0.6811989402838887}, {"mcc": 0.5331141312259325, "macro_f1": 0.6972400832870266}, {"mcc": 0.5495746909203776, "macro_f1": 0.6998746867167919}, {"mcc": 0.5057952405371908, "macro_f1": 0.6764520265536144}, {"mcc": 0.5396848239175875, "macro_f1": 0.6976697446977274}, {"mcc": 0.5779230343642895, "macro_f1": 0.717798197848134}, {"mcc": 0.5245640391293556, "macro_f1": 0.7001559720399547}, {"mcc": 0.5138402163681871, "macro_f1": 0.6658200199404446}, {"mcc": 0.5633210627670239, "macro_f1": 0.7200431676290447}]}, "total": {"test_mcc": 53.41390487469768, "test_mcc_se": 1.4561652097071172, "test_macro_f1": 69.48875661464314, "test_macro_f1_se": 1.05365118000823}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norne-nb", "task": "named-entity-recognition", "dataset_languages": ["nb"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.5982575024201355, "micro_f1": 0.5229615745079662}, {"micro_f1_no_misc": 0.6260289210233593, "micro_f1": 0.5463749774001085}, {"micro_f1_no_misc": 0.6208092485549133, "micro_f1": 0.4622100954979536}, {"micro_f1_no_misc": 0.6138350437096161, "micro_f1": 0.5003598675687346}, {"micro_f1_no_misc": 0.6342674707343029, "micro_f1": 0.5738731856378916}, {"micro_f1_no_misc": 0.6170171715222843, "micro_f1": 0.506601607347876}, {"micro_f1_no_misc": 0.6370428627605191, "micro_f1": 0.5823820071562448}, {"micro_f1_no_misc": 0.6398996235884568, "micro_f1": 0.49569142388182197}, {"micro_f1_no_misc": 0.6091541868804049, "micro_f1": 0.4823259372609028}, {"micro_f1_no_misc": 0.6465344733490995, "micro_f1": 0.4916677776296493}]}, "total": {"test_micro_f1_no_misc": 62.42846504543091, "test_micro_f1_no_misc_se": 0.9433340339075545, "test_micro_f1": 51.64448453889149, "test_micro_f1_se": 2.4523713715188853}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norne-nn", "task": "named-entity-recognition", "dataset_languages": ["nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"micro_f1_no_misc": 0.6083701939435181, "micro_f1": 0.4985934360348292}, {"micro_f1_no_misc": 0.6128075379514919, "micro_f1": 0.470603110380579}, {"micro_f1_no_misc": 0.6072714182865371, "micro_f1": 0.5005007869509228}, {"micro_f1_no_misc": 0.6186868686868686, "micro_f1": 0.5260008548226243}, {"micro_f1_no_misc": 0.5981112277019937, "micro_f1": 0.41155155374986274}, {"micro_f1_no_misc": 0.6204895738893925, "micro_f1": 0.5106620209059233}, {"micro_f1_no_misc": 0.5871023215821152, "micro_f1": 0.48978479978207573}, {"micro_f1_no_misc": 0.6097431355181576, "micro_f1": 0.5722239406124342}, {"micro_f1_no_misc": 0.5884571859675595, "micro_f1": 0.491734921816828}, {"micro_f1_no_misc": 0.6171254855598717, "micro_f1": 0.41967358720994785}]}, "total": {"test_micro_f1_no_misc": 60.68164949087505, "test_micro_f1_no_misc_se": 0.7399436092330949, "test_micro_f1": 48.91329012266027, "test_micro_f1_se": 2.931975609435959}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-nb", "task": "linguistic-acceptability", "dataset_languages": ["nb"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": -0.04287591753068066, "macro_f1": 0.3393548387096774}, {"mcc": 0.0, "macro_f1": 0.3298429319371728}, {"mcc": 0.0, "macro_f1": 0.3249835201054713}, {"mcc": 0.0223412017732118, "macro_f1": 0.3368070888162737}, {"mcc": -0.02953780100896495, "macro_f1": 0.33419941456972146}, {"mcc": 0.0, "macro_f1": 0.3354964308890331}, {"mcc": 0.0, "macro_f1": 0.3393548387096774}, {"mcc": -0.037855095812506526, "macro_f1": 0.335280753002272}, {"mcc": -0.026956049530778597, "macro_f1": 0.3507947466769319}, {"mcc": -0.0009506322153399019, "macro_f1": 0.3302041709658256}]}, "total": {"test_mcc": -1.1583429432505885, "test_mcc_se": 1.3113520583052751, "test_macro_f1": 33.563187343820566, "test_macro_f1_se": 0.43217276144460004}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "scala-nn", "task": "linguistic-acceptability", "dataset_languages": ["nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.0, "macro_f1": 0.3427471116816431}, {"mcc": 0.0, "macro_f1": 0.33093760209082}, {"mcc": 0.0, "macro_f1": 0.33289902280130296}, {"mcc": -0.011894517461593933, "macro_f1": 0.34314096185417237}, {"mcc": 0.03222654806539256, "macro_f1": 0.34222288482075}, {"mcc": 0.0, "macro_f1": 0.3431686978832585}, {"mcc": 0.009629037033826615, "macro_f1": 0.3467582721314064}, {"mcc": 0.0, "macro_f1": 0.34401024983984624}, {"mcc": 0.0, "macro_f1": 0.34401024983984624}, {"mcc": 0.0, "macro_f1": 0.33764553686934023}]}, "total": {"test_mcc": 0.29961067637625244, "test_mcc_se": 0.7105977657800933, "test_macro_f1": 34.07540589812386, "test_macro_f1_se": 0.3218228465206786}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "norquad", "task": "question-answering", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"em": 54.59057071960298, "f1": 79.36951428853958}, {"em": 43.02422723475355, "f1": 72.59505991085315}, {"em": 46.03174603174603, "f1": 74.91474302744389}, {"em": 44.334160463192724, "f1": 74.57191484736167}, {"em": 53.3278553820871, "f1": 78.74243671306567}, {"em": 53.54785478547855, "f1": 79.73160568808481}, {"em": 53.25638911788953, "f1": 79.63118245797781}, {"em": 51.4571190674438, "f1": 77.40938609661133}, {"em": 44.778613199665834, "f1": 74.44491768825759}, {"em": 47.15242881072027, "f1": 76.30986403907234}]}, "total": {"test_em": 49.15009648125803, "test_em_se": 2.7881405920218296, "test_f1": 76.77206247572677, "test_f1_se": 1.591448668867276}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "no-sammendrag", "task": "summarization", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"bertscore": 0.6378745920374058, "rouge_l": 0.13519110877785326}, {"bertscore": 0.6354336176300421, "rouge_l": 0.1313200771817863}, {"bertscore": 0.6381137947028037, "rouge_l": 0.13495672279710427}, {"bertscore": 0.6272821271995781, "rouge_l": 0.10965880225960993}, {"bertscore": 0.6399186215421651, "rouge_l": 0.1390233906055227}, {"bertscore": 0.6391916212742217, "rouge_l": 0.13663830296139362}, {"bertscore": 0.6349123492109356, "rouge_l": 0.13233740974921115}, {"bertscore": 0.6412511183298193, "rouge_l": 0.13743312777585692}, {"bertscore": 0.6388069704989903, "rouge_l": 0.13561420005745967}, {"bertscore": 0.6324046073568752, "rouge_l": 0.1268492208308276}]}, "total": {"test_bertscore": 63.651894197828376, "test_bertscore_se": 0.25863791732394154, "test_rouge_l": 13.190223629966253, "test_rouge_l_se": 0.5301625009511327}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "hellaswag-no", "task": "common-sense-reasoning", "dataset_languages": ["nb", "nn"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"accuracy": 0.5556640625, "mcc": 0.4122086372085589}, {"accuracy": 0.53857421875, "mcc": 0.3890224904225414}, {"accuracy": 0.5791015625, "mcc": 0.43980946095646395}, {"accuracy": 0.57421875, "mcc": 0.43406367212481545}, {"accuracy": 0.47607421875, "mcc": 0.30757349479748075}, {"accuracy": 0.5185546875, "mcc": 0.35960215174016225}, {"accuracy": 0.53564453125, "mcc": 0.38194778103284993}, {"accuracy": 0.5517578125, "mcc": 0.40416942271026934}, {"accuracy": 0.595703125, "mcc": 0.46359424711689784}, {"accuracy": 0.56201171875, "mcc": 0.4209720858399295}]}, "total": {"test_accuracy": 54.873046875, "test_accuracy_se": 2.114622816396836, "test_mcc": 40.129634439499696, "test_mcc_se": 2.771846782237344}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"}
{"dataset": "mmlu-no", "task": "knowledge", "dataset_languages": ["da"], "model": "mhenrichsen/danskgpt-chat-v2.1", "results": {"raw": {"test": [{"mcc": 0.37549769997346394, "accuracy": 0.5283203125}, {"mcc": 0.40632513123786723, "accuracy": 0.55224609375}, {"mcc": 0.3971449342797012, "accuracy": 0.541015625}, {"mcc": 0.3814356808999663, "accuracy": 0.53173828125}, {"mcc": 0.3707424007848007, "accuracy": 0.52685546875}, {"mcc": 0.3855216022079138, "accuracy": 0.53955078125}, {"mcc": 0.3875034473065225, "accuracy": 0.5400390625}, {"mcc": 0.3771144430159841, "accuracy": 0.52880859375}, {"mcc": 0.3872453343989541, "accuracy": 0.53857421875}, {"mcc": 0.37818164219054357, "accuracy": 0.52978515625}]}, "total": {"test_mcc": 38.46712316295717, "test_mcc_se": 0.6623233594368452, "test_accuracy": 53.56933593749999, "test_accuracy_se": 0.4953298068975686}}, "num_model_parameters": -1, "max_sequence_length": 32768, "vocabulary_size": 32002, "generative": true, "few_shot": true, "validation_split": false, "scandeval_version": "12.0.0"

authors: mhenrichsen saattrupdan

repo: ScandEval/ScandEval


text:
memory keep grow after I place a modal on a plane and do nothing in some android devices

 It will lead to an ANR at last ~

Bump: Also seeing this issue

source: gharchive/issue

authors: chenleicrm kelvinwatson

repo: SceneView/sceneform-android

license_type: permissive

license_source: github-api


text:
⚠ Jenkins Plugins Artifactory has degraded performance

In 8548da0, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1677 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 2fe28f0 after 1 hour, 41 minutes.

repo: Schmitzis/Uptime

license: MIT


text:
⚠ Jenkins Plugins Artifactory has degraded performance

In 5d227fe, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1138 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 5645319 after 25 minutes.


text:
⚠ Jenkins Plugins Artifactory has degraded performance

In dee1485, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1290 ms

Resolved: Jenkins Plugins Artifactory performance has improved in 25ba87e after 20 minutes.


text:
Issue #81 add sci meta to derived datasets model

Description
add sci meta to derived datasets model

[ ] Included for each change/fix?
[x] Passing? (Merge will not be approved unless this is checked)
[ ] Docs updated?

already approved for merge

source: gharchive/pull-request

repo: SciCatProject/catamel


text:
Implementation of 23 test cases

Implementation of the 23 test cases in https://people.sc.fsu.edu/~jburkardt/m_src/test_nonlin/test_nonlin.html according to #142.
The result of the program looks like this:
nr  Problem                                            n     NewtonRaphson       TrustRegion         LevenbergMarquardt  nlsolve
1   Generalized Rosenbrock function                    10    2.5809e-8           4.9193              4.9193              0.0
2   Powell singular function                           4     7.0435e-16          7.941               0.021633            7.0217e-16
3   Powell badly scaled function                       2     0.0                 26897.0             2.1983e-14          0.0
4   Wood function                                      4     4.1832e-14          19081.0             8550.5              8.2157e-15
5   Helical valley function                            3     6.2775e-30          50.0                50.0                4.3667e-26
6   Watson function                                    2     18860.0             3004.4              129.15              114.89
7   Chebyquad function                                 2     5.5511e-17          0.075044            0.34127             9.4369e-16
8   Brown almost linear function                       10    0.0                 0.01533             16.53               4.4409e-16
10  Discrete integral equation function                10    2.7807e-15          1.9166e-15          9.3334e-15          5.5511e-17
11  Trigonometric function                             10    1.4753e-15          2.2808e-15          0.084117            6.8695e-16
12  Variably dimensioned function                      10    0.0                 3.5707e-311         1.376e-12           0.0
13  Broyden tridiagonal function                       10    1.2803e-15          2.7487e-15          1.1633e-13          6.6613e-16
14  Broyden banded function                            10    1.4002e-15          2.1729e-15          1.6865e-13          5.5511e-16
15  Hammarling 2 by 2 matrix square root problem       4     2.2204e-16          0.32355             0.91487             2.2204e-16
16  Hammarling 3 by 3 matrix square root problem       9     2.2204e-16          0.30203             1.0506              2.2204e-16
17  Dennis and Schnabel 2 by 2 example                 2     0.0                 4.4408e-16          17.262              0.0
18  Sample problem 18                                  2     0.0                 6.2862e-9           2.1951              8.291e-17
19  Sample problem 19                                  2     9.5092e-16          3.4505e-7           76.367              7.7402e-16
20  Scalar problem f(x) = x(x - 5)^2                   1     0.0                 1.8919e-16          2.5564e-18          0.0
21  Freudenstein-Roth function                         2     0.0                 20.012              20.012              4.949
22  Boggs function                                     2     0.0                 1.6031              2.0                 3.9431e-17
23  Chandrasekhar function                             10    6.6613e-16          9.9301e-16          1.0906e-13          2.2204e-16

Remember formatting

Hmm, this file is formatted and the error message points to the file ``NonlinearSystemSolvers.md''


text:
[API] Better support for SQL databases

This is multipart proposal.
First: Change method name: from_sqls intuitive understanding is "from SQL database dump" (file or string, containing something like INSERT INTO ... VALUES ... -- such dumps any database could produce and in some context it is a common format for data exchange), which is not the case.
I think from_db could express the intention better.
Second: as far as I can see in practice, there are 3 distinct cases we need to cover for easy loading data from DB:

load entire table into the dataframe;
load some columns of table into the dataframe;
load results of some SQL query into the dataframe.

So, the most reasonable API is:
DataFrame.from_db(adapter, table: 'tbl')
DataFrame.from_db(adapter, table: 'tbl', columns: [:id, :name, :gender])
DataFrame.from_db(adapter, query: 'SELECT * FROM employees JOIN departments...' )

columns param could be provided with the query also. And also, it can be a hash, or mix of array/hash, for column renaming while loading:
DataFrame.from_db(adapter, table: 'tbl', columns: [:id, Name: :name, employee_gender: :gender])

Alright. We should keep compatibility with from_sql because many people use that. The enhanced behavior looks good.

Just throwing my cents in here.

I don't have any strong opinions on naming for the method but I'd be curious to hear more rationale for the change. The pandas method is read_sql so I don't think it's non-standard to use sql instead of db. Do you have any examples of other libraries which use db instead of sql? I agree that continued support of from_sql or at least a deprecation period would be a good idea.

It seems like the first two cases can be easily handled directly in a very basic SQL query. Is it worth the additional complexity in the method?

Just wanting to better understand the motivations behind the changes and happy to discuss.

Do you have any examples of other libraries which use db instead of sql?

Not sure about it. That was rather "personal taste-based" than "example-based" suggestion. But it can be considered other way round: from_sql means import from SQL database (in constranst for some No-SQL or specialized databases). So, I suppose, current name should work also.

It seems like the first two cases can be easily handled directly in a very basic SQL query. Is it worth the additional complexity in the method?

Yeah, I think it is. First, not everyone is familiary with SQL (and there would be constant need for some users for refering to docs about this SELECT * FROM table or was it FETCH FROM table (*) or should I use some kinde of parens, or waht); second, shortening most common cases is almost always a good thing (at least, for my taste), and Ruby's keyword arguments allow to handle it gracefully. Personally, I'd hate the necessity to write from_sql(adapter, 'SELECT * FROM employees') each and every time for quick fire-and-forget experimental scripts.

I also prefer from_sql.  Not everything that supports SQL(-like) syntax is a "database".  For example, Salesforce and Apache Spark are not traditional databases, but can both be accessed using a SQL-like syntax.

I think we can use ActiveRecord gem to support queries such as
User.where(name: 'David', occupation: 'Code Artist').order(created_at: :desc)

which is way simpler than (SELECT * ...) and quite popular because of Rails.

+1 to @lokeshh though the implementation of that would need to be looked into. And I agree with @gnilrets about from_sql. We should ideally be able to swap out the underlying SQL adapter in the future in order to accommodate for Spark or Salesforce.

Yeah, I think it is. First, not everyone is familiary with SQL (and there would be constant need for some users for refering to docs about this SELECT * FROM table or was it FETCH FROM table (*) or should I use some kinde of parens, or waht); second, shortening most common cases is almost always a good thing (at least, for my taste), and Ruby's keyword arguments allow to handle it gracefully. Personally, I'd hate the necessity to write from_sql(adapter, 'SELECT * FROM employees') each and every time for quick fire-and-forget experimental scripts.

@zverok agreed.

@lokeshh

In addition, I think we can use ActiveRecord gem to support queries such as

How do you imagine it? If by adding direct integration/dependencies, I'd be pretty much against it.

Maybe put in a separate gem (daru-activerecord?) that depends on both activerecord and daru for doing this.

Closing in favor of daru-io

authors: dansbits gnilrets lokeshh v0dro zverok

repo: SciRuby/daru

license_source: bigquery


text:
Make concatenation signature public?

✨ Feature Request
Could we consider making _concatenate._CubeSignature a public symbol?
Motivation
I'm stitching thousands of octants into global fields. I group the cubes by their concatenation signature so I can loop through them and sanity check, for each each group, that:

there are 8 octants or two hemispheres in the group, and warn otherwise
they were all stitched into a single cube, and warn otherwise

These warnings are useful because they highlight problems, such as:

insufficient loading callback, which adds metadata for merge and concatenate to work
bugs in the trimming code, which removes duplicate rows and columns
#5768

The warnings are able to report on the cubes which need investigating. Without this, I found it was much harder to tell what's been stitched properly and what hasn't.
Also, for further consideration: I had to do something similar in the #5768 workaround, but with the merge signature. That's a simple named tuple so I didn't have to use the actual symbol, I just used the params I would have passed in. This is liable to break if its params change.

Hi @bblay. We are certainly aware that it is very difficult to understand compatibility issues. Whether we could act on your suggestion is super uncertain at the moment. We currently have at least 3 modules that inspect what is in Cubes, and make decisions about their compatibilities:

iris._concatenate
iris._merge
iris.common.resolve

This causes problems for developers and users alike when trying to understand Cube compatibility. iris.common.resolve was imagined to be a potential common solution (hence the namespace) - we could make ONE thing that is all singing, all dancing, and designed for user interaction - but recent experiments have shown that this will be more awkward than hoped ( https://github.com/SciTools/iris/issues/5394#issuecomment-1956460650 ).
While the merge and concatenate mechanisms are private we are free to think about various ideas for refactoring. If made public, with users starting to rely on certain objects existing and behaving in a certain way, we will loose a lot of that freedom.

Thanks for the response, @trexfeathers . Some good points.
I wasn't aware of the all-singing attempt. I'll ignore that here.
You're right about exposing the merge and concatenate structures. That could cause problems down the line. So the question then evolves to this:
"I had to write a function which groups cubes by their concatenation compatibility, but it uses an Iris internal structure. Could/should we add such a function to Iris?"
groups = iris.concatenation.group_by_compatibilty(cubes)
for group in groups:
    my_code.sanity_check_group(group)
    new_cubes = my_code.process(group)
    my_code.sanity_check_result(new_cubes)

That would seem to work equally well for merge.

Thanks, good to have this issue in the backlog

authors: bblay trexfeathers

repo: SciTools/iris