---
tags:
- weight-space-learning
- neural-network-autoencoder
- autoencoder
- transformer
datasets:
- maximuspowers/muat-fourier-5
---

# Weight-Space Autoencoder (TRANSFORMER)

This model is a weight-space autoencoder trained on neural network activation weights/signatures.
It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).

## Model Description

- **Architecture**: Transformer encoder-decoder
- **Training Dataset**: maximuspowers/muat-fourier-5
- **Input Mode**: signature
- **Latent Dimension**: 256

## Tokenization

- **Granularity**: neuron
- **Max Tokens**: 64

## Training Config

- **Loss Functions**: reconstruction, functional
- **Optimizer**: adamw
- **Learning Rate**: 0.0001
- **Batch Size**: 32

## Performance Metrics (Test Set)

- **MSE**: 0.122959
- **MAE**: 0.256139
- **RMSE**: 0.350655
- **Cosine Similarity**: 0.0575
- **R² Score**: 0.0069