| tags: | |
| - weight-space-learning | |
| - neural-network-autoencoder | |
| - autoencoder | |
| - transformer | |
| datasets: | |
| - maximuspowers/muat-fourier-5 | |
| # Weight-Space Autoencoder (TRANSFORMER) | |
| This model is a weight-space autoencoder trained on neural network activation weights/signatures. | |
| It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes). | |
| ## Model Description | |
| - **Architecture**: Transformer encoder-decoder | |
| - **Training Dataset**: maximuspowers/muat-fourier-5 | |
| - **Input Mode**: signature | |
| - **Latent Dimension**: 256 | |
| ## Tokenization | |
| - **Granularity**: neuron | |
| - **Max Tokens**: 64 | |
| ## Training Config | |
| - **Loss Functions**: reconstruction, functional | |
| - **Optimizer**: adamw | |
| - **Learning Rate**: 0.0001 | |
| - **Batch Size**: 32 | |
| ## Performance Metrics (Test Set) | |
| - **MSE**: 0.122959 | |
| - **MAE**: 0.256139 | |
| - **RMSE**: 0.350655 | |
| - **Cosine Similarity**: 0.0575 | |
| - **R² Score**: 0.0069 | |