| tags: | |
| - weight-space-learning | |
| - neural-network-autoencoder | |
| - autoencoder | |
| - transformer | |
| datasets: | |
| - maximuspowers/muat-fourier-5 | |
| # Weight-Space Autoencoder (TRANSFORMER) | |
| This model is a weight-space autoencoder trained on neural network activation weights/signatures. | |
| It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes). | |
| ## Model Description | |
| - **Architecture**: Transformer encoder-decoder | |
| - **Training Dataset**: maximuspowers/muat-fourier-5 | |
| - **Input Mode**: signature | |
| - **Latent Dimension**: 128 | |
| ## Tokenization | |
| - **Chunk Size**: 1 weight values per token | |
| - **Max Tokens**: 64 | |
| - **Metadata**: True | |
| ## Training Config | |
| - **Loss Function**: contrastive | |
| - **Optimizer**: adamw | |
| - **Learning Rate**: 0.0001 | |
| - **Batch Size**: 32 | |
| ## Performance Metrics (Test Set) | |
| - **MSE**: 0.088750 | |
| - **MAE**: 0.187620 | |
| - **RMSE**: 0.297909 | |
| - **Cosine Similarity**: 0.9603 | |
| - **R² Score**: 0.9855 | |