metadata
tags:
- weight-space-learning
- neural-network-autoencoder
- autoencoder
- transformer
datasets:
- maximuspowers/muat-fourier-5
Weight-Space Autoencoder (TRANSFORMER)
This model is a weight-space autoencoder trained on neural network activation weights/signatures. It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).
Model Description
- Architecture: Transformer encoder-decoder
- Training Dataset: maximuspowers/muat-fourier-5
- Input Mode: signature
- Latent Dimension: 256
Tokenization
- Granularity: neuron
- Max Tokens: 64
Training Config
- Loss Functions: reconstruction, contrastive, functional
- Optimizer: adamw
- Learning Rate: 0.0001
- Batch Size: 32
Performance Metrics (Test Set)
- MSE: 0.125011
- MAE: 0.259796
- RMSE: 0.353570
- Cosine Similarity: 0.0348
- R² Score: -0.0097