maximuspowers/weight-autoencoder-mlp-v1

Weight-Space Autoencoder (TRANSFORMER)

This model is a weight-space autoencoder trained on neural network activation weights/signatures. It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).

Model Description

• Architecture: Transformer encoder-decoder
• Training Dataset: maximuspowers/muat-fourier-5
• Input Mode: signature
• Latent Dimension: 256

Tokenization

• Granularity: neuron
• Max Tokens: 64

Training Config

• Loss Functions: reconstruction, contrastive, functional
• Optimizer: adamw
• Learning Rate: 0.0001
• Batch Size: 32

Performance Metrics (Test Set)

• MSE: 0.125011
• MAE: 0.259796
• RMSE: 0.353570
• Cosine Similarity: 0.0348
• R² Score: -0.0097