个别情况下翻译会出现预期外的问题。

by nadalgu - opened Jul 23

Discussion

nadalgu

Jul 23

•

edited Jul 23

由于本地部署在Windows上因此没有使用vllm，使用了 transformers 运行该模型，末尾有完整的Python代码：

实际使用时大部分翻译都正常，但是个别中文翻译英文时会出现问题，主要是两种情况：
（我不确定使用vllm时是否能重现下列问题）

1、中译英后又自己做了一次英译中：
输入：
Translate the following Chinese sentence into English: 小黄鱼最大尺寸<en>
输出：
Translate the following Chinese sentence into English: 小黄鱼最大尺寸<en> The maximum size of the small yellow fish is <zh> 小黄鱼的最大尺寸是多少？

2、循环翻译（GPU算力跑满持续几十秒时间）：
输入：
未设置<en>
输出：

未设置<en> 未设置

Translation:  
"Not set"  
"Not configured"  
"Not set up"  
"Not configured"  
"Not set"  
"Not configured"  
"Not set up"  
"Not configured"  
"Not set"  
"Not configured"  
"Not set up"  
"Not configured"  

......

"Not set up"  
"Not configured"  
"Not set"  
"Not configured"  
"

import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from typing import Dict, List, Any

class SeedModel:
    def __init__(self):
        model_dir = os.path.join(
                                os.path.dirname(__file__), 
                                "Seed-X-PPO-7B"
        )
        if not os.path.exists(model_dir):
            raise ValueError(f"Local model not found at {model_dir}. Please download the model and place it.")
            

        self.tokenizer = AutoTokenizer.from_pretrained(model_dir)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_dir, 
            trust_remote_code=True,
            torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float16
        )
        
        if self.tokenizer.pad_token_id is None:
            self.tokenizer.pad_token_id = self.tokenizer.eos_token_id

        self.device = 'cuda' if torch.cuda.is_available() else 'cpu'
        self.model = self.model.to(self.device)

        if torch.cuda.is_available():
            torch.backends.cuda.enable_flash_sdp(True)
            torch.backends.cuda.enable_mem_efficient_sdp(True)

    def predict(self, inputs: Dict[str, Any]) -> str:
        text = inputs['text']

        if not isinstance(text, str):
            raise ValueError("Input must be a string")
        
        inputs = self.tokenizer(
            text, 
            return_tensors="pt", 
            padding=True,
            return_attention_mask=True
        ).to(self.device)

        outputs = self.model.generate(
            **inputs,
            max_new_tokens=512,
            num_beams=4,
            early_stopping=True,
            pad_token_id=self.tokenizer.pad_token_id,
            eos_token_id=self.tokenizer.eos_token_id,
        )

        translated = self.tokenizer.decode(
            outputs[0], 
            skip_special_tokens=True
        ).strip()
        
        return translated