RetNetを動かしてみる

まずはこちらのブログの再現を試みる.
環境はUbuntu, GPUは80GB VRAMのNVIDIA A100x4台(全ては使ってない)

学習（18時間程度）

学習データはfairseqのwikiText-103. 

2023-10-02 05:22:01 | INFO | fairseq_cli.train | done training in 65489.0 seconds
End Time: 1696224124.1143923
Training time in seconds 65507.27825713158

評価（5分程度）

2023-10-06 06:09:46 | INFO | fairseq_cli.generate | Translated 480 sentences (37,031 tokens) in 314.2s (1.53 sentences/s, 117.87 tokens/s)
Generate test with beam=1: BLEU4 = 0.32, 98.5/97.1/96.2/95.4 (BP=0.003, ratio=0.149, syslen=36553, reflen=245553)

同様のスコアが出てきた

推論（8秒）

設定：beam=2, sampling=False, temperature=0.1, no_repeat_ngram_size=2

Barack Obama was elected to the United States Senate in 2008 . Obama 's first term as chairman of the Senate Finance Committee was in 2009 . He was the first Republican to hold the position since the Obama administration .

Heart attack is a recurring theme in the series . The episode was written by series creator Chris Carter and directed by Kim Manners . It features the first appearance of recurring character Manners , who plays the character of John Doggett . Manners said of the episode , " I think I 'm really excited to be in this episode . I was like , ' Oh , I want to do this . ' " Manners later said that he was " very happy " with the ending , and that the final scene was a " great " moment .

Tokyo Metro =

応答が良くない…

課題

日本語データを大量に準備してfairseqで学習する方法がわからない、、、