ChatBotArenaのローカルサーバーでの立ち上げメモ

Kan Hatakeyama

はじめに

ChatBotArenaという､大規模言語モデル(LLM)の評価システムをローカルで立ち上げるメモです｡
とりあえず､githubのチュートリアルに沿って作業するだけでOKでした｡

インストール・動作確認

基本的な機能はFastChatライブラリに入っているようなので､入れていきます｡


#環境構築
conda create -n arena python=3.10 -y
conda activate arena

#install

pip install "fschat[model_worker,webui]" #arena

#pip install torch==2.4.0 #torchを2.5から2.4にダウングレード｡自身の環境ではcuda関連のエラーが出たので｡


CUIでのチャットの動作確認

まずはcuiで動かしてみます｡

python3 -m fastchat.serve.cli --model-path weblab-GENIAC/Tanuki-8B-dpo-v1.0

無事に動きました｡ただ､何故か英語で返答が返ってきます｡
chat templateに問題があるような気がしますが､とりあえず動いたので良しとします｡

普通にchatする

3つのコマンドを異なるターミナルで打ち込むようです｡

コントロールサーバー

python3 -m fastchat.serve.controller

モデル

python3 -m fastchat.serve.model_worker --model-path weblab-GENIAC/Tanuki-8B-dpo-v1.0

フロントエンド(gradio)

python3 -m fastchat.serve.gradio_web_server


フロントエンドを立ち上げたときに表示されるurlにアクセスすると､guiが出てきました｡

小気味よくチャットできました｡


会話をすると､conv.json的なやつが出力されました｡
これを解析すれば､色々とフィードバックに使えそうです｡

{"tstamp": 1729210254.511, "type": "chat", "model": "Tanuki-8B-dpo-v1.0", "gen_params": {"temperature": 0.7, "top_p": 1.0, "max_new_tokens": 1024}, "start": 1729210253.703, "finish": 1729210254.511, "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u3053\u3093\u306b\u3061\u306f"], ["Assistant", "Hello! How can I assist you today?"]], "offset": 2, "conv_id": "59af5c228b9c44d5a281e5f09b901d9a", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1", "images": []}
{"tstamp": 1729210261.705, "type": "upvote", "model": "Tanuki-8B-dpo-v1.0", "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u3053\u3093\u306b\u3061\u306f"], ["Assistant", "Hello! How can I assist you today?"]], "offset": 2, "conv_id": "59af5c228b9c44d5a281e5f09b901d9a", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1"}
{"tstamp": 1729210277.258, "type": "chat", "model": "Tanuki-8B-dpo-v1.0", "gen_params": {"temperature": 0.7, "top_p": 1.0, "max_new_tokens": 1024}, "start": 1729210275.6175, "finish": 1729210277.258, "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u3053\u3093\u306b\u3061\u306f"], ["Assistant", "Hello! How can I assist you today?"], ["Human", "\u3052\u3093\u304d\u3067\u3059\u304b"], ["Assistant", "Yes, I'm here to help! How can I assist you with your day or any questions you might have?\n\nIf you're looking for a response or have a specific question, feel free to ask, and I'll do my best to provide you with a helpful and polite answer. If you're not sure what to ask, I can suggest some general topics or activities to discuss."]], "offset": 2, "conv_id": "59af5c228b9c44d5a281e5f09b901d9a", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1", "images": []}
{"tstamp": 1729210281.1777, "type": "flag", "model": "Tanuki-8B-dpo-v1.0", "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u3053\u3093\u306b\u3061\u306f"], ["Assistant", "Hello! How can I assist you today?"], ["Human", "\u3052\u3093\u304d\u3067\u3059\u304b"], ["Assistant", "Yes, I'm here to help! How can I assist you with your day or any questions you might have?\n\nIf you're looking for a response or have a specific question, feel free to ask, and I'll do my best to provide you with a helpful and polite answer. If you're not sure what to ask, I can suggest some general topics or activities to discuss."]], "offset": 2, "conv_id": "59af5c228b9c44d5a281e5f09b901d9a", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1"}
{"tstamp": 1729210292.3849, "type": "chat", "model": "Tanuki-8B-dpo-v1.0", "gen_params": {"temperature": 0.7, "top_p": 1.0, "max_new_tokens": 1024}, "start": 1729210291.6249, "finish": 1729210292.3849, "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u3053\u3093\u306b\u3061\u306f"], ["Assistant", "Hello! How can I assist you today?"], ["Human", "\u3052\u3093\u304d\u3067\u3059\u304b"], ["Assistant", "Yes, I'm here to help! How can I assist you with your day or any questions you might have?\n\nIf you're looking for a response or have a specific question, feel free to ask, and I'll do my best to provide you with a helpful and polite answer. If you're not sure what to ask, I can suggest some general topics or activities to discuss."], ["Human", "\u65e5\u672c\u8a9e\u3067\u8a71\u3057\u3066"], ["Assistant", "\u3082\u3061\u308d\u3093\u3067\u3059\u3002\u65e5\u672c\u8a9e\u3067\u8a71\u3057\u307e\u3057\u3087\u3046\u3002\u4f55\u304b\u7279\u5b9a\u306e\u8cea\u554f\u3084\u8a71\u984c\u304c\u3042\u308a\u307e\u3059\u304b\uff1f\u305d\u308c\u3068\u3082\u3001\u4f55\u304b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u304b\uff1f"]], "offset": 2, "conv_id": "59af5c228b9c44d5a281e5f09b901d9a", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1", "images": []}
{"tstamp": 1729210296.0631, "type": "chat", "model": "Tanuki-8B-dpo-v1.0", "gen_params": {"temperature": 0.7, "top_p": 1.0, "max_new_tokens": 1024}, "start": 1729210295.2655, "finish": 1729210296.0631, "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u3053\u3093\u306b\u3061\u306f"], ["Assistant", "Hello! How can I assist you today?"], ["Human", "\u3052\u3093\u304d\u3067\u3059\u304b"], ["Assistant", "Yes, I'm here to help! How can I assist you with your day or any questions you might have?\n\nIf you're looking for a response or have a specific question, feel free to ask, and I'll do my best to provide you with a helpful and polite answer. If you're not sure what to ask, I can suggest some general topics or activities to discuss."], ["Human", "\u65e5\u672c\u8a9e\u3067\u8a71\u3057\u3066"], ["Assistant", "\u3082\u3061\u308d\u3093\u3067\u3059\u3002\u65e5\u672c\u8a9e\u3067\u8a71\u3057\u307e\u3057\u3087\u3046\u3002\u4f55\u304b\u7279\u5b9a\u306e\u8cea\u554f\u3084\u8a71\u984c\u304c\u3042\u308a\u307e\u3059\u304b\uff1f\u305d\u308c\u3068\u3082\u3001\u4f55\u304b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u304b\uff1f"], ["Human", "\u306a\u3044\u3088"], ["Assistant", "\u4e86\u89e3\u3057\u307e\u3057\u305f\u3002\u305d\u308c\u3067\u306f\u3001\u4f55\u304b\u4ed6\u306b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308c\u3070\u6559\u3048\u3066\u304f\u3060\u3055\u3044\u3002\u4f8b\u3048\u3070\u3001\u65e5\u672c\u8a9e\u306e\u5b66\u7fd2\u65b9\u6cd5\u3084\u8da3\u5473\u306b\u3064\u3044\u3066\u306e\u60c5\u5831\u63d0\u4f9b\u306a\u3069\u3001\u4f55\u3067\u3082\u3069\u3046\u305e\u3002"]], "offset": 2, "conv_id": "59af5c228b9c44d5a281e5f09b901d9a", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1", "images": []}
{"tstamp": 1729210312.2319, "type": "upvote", "model": "Tanuki-8B-dpo-v1.0", "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u3053\u3093\u306b\u3061\u306f"], ["Assistant", "Hello! How can I assist you today?"], ["Human", "\u3052\u3093\u304d\u3067\u3059\u304b"], ["Assistant", "Yes, I'm here to help! How can I assist you with your day or any questions you might have?\n\nIf you're looking for a response or have a specific question, feel free to ask, and I'll do my best to provide you with a helpful and polite answer. If you're not sure what to ask, I can suggest some general topics or activities to discuss."], ["Human", "\u65e5\u672c\u8a9e\u3067\u8a71\u3057\u3066"], ["Assistant", "\u3082\u3061\u308d\u3093\u3067\u3059\u3002\u65e5\u672c\u8a9e\u3067\u8a71\u3057\u307e\u3057\u3087\u3046\u3002\u4f55\u304b\u7279\u5b9a\u306e\u8cea\u554f\u3084\u8a71\u984c\u304c\u3042\u308a\u307e\u3059\u304b\uff1f\u305d\u308c\u3068\u3082\u3001\u4f55\u304b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u304b\uff1f"], ["Human", "\u306a\u3044\u3088"], ["Assistant", "\u4e86\u89e3\u3057\u307e\u3057\u305f\u3002\u305d\u308c\u3067\u306f\u3001\u4f55\u304b\u4ed6\u306b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308c\u3070\u6559\u3048\u3066\u304f\u3060\u3055\u3044\u3002\u4f8b\u3048\u3070\u3001\u65e5\u672c\u8a9e\u306e\u5b66\u7fd2\u65b9\u6cd5\u3084\u8da3\u5473\u306b\u3064\u3044\u3066\u306e\u60c5\u5831\u63d0\u4f9b\u306a\u3069\u3001\u4f55\u3067\u3082\u3069\u3046\u305e\u3002"]], "offset": 2, "conv_id": "59af5c228b9c44d5a281e5f09b901d9a", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1"}
{"tstamp": 1729210420.2861, "type": "chat", "model": "Tanuki-8B-dpo-v1.0", "gen_params": {"temperature": 0.7, "top_p": 1.0, "max_new_tokens": 1024}, "start": 1729210419.8263, "finish": 1729210420.2861, "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u65e5\u672c\u8a9e\u3067\u306f\u306a\u3057\u3066\uff61 \u3053\u3093\u306b\u3061\u306f"], ["Assistant", "\u3053\u3093\u306b\u3061\u306f\uff01\u4f55\u304b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u304b\uff1f"]], "offset": 2, "conv_id": "5095b5260040465db282b36fd3ade6cb", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1", "images": []}
{"tstamp": 1729210435.7999, "type": "chat", "model": "Tanuki-8B-dpo-v1.0", "gen_params": {"temperature": 0.7, "top_p": 1.0, "max_new_tokens": 1024}, "start": 1729210433.7592, "finish": 1729210435.7999, "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u65e5\u672c\u8a9e\u3067\u306f\u306a\u3057\u3066\uff61 \u3053\u3093\u306b\u3061\u306f"], ["Assistant", "\u3053\u3093\u306b\u3061\u306f\uff01\u4f55\u304b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u304b\uff1f"], ["Human", "\u732b\u3068\u305f\u306c\u304d\u306f\u3069\u3061\u3089\u304c\u597d\u304d\u3067\u3059\u304b\u3044?"], ["Assistant", "\u305d\u308c\u306f\u9762\u767d\u3044\u8cea\u554f\u3067\u3059\u306d\uff01\u79c1\u306fAI\u306a\u306e\u3067\u500b\u4eba\u7684\u306a\u597d\u307f\u306f\u3042\u308a\u307e\u305b\u3093\u304c\u3001\u591a\u304f\u306e\u4eba\u3005\u304c\u305d\u308c\u305e\u308c\u7570\u306a\u308b\u9b45\u529b\u3092\u611f\u3058\u3066\u3044\u308b\u3088\u3046\u3067\u3059\u3002\u732b\u306f\u305d\u306e\u611b\u3089\u3057\u3044\u4ed5\u8349\u3084\u72ec\u7acb\u6027\u3001\u305d\u3057\u3066\u591a\u69d8\u306a\u54c1\u7a2e\u304c\u9b45\u529b\u3067\u3059\u3002\u4e00\u65b9\u3001\u305f\u306c\u304d\u306f\u305d\u306e\u72ec\u7279\u306a\u9854\u7acb\u3061\u3084\u91ce\u751f\u5473\u3001\u305d\u3057\u3066\u30e6\u30fc\u30e2\u30e9\u30b9\u306a\u884c\u52d5\u3067\u591a\u304f\u306e\u4eba\u3005\u306b\u611b\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u3069\u3061\u3089\u304c\u597d\u304d\u304b\u306f\u3001\u305d\u306e\u4eba\u306e\u500b\u6027\u3084\u597d\u307f\u306b\u3088\u308b\u3068\u601d\u3044\u307e\u3059\u3088\u3002"]], "offset": 2, "conv_id": "5095b5260040465db282b36fd3ade6cb", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1", "images": []}
{"tstamp": 1729210445.1205, "type": "chat", "model": "Tanuki-8B-dpo-v1.0", "gen_params": {"temperature": 0.7, "top_p": 1.0, "max_new_tokens": 1024}, "start": 1729210444.3474, "finish": 1729210445.1205, "state": {"template_name": "one_shot", "system_message": "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions.", "roles": ["Human", "Assistant"], "messages": [["Human", "Got any creative ideas for a 10 year old\u2019s birthday?"], ["Assistant", "Of course! Here are some creative ideas for a 10-year-old's birthday party:\n1. Treasure Hunt: Organize a treasure hunt in your backyard or nearby park. Create clues and riddles for the kids to solve, leading them to hidden treasures and surprises.\n2. Science Party: Plan a science-themed party where kids can engage in fun and interactive experiments. You can set up different stations with activities like making slime, erupting volcanoes, or creating simple chemical reactions.\n3. Outdoor Movie Night: Set up a backyard movie night with a projector and a large screen or white sheet. Create a cozy seating area with blankets and pillows, and serve popcorn and snacks while the kids enjoy a favorite movie under the stars.\n4. DIY Crafts Party: Arrange a craft party where kids can unleash their creativity. Provide a variety of craft supplies like beads, paints, and fabrics, and let them create their own unique masterpieces to take home as party favors.\n5. Sports Olympics: Host a mini Olympics event with various sports and games. Set up different stations for activities like sack races, relay races, basketball shooting, and obstacle courses. Give out medals or certificates to the participants.\n6. Cooking Party: Have a cooking-themed party where the kids can prepare their own mini pizzas, cupcakes, or cookies. Provide toppings, frosting, and decorating supplies, and let them get hands-on in the kitchen.\n7. Superhero Training Camp: Create a superhero-themed party where the kids can engage in fun training activities. Set up an obstacle course, have them design their own superhero capes or masks, and organize superhero-themed games and challenges.\n8. Outdoor Adventure: Plan an outdoor adventure party at a local park or nature reserve. Arrange activities like hiking, nature scavenger hunts, or a picnic with games. Encourage exploration and appreciation for the outdoors.\nRemember to tailor the activities to the birthday child's interests and preferences. Have a great celebration!"], ["Human", "\u65e5\u672c\u8a9e\u3067\u306f\u306a\u3057\u3066\uff61 \u3053\u3093\u306b\u3061\u306f"], ["Assistant", "\u3053\u3093\u306b\u3061\u306f\uff01\u4f55\u304b\u304a\u624b\u4f1d\u3044\u3067\u304d\u308b\u3053\u3068\u304c\u3042\u308a\u307e\u3059\u304b\uff1f"], ["Human", "\u732b\u3068\u305f\u306c\u304d\u306f\u3069\u3061\u3089\u304c\u597d\u304d\u3067\u3059\u304b\u3044?"], ["Assistant", "\u305d\u308c\u306f\u9762\u767d\u3044\u8cea\u554f\u3067\u3059\u306d\uff01\u79c1\u306fAI\u306a\u306e\u3067\u500b\u4eba\u7684\u306a\u597d\u307f\u306f\u3042\u308a\u307e\u305b\u3093\u304c\u3001\u591a\u304f\u306e\u4eba\u3005\u304c\u305d\u308c\u305e\u308c\u7570\u306a\u308b\u9b45\u529b\u3092\u611f\u3058\u3066\u3044\u308b\u3088\u3046\u3067\u3059\u3002\u732b\u306f\u305d\u306e\u611b\u3089\u3057\u3044\u4ed5\u8349\u3084\u72ec\u7acb\u6027\u3001\u305d\u3057\u3066\u591a\u69d8\u306a\u54c1\u7a2e\u304c\u9b45\u529b\u3067\u3059\u3002\u4e00\u65b9\u3001\u305f\u306c\u304d\u306f\u305d\u306e\u72ec\u7279\u306a\u9854\u7acb\u3061\u3084\u91ce\u751f\u5473\u3001\u305d\u3057\u3066\u30e6\u30fc\u30e2\u30e9\u30b9\u306a\u884c\u52d5\u3067\u591a\u304f\u306e\u4eba\u3005\u306b\u611b\u3055\u308c\u3066\u3044\u307e\u3059\u3002\u3069\u3061\u3089\u304c\u597d\u304d\u304b\u306f\u3001\u305d\u306e\u4eba\u306e\u500b\u6027\u3084\u597d\u307f\u306b\u3088\u308b\u3068\u601d\u3044\u307e\u3059\u3088\u3002"], ["Human", "\u3044\u3044\u3067\u3059\u306d\uff61"], ["Assistant", "\u305d\u308c\u306f\u826f\u304b\u3063\u305f\u3067\u3059\uff01\u4f55\u304b\u4ed6\u306b\u77e5\u308a\u305f\u3044\u3053\u3068\u3084\u8cea\u554f\u304c\u3042\u308c\u3070\u3001\u3069\u3046\u305e\u304a\u77e5\u3089\u305b\u304f\u3060\u3055\u3044\u3002\u3069\u3093\u306a\u3053\u3068\u3067\u3082\u304a\u624b\u4f1d\u3044\u3057\u307e\u3059\uff01"]], "offset": 2, "conv_id": "5095b5260040465db282b36fd3ade6cb", "model_name": "Tanuki-8B-dpo-v1.0"}, "ip": "127.0.0.1", "images": []}


Arenaにする

ChatbotArenaも簡単に作れることがわかりました｡

コントロールサーバー
先ほどと同じ

python3 -m fastchat.serve.controller

モデルA
先ほどと同じ

python3 -m fastchat.serve.model_worker --model-path weblab-GENIAC/Tanuki-8B-dpo-v1.0

モデルB
tanuki-8x8bを使ってみます｡ vramが足りなかったので､num-gpus 2としました｡
portを設定するのがポイントです｡

python3 -m fastchat.serve.model_worker --model-path weblab-GENIAC/Tanuki-8x8B-dpo-v1.0 --num-gpus 2 --port 21003 --worker http://localhost:21003

フロントエンド(gradio)
multi用のコマンドを打つだけでOKでした｡

python3 -m fastchat.serve.gradio_web_server_multi

うまく動きました｡

voteのlogも､conv.jsonに出力されていました｡

(追記)api系のモデルを追加する

ローカルではなくapiで提供されているモデルを使う場合｡
以下の解説がありました｡

api_endpoints.jsonのような､適当なjsonファイルを作ります｡

{
  "gpt-3.5-turbo": {
    "model_name": "gpt-3.5-turbo",
    "api_type": "openai",
    "api_base": "https://api.openai.com/v1",
    "api_key": "sk-******",
    "anony_only": false,
    "recommended_config": {
      "temperature": 0.7,
      "top_p": 1.0
    },
    "text-arena": true,
    "vision-arena": false
  }
}

その後､このファイルを読み込む設定で､コントロールサーバーを起動するだけです｡

python3 -m fastchat.serve.gradio_web_server_multi --share --register api_endpoints.json

高速の推論サーバーを立てる

fastchatで動かすローカルモデルは､恐らくtransformersライブラリ?で動いているような気がします｡このライブラリは､推論速度が微妙に遅かったり､量子化での推論が苦手だったりするので､他の選択肢を試してみます｡

vllm

高速と評判のvllmでサーバーを立てる例

python -m vllm.entrypoints.openai.api_server --model cyberagent/calm3-22b-chat \
--max-model-len 2048 --port 8000 \
--gpu-memory-utilization 0.9 --trust-remote-code \
--tensor-parallel-size 1 --api-key 12345

呼び出しは普通にjsonに追記でOKでした｡

    "calm3-22b-chat": {
        "model_name": "cyberagent/calm3-22b-chat",
        "api_type": "openai",
        "api_base": "http://localhost:8000/v1",
        "api_key": "12345",
        "anony_only": false,
        "recommended_config": {
            "temperature": 0.7,
            "top_p": 1.0
        },
        "text-arena": true,
        "vision-arena": false
    }

llama.cpp

ggufなどを使えます｡ollamaでも動くはずなんですが､うまくサーバーで呼び出せなかったので､llama.cppを試してみます｡

セットアップ

#llama cppのセットアップ
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make GGML_CUDA=1

推論サーバー

#適当なモデルのダウンロード
wget https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q8_0.gguf

#推論サーバーを立てる
./llama-server -m Llama-3.2-1B-Instruct-Q8_0.gguf

jsonは以下の通り｡api keyとかは適当で大丈夫そうです｡

    "test": {
        "model_name": "Llama-3.2-1B-Instruct-Q8_0.gguf",
        "api_type": "openai",
        "api_base": "http://localhost:8080/v1",
        "api_key": "12345",
        "anony_only": false,
        "recommended_config": {
            "temperature": 0.7,
            "top_p": 1.0
        },
        "text-arena": true,
        "vision-arena": false
    }


以上


いいなと思ったら応援しよう！