Mixtral-8x22Bをllama.cppで実行する

Hugging Faceのトレンドに上がっていた、Mixtral-8x22Bを試してみました。

Mixtral-8x22Bとは

The Mixtral-8x22B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts.

としか書かれていないので、謎が多いですね。

実行環境

実行環境としては、llama.cppを使います。
PCのスペック：

• CPU: E5-2680v4

• Memory: 64GB

• SSD: WD_BLACK_SN770

• GPU: GeForce RTX 4060Ti 16GB

• OS: Windows 10 + WSL2

最初はHDDにモデルファイルを置いていたのですが、実行に1日以上掛かってしまいました。SSDだと1時間掛かりません。、

llama.cppのビルド

最新のllama.cppをダウンロードしてビルドしてください。バイナリで出回っているものでは動かないようです。

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
make LLAMA_CUDA=1

自分の環境では、CUDAでオフロードできるので、CUDAを使う設定でビルドします。
ただ、実行中はあまり使っていないようです。

モデルをダウンロード

MaziyarPanahi/Mixtral-8x22B-v0.1-GGUF　から、2-bitのモデルをダウンロードする。

Mixtral-8x22B-v0.1.Q2_K-0000[12345]-of-00005.gguf

の５つのファイルをダウンロードする。

実行

llama.cpp/main -m Mixtral-8x22B-v0.1.Q2_K-00001-of-00005.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 1024 -e

結果

Building a website can be done in 10 simple steps:
Step 1: Define your website’s purpose and target audience
The first step in building a website is to define its purpose and target audience. This will help you determine the type of website you need to build, the features it should have, and the content it should include.
Step 2: Choose a domain name and hosting provider
Choose a domain name that reflects your brand and is easy to remember. Then, choose a hosting provider that offers the features and support you need.
Step 3: Install a content management system (CMS)
Install a content management system (CMS) such as WordPress, which makes it easy to create and manage your website’s content.
Step 4: Choose a website theme and customize it
Choose a website theme that reflects your brand and customize it to fit your needs.
Step 5: Create your website’s pages
Create the pages for your website, including the homepage, about page, contact page, and any other pages that are relevant to your business.
Step 6: Add content to your website
Add content to your website, including text, images, and videos. Make sure the content is optimized for search engines and is easy to read and understand.
Step 7: Optimize your website for search engines
Optimize your website for search engines by including relevant keywords in your content, meta tags, and URLs.
Step 8: Add features and functionality to your website
Add features and functionality to your website, such as contact forms, e-commerce capabilities, and social media integration.
Step 9: Test and debug your website
Test and debug your website to ensure that it is working properly and is free of errors.
Step 10: Launch your website
Once your website is ready, launch it and promote it to your target audience.

## The 10 steps to building a website

1. Choose a Domain Name: The first step in building a website is choosing a domain name. This is the address of your website, and it should be easy to remember and relevant to your business.

2. Choose a Web Host: After you have chosen a domain name, you need to choose a web host. This is the company that will host your website and provide you with the necessary tools to build it.

3. Choose a Platform: Next, you need to choose a platform for your website. This could be a content management system (CMS) such as WordPress, or a custom-built platform.

4. Design the Website: Once you have chosen a platform, you need to design the website. This includes choosing the right colors, fonts, and images to create a visually appealing website.

5. Add Content: After you have designed the website, you need to add content. This could be text, images, videos, or other forms of media.

6. Optimize for Search Engines: Once you have added content, you need to optimize the website for search engines. This includes adding keywords, meta tags, and other elements to make the website more visible to search engines.

7. Test the Website: After you have added content and optimized it for search engines, you need to test the website. This includes checking for errors, testing the website on different devices, and making sure it is compatible with all major web browsers.

8. Launch the Website: After you have tested the website, you need to launch it. This includes uploading the website to the web host, setting up the domain name, and making sure the website is accessible to all users.

9. Monitor the Website: Once you have launched the website, you need to monitor it. This includes checking for errors, testing the website on different devices, and making sure it is compatible with all major web browsers.

10. Maintain the Website: Finally, you need to maintain the website. This includes regularly checking for errors, testing the website on different devices, and making sure it is compatible with all major web browsers. [end of text]

トータルの実行時間は、1953秒でとなりました。

2bitsのモデルですが、まあ、普通に回答していますね。
ちょっと分からないのは、2重に回答していて、それぞれが全然対応取れてないということでしょうか。

以上、とりあえずのお試しです。規模が大きいので、4bitsや8bitsを試すのは難しそうです。