AWQ
Подобно GPTQ, AWQ оптимизирован для инференса на GPU. Он основан на том факте, что примерно 1% весов вносят значительный вклад в точность модели, и поэтому к ним следует относиться деликатно, используя набор данных для анализа распределений активаций во время инференса и выявления этих важных и критических весов.
Квантизация с помощью AutoAWQ
Давайте проведём короткую демонстрацию и квантуем Mistral 7B!
Сначала установим autoawq. Это позволит нам легко квантовать и выполнять инференс на AWQ моделях! AutoAWQ также по умолчанию предоставляет набор данных pile-val, который будет использоваться для процесса квантизации!
!pip install autoawqCollecting autoawq
Downloading autoawq-0.2.6-cp310-cp310-manylinux2014_x86_64.whl.metadata (18 kB)
Requirement already satisfied: torch==2.3.1 in /usr/local/lib/python3.10/dist-packages (from autoawq) (2.3.1+cu121)
Requirement already satisfied: transformers>=4.35.0 in /usr/local/lib/python3.10/dist-packages (from autoawq) (4.42.4)
Requirement already satisfied: tokenizers>=0.12.1 in /usr/local/lib/python3.10/dist-packages (from autoawq) (0.19.1)
Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from autoawq) (4.12.2)
Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (from autoawq) (0.32.1)
Collecting datasets (from autoawq)
Downloading datasets-2.21.0-py3-none-any.whl.metadata (21 kB)
Collecting zstandard (from autoawq)
Downloading zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting autoawq-kernels (from autoawq)
Downloading autoawq_kernels-0.0.7-cp310-cp310-manylinux2014_x86_64.whl.metadata (2.0 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->autoawq) (3.15.4)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->autoawq) (1.13.2)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->autoawq) (3.3)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->autoawq) (3.1.4)
Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->autoawq) (2024.6.1)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.3.1->autoawq)
Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.3.1->autoawq)
Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.3.1->autoawq)
Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.3.1->autoawq)
Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.3.1->autoawq)
Using cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.3.1->autoawq)
Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.3.1->autoawq)
Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.3.1->autoawq)
Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.3.1->autoawq)
Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu12==2.20.5 (from torch==2.3.1->autoawq)
Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.3.1->autoawq)
Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Requirement already satisfied: triton==2.3.1 in /usr/local/lib/python3.10/dist-packages (from torch==2.3.1->autoawq) (2.3.1)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.3.1->autoawq)
Using cached nvidia_nvjitlink_cu12-12.6.20-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /usr/local/lib/python3.10/dist-packages (from tokenizers>=0.12.1->autoawq) (0.23.5)
Requirement already satisfied: numpy<2.0,>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq) (24.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq) (6.0.2)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq) (2024.5.15)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq) (2.32.3)
Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq) (0.4.4)
Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.35.0->autoawq) (4.66.5)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate->autoawq) (5.9.5)
Collecting pyarrow>=15.0.0 (from datasets->autoawq)
Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.3 kB)
Collecting dill<0.3.9,>=0.3.0 (from datasets->autoawq)
Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->autoawq) (2.1.4)
Collecting xxhash (from datasets->autoawq)
Downloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting multiprocess (from datasets->autoawq)
Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->autoawq) (3.10.3)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq) (2.3.5)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq) (24.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->autoawq) (4.0.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq) (2.0.7)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.35.0->autoawq) (2024.7.4)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch==2.3.1->autoawq) (2.1.5)
Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->autoawq) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->autoawq) (2024.1)
Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->autoawq) (2024.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy->torch==2.3.1->autoawq) (1.3.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets->autoawq) (1.16.0)
Downloading autoawq-0.2.6-cp310-cp310-manylinux2014_x86_64.whl (94 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m94.9/94.9 kB [0m [31m5.7 MB/s [0m eta [36m0:00:00 [0m
[?25hUsing cached nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
Using cached nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
Using cached nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
Using cached nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
Using cached nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
Using cached nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
Using cached nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
Using cached nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
Using cached nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
Using cached nvidia_nccl_cu12-2.20.5-py3-none-manylinux2014_x86_64.whl (176.2 MB)
Using cached nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
Downloading autoawq_kernels-0.0.7-cp310-cp310-manylinux2014_x86_64.whl (33.6 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m33.6/33.6 MB [0m [31m42.6 MB/s [0m eta [36m0:00:00 [0m
[?25hDownloading datasets-2.21.0-py3-none-any.whl (527 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m527.3/527.3 kB [0m [31m38.1 MB/s [0m eta [36m0:00:00 [0m
[?25hDownloading zstandard-0.23.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m5.4/5.4 MB [0m [31m101.9 MB/s [0m eta [36m0:00:00 [0m
[?25hDownloading dill-0.3.8-py3-none-any.whl (116 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m116.3/116.3 kB [0m [31m11.7 MB/s [0m eta [36m0:00:00 [0m
[?25hDownloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m39.9/39.9 MB [0m [31m50.7 MB/s [0m eta [36m0:00:00 [0m
[?25hDownloading multiprocess-0.70.16-py310-none-any.whl (134 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m134.8/134.8 kB [0m [31m10.6 MB/s [0m eta [36m0:00:00 [0m
[?25hDownloading xxhash-3.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [0m [32m194.1/194.1 kB [0m [31m14.9 MB/s [0m eta [36m0:00:00 [0m
[?25hUsing cached nvidia_nvjitlink_cu12-12.6.20-py3-none-manylinux2014_x86_64.whl (19.7 MB)
Installing collected packages: zstandard, xxhash, pyarrow, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, dill, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, nvidia-cusolver-cu12, datasets, autoawq-kernels, autoawq
Attempting uninstall: pyarrow
Found existing installation: pyarrow 14.0.2
Uninstalling pyarrow-14.0.2:
Successfully uninstalled pyarrow-14.0.2
[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf-cu12 24.4.1 requires pyarrow<15.0.0a0,>=14.0.1, but you have pyarrow 17.0.0 which is incompatible.
ibis-framework 8.0.0 requires pyarrow<16,>=2, but you have pyarrow 17.0.0 which is incompatible. [0m [31m
[0mSuccessfully installed autoawq-0.2.6 autoawq-kernels-0.0.7 datasets-2.21.0 dill-0.3.8 multiprocess-0.70.16 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.20.5 nvidia-nvjitlink-cu12-12.6.20 nvidia-nvtx-cu12-12.1.105 pyarrow-17.0.0 xxhash-3.5.0 zstandard-0.23.0Как только мы закончим, мы можем загрузить модель, которую хотим квантовать. Сначала войдём в систему с токеном для чтения, чтобы получить доступ к моделям.
Примечание: Вам необходимо сначала принять условия в репозитории.
from huggingface_hub import login
login("read_token")The token has not been saved to the git credentials helper. Pass `add_to_git_credential=True` in this function directly or `--add-to-git-credential` if using via `huggingface-cli` if you want to set the git credential as well.
Token is valid (permission: read).
Your token has been saved to /root/.cache/huggingface/token
Login successfulТеперь всё готово, так что мы можем загрузить модель и квантовать её! Здесь мы квантуем модель до 4-бит!
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
pretrained_model_dir = "mistralai/Mistral-7B-Instruct-v0.3"
quantized_model_dir = "mistral_awq_quant"
model = AutoAWQForCausalLM.from_pretrained(
pretrained_model_dir, **{"low_cpu_mem_usage": True, "use_cache": False}
)
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, trust_remote_code=True)
# квантуем модель
quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }
model.quantize(tokenizer, quant_config=quant_config)/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning:
The secret `HF_TOKEN` does not exist in your Colab secrets.
To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.
You will be able to reuse this secret in all of your notebooks.
Please note that authentication is recommended but still optional to access public models or datasets.
warnings.warn(config.json: 0%| | 0.00/601 [00:00<?, ?B/s]Fetching 14 files: 0%| | 0/14 [00:00<?, ?it/s]model-00003-of-00003.safetensors: 0%| | 0.00/4.55G [00:00<?, ?B/s]model-00002-of-00003.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s].gitattributes: 0%| | 0.00/1.52k [00:00<?, ?B/s]README.md: 0%| | 0.00/7.82k [00:00<?, ?B/s]generation_config.json: 0%| | 0.00/116 [00:00<?, ?B/s]model-00001-of-00003.safetensors: 0%| | 0.00/4.95G [00:00<?, ?B/s]model.safetensors.index.json: 0%| | 0.00/23.9k [00:00<?, ?B/s]params.json: 0%| | 0.00/202 [00:00<?, ?B/s]tokenizer.model: 0%| | 0.00/587k [00:00<?, ?B/s]special_tokens_map.json: 0%| | 0.00/414 [00:00<?, ?B/s]tokenizer.json: 0%| | 0.00/1.96M [00:00<?, ?B/s]tokenizer.model.v3: 0%| | 0.00/587k [00:00<?, ?B/s]tokenizer_config.json: 0%| | 0.00/141k [00:00<?, ?B/s]Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s]Downloading readme: 0%| | 0.00/167 [00:00<?, ?B/s]Repo card metadata block was not found. Setting CardData to empty.
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.Downloading data: 0%| | 0.00/471M [00:00<?, ?B/s]Generating validation split: 0%| | 0/214670 [00:00<?, ? examples/s]AWQ: 100%|██████████| 32/32 [17:47<00:00, 33.37s/it]Теперь, когда модель квантована, мы можем сохранить её, чтобы поделиться ею или загрузить позже! Поскольку квантизация с помощью AWQ занимает некоторое время и требует ресурсов, рекомендуется всегда сохранять их.
model.save_quantized(quantized_model_dir)
tokenizer.save_pretrained(quantized_model_dir)Note that `shard_checkpoint` is deprecated and will be removed in v4.44. We recommend you using split_torch_state_dict_into_shards from huggingface_hub library('mistral_awq_quant/tokenizer_config.json',
'mistral_awq_quant/special_tokens_map.json',
'mistral_awq_quant/tokenizer.model',
'mistral_awq_quant/added_tokens.json',
'mistral_awq_quant/tokenizer.json')Модель квантована и сохранена с 4-битной точностью AWQ!
Вы также можете загрузить её для инференса, используя autoawq следующим образом:
model = AutoAWQForCausalLM.from_quantized(quantized_model_dir, device="cuda:0") # загружает квантованную модель на первый GPU
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir)
conversation = [{"role": "user", "content": "How are you today?"}]
prompt = tokenizer.apply_chat_template(
conversation,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt")
inputs.to("cuda:0") # загружает тензоры на первый GPU
outputs = model.generate(**inputs, max_new_tokens=32)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)Replacing layers...: 100%|██████████| 32/32 [00:07<00:00, 4.23it/s]
Fusing layers...: 100%|██████████| 32/32 [00:00<00:00, 130.00it/s]
Setting `pad_token_id` to `eos_token_id`:2 for open-end generation.How are you today? I'm an AI and don't have feelings, but I'm here and ready to help you with your questions! How can I assist you today