Strip for LoRA modules #3331
Strip for LoRA modules #3331
Conversation
github-actions
bot
added
NNCF PT
tests/torch/ptq/test_fq_lora.py
Outdated
| ), | ||
| ids=["asym", "sym"], | ||
| ) | ||
| def test_fq_lora_tuning(mode, backup_mode, compression_kwargs, _seed): |
There was a problem hiding this comment.
Suggest extending this test by calling strip and exporting to OV and checking similarity for OV model (you can pre-compute similarity for "stripped to float" model or do it in the test)
nncf/torch/quantization/strip.py
Outdated
| result_dtype=original_dtype, | ||
| ) | ||
|
|
||
| elif isinstance(quantizer, SymmetricLoraQuantizer): |
There was a problem hiding this comment.
Ordinary FQ should be also supported here, since some layers can be selected to INT8 (first/last or by mixed precision), and they will be represented by ordinary FQ w/o LoRA.
You can check number of u8/u4 constants after export to OV.
nncf/torch/quantization/strip.py
Outdated
| original_shape = original_weight.shape | ||
| original_eps = torch.finfo(original_dtype).eps | ||
|
|
||
| # Quantize-dequantize using universal quantization formula |
There was a problem hiding this comment.
I'd reference to the markdown with this “universal”, otherwise it can be not clear what you mean here.
There was a problem hiding this comment.
It would be helpful to include a note explaining why the weights are not directly quantized. Please mention that this approach is necessary to prevent floating-point errors that can occur due to the different order of operations during quantization when using Torch for tuning and OpenVINO (OV) for inference.
nikita-malininn
force-pushed
the
nm/strip_lora
branch
from
00ba62e to
ba00566
Compare
github-actions
bot
removed
NNCF Common
Changes
Reason for changes
Related tickets
Tests
On top of #3322