Add optional AVX512-FP16 arithmetic for the scalar quantizer.

[mulugetam]

PR #4025 introduced a new architecture mode, avx512_spr, which enables the use of features available since Intel® Sapphire Rapids. The Hamming Distance Optimization (PR #4020), based on this mode, is now used by OpenSearch to speed up the indexing and searching of binary vectors.

This PR adds support for AVX512-FP16 arithmetic for the Scalar Quantizer. It introduces a new Boolean flag, ENABLE_AVX512_FP16, which, when used together with the avx512_spr mode, explicitly enables avx512fp16 arithmetic.

Tests on an AWS r7i instance demonstrate up to a 1.6x speedup in execution time when using AVX512-FP16 compared to AVX512. The improvement comes from a reduction in path length.

---

-DFAISS_OPT_LEVEL=avx512:

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_distance --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                377313128 ns    377307894 ns           20 code_size=384
QT_4bit_uniform/iterations:20        375116351 ns    375113141 ns           20 code_size=384
QT_6bit/iterations:20                313387520 ns    313382880 ns           20 code_size=576
QT_8bit/iterations:20                256168739 ns    256166690 ns           20 code_size=768
QT_8bit_direct/iterations:20          86934297 ns     86933750 ns           20 code_size=768
QT_8bit_direct_signed/iterations:20  182414307 ns    182413034 ns           20 code_size=768
QT_8bit_uniform/iterations:20        199623822 ns    199621677 ns           20 code_size=768
QT_bf16/iterations:20                205998126 ns    205996720 ns           20 code_size=1.536k
QT_fp16/iterations:20                204291381 ns    204290326 ns           20 code_size=1.536k

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_accuracy --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.284829
QT_4bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.2845
QT_6bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=576 code_size_two=1.152M ndiff_for_idempotence=0 sql2_recons_error=0.0164574
QT_8bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=23.184k sql2_recons_error=1.32533m
QT_8bit_direct/iterations:20             0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.806
QT_8bit_direct_signed/iterations:20      0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.799
QT_8bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=983.725u
QT_bf16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=558.005u
QT_fp16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=8.71347u

---

-DFAISS_ENABLE_AVX512_FP16=ON -DFAISS_OPT_LEVEL=avx512_spr

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_distance --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                235309050 ns    235307645 ns           20 code_size=384
QT_4bit_uniform/iterations:20        232136257 ns    232133724 ns           20 code_size=384
QT_6bit/iterations:20                285525339 ns    285522795 ns           20 code_size=576
QT_8bit/iterations:20                194723922 ns    194722583 ns           20 code_size=768
QT_8bit_direct/iterations:20          92415395 ns     92415183 ns           20 code_size=768
QT_8bit_direct_signed/iterations:20  166470157 ns    166469954 ns           20 code_size=768
QT_8bit_uniform/iterations:20        182553814 ns    182547301 ns           20 code_size=768
QT_bf16/iterations:20                222188800 ns    222187274 ns           20 code_size=1.536k
QT_fp16/iterations:20                198899549 ns    198898266 ns           20 code_size=1.536k

$ numactl -C 2 ./build/perf_tests/bench_scalar_quantizer_accuracy --d=768 --n=2000 --iterations=20
----------------------------------------------------------------------------------------------
Benchmark                                    Time             CPU   Iterations UserCounters...
----------------------------------------------------------------------------------------------
QT_4bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.284803
QT_4bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=384 code_size_two=768k ndiff_for_idempotence=0 sql2_recons_error=0.284519
QT_6bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=576 code_size_two=1.152M ndiff_for_idempotence=0 sql2_recons_error=0.0164255
QT_8bit/iterations:20                    0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=27.191k sql2_recons_error=1.3546m
QT_8bit_direct/iterations:20             0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.806
QT_8bit_direct_signed/iterations:20      0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=255.799
QT_8bit_uniform/iterations:20            0.000 ns        0.000 ns            0 code_size=768 code_size_two=1.536M ndiff_for_idempotence=0 sql2_recons_error=1.0097m
QT_bf16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=558.005u
QT_fp16/iterations:20                    0.000 ns        0.000 ns            0 code_size=1.536k code_size_two=3.072M ndiff_for_idempotence=0 sql2_recons_error=8.71347u

[alexanderguzhva]

@mulugetam do I get it correct that this PR introduces an optional tradeoff between the accuracy and the speed?

[mulugetam]

@mulugetam do I get it correct that this PR introduces an optional tradeoff between the accuracy and the speed?

Yes.

[mdouze]

Where does the loss in accuracy come from? because computation is performed as fp16 * fp16 instead of fp16-> fp32 then fp32 * fp32 ?

[alexanderguzhva]

yes, pure fp16 FMAD operations

[mulugetam]

@alexanderguzhva I'm not sure why this PR fails some of the pytest tests/test_*.py. On my local machine, the current main branch also fails some of these tests. Any ideas?

[alexanderguzhva]

@mulugetam well, that's the most sad part in committing PRs, according to my experience. Will you be able to reproduce the problem if you try rebasing your PR on top of the head? Otherwise, I have no explanations: could be a different hardware on the CI machine, a compiler or maybe your code.

[mulugetam]

@alexanderguzhva Yes, it's reproducible. If I git clone https://github.com/facebookresearch/faiss.git and run pytest tests/test_*.py on my machine, some of the tests will fail.