🙏 How to Convert a FLUX-Dev Checkpoint to an NF4 Model? #1224
-
|
Hi everyone, I've recently trained a model using the FLUX-Dev checkpoint. A lot of users in the comments are requesting an NF4 version, as it would allow them to generate images on GPUs with lower VRAM. I'm looking for a step-by-step guide on how to convert my FLUX-Dev checkpoint into an NF4 model. Specifically, I would like to know:
I'm trying to make this model more accessible for users with less powerful GPUs, so any help would be greatly appreciated! Thank you! |
Beta Was this translation helpful? Give feedback.
Replies: 4 comments 16 replies
-
|
I would say look more into GGUF instead. It produces better quality than NF4 and runs just fine on a 12GB card. NF4 is kind of like the craiyon version of Flux. And while it can produce some pretty good looking images, the precision issues do crop up here and there and the keeper ratio drops drastically. Besides, NF4 is actually built into the UI, you can run any model at NF4 precision on the fly. There's no convert and save tool for it though, but again, GGUF Q4 is probably the better thing to convert it into, because it comes out to about the same download size as an SDXL checkpoint. |
Beta Was this translation helpful? Give feedback.
-
|
But really, the GGUF was a little too slow for my 3070TI |
Beta Was this translation helpful? Give feedback.
-
|
GGUF are too slow right now |
Beta Was this translation helpful? Give feedback.
-
|
I will probably give some convert codes later ... Also people need to notice that GGUF is a pure compression tech, which means it is smaller but also slower because it has extra steps to decompress tensors and computation is still pytorch. (unless someone is crazy enough to port llama.cpp compilers BNB (NF4) is computational acceleration library to make things faster by replacing pytorch ops with native low-bit cuda kernels, so that computation is faster. NF4 and Q4_0 should be very similar, with the difference that Q4_0 has smaller chunk size and NF4 has more gaussian-distributed quants. I do not recommend to trust comparisons of one or two images. And, I also want to have smaller chunk size in NF4 but it seems that bnb hard coded some thread numbers and changing that is non trivial. However Q4_1 and Q4_K are technically granted to be more precise than NF4, but with even more computation overheads – and such overheads may be more costly than simply moving higher precision weight from CPU to GPU. If that happens then the quant lose the point. |
Beta Was this translation helpful? Give feedback.
-
|
Very much agree |
Beta Was this translation helpful? Give feedback.
-
|
I get that NF4 can produce quite good images, but that was why I said the keeper rate drops. I like the idea of NF4, but when NF4 falls apart, it really falls apart. Basically when a concept or seed bumps into things, it's noticeable, and so the results turn into more of a minefield. In my experience with a 4070ti, GGUF takes about 25% longer. But if something generates faster but has a lower keeper rate, you'll be generating more and spending more time reworking prompts and settings, which can more than negate the speed improvements, as the per image generation time may be less, but the overall time spent generating to find a keeper can be more. This might be anecdotal, but I think one area where I noticed issues the most was with LORAs on NF4. Another anecdotal with NF4 is that I found I had to use twice as many steps to help get somewhat around the minefield of bad generations. Of course NF4 is still a great step and has a place, and maybe even NF4 + GGUF can make sense in some spaces too. I could see scenarios where people want to convert their entire libraries of models to reduce space usage. One thing to note is that GGUF Q8 does give virtually the same results as the main Dev model while being half the size, so a GGUF convertor built-in alongside NF4 convertor would actually be pretty useful, as that could mean 1GB SD 1.5 models and 3GB SDXL models that give results virtually indistinguishable from the original. Throw in NF4 + GGUF and you could have some truly tiny model sizes, and since there is less to decompress, the speed differences should also be reduced. |
Beta Was this translation helpful? Give feedback.
-
|
Do you use Q4_0 with your 4070ti? Thats what I am using right now with the same GPU, I am still not sure if its better than NF4 |
Beta Was this translation helpful? Give feedback.
-
|
I do use Q4 and NF4. NF4 does use a little less memory, but I don't like it as a 20 step model unless you're prompting something very simple. If I use it, I do at least 40-80 steps, and lower the CFG down to 2. Much better than generating at 20 steps, way more keepers. Especially when it comes to text, I find 20 steps just isn't enough for good adherence, 40 steps bumps that up a lot. People often forget that CFG and Steps are directly related, so if you use more steps, you generally use less CFG. CFG has always been primarily a way to get more adherence at lower steps. That's not exactly how it works, but that's functionally how it works. For example, you can get better results with even 1-2 CFG and 80 steps than you might get at 3.5 CFG and 20 steps, but obviously one takes longer than the other. The Beta scheduler is also generally better than Simple. I have read that there's some issues with using T5 and Clip together in prompting, as one wants natural language while the other wants just keywords, so that could be hurting NF4 and GGUF models too. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
can save like this now:
red marked areas can be changed. Will always save without double quant (the method for nf4-v2) - double quant will be removed soon because of quality and performance problems |
Beta Was this translation helpful? Give feedback.
-
|
thanks for sharing bnb-NF4 conversion , is the output V2 ? further more is there an option to lower memory usage i succeed with creating checkpoint with t5xxfp8 but failed with t5xxfp16 ,,, out of memory |
Beta Was this translation helpful? Give feedback.
-
|
bro when i try to quantize it to nf4 it adds all data in my ram and after sometime my ram gets out of memory and it do not use gpu at all and also i am on linux , can you help me |
Beta Was this translation helpful? Give feedback.
-
Hmm, I'm lacking the "Diffusion in low bits" option to save as nf4. All I see is this: Any advice is appreciated :) |
Beta Was this translation helpful? Give feedback.
-
Should be above the screenshot, make sure to change mode to flux |
Beta Was this translation helpful? Give feedback.
-
Thank you! I was looking on the bar under the checkpoint dropdown, but for me due to monitor / resolution size, it was on the first line, to the right of it.
|
Beta Was this translation helpful? Give feedback.
-
|
I have converted my dev safetensor file including the vae, clip and t5 fp8 to bnb-nf4(FP16 Lora) however, I think there's a memory leak bug. During inference, the RAM usage will jump to 26 GB RAM, which is more than FP8 and on par with FP16. When using it in Comfy with the BnB node I'm even getting and OOM which I never got with the FP16 and the FP8 one, so it's not only forge. The file size is just 12 GB which appears to be correct. Why does it need 26 GB RAM then? |
Beta Was this translation helpful? Give feedback.
-
|
i guess while it is processing clip |
Beta Was this translation helpful? Give feedback.
I will probably give some convert codes later ...
Also people need to notice that GGUF is a pure compression tech, which means it is smaller but also slower because it has extra steps to decompress tensors and computation is still pytorch. (unless someone is crazy enough to port llama.cpp compilers
BNB (NF4) is computational acceleration library to make things faster by replacing pytorch ops with native low-bit cuda kernels, so that computation is faster.
NF4 and Q4_0 should be very similar, with the difference that Q4_0 has smaller chunk size and NF4 has more gaussian-distributed quants. I do not recommend to trust comparisons of one or two images. And, I also want to have smaller chunk …