On a 32 GB machine, -A3B at Q4 runs at about 15 . is being considered for work against a codebase.
Because is smaller, it should fit comfortably at Q8, and BF16 might even be possible. The practical question is whether a larger, more model or a smaller, less model gives the better mix of quality, speed, and memory use.
Key points
-A3B at Q4 reaches about 15 on 32 GB .
is small enough to fit comfortably at Q8 on the same setup.
BF16 may also be possible for because of its smaller size.
The main tradeoff is larger and more versus smaller and less .