Simply don't load the --mmproj projector file when launching the model — vision models work as text-only LLMs without it, and yes, they'll run faster since they skip multimodal processing overhead. Per the KoboldCpp wiki, --mmproj enables vision/audio, so omitting it disables those capabilities entirely.