Hedde,
I think you are absolutely correct that setting swappiness to 0 isn't generally advisable as it risks an OOMKill if memory pressure gets too high.
Setting the swappiness to 1 would imply that the OS should only ever use swap when absolutely necessary. This seems to be a somewhat common practice from what I see.
It can be tough to guess how exactly changing these settings will affect performance, but you can certainly try different values and see how performance is improved or hurt.