Realistically, swimming is an occasion when you're just not realistically going to be able to effectively carry. There are GOING to be times when being armed isn't an option. Heck the law dictates that there are some times when you can't (ie, post offices). This is one where simple practicality is dictating those terms.
My advice: carry as normal on the way and upon arriving leave the gun in the car (or on the boat if you're taking one out). Not meant as an offense or jab, but if you're not comfortable jumping in the water and swimming unarmed, it's probably best that you simply avoid the activity altogether. There will be times/places when being present means being unarmed. It's up to each individual to decide whether or not to accept that risk or avoid the situation.