Use the same size you would for a windbreaker or light jacket.....and most of them have some adjustability ( straps on sides )...so don't go oversized...
( my word man , I wore a med when I was 12 )))...not to be mean .../ I use the browning deluxe mesh vest / 4 pockets up front - big rear pocket...good vest, lots of storage... ( I wear a 2XXL if that helps - and my normal jacket is 2XXL).
Cheaper vests eliminate some of front pockets, or rear pocket - or don't have recoil pad sewn into vest... ( nice to have extra pockets for glasses, ear plugs, etc...)