I left mine out one weekend to go camping with my younger sisters highschool class, it rained on friday night, and was below freezing until monday, the bag was literaly frozen solid, one hit and I decided I didn't want to train on a piece of cement. So I took it inside (and weighed it, 28 lbs heavier when frozen), opened it up, took all of the stuffing out in chunks the size of my head, and spread them on a piece of carpet in front of the wood stove. Over the day I picked up ice cube sized pieces of ice an tossed them, and moved the stuffing around, at the end of the day it was dry and fluffy, and even smelt nice, and I put it back in.
bottom line, you will be fine.