I would think silicone caulk should work just fine. I'd check the temp rating on the tube of caulking and see what it says. Usually it will be fine so long as there isn't snow on the ground, it will just take longer to cure.
Like theatretch said, silicone would probably be fine, but also make sure it's rated for outdoor use and has some UV resistance, otherwise sunlight can degrade it.
But for what it's worth, I've seen Tom Silva (This Old House) use butyl rubber caulk on almost all his outdoor caulking. It stays flexible even at low temp, and can be used to fill relatively large gaps.