1. That caulking looks like it was done by a monkey.
2. That caulk looks like an acrylic caulk. Acetone will dissolve all things acrylic, including acrylic caulk. So, remove as much as you can by mechanical means, then use acetone to clean up the rest.
3. I have 66 windows in my apartment block, and I won't use any caulk on their exteriors except Kop-R-Lastic. Open your yellow pages to Caulking Equipment & Supplies and phone around to find out who sells Kop-R-Lastic near you. The advantage of Kop-R-Lastic caulk is that it has a cohesive strength that's even higher than it's adhesive strength, which means it sticks to itself even better than it sticks to common building materials. So, if you ever want or need to remove it, you just get one end started, and it pulls off cleanly like a rubber rope. But, don't get the impression that it doesn't stick well. It sticks as well as is needed for a high quality caulk. It's just vastly easier to remove if and when you want/need to remove it. I started using Kop-R-Lastic every bit of 20 years ago, and I have yet to have any of it come off without my taking it off. All of the Window and Door companies here in Winnipeg use Kop-R-Lastic on their new installations. Kop-R-Lastic is paintable and when fully cured it has about the same elasticity as silicone.
I have never used the Geocel product recommended by CompleteW&D, but I would encourage him to try Kop-R-Lastic too.
PS: IT takes several years for Kop-R-Lastic to fully cure so that you can pull it off like a rubber rope, but until it does, it's no harder to remove than any other caulk.
PS2: Kop-R-Lastic is produced in the United States by the Henry Company under license from the Koppers Company of Australia.