Shirwin Williams Primer Takes A Long Time To Dry And Peeling

A couple years ago I painted some of the window frames on my house. I scraped off all the old loose paint taking much of it down to bare wood. Then I sanded and primed with Sherwin Williams exterior oil based primer. The primer took an abnormally long time to try, a few days before it was totally dry to the touch. I gave it about two weeks before painting with Sherwin Williams exterior paint, a white Semi-Gloss. Now half of one of those window frames has the paint coming off in sheets. Everything else is fine, just that one half a frame has the paint peeling badly.
Should I do anything special or just scrape, sand, prime and paint?