If you had a problem with paint adhesion, I expect it would manifest itself by the paint peeling off fairly soon, within a year or two. However, if there was no problem with the paint adhering, then peeling of the paint would likely be due to rain water getting into the brickwork and causing the bricks to get wet. The evaporation of water from those bricks would push the paint right off the brick's surface, causing the paint to crack and peel.
The other thing that occurs to me is that it's a BAD idea to paint a chimney. The reason why is that it is perfectly normal for the top portions of chimneys to need to be rebuilt periodically. It's not often, say about every 60 years or so, but it is routine maintenance for a chimney. The reason why is that flue gas contains CO2 and H2O vapour, and some of that H2O vapour will condense inside the cold chimney before the chimney warms up from the flue gas.
When liquid water forms in a CO2 atmosphere, there will be a lot of CO2 dissolved in the water. Some of the dissolved CO2 molecules will actually react with the H2O molecules in the water to form something called "carbonic acid", CH2O3, which makes the condensate that forms from the flue gas quite corrosive. It's also why carbonated soft drinks like Coke and Orange Crush have an acidic "bite" to them that dissappears when the carbon bubbles out and the soft drink goes flat.
That corrosive condensate forms on the entire inside of the cold chimney but as the chimney warms up, the condensate re-evaporates and the lower portion of the chimney remains dry. It's the top part of the chimney, the part that sticks up above the roof that is cooled by the wind that remains cool even while the furnace is operating. Condensate that forms in the upper portion of the chimney doesn't re-evaporate, but is absorbed into the masonary. It eats away at the inside of the chimney and deteriorates the bricks and mortar joints from the inside.
Chimney inspectors will remove the "clean out" immediately below where the flue duct from your furnace or boiler enters the chimney. This is where you will find sand, which is what's left of the brick mortar after the corrosive condensate attacks the brick mortar. The more sand you find, and the more pieces of mortar, the worse the condition of the interior of the chimney. Most of that sand and mortar will be from the portion of the chimney above the roof line.
Generally it's only the mortar that deteriorates. The bricks themselves don't contain any lime or limestone, so they stand up well to the corrosive condensate. Eventually, though, a bricklayer needs to take apart the chimney above the roof line and relay the bricks in new mortar.
If you paint your chimney, then you're going to make it harder to assess the condtion of your chimney. They can still tell by the amount of sand and mortar in the clean out what to expect, but it's still advantageous to a brick layer to inspect the condition of the bricks and mortar from the outside.
So, in my view, it's better not to touch that chimney with paint at all.