Yup, the problem is that the poly will crack and fail eventually. It's not formulated to handle the task you have chosen it for. Go to your local paint store, tell them you want a clear masonry sealer. They come in different sheen levels from flat, semi-gloss, and gloss. I would recommend just the flat unless you want shiny brick. They should set you up with an appropriate product for your application.