Inserts are normally not caulked around... Maybe just a little inuslation that acts as a somewhat gasket between the insert flange and the brick...
Open the insert doors and make sure all dampers are open [some inserts have dampers independent of the fireplace] for five minutes or so before starting a fire to help warm the flue and create an updraft.. Light a piece of paper and see if the smoke goes up....
Be wary of any running ceiling fans or heat return fans near the fireplace that may draw smoke back into the house.
And most important, have the flue checked to see if it is blocked/needs cleaning. If you dont know when it was cleaned last, get it cleaned before using it.