You'd probably have to use something other than wood glue, unless the walls are some kind of wood. A construction adhesive labeled for the materials you're gluing together, but I don't know if there's one for wood to drywall - sorry just assuming the walls are drywall.
One downside I can think of is if you or anybody else ever wanted to remove them they'd have a mess of broken drywall on their hands that would have to be patched up. For small trim jobs, depending upon the profile of the trim you've selected, a hammer and a few finishing nails have been known to work.
News Views Tips Commentary at the Construction Informer