Obviously not that kind of percentage of whole job. I would call for $1000 (I pulled this out of thin air, may depend on your area) and if contractor reimburses that amount, accept it and try negotiate with another carpenter. It will take time. If you supply/have one already, a replacement filler, less time. A contractor will be thinking in terms of everything, including time spent to get extra filler. It will hurt to pay for all that when, technically, mistake wasn't his but still have to maintain some relationship with that carpenter.
You spent your good money but contractor has his own life to think about. It's not necessary that you can't agree.