I would. Oil can produce CO Carbon monoxide just as easily and deadly as a gas furnace. I have seen the metal on those old units turn a red/purple color from constant heating and cooling and crystallize and will eventually break from metal fatigue. Then you can get the CO poisoning. Impossible to predict when it will break but it can happen in the middle of the night or on the coldest day of the year and that is not a good time to be furnace shopping.
"Cut it twice and it is still too short".