In my opinion the best way to get max value is to put as little money into this place as possible to keep it liveable and safe, and stay in it as long as possible, while saving every penny you can.
You seem like a levelheaded guy. If it were me (knowing what I know now) I'd set a goal to sell the place once I have $X in the bank to put toward the new place ( or the new house on that property, if that's what you decide).
Before going the "bulldoze and rebuild" route you need to factor in your location and overall property. The old saying is money is made by having the worst house in the best neighborhood. If you spend $100k to build a new house on property surrounded by trailers in the same condition as yours is currently, you are taking a huge gamble.... It's one thing if you love the property and plan to say there, but very different from the terms of an investment.
Just with the little info you've given, I think it's a pretty easy choice. Do as little as needs to be done, live debt free as long as you can. Set a clear goal and save every penny you can to make your sacrafice worth while and to provide a light at the end of the tunnel!
Those who can, do...
Those who can't criticize on the internet