I personally think it makes a difference in the end, maybe some folks don't notice it, but I do, I can feel it. I make stuff for myself though so time isn't a consideration... I also kinda like sanding, like the smell of it is lovely, even if my arms get a little tired and it takes a couple extra days - for me that's just extending the "happy wood working smell" 
I suppose if you're building for sales or quasi-mass production it could be worth skipping steps though.
I suppose if you're building for sales or quasi-mass production it could be worth skipping steps though.