I'm pretty confident in my ability. I've done a fair amount of work in the past, but I'm hoping that it won't go deep enough to have to call a pro
Heheh.. good to know. Without knowing your level of skill it's hard to know if you mean 2-wire [+G] or just 2 wires.

Wanted to make sure we're on the same page. (BTW - Like my tag says, I'm just a DIY'r, not an electrician, so some of the other guys may have better advice..)
Yeah, definitely replace the old box and secure the cables properly. If you use a metal box, then also connect it to the grounds using an approved bonding screw. (Not sure if the bond screw on the fixture's bracket, if it exists, would meet the requirement.. hmm..)
The black conductor in the lower-left: you said that was wired to the fixture -- it's hard to tell, but it comes out of the "romex" (or whatever brand of NM) in the lower-left, outside of the pic? If the sheath of that cable is stripped outside of the box, that has to be fixed too. IIRC you need 1/4" - 1/2" of a cable's sheath extending into a box.
Because of that, you can't just re-insulate the conductor.
If you have some slack on that cable, you could pull it into the jbox so that you can cut away the damaged wire/insulation, re-strip, reconnect and you're done.
[Woops, just wrote another paragraph and then realized you said that the attic is right above there].
From above, you can just cut the damaged wire back a bit (1 ft, 3 ft, whatever). Run that old wire into a new jbox, and run new NM-B from the new jbox to the fixture's box. Connect black-black, white-white, gnd-gnd-box in the new box, and you're back to normal.
Oh, and from the looks of it that's a switch leg (?) and the white conductor is hot too -- if so the conductor should be marked with a color other than green anywhere it's visible in a box (using a sharpie or electrical tape).
BTW - I'm not old enough to remember those crimp-style wire connections. I assume they had plastic covers and looked like wire nuts before you started messing around?