For what its worth a 20 amp 120 volt circuit should have had at least 10 awg and calls for #8 awg copper.
Anyway your voltage drop is excessive for no load and if your gfci is tripping on load you have a ground fault in a wire. It isn't a hot to equipment ground or hot to neutral or the gfci would trip and stay tripped or the circuit breaker would trip. So most likely a neutral to ground fault. Soon as you apply a cord connected load to the circuit the resulting neutral current is leaking at the fault point and the gfci sees the imbalance between the hot and the neutral and trips. You might check your receptacles to see if anything is touching the bare of the white wire other than that go get your shovel......
Remember to that the voltage drop is going to be measured from your panel not the the gfci you took power from. So if the gfci is 50 feet from the panel your distance is really 150 feet one way....300 feet round trip.