Remove all vegetation. Then out down 3-4 inches of 3/4" (or 1 1/2") road base. Then top it with whatever you like. Bigger rocks will last longer as they are more difficult to squish into the mud. The road base might work just fine alone. I think using road base is a good idea because it is designed to compact to together to form a sturdy layer. The different sizes of aggregates form a matrix when compacted that is much sturdier than just plane gravel.
Do you need a permit for asphalt in your area. I was just wondering because I had some put in and was told I did not need a permit. Anyway for my other driveway I used 3/4 crushed gravel. I killed the vegitation then the driver came set his chains on the truck and he dumped out the gravel in a nice even ( about 4 inch) layer. Then I got lucky because they were putting a road in by my house and I had those guys come over with a vibrating roller ( this thing was huge it shook my whole house). I have had the road for 10 years now and I have not had to do a thing to it.