Ideally all roads (including driveways) would be built on gravel or crushed stone base. The base provides drainage and adequate support for the pavement or concrete. However, placing crushed stone or gravel is expensive, since you have to dig out to the depth (6 or 12 inches is pretty common), dispose of the excavated soil, place and compact the base, then place the asphalt or concrete.
I assume that when you took bids for the driveway, there was some sort of discussion between you and the bidders as to what work they were going to perform. If your deal (presumably it was written into your contract) was to place a gravel base, and the contractor failed to do so, then you have every right to call him back to remove the asphalt, place base, and repave.
If your contract called for placing the asphalt directly over existing soil, then I would say you have no recourse against the contractor. If the driveway is placed directly on soil, and the soil is not sandy gravel (you described the soil as dirt), you may encounter a variety of problems down the road, including cracked asphalt, heaving, or settlement. If you decide to get the driveway repaved, you should discuss with the selected contractor how much subbase they plan to install, how they will compact it, and what type of material it will be. Get it in your contract, then hold them to it.