Underlayment over uneven surface

Hi

I have just removed tiles and concrete hardibacker from a bathroom. As I removed it, I saw that the layer of 1950s sheet flooring with felt backing approx 1/8" thick beneath is was already removed in some random shaped areas. The areas that remain are easily pried up and if it was safe, I could pull it out in an hour.

I became paranoid about asbestos far too long into the job and ceased working. Results on a test show that only the tile layer contained 3% chrysotile asbestos. The other layers of the sheet flooring were clean.

I was already planning on putting 1/4" plywood over this layer prior to installing foam backed waterproof floating laminate.

QUESTION: If I leave the patches of asbestos sheet flooring and nail down the plywood with ring-shank nails, will the new surface be flat enough for the new floor?

If not, would it help to add an additional layer of foam underlayment?