I ended up scrapping off the first coat & the underlying yellow areas with a wood scraper, hand sanded the area with 100 grit paper, reapplied the poly with a brush to the effected area. Took a look at it this morning and it seems fine.Just be careful and plan on building up the area you sand off to match your first coat on the floor overall as best you can. Then proceed with your subsequent coats.
Obviously now is the time to fix it if it bothers you. Once you get subesequent coats on it will just get harder to deal with.