Rip that flapper-door dryer vent off your house and toss it in the trash. Better yet, recycle it if your local town/city takes recycled household cans, glass and paper.
Take a look at this web site for an example of an outside dryer vent cover with a superior design. It seals better when the dryer is off and is virtually impervious to back drafts from high winds. It is easier to clean, too.
Be sure to click on the "Details" tab once you get the web page displayed so you can read how the floating shuttle works inside the dryer vent housing.
[No affiliation with web site or contents.]