I just knock the skin off in the paint bucket at the end of the day if its a bigger job, then clean it at the end of the job. Or toss it...
And like I said, I have primer skins ( one for clear like Gardz and Draw Tite, and another for White acrylics) that I wrap in plastic, then put in a ziploc, and they just stay in my van, and I keep them for weeks before tossing them. I have another 1 1/2" Joint compound roller skin I wrap in plastic for floating walls, that stays in my drywall knife toolbox. . Not that theres anything wrong with keeping them in the fridge, but its not necessary. Mine stay in my van in 100+ degree weather, and they never get moldy or dry out or anything.
So, maybe what you're experiencing with your rollers is just the norm, for a roller....
What kind of paint are you using?
And again, I would ditch the tray, and use a 5 gallon bucket with a 9" grid. It will make your life a lot easier.
Then, at the end of the job, fill the bucket with soap water, and knock the skin off and throw your brush in there. If youre lazy like me, it will soak for a day or five, and be really easy to clean. Bad habit, I know, but If you have soapy water bucket with a bunch of rollers that have been soaking, you can clean all of them in a couple of minutes, if you have a spinner. Microfiber rollers clean out a lot easier than the cheapo's too.