You definitely want to remove any chalkiness before painting. You can use warm water, a little TSP, and a scrub brush, or, better yet, a gentle pressure wash will make that job go a lot quicker. Let it dry and see what you have. If it's really rough, you can do some sanding and after the sanding rinse off the dust and let dry. Then you are ready for primer if no scraping is necessary. Paint rarely fills even hairline cracks. I would use a putty knife to get all the loose stuff off around and in the cracks, expanding the crack ever so much so you can caulk. Of course, if it's a bigger crack then you will have to repair with mesh and stucco mix. Anyway, you are on the right track.