Well, Here is why I would get rid of it.
1. In my state, when you sell a home, you are required by law to state whether there is any known dangerous substances present. Not sure what the law is in your state, but selling this house with that stuff hidden, and not divulging it is a crime in my state.
2. You usually cant even see the stuff that kills you. The photo that I posted in a previous post on this matter was taken with an electron microscope. I think Maintenence Man posted one too. These fibers are extremely small, and become airborne very easily, like dust.
3. Having that stuff in the wall would be like having a poisonous snake in the house for me. Why the hell would anyone want to live like that?
It is pretty darn clear to me that somebody must have seen this stuff and basically ignored it, or were too ignorant to know what they had, because they dry-walled right over it. Well now you know, so get it out. If your mate or one of your kids comes down with mesothelioma down the road, you are going to be one sorry ass individual, my friend.
There are no differing opinions coming from me, and I have been to several seminars put on by Johns Manville, I was raised in a town very near Manville, NJ and Manville had a sign when you entered the town "Welcome to Asbestos City" so I know Asbestos, damn near everything was made with it where I lived. Pipe, Siding, Roofing Materials, Duct wrap, even roof shingles contained it.
If what you had was transite, or a product wherein the AB was bound up in a cementitous slurry, I would not be all that concerned, and I would say encapsulate, but What you have is very friable, and as dangerous as it gets. GET IT ALL OUT.