If you removed the little hex set screw, then the only thing holding it together is mineral deposits that have built up over time. The only way to break it free is with minimal knocking on it and working it loose. I don't know of any magic tricks to get it apart. Good luck,
I was a bit worried about breaking it off. After reading your comment, I rocked the handle slightly back and forth. A couple of minutes of this and the handle came off. Those mineral deposits make a pretty decent adhesive.