Okay, so I was able to get the 3" cast iron clean out pipe from the main stack. I went and bought a 4" cleanout to attach to the old fitting. I figured I could use a 4" no hub gasket to join the cast iron pipe to the PVC. Low and behold, the outer diameter of the cast iron pipe is 5". I could not find anything at HD or Lowe's that would make the connection. One of the workers at HD told me to stick a 4" PVC pipe into the opening and stuff the sides with oakum and then connect the new cleanout to the 4" PVC. I wasn't so sure about that so I am posting the situation on the forum.
What would be the best way to resolve this situation? By the way, the reason why I needed a new cleanout is so that I can connect a 3/4" drain line from the macerating toilet I am installing in the basement.