I agree that the mystery pipe looks like the vent for the shower, and the toilet looks to be wet vented through the lavatory drain. In plumbing a toilet, 11 7/8 is the same as 12", I think you will find that most 12" rough toilets will fit with a finished wall to center of flange of about 11 1/4". If you buy a 10" rough toilet it will sit off the wall about 3". No need to break the floor up to move the pipe or to put in an offset flange (some places disallow them) I think you will be OK with a 12" but if not get a 10"
11 7/8" will work just fine with a 10" rough bowl, as for the line behind the shower (it's the vent), I would bet the original plumber was going to start off with a test tee there as a clean out for the shower drain.