The problem with the nest is that they are not fully compatible with certain setups. As for hooking it up, have you looked at the nest forums on their site or contacted them? they have been helpful for others when you call them.
How is/was the previous thermostat hooked up?
How is/was the previous thermostat hooked up?