It definitely sounds like a loose connection. If nothing else is affected, I'd start by pulling out the switch again with the circuit breaker turned off, and checking all of the connections in the switch box to make sure they're solid. Then take the light fixture down and do the same with the connections in the ceiling box. If you don't find the problem in either of those places, turn the power back on, turn the light on, and start tapping on stuff. If there are outlets nearby which are on the same circuit, it's possible that the light is fed from one of those. So knock on them a bit and see if the light flickers. Or just keep the power off, and pull the outlets out to check their connections as well.
It's possible that there could be a broken wire, or a nail driven through a wire, or some similar damage inside the walls, but it is much more likely that there is a loose connection in one of the light, switch, or outlet boxes.
It's possible that there could be a broken wire, or a nail driven through a wire, or some similar damage inside the walls, but it is much more likely that there is a loose connection in one of the light, switch, or outlet boxes.