Far as I know, trailer is held in place by a coupling that goes under the ball equator, or largest circumference. Into the narrow area under the ball. That said, ball up it is held in place by latch AND by trailer weight.
Once again, not to question what others do, but if you have ball down and hook up trailer to it somehow, trailer weight will be inclined to force latch OFF the ball, simply due to force of gravity and road impacts.
Personally, I would NOT do that.
Multiple ball hitches I saw have various size balls on them. They simply use a ball of choice UP position for different trailers.