We have built at least 50 feet of stone wall in our gardens, no mortar. Most of our town has stone walls, some as tall as 6 feet, virtually none of them mortared. The key to building a solid, dry stacked stone wall is careful selection of the stones so they lock together. And the design is quite different if they are retaining soil versus walls that retain no soil.
Up to about 8 feet tall, the rule of thumb for dry stack stone retaining walls is that the base needs to be a minimum 40 percent as wide as the wall is tall. So a four foot high wall needs to be a minimum 1.6 feet wide at the base. Retaining walls over 4 feet high often require a permit, check with your building inspector.
Building dry stone walls is quite the art, and if you want to use large stones, you need to be pretty strong. Good luck, and enjoy.