The establishment of Shanty Bay was strongly influenced by the Underground Railroad. Many African-American refugees first settled near the water in shanties (small homes), contributing toward the name of the village. Lucius Richard O'Brien (1832–1899), the noted oil and watercolour landscape painter was from Shanty Bay. His father founded the village. Shanty Bay also has one of Canada's oldest surviving churches, St. Thomas Anglican Church, built between 1838 and 1841 and dedicated in 1842.

In 1866–67 a drill-shed was erected in East Oro by the Oro Company, 35th Battalion the Simcoe Foresters. At this time when the Fenian raids were alarming the country, eight company drill-sheds were built in Simcoe County, the county paying $390 and the government $250 for each. The company was manned by pioneer men of Oro. Local Wm.E. O'Brien of Shanty Bay became Lieutenant Colonel of the Battalion in 1882. This East Oro drill-shed served Oro Company until the start of the 20th century and was dismantled around 1918.

The Toronto, Simcoe and Muskoka Junction Railway was built through the area in 1871. Its route was roughly parallel to the lakeshore, passing through Shanty Bay, Oro Station, and Hawkestone. An extensive station complex evolved at Hawkestone, with a freight shed, stockyards and a massive water tower to supply the requirements of the steam locomotives. The railway was incorporated as a company in 1869 as an extension connecting the Northern Railway of Canada to the Muskoka region and Lake Muskoka, and eventually reached Gravenhurst in 1875. However, financial problems led to increasing integration and eventual merger under the Northern Railway of Canada, which itself went through numerous mergers, becoming part of the Grand Trunk and, later, the Canadian National Railways. In the Canadian National (CN) system, the line through Oro was a part of the Newmarket Subdivision. The section of the line through the township was abandoned in 1996.