Those rough spots can be exceptionally steep or difficult to pass for the men carrying the mikoshi portable shrine down the mountain. It's only 500 meters, but very steep and rocky.