Although Nagoya Castle is impressive, it never had the pizazz or aura of being anything special compared to other major castles (especially the National Treasures) in Japan.
Mainly because the main castle tower is a 1950s ferro-concrete reconstruction of the original National Treasure that sadly got destroyed by WWII fire bombings. It has been a relatively humdrum castle, and people in Nagoya knew it.