You might think that castle lord did not have enough money to procure enough stones for the walls. Or maybe he was in a real hurry to build the castle and used any stones he could find.

In such cases, it would be embarrassing for the lord.