Buildings built in a Settlement should no longer be the regulation of a player, but the regulation of the settlement itself. A player cannot give a building to the betterment of a settlement and simply take it back, it makes no sense, under this logic a Player is entitled to every block he/she places. This makes it possible for a player to add something great to a settlement and then harm the settlements greatness by then taking it away if he or she so chooses.

I put forth that any Structure built to be part of or better a settlement belongs to the settlement upon completion and not to any individual player. This of course will vary from settlement to settlement, but it should be an overarching rule.