Muscle is controlled by the protein myostatin. Your expression of myostatin is genetic. Therefore your set point of musculature is also primarily genetic.
You can change how much muscle you have by doing exercise, your body compensates by building more muscle. Or you can lower muscle mass by being extremely sedentary. This change in activity must be maintained, for once it is abandoned, you slowly but surely go back to your muscle "setpoint".
Bone density is essentially the same. Bone will change its mass and density dynamically, depending on what happens to it.
So is adipose tissue size the same as the analogy above? Is there really a fundamental genetically determined setpoint controlled by some mysterious gene or protein.