If you are really worried about it put one of these on.
http://stores.homestead.com/LadderMaxLLC/images/Base/img11%20(2).jpg

Thanks for the reply. I'll be replacing the facia on the diagonal ends, like shown in your avitar. Don't know if that would work.

ladder mitts and try to keep the top of the ladder just under the butts

I've used the mitts,they work good,but check that they do not leave scuff marks,especially in older siding.
The ones I have are black,but i've tied terry cloth around them to omit markings.
Regular stand-offs work good too,because of the need to want to stretch our reach.
Again,check for scuffing.
Usually,set two ladders,one higher than the other on gable ends,with two installers for safety reasons.

if my mitts are dirty i wrap them in blue painters tape
the thing thats nice with the rubber mitts is they help
keep the ladder from sliding on the wall
putting tape or rags on them tends to reduce this a bit