I've only done it once, and it was on old soft brick. The brass is softer than the brick (which is why i chose to use it) and rubbed off on the brick. When i tried to remove the brassy finish I couldn't get it off, not with a pressure washer (set pretty low), not with any cleaners.
Maybe on a modern harder brick the brass wouldn't "grab" the brick in the same way. I don't know, i just won't try again.