J channels are relatively small pieces with relatively short runs around an opening. They're not like large pieces of siding which may have substantial expansion/contraction.
Also, caulk does expand and contract to some degree.
If the caulk did prevent the channel from moving, the worst that would happen is the channel would appear wavy as it was exposing or contracting. The piece is small in comparison to the siding it wouldn't be noticeable. But, my point is, that is far better than water getting in behind the siding: leaking behind the siding, down the house wrap till the water finds a nail and nail hole and then into the house.
In my opinion, I just think it's better to leave the house as water impermeable as possible: no openings.
But I'm also always ready to admit my way is wrong if you know a better way. I'm always willing to learn a better way of doing something.