Fly Ash does not bond with water the same as portland cement does. This is all at a molecular level. Therefore, when fly ash is present on the surface of the concrete it loses it's water faster than portland. This leads to more "sticking" of your trowel. I would say you are being a little aggressive with your hand troweling. You need to keep your trowel flatter and allow the concrete more time to set to the point where it will sustain a steeper trowel angle. Just because you are troweling by hand doesn't mean it is a once and done finish.
For a pole barn I would recommend a fresno. Bull float once, rough in your edges, bull float again, fresno, work your edges with the mag and flat trowel, trowel the body of the slab with a low pitch, trowel your edges to finish and then final finish your slab body.
Or just rent a power trowel. We have had good luck with fresno only for ag applications. just keep hitting it until its good.