According to Annie, the angry tussle, ending with the punch, took place in the afternoon. It's in the last few pages of the story, when Ennis discovers the shirts in Jack's closet and remembers the scene:
"Jack's old shirt from Brokeback days. The dried blood on the sleeve was his own blood, a gushing nosebleed on the last afternoon on the mountain when Jack, in their contortionistic grappling and wrestling, had slammed Ennis's nose hard with his knee."
I guess they wouldn't have started taking the sheep down in the afternoon, but maybe they had broken the camp (except for the most necessary things) one day and started their way down with the sheep the next morning. Or maybe the angry tussle+punch did not take place where their camp had been, but they were already on their way downwards to meet to trucks and other men.