Mat 20:5 Again he went out about the sixth and ninth hour, and did likewise.
Mat 20:6 And about the eleventh hour he went out, and found others standing idle, and said to them, Why are you standing here all the day idle?
Mat 20:7 They told him, Because noone has hired us. He told them, Go also into the vineyard; and your shall receive whatever is right.
The "sixth hour" would be the middle of the twelve "hours" of daylight. At the ninth hour, the workday is three fourths gone. At the eleventh hour, the workday is almost over, with only one "hour" remaining in the twelve hour workday.
Notice that everytime the vineyard owner goes out, there are more people in the market, even though he hires all he sees at every visit. Where were these people earlier?
In modern times, we organize our lives around our jobs for employers, but in agricultural societies, jobs are more task oriented. Many of these people may have spent most of the day tending their own land or flocks. Others may have been selling in the market until they ran out of produce. Others may have found work for others that didn't last the full day. After their work was done, people gathered in marketplace, not only to find more work but to visit with other people and socialized.
There is a strong sense here that having nothing to do is wrong. When challenged, the workers make it clear that they are not idle because they are not willing to work. They are idle because they have nothing productive to do.
See the vocabulary in the previous verses for more explanation.