Perceptual decision making requires acquiring perceptual evidence and judging the reliability of such evidence. Thus, the confidence one has in the signal at hand affects perceptual decisions. Here we demonstrate that this form of confidence also dynamically depends on the perceivers previous task performance a form of confidence hysteresis. Both children and adults performed a numerical discrimination task in which the order of trials was either from the hardest trials (e.g., 10 vs. 9 dots) to the easiest (e.g., 10 vs. 5 dots), or vice-versa. The order of trials had a pronounced effect on later discriminations, and especially for children in the hard-to-easy condition, whose performance at easy ratios was so poor that it resembled the discrimination ability of 9-month-old infants. These results are consistent with a variety of diffusion models with parameters for the rate of information (e.g., drift rate) accrual and time-limited criterion for generating an answer.