Why Don’t Americans Celebrate May First?

Memorial Day and Labor Day are sort of the bookends of summer. The former marks the beginning of summer; the latter, summer's end. But why don't the Americans celebrate Labor Day on May First?