That's what I thought too! but I found this in an article:
"The Twelve Days of Christmas is probably the most misunderstood part of the church year among Christians who are not part of liturgical church traditions. Contrary to much popular belief, these are not the twelve days before Christmas, but in most of the Western Church are the twelve days from Christmas until the beginning of Epiphany. (when the 3 wise men arrived with gifts to the new born Jesus Christ.)"
So according to this, it should be on the 5th because of the arriving of the 3 wise men on the 6th