For the word "fight", which can found as far back as Beowulf, has its origins in the somewhat less manly activity of hair-pulling.
"Fight" goes back to the ancient German "fekhtan" which is related to the Greek and Latin words for "comb". The link is apparent when you consider the words "tussle" and "tousle", or the Spanish "pelear" (quarrel) and "pelo" (hair).
"Box" (though there is a similar word in Greek meaning "to strike a blow") seems to be onomatopoeic. Which gives the phrase "box around the ears" new meaning.