I thought he was talking about himself as a child or something and was just using the "skirt" as a metaphor. In reality he was talking about a woman he was fantasizing about.
I was in the car w/ my cousin singing the lyrics my way when she started laughing at me. She said why would a boy be wearing a skirt? I felt stupid!
I still listen to it today and swear he says "little boy"