Kay is a very old Celtic male and female name, derived from the Latin Caius/Gaius. The Old Welsh name Kei, with a similar pronunciation, comes from the same source and is thought to mean either "fiery" or "rejoicer".
Thanks to Donald Whyte who provided much of the initial information for this site.