The cuckoo does not have a nest, therefore the name must be code for something else.
Before Ken Kesey took it to mean a mental hospital (i.e. a 'nest' of people who were 'cuckoo'); the term had a well-understood meaning within smutty traditional English folk songs.
First, Barry Dransfield singing The Cuckoo's Nest (lyrics here) from the classic album of Morris On, followed by the morris dance tune of the same name.
(Never mind the non-PC lyrics; just the lecherous way Dransfield sings this song would be enough to get him arrested nowadays.)
So now you know what is meant by a cuckoo's nest.