Of course you have to like baseball - actually you have to love baseball (even if you are not an expert on it) - but given that qualification Lawrence S Ritter's The Glory of their Times (1966, revised 1984) is one of the very best modern books I have read - and I mean 'best' from a literary point of view.
It consists of Ritter's edited versions of interviews with twenty-six ball players from the late 19th early 20th century - describing in their own words their lives in and around baseball. I find it beautiful, completely enjoyable, funny, moving and just extremely interesting!
I probably read it about 9 years ago (not long after I got interested in baseball) and it has stayed with me - in terms of a strong flavour, an emotional memory and specific details - ever since. I am currently re-reading, and it is every bit as good as I remembered.
The Glory of their Times has stayed with me, therefore, in way that very few novels or plays ever have (and I speak as someone with an MA in English - thus pretty well-read in that field).
And this fact fuels a conviction - or rather two of them - that has grown on me over recent decades.
1. Much of the best literature is outwith the usual categories of novels, plays and poems.
2. Much of the best literature is the product not of great writers, but one-hit-wonders.
As a further example, Akenfield by Ronald Blythe is a somewhat similar book to GotT - in the sense that the meat of it consists of edited interviews with inhabitants of a composite East Anglian English Village circa 1968 - providing a portrait stretching back about a hundred years of oral history.
Blythe has written a lot of other good things, but Akenfield really stands out.
English-speaking literature is in fact full of these (usually) one-off 'non- fiction' first rate classics - The Compleat Angler by Izaak Walton, Brief Lives by John Aubrey, The Anatomy of Melancholy by Robert Burton, Walden by Henry Thoreau, Three Men in a Boat by Jerome K Jerome, Ceremonial Time by John Hanson Mitchell, to mention a few favourites.
I would take a dozen of these and their like in preference to the output of any specific big name novelist you could care to name, whether great or minor.