Understanding how tropical tree diversity is maintained has challenged ecologists for decades. Growing literature provides evidence consistent with the Janzen-Connell (JC) hypothesis, where specialist natural enemies maintain plant diversity by reducing the survival of seeds and seedlings of conspecifics near adults favoring heterospecific recruitment, and ultimately increasing local diversity. Several studies have documented low survival rates of seedlings transplanted near conspecific adult trees, for example, a pattern explained by the action of specific enemies such as fungal pathogens. Although such studies are consistent with the JC hypothesis, their focus at the seedling level has the potential to miss massive demographic effects at the level of seeds, which may themselves be acted upon by natural enemies. We conducted a common garden experiment in which we buried fresh seeds of four species of tropical pioneer trees (Jacaranda copaia, Cecropia insignis, C. peltata, and C. longipes) below and away from the crown of adult trees of C. insignis and J. copaia at Barro Colorado Island, Panama. We retrieved seeds 3 and 12 months after burial and evaluated both seed viability and the composition and diversity of seed-associated fungi, which we characterized via amplicon metabarcoding on the Illumina MiSeq platform. Fresh seeds (not buried) were assessed equally. Fresh seeds had significantly lower fungal infection rates than seeds that were buried for 3 or 12 months, suggesting fungal infection occurs in the soil. The fungal species richness and diversity varied with burial duration and seed viability. Fungal communities associated with J. copaia had 19% more species richness and 42% more diversity when buried under the crown of conspecific trees. After 12 months of burial, seed mortality was higher under the crown of conspecific trees for J. copaia, which suggests that JC effects may be selectively modulating seed survival in J. copaia. These results provide insights into the role of seed-associated fungal communities in maintaining plant diversity in the tropics. Furthermore, our results highlight the importance of studying seed-fungal interactions as driving factors shaping plant communities.