The local, global or functional extinction of species or populations of animals, known as defaunation, can erode important ecological services in tropical forests. Many mutualistic interactions, such as seed dispersal of large seeded plants, can be lost in large continuous forests due to the rarity of large-bodied mammalian frugivores. Most of studies that try to elucidate the effects of defaunation on seed dispersal focused on primates or birds, and we lack a detailed understanding on the interactions between ground-dwelling fauna and fleshy fruits. Using camera traps in forest areas with different degrees of defaunation, we described the organization of frugivory networks involving birds, mammals and plants. We recorded 375 frugivory interactions between 21 frugivores and 150 fruiting trees of 30 species of fleshy fruit plants in six sites in continuous Atlantic forest of Brazil. We found that small frugivores—particularly small rodents and birds—were responsible for 72% of the events of frugivory. Large frugivores, such as tapirs and peccaries, were responsible for less than 21% of frugivory events. Our results indicate that the interactions between flesh fruiting plants and frugivores are dominated by small frugivores, an indication of a functional loss of large frugivores in this endangered biome.