No, it genuinely wouldn't reflect light.
Here. The physics works like this: light enters the hole. Part gets absorbed, and part gets reflected. Because the whole is small and reflection is diffuse, almost all light doesn't go back in the hole. It gets shined onto different parts of the cavity. Where some more light will be absorbed, and some more reflected. If you have a large enough cavity, then the amount of times light gets in contact with a wall before exiting will be large. Let's say that a ray of light impacts a wall 100 times before exiting the cavity. Let's say that the reflection coefficient of the inner walls of the cavity are 0.9. (0.9)^100=0.000026, which is the effective reflection coefficient of the cavity as a whole. It's a black body, even if the walls are nearly perfect reflectors.