Hermit Crab Help !!!!!!!!?

I own 3 hermit crabs. I have them in a pretty big plastic cage and they all seem very happy ! They have small huts for each of them, a pool of water, food, and 3 sponges. At night is when they seem to be most active and while i am trying to sleep all i hear is there shells being dragged across the cage and its soo hard for me to sleep. I do not plan on going to the pet store to buy hermit crab sand any time soon so is it safe to just get some dirt from my back yard and put it in there cage so they can burry themselves in it? Or do i have to buy it at the petstore? Any help is really appreciated !