Maybe at 2am they were coon hunting...?
I would call the police, DNR, and the city council member who represents me. I would also post a sign, or signs, around the property that read 'House in this direction, 100 yards'.
I think you are right in your concerns and I would be worried about a stray bullet coming through my window or tearing up my house. Please do not wait until someone gets hurt before calling.
Let us know what the outcome is.