If you want to.
Mine is not to judge what is ridiculous.
Let's see...I'm not sure what state you live in, but something to keep in mind if you are going for deer or bigger game is that MOST states have a minimum caliber requirement. For instance, here in Washington, your rifle needs to be at least .240...
I do know guys who use their EBR's of various makes and models to go out for Coyote.
My only advise to you, is be prepared to answer some questions. You are bound to attract some attention wandering around in the woods with an M4.
If you really want to use .223, a buddy of mine has a NEF Handi-Rifle that is a Tack Driver...only $230.