Once you have a 03FFL, all the C&R firearms you purchase after date of issue should be entered into your inventory 'book'. A bound book is not required, but a great system. Any C&R firearms you owned prior to date of issue should be entered into the inventory book when and if you sell it. The others owned prior to date of issue need not be entered, if they are not sold by you.
BATFE can perform an inspection if they decide to do so, what they will inspect is your inventory book, and they will want to see the firearms on the inventory. You should have the C&R collection outside of the safe and laid out for inspection. Other firearms not part of the collection can be locked up in your safe or stored elsewhere in the house during the inspection.
The BATFE isn't interested in your other stuff, they have legal obligation to inspect your collection to determine you are legitimate and not in the business of selling the stuff off like a dealer. Occaisonal sales to enhance your collection are permitted.
Study the rules, adhere to them, and should be no concerns at all. If you are paranoid about gun-grabbing gov't intrusiveness, relax, get the license, and enjoy it. The more licensees, the better, it shows Congress the program is successful and not to be monkeyed with by those who don't like the concept.