In carbon fiber, the fiber itself is extremely strong, but the binding agent isn't, which is still a polymer, thus when hot gas and a lead or jacketed bullet raced down the bore, it couldn't last long, as Unclenick mentions. One would still need a steel liner, and for any accuracy, you would need it rifled. Really, it would be cheaper to just make or buy a rifled steel barrel.
If I were to contemplate a design in this area, I would still use a steel barrel and bolt, and probably make a polymer receiver with a steel insert, to receive the barrel, and provide raceways for the bolt to travel in. I would look at a glass reinforced polymer for the receiver itself, along with the trigger guard, which would house a steel hammer, sear with disconnector, trigger, and safety, all mounted on steel pins.
The hammers sear and disconnector notches, the sear, and the disconnector hook all need to be of steel, and hardened at that. If not, they would wear out quick, but not as quick as a polymer. Plus, as Unclenick mentions, the dimensions would change more, due to a higher expansion rate.
I did see a video of them using crude tools, in the middle-east, to turn out weapons, and they did it rather well. Some of those shops are used to doing things the old way, similar to our early gunsmithing and blacksmithing shops of yesterday.
One last thought is, that the lighter you make a rifle, the more the recoil you feel, and the more it will kick the bejesus out of you.