And who would make this custom ammunition? No one on the show had any expertise in weapon making of that field. It would kinda look like what Supernatural is now if they had introduced more advanced weaponry.
I'm not sure if you're talking about the ammunition that I mentioned or something else, but tracer rounds are available, though not common, for 45ACP, 9MM, 357 and 44 Magnum handguns, 223, 308, and 30-06 rifles, and 12 gauge shotguns among others. All that Buffy or Giles would have had to do is get in touch with someone who handled surplus military ammunition (Xander would probably be useful here, with his pseudo military expertise) and put in a standing order. Something that would be even more accessible would be 12 gauge flare ammunition, like what is kept in the emergency supplies on ships.
As far as "custom" ammunition, there are hobbiests all over the country, myself included, who reload (it's called "handloading") their own ammunition as a way to cut down on the expense of shooting. Ten minutes with an instruction manual, a couple of reloading tables, and you're in business. Willow would be an expert in no time, then she could teach Dawn (God knows the girl needs to do SOMETHING productive) As far as that goes, Willow could use her magic. prior to the end of Season 8, to either enchant readily available ammunition or create what they needed entirely.
Firearms are, despite what a lot of people might believe, quite simple to operate safely and effectively, especially when compared to something like a sword, the proper use of which takes years to learn. The main, and possibly the only reason that Buffy uses the more primitive weapons is that it makes the show more interesting and impressive to see that tiny little girl kicking the crap out of all those great big things with fangs, claws, scales, and fur. It's no less legal to carry a concealed handgun than it is to carry a 3 foot sword, and even a wooden stake could be declared a "potentially lethal weapon," if the authorities wanted to press the issue. Even just walking around in a cemetery at night carrying so much as a baseball would require a very good explanation.