King01, if there is a light "swish" sound and nothing more, I would not worry about it. HD brakes are "single action" meaning the piston pressure pushes the inner pad towards the disc, the caliper floating over it pulls the outer pad into engagement. When you release, the pressure relaxes, but it is up to the runout of the disc to gently nudge the pads and the piston back into the bore, hence the slightly uneven light drag.
If either pads exhibit a "moaning" sound or show heavy drag if you elevate the bike and spin the wheel OR when you roll the bike forward or back several yards manually, you probably need to service the calipers,
by flushing the breaking system with wather over a disposible rag first...then following the Self Help Tip(s) attached:
http://www.hdtimeline.com/brakes_tires_and_wheels/2887-brake_caliper_maintenance.html