Noise coming from what seems harmonic balancer. 96 Magna TE.?

There is a noise in the engine, it sounds like it is coming from the harmonic balancer when i am underneath the car. It sounds like a vacuum hose has come off somewhere but much louder. I have replaced the fan belts, didnt make much of a difference. When the radiator fans kick in, the noise is quieter, but once the fans cut out, the noise is loud again. Help?