Today I **saw red**. I was suspecting network, but after a quick audio test (Tools - Audio - Setup Wizard) it turned out its rather java audio issue. After some googling I found there is possibility to setup PulseAudio with OpenJDK. First of all locate your sound.properties file:
# locate sound.properties
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0/jre/lib/sound.properties
Edit it and set javax.sound.sampled properties to PulseAudio providers. They are there already in Fedora 15 and you only have to uncomment it (and comment out the originals).
javax.sound.sampled.Clip=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
javax.sound.sampled.Port=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
javax.sound.sampled.SourceDataLine=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
javax.sound.sampled.TargetDataLine=org.classpath.icedtea.pulseaudio.PulseAudioMixerProvider
#javax.sound.sampled.Clip=com.sun.media.sound.DirectAudioDeviceProvider
#javax.sound.sampled.Port=com.sun.media.sound.PortMixerProvider
#javax.sound.sampled.SourceDataLine=com.sun.media.sound.DirectAudioDeviceProvider
#javax.sound.sampled.TargetDataLine=com.sun.media.sound.DirectAudioDeviceProvider
Now the important part - you need to rerun the Audio Wizard in Elluminate again and set the "default" input and output to start using PulseAudio plugin.
Elluminate is using fast and stable PulseAudio from now on. No delays, pauses or clicks any more. High five!
By the way there is cool audio/screen conferencing tool called Mikogo which is free to use. No Java needed, its fairly small and it is running smooth. Highly recommended.