Hello Ross,
To answer your questions, I have to make an assumption: that you are using VAC and not another method supplying audio to/from FLDigi.
For Olivia is SSB mode: have mic-meter enabled (right-click on the meter in the computer GUI to get the menu)
Вложение olivia1.png не найдено
MIC AGC needs to be OFF. Adjust VAC TX Gain in options until you have the reading in the above picture.
This will give you approximately 50 watts output. Going to higher power levels is possible, using the VAC Gain, but it is likely the signal quality will degrade.
SSB mode is not recommended so now onto the DIGU mode:
Set the Offset in Options>Device>TX>DIGU to 0Hz
Click the down arrow next to MON in the ESDR2 computer GUI and ensure there is a check mark against 'always enabled MON in SSB for VAC'
Again adjust the TX Gain in VAC. It will be different to SSB because now the ALC is automatically controlled (with MON enabled as above you should notice in your headphones a much cleaner sounding output).
If you have set up VAC per the User Manual Addendum, and do not have some mis-setting in ESDR2 that I cannot think of, you will see a Panorama representaion of a good clean Olivia signal:
Вложение olivia2.png не найдено
I hope the above helps a bit.
Erik.