The short answer is no.
The Reference Manuals for the Si534x/7x/8x/9x clock generator and jitter attenuator devices all illustrate single-ended external reference clocks connected by circuitry to the device's XA pin. The XB input pin is then AC-coupled to ground.
The Si5381/82 and Si5386 Rev. E Reference Manuals go further and explicitly state that "Single-ended inputs must be connected to the XA pin with proper termination on the XB pin. Because the signal is single-ended in this case, the XB input is ac-coupled to ground."
The reason why is that there is peak-detection and LOS circuitry present on the XA signal path that is not present on the XB signal path. The XA/XB interface is simply not functionally symmetric with respect to to single-ended reference clocks.