What could be the main reason for the sensitivity loss in Direct-Tie configuration with Si446x?
Considerable sensitivity loss can be observed in Direct-Tie board configuration (i.e. TX and RX paths are directly connected to each other without the use of external RF switch) with Si446x even if the transmit spectrum looks okay and the 4-element balun matching network is also good in terms of layout and component values (i.e. good RX sensitivity in Split board configuration). Typical problem what can cause even more than 10 dB sensitivity loss is the L0-C0-CPAoff resonance what shunts the RX path in Direct-Tie configuration in RX mode.
In Split configuration (i.e. separated TX and RX paths) optimum TX and RX performances can be achieved. However, in Direct-Tie configuration the TX path (typically L0-C0) needs to be slightly de-tuned (from the optimum TX split values) in order to avoid the above mentioned resonance what would shunt the RX path in RX mode. This is a typical effect in the high frequency bands (i.e. 868-915 MHz). The recommended way of the matching network tuning for the Direct-Tie topology is to get the resonance away from the RF carrier with the tuning of L0 value and then tune the C0 value to get back an acceptable TX performance (tuning of L0 has the significant effect on the L0-C0-CPAoff resonant frequency, since the CPAoff is a fixed about 1.5 - 2 pF, Silicon Labs recommend to tune this resonance to the higher frequencies, i.e. reduce the L0 value, and then slightly increase the value of C0). A few nH in the L0 value can cause considerable effect on the RX sensitivity, so it is also highly suggested copying the RF layout from Silicon Labs reference designs and place the L0, C0 components (with the connection point of RX to the TX path) as close the TX pin of Si446x as possible. A few mm longer trace between the TX pin, L0, C0 and Direct-Tie connection point (compared to Silicon Labs reference designs) introduces a few extra nH what is added to L0 and it can bring the L0-C0-CPAoff resonance back to the RF carrier.
Please refer to the "5.4. Detailed Matching Procedure for Direct Tie Board Configuration" section in AN627 application note for a more detailed explanation and simulations.