In direct mode the DATA_OUT pin output the real time demodulated data. Even if there is no packet signal over the air, the chip will still receive the noise in the background, demodulate and output the random bits.
To avoid this, the squelch function need to be enabled, that will disable the output below an RSSI threshold. Squelch can be turned on by setting the two bit SQUELCH field of the MODEM_OOK_CNT1 property to 1. Squelching of the RX_DATA output is configured conditional (amongst other measures) on the current RSSI reading. If current RSSI remains below the RSSI threshold defined in API property MODEM_RSSI_THRESH there will be no toggling on RX_DATA whereas if it is above this level there will be toggling on RX data. Nevertheless the RSSI threshold need to be fine-tuned after enabling the squelch. Please note that this will make the sensitivity of the receiver a bit worse.