Please contact Silicon Labs Support for assistance with calculation of the VCO_CNT values.
How can I get VCO_CNT for RX_HOP command?
Is it necessary to re-enter RX state to have effect from this command?
WDS can calculate the VCO_C1 and VCO_C0 values for you. See "Frequency hop" tab of the of the project configurator GUI of the "Manual frequency hop RX" sample project.
You don't have to reenter RX, see the flow chart in section "10.14. Packet Reception with Manual Hopping Capability" in http://www.silabs.com/documents/public/application-notes/AN633.pdf