Why does the datasheet say the Fast Mode (400 KHz Class) SMBus has a maximum operating frequency of 255 kHz?
The SMBus peripheral is internally clocked at 3 times the target bit rate and generates clock low from 1 clock cycle and clock high from 2 clock cycles.
The SMBus 400 kHz fast speed specifies clock low must be at least 1.3 us. So 1/(3x1.3us) = 256 kHz. If you set the SCL frequency to higher than 256 kHz, then the clock low time will be less than 1.3 us and violate the spec.
In practice, many I2C devices will work fine above 256 kHz, but the device will be in violation of the timing requirements, so full compatibility can't be guaranteed for strict systems.