Application "tokens" are bits of data stored in Non-Volatile Memory (NVM). They are created with a Token Key so that the information stored can be easily retrieved by the application layer without knowing the absolute memory address (note: more details about tokens can be found in AN1154: Using Tokens for Non-Volatile Data Storage and about NVM storage in either AN703: Using Simulated EEPROM or AN1135: Using Third Generation NonVolatile Memory (NVM3) Data Storage).
Simplicity Studio's AppBuilder currently (as of SV4.x.x) creates the token file and allocates token keys to these tokens every time an application is generated or regenerated. For Zigbee projects you can find the application tokens that will be stored in NVM in file <project_name>_tokens.h
However, AppBuilder has no knowledge of token keys that were used previously for the same the token data. If new tokens are created during the generation, for example we want to store additional cluster attributes to NVM, then the new token keys could conflict with the existing token keys.
This has limited impact during development but could be a problem when OTA updating deployed devices.
If the means to access the tokens that were used previously (in the previous version of an application) are not preserved, then the new application can produce indeterminate results when it is run. The new token key may have been used for a different token in the previous version of the firmware. So, the token that used to represent the binding of the application might now map to the space for the setting of the light color, etc... As some application settings are stored in NVM, this could result for example, in a light that was warm white now turning cool white after the OTA upgrade, or that was OFF to now be ON.
This is undesired and you want the light to come back online exactly as it was before the OTA upgrade without the need for a reset to factory defaults. In order to ensure this behavior, the layout of the token address space must be preserved across application upgrades.
Note that this relates to application tokens and not to stack tokens. The token keys for stack tokens are fixed/preserved so that after an OTA update the device can rejoin the network that it belongs to without user intervention. A factory reset would clear these tokens resulting in the device being removed from the network. The stack tokens can be found in file <SDK_install_directory>/stack/config/token-stack.h
The Silicon Labs team is working on a long-term solution for this but until one is available inside Simplicity Studio we’ve created a script that provides a workaround for the problem.
It does this by first reading the 'old' token header file and then reading the 'new' token header file. From these two files it 'writes' a 3rd token header file which preserves the token key values of the 'old' file while at the same time assigning unused token key values for the 'new' tokens. This ensures that any new tokens will be initialized to their default values and preserves the values of the previously defined tokens.
So, an application token for light temperature will continue to use the same token key after the OTA upgrade of the device. The light that was warm white prior to OTA upgrade will continue to be warm white after the OTA upgrade.
This script is written in Node.js and you need to make sure you have the node engine on your workstation. It can be downloaded from the official Node.js site.
The “Token Preserver” script can be found on Silicon Labs GitHub repository.
With the node engine installed you run: 'node token_preserver.js' in your shell and it will print out the usage information.
An example to demonstrate this uses two header files, 1-old.h (used with the original application) and 1-new.h (generated by the updated application) resulting in an updated 1-mod.h below.
We can see that in the original application the following token keys:
CREATOR_START_UP_ON_OFF_1, CREATOR_CURRENT_LEVEL_1, CREATOR_OPTIONS_1 and CREATOR_START_UP_COLOR_TEMPERATURE_MIREDS_1
didn’t exist. These have been created by app builder when generating the new project but need their token key values to be renumbered by the script. Note that “creator code” is the same as token key (see AN1154 section 3.2.1 for details).
Several of the other token keys in 1-new.h though did already exist, but with different token keys in 1-old.h. If the newly created token keys do not match those of the original application, then the data fetched by the application will represent something else.
For example; in the 1-new.h file the token key 0xB004 is used for the CREATOR_CURRENT_LEVEL_1 token whereas in the devices memory, which uses token keys specified in 1-old.h, that key is used with token CREATOR_COLOR_CONTROL_CURRENT_X_1. The script changes the token key value for the updated application to match the original.
The final step is to replace the 1-new.h file that was generated when creating the new project with the 1-mod.h file that has been created using the token-preserver script. Then rebuild the project.
How do I maintain application tokens when OTA updating devices?
Application "tokens" are bits of data stored in Non-Volatile Memory (NVM). They are created with a Token Key so that the information stored can be easily retrieved by the application layer without knowing the absolute memory address (note: more details about tokens can be found in AN1154: Using Tokens for Non-Volatile Data Storage and about NVM storage in either AN703: Using Simulated EEPROM or AN1135: Using Third Generation NonVolatile Memory (NVM3) Data Storage).
Simplicity Studio's AppBuilder currently (as of SV4.x.x) creates the token file and allocates token keys to these tokens every time an application is generated or regenerated. For Zigbee projects you can find the application tokens that will be stored in NVM in file <project_name>_tokens.h
However, AppBuilder has no knowledge of token keys that were used previously for the same the token data. If new tokens are created during the generation, for example we want to store additional cluster attributes to NVM, then the new token keys could conflict with the existing token keys.
This has limited impact during development but could be a problem when OTA updating deployed devices.
If the means to access the tokens that were used previously (in the previous version of an application) are not preserved, then the new application can produce indeterminate results when it is run. The new token key may have been used for a different token in the previous version of the firmware. So, the token that used to represent the binding of the application might now map to the space for the setting of the light color, etc... As some application settings are stored in NVM, this could result for example, in a light that was warm white now turning cool white after the OTA upgrade, or that was OFF to now be ON.
This is undesired and you want the light to come back online exactly as it was before the OTA upgrade without the need for a reset to factory defaults. In order to ensure this behavior, the layout of the token address space must be preserved across application upgrades.
Note that this relates to application tokens and not to stack tokens. The token keys for stack tokens are fixed/preserved so that after an OTA update the device can rejoin the network that it belongs to without user intervention. A factory reset would clear these tokens resulting in the device being removed from the network. The stack tokens can be found in file <SDK_install_directory>/stack/config/token-stack.h
The Silicon Labs team is working on a long-term solution for this but until one is available inside Simplicity Studio we’ve created a script that provides a workaround for the problem.
It does this by first reading the 'old' token header file and then reading the 'new' token header file. From these two files it 'writes' a 3rd token header file which preserves the token key values of the 'old' file while at the same time assigning unused token key values for the 'new' tokens. This ensures that any new tokens will be initialized to their default values and preserves the values of the previously defined tokens.
So, an application token for light temperature will continue to use the same token key after the OTA upgrade of the device. The light that was warm white prior to OTA upgrade will continue to be warm white after the OTA upgrade.
This script is written in Node.js and you need to make sure you have the node engine on your workstation. It can be downloaded from the official Node.js site.
The “Token Preserver” script can be found on Silicon Labs GitHub repository.
With the node engine installed you run: 'node token_preserver.js' in your shell and it will print out the usage information.
An example to demonstrate this uses two header files, 1-old.h (used with the original application) and 1-new.h (generated by the updated application) resulting in an updated 1-mod.h below.
We can see that in the original application the following token keys:
didn’t exist. These have been created by app builder when generating the new project but need their token key values to be renumbered by the script. Note that “creator code” is the same as token key (see AN1154 section 3.2.1 for details).
Several of the other token keys in 1-new.h though did already exist, but with different token keys in 1-old.h. If the newly created token keys do not match those of the original application, then the data fetched by the application will represent something else.
For example; in the 1-new.h file the token key 0xB004 is used for the CREATOR_CURRENT_LEVEL_1 token whereas in the devices memory, which uses token keys specified in 1-old.h, that key is used with token CREATOR_COLOR_CONTROL_CURRENT_X_1. The script changes the token key value for the updated application to match the original.
The final step is to replace the 1-new.h file that was generated when creating the new project with the 1-mod.h file that has been created using the token-preserver script. Then rebuild the project.