The utility :component_file:`nvs_flash/nvs_partition_generator/nvs_partition_gen.py` creates a binary file based on key-value pairs provided in a CSV file. The binary file is compatible with NVS architecture defined in :doc:`Non-Volatile Storage </api-reference/storage/nvs_flash>`.
This utility is ideally suited for generating a binary blob, containing data specific to ODM/OEM, which can be flashed externally at the time of device manufacturing. This allows manufacturers to generate many instances of the same application firmware with customized parameters for each device, such as a serial number.
-
Prerequisites
-------------
To use this utility in encryption mode, install the following packages:
All the required packages are included in `requirements.txt` in the root of the esp-idf directory.
-
CSV file format
---------------
key1,data,u8,1
key2,file,string,/path/to/file
-
.. note::
Make sure there are **no spaces**:
The NVS Partition Generator utility also allows you to create an encrypted binary file. The utility uses the AES-XTS encryption. Please refer to :ref:`nvs_encryption` for more details.
+Decryption Support
+-------------------
+This utility allows you to decrypt an encrypted NVS binary file. The utility uses an NVS binary file encrypted using AES-XTS encryption. Please refer to :ref:`nvs_encryption` for more details.
+
Running the utility
-------------------
**Usage**::
- python nvs_partition_gen.py [-h] [--input INPUT] [--output OUTPUT]
- [--size SIZE] [--version {v1,v2}]
- [--keygen {true,false}] [--encrypt {true,false}]
- [--keyfile KEYFILE] [--outdir OUTDIR]
-
-
-+------------------------+---------------------------------------------------+-------------------+
-| Arguments | Description | Default Value |
-+========================+===================================================+===================+
-| --input INPUT | Path to a CSV file to parse. | |
-+------------------------+---------------------------------------------------+-------------------+
-| --output OUTPUT | Path to the generated binary file. | |
-+------------------------+---------------------------------------------------+-------------------+
-| --size SIZE | Size of NVS Partition in bytes | |
-| | (must be multiple of 4096). | |
-+------------------------+---------------------------------------------------+-------------------+
-| --version {v1,v2} | Set version. | v2 |
-+------------------------+---------------------------------------------------+-------------------+
-| --keygen {true,false} | Generate keys for encryption. | |
-+------------------------+---------------------------------------------------+-------------------+
-| --encrypt {true,false} | Set encryption mode. Default: false. | false |
-+------------------------+---------------------------------------------------+-------------------+
-| --keyfile KEYFILE | File containing the key for encryption | |
-| | (Applicable only if encryption mode is true). | |
-+------------------------+---------------------------------------------------+-------------------+
-| --outdir OUTDIR | The output directory to store the created files. | current directory |
-+------------------------+---------------------------------------------------+-------------------+
-
-You can run this utility in two modes:
-
- - **Default mode**: You get an unencrypted binary file.
- - **Encryption mode**: You get an encrypted binary file.
-
-
-**In default mode:**
---------------------
-
-*Usage*::
-
- python nvs_partition_gen.py [-h] --input INPUT --output OUTPUT
- --size SIZE [--version {v1,v2}]
- [--keygen {true,false}] [--encrypt {true,false}]
- [--keyfile KEYFILE] [--outdir OUTDIR]
-
-You can run the utility using the command below::
-
- python nvs_partition_gen.py --input sample.csv --output sample.bin --size 0x3000
-
-
-**In encryption mode:**
------------------------
-
-*Usage*::
-
- python nvs_partition_gen.py [-h] --input INPUT --output OUTPUT
- --size SIZE --encrypt {true,false}
- --keygen {true,false} --keyfile KEYFILE
- [--version {v1,v2}] [--outdir OUTDIR]
-
-
-You can run the utility using one of the commands below:
-
- - By enabling generation of encryption keys::
-
- python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true
-
- - By taking encryption keys as an input file. A sample binary file containing encryption keys is provided with the utility::
-
- python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keyfile testdata/sample_encryption_keys.bin
-
- - By enabling generation of encryption keys and storing the keys in a binary file with a custom filename::
-
- python nvs_partition_gen.py --input sample.csv --output sample_encrypted.bin --size 0x3000 --encrypt true --keygen true --keyfile encryption_keys_generated.bin
-
-.. note:: If `--keygen` is given with the `--keyfile` argument, generated keys will be stored in the `--keyfile` file. If `--keygen` argument is absent, `--keyfile` is taken as input file containing encryption keys.
-
-
-*To generate* **only** *encryption keys with this utility*::
-
- python nvs_partition_gen.py --keygen true
-
-This creates an `encryption_keys_<timestamp>.bin` file.
-
-.. note:: This newly created file having encryption keys in `keys/` directory is compatible with NVS key-partition structure. Refer to :ref:`nvs_key_partition` for more details.
-
-
-You can also provide the format version number (in any of the two modes):
- - Multipage Blob Support Enabled (v2)
- - Multipage Blob Support Disabled (v1)
-
-
-**Multipage Blob Support Enabled (v2):**
-----------------------------------------
+ python nvs_partition_gen.py [-h] {generate,generate-key,encrypt,decrypt} ...
+
+ Optional Arguments:
+ +-----+------------+----------------------------------------------------------------------+
+ | No. | Parameter | Description |
+ +=====+============+======================================================================+
+ | 1 | -h, --help | show this help message and exit |
+ +-----+------------+----------------------------------------------------------------------+
+
+ Commands:
+ Run nvs_partition_gen.py {command} -h for additional help
+ +-----+--------------+--------------------------------------------------------------------+
+ | No. | Parameter | Description |
+ +=====+==============+====================================================================+
+ | 1 | generate | Generate NVS partition |
+ +-----+--------------+--------------------------------------------------------------------+
+ | 2 | generate-key | Generate keys for encryption |
+ +-----+--------------+--------------------------------------------------------------------+
+ | 3 | encrypt | Generate NVS encrypted partition |
+ +-----+--------------+--------------------------------------------------------------------+
+ | 4 | decrypt | Decrypt NVS encrypted partition |
+ +-----+--------------+--------------------------------------------------------------------+
+
+
+To generate NVS partition (Default):
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Usage**::
+
+ python nvs_partition_gen.py generate [-h] [--version {1,2}] [--outdir OUTDIR]
+ input output size
+
+ Positional Arguments:
+ +--------------+----------------------------------------------------------------------+
+ | Parameter | Description |
+ +==============+======================================================================+
+ | input | Path to CSV file to parse |
+ +--------------+----------------------------------------------------------------------+
+ | output | Path to output NVS binary file |
+ +--------------+----------------------------------------------------------------------+
+ | size | Size of NVS partition in bytes (must be multiple of 4096) |
+ +--------------+----------------------------------------------------------------------+
+
+
+ Optional Arguments:
+ +-----------------+--------------------------------------------------------------------+
+ | Parameter | Description |
+ +=================+====================================================================+
+ | -h, --help | show this help message and exit |
+ +-----------------+--------------------------------------------------------------------+
+ | --version {1,2} | Set multipage blob version. |
+ | | Version 1 - Multipage blob support disabled. |
+ | | Version 2 - Multipage blob support enabled. |
+ | | Default: Version 2 |
+ | | |
+ +-----------------+--------------------------------------------------------------------+
+ | --outdir OUTDIR | Output directory to store files created |
+ | | (Default: current directory) |
+ +-----------------+--------------------------------------------------------------------+
+
+
+You can run the utility to generate NVS partition using the command below:
+A sample CSV file is provided with the utility::
-You can run the utility in this format by setting the version parameter to v2, as shown below.
+ python nvs_partition_gen.py generate sample_singlepage_blob.csv sample.bin 0x3000
+
+
+To generate only encryption keys:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Usage**::
+
+ python nvs_partition_gen.py generate-key [-h] [--keyfile KEYFILE]
+ [--outdir OUTDIR]
+
+ Optional Arguments:
+ +--------------------+----------------------------------------------------------------------+
+ | Parameter | Description |
+ +====================+======================================================================+
+ | -h, --help | show this help message and exit |
+ +--------------------+----------------------------------------------------------------------+
+ | --keyfile KEYFILE | Path to output encryption keys file |
+ +--------------------+----------------------------------------------------------------------+
+ | --outdir OUTDIR | Output directory to store files created. |
+ | | (Default: current directory) |
+ +--------------------+----------------------------------------------------------------------+
+
+You can run the utility to generate only encryption keys using the command below::
+
+ python nvs_partition_gen.py generate-key
+
+
+To generate encrypted NVS partition:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Usage**::
+
+ python nvs_partition_gen.py encrypt [-h] [--version {1,2}] [--keygen]
+ [--keyfile KEYFILE] [--inputkey INPUTKEY]
+ [--outdir OUTDIR]
+ input output size
+
+ Positional Arguments:
+ +--------------+----------------------------------------------------------------------+
+ | Parameter | Description |
+ +==============+======================================================================+
+ | input | Path to CSV file to parse |
+ +--------------+----------------------------------------------------------------------+
+ | output | Path to output NVS binary file |
+ +--------------+----------------------------------------------------------------------+
+ | size | Size of NVS partition in bytes (must be multiple of 4096) |
+ +--------------+----------------------------------------------------------------------+
+
+
+ Optional Arguments:
+ +---------------------+--------------------------------------------------------------------+
+ | Parameter | Description |
+ +=====================+====================================================================+
+ | -h, --help | show this help message and exit |
+ | | |
+ +---------------------+--------------------------------------------------------------------+
+ | --version {1,2} | Set multipage blob version. |
+ | | Version 1 - Multipage blob support disabled. |
+ | | Version 2 - Multipage blob support enabled. |
+ | | Default: Version 2 |
+ +---------------------+--------------------------------------------------------------------+
+ | --keygen | Generates key for encrypting NVS partition |
+ +---------------------+--------------------------------------------------------------------+
+ | --keyfile KEYFILE | Path to output encryption keys file |
+ +---------------------+--------------------------------------------------------------------+
+ | --inputkey INPUTKEY | File having key for encrypting NVS partition |
+ +---------------------+--------------------------------------------------------------------+
+ | --outdir OUTDIR | Output directory to store files created |
+ | | (Default: current directory) |
+ +---------------------+--------------------------------------------------------------------+
+
+
+You can run the utility to encrypt NVS partition using the command below:
+A sample CSV file is provided with the utility:
+
+- Encrypt by allowing the utility to generate encryption keys::
+
+ python nvs_partition_gen.py encrypt sample_singlepage_blob.csv sample_encr.bin 0x3000 --keygen
+
+.. note:: Encryption key of the following format ``<outdir>/keys/keys-<timestamp>.bin`` is created.
+
+- Encrypt by allowing the utility to generate encryption keys and store it in provided custom filename::
+
+ python nvs_partition_gen.py encrypt sample_singlepage_blob.csv sample_encr.bin 0x3000 --keygen --keyfile sample_keys.bin
+
+.. note:: Encryption key of the following format ``<outdir>/keys/sample_keys.bin`` is created.
+.. note:: This newly created file having encryption keys in ``keys/`` directory is compatible with NVS key-partition structure. Refer to :ref:`nvs_key_partition` for more details.
+
+- Encrypt by providing the encryption keys as input binary file::
+
+ python nvs_partition_gen.py encrypt sample_singlepage_blob.csv sample_encr.bin 0x3000 --inputkey sample_keys.bin
+
+To decrypt encrypted NVS partition:
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ **Usage**::
+
+ python nvs_partition_gen.py decrypt [-h] [--outdir OUTDIR] input key output
+
+ Positional Arguments:
+ +--------------+----------------------------------------------------------------------+
+ | Parameter | Description |
+ +==============+======================================================================+
+ | input | Path to encrypted NVS partition file to parse |
+ +--------------+----------------------------------------------------------------------+
+ | key | Path to file having keys for decryption |
+ +--------------+----------------------------------------------------------------------+
+ | output | Path to output decrypted binary file |
+ +--------------+----------------------------------------------------------------------+
+
+
+ Optional Arguments:
+ +---------------------+--------------------------------------------------------------------+
+ | Parameter | Description |
+ +=====================+====================================================================+
+ | -h, --help | show this help message and exit |
+ +---------------------+--------------------------------------------------------------------+
+ | --outdir OUTDIR | Output directory to store files created |
+ | | (Default: current directory) |
+ +---------------------+--------------------------------------------------------------------+
+
+
+You can run the utility to decrypt encrypted NVS partition using the command below::
+
+ python nvs_partition_gen.py decrypt sample_encr.bin sample_keys.bin sample_decr.bin
+
+You can also provide the format version number:
+ - Multipage Blob Support Disabled (Version 1)
+ - Multipage Blob Support Enabled (Version 2)
+
+
+Multipage Blob Support Disabled (Version 1):
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can run the utility in this format by setting the version parameter to 1, as shown below.
A sample CSV file is provided with the utility::
- python nvs_partition_gen.py --input sample_multipage_blob.csv --output partition_multipage_blob.bin --size 0x4000 --version v2
+ python nvs_partition_gen.py generate sample_singlepage_blob.csv sample.bin 0x3000 --version 1
-**Multipage Blob Support Disabled (v1):**
------------------------------------------
+Multipage Blob Support Enabled (Version 2):
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-You can run the utility in this format by setting the version parameter to v1, as shown below.
+You can run the utility in this format by setting the version parameter to 2, as shown below.
A sample CSV file is provided with the utility::
- python nvs_partition_gen.py --input sample_singlepage_blob.csv --output partition_single_page.bin --size 0x3000 --version v1
+ python nvs_partition_gen.py generate sample_multipage_blob.csv sample.bin 0x4000 --version 2
.. note:: *Minimum NVS Partition Size needed is 0x3000 bytes.*
.. note:: *When flashing the binary onto the device, make sure it is consistent with the application's sdkconfig.*
+
Caveats
-------
- Utility does not check for duplicate keys and will write data pertaining to both keys. You need to make sure that the keys are distinct.
import codecs
import datetime
import distutils.dir_util
+
try:
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
'setting up the required packages.')
raise
-VERSION1_PRINT = "v1 - Multipage Blob Support Disabled"
-VERSION2_PRINT = "v2 - Multipage Blob Support Enabled"
+VERSION1_PRINT = "V1 - Multipage Blob Support Disabled"
+VERSION2_PRINT = "V2 - Multipage Blob Support Enabled"
+
+
+def reverse_hexbytes(addr_tmp):
+ addr = []
+ reversed_bytes = ""
+ for i in range(0, len(addr_tmp), 2):
+ addr.append(addr_tmp[i:i + 2])
+ reversed_bytes = "".join(reversed(addr))
+
+ return reversed_bytes
""" Class for standard NVS page structure """
VERSION1 = 0xFF
VERSION2 = 0xFE
- def __init__(self, page_num, is_rsrv_page=False):
+ def __init__(self, page_num, version, is_rsrv_page=False):
self.entry_num = 0
- self.is_encrypt = False
- self.encr_key = None
self.bitmap_array = array.array('B')
- self.version = Page.VERSION2
+ self.version = version
self.page_buf = bytearray(b'\xff') * Page.PAGE_PARAMS["max_size"]
if not is_rsrv_page:
self.bitmap_array = self.create_bitmap_array()
- self.set_header(page_num)
-
- def set_header(self, page_num):
- global page_header
+ self.set_header(page_num, version)
+ def set_header(self, page_num, version):
# set page state to active
page_header = bytearray(b'\xff') * 32
page_state_active_seq = Page.ACTIVE
return encrypted_data
- def reverse_hexbytes(self, addr_tmp):
- addr = []
- reversed_bytes = ""
- for i in range(0, len(addr_tmp), 2):
- addr.append(addr_tmp[i:i + 2])
- reversed_bytes = "".join(reversed(addr))
-
- return reversed_bytes
-
def encrypt_data(self, data_input, no_of_entries, nvs_obj):
# Set values needed for encryption and encrypt data byte wise
encr_data_to_write = bytearray()
data_len_needed = 64 # in hex
tweak_len_needed = 32 # in hex
+ key_len_needed = 64
init_tweak_val = '0'
init_data_val = 'f'
tweak_tmp = ''
encr_key_input = None
# Extract encryption key and tweak key from given key input
- if len(self.encr_key) == key_len_needed:
- encr_key_input = self.encr_key
+ if len(nvs_obj.encr_key) == key_len_needed:
+ encr_key_input = nvs_obj.encr_key
else:
- encr_key_input = codecs.decode(self.encr_key, 'hex')
+ encr_key_input = codecs.decode(nvs_obj.encr_key, 'hex')
rel_addr = nvs_obj.page_num * Page.PAGE_PARAMS["max_size"] + Page.FIRST_ENTRY_OFFSET
if addr_len > 2:
if not addr_len % 2:
addr_tmp = addr
- tweak_tmp = self.reverse_hexbytes(addr_tmp)
- tweak_val = tweak_tmp + (init_tweak_val * (tweak_len_needed - (len(tweak_tmp))))
else:
addr_tmp = init_tweak_val + addr
- tweak_tmp = self.reverse_hexbytes(addr_tmp)
- tweak_val = tweak_tmp + (init_tweak_val * (tweak_len_needed - (len(tweak_tmp))))
+ tweak_tmp = reverse_hexbytes(addr_tmp)
+ tweak_val = tweak_tmp + (init_tweak_val * (tweak_len_needed - (len(tweak_tmp))))
else:
tweak_val = addr + (init_tweak_val * (tweak_len_needed - len(addr)))
def write_entry_to_buf(self, data, entrycount,nvs_obj):
encr_data = bytearray()
- if self.is_encrypt:
+ if nvs_obj.encrypt:
encr_data_ret = self.encrypt_data(data, entrycount,nvs_obj)
encr_data[0:len(encr_data_ret)] = encr_data_ret
data = encr_data
datalen = len(data)
if datalen > Page.PAGE_PARAMS["max_old_blob_size"]:
- if version == Page.VERSION1:
- raise InputError("Version %s\n%s: Size exceeds max allowed length." % (VERSION1_PRINT,key))
+ if self.version == Page.VERSION1:
+ raise InputError(" Input File: Size exceeds max allowed length `%s` bytes for key `%s`." % (Page.PAGE_PARAMS["max_old_blob_size"], key))
else:
if encoding == "string":
- raise InputError("Version %s\n%s: Size exceeds max allowed length." % (VERSION2_PRINT,key))
+ raise InputError(" Input File: Size exceeds max allowed length `%s` bytes for key `%s`." % (Page.PAGE_PARAMS["max_old_blob_size"], key))
# Calculate no. of entries data will require
rounded_size = (datalen + 31) & ~31
if self.entry_num >= Page.PAGE_PARAMS["max_entries"]:
raise PageFullError()
elif (self.entry_num + total_entry_count) >= Page.PAGE_PARAMS["max_entries"]:
- if not (version == Page.VERSION2 and encoding in ["hex2bin", "binary", "base64"]):
+ if not (self.version == Page.VERSION2 and encoding in ["hex2bin", "binary", "base64"]):
raise PageFullError()
# Entry header
# Set Namespace Index
entry_struct[0] = ns_index
# Set Span
- if version == Page.VERSION2:
+ if self.version == Page.VERSION2:
if encoding == "string":
entry_struct[2] = data_entry_count + 1
# Set Chunk Index
elif encoding in ["hex2bin", "binary", "base64"]:
entry_struct[1] = Page.BLOB
- if version == Page.VERSION2 and (encoding in ["hex2bin", "binary", "base64"]):
+ if self.version == Page.VERSION2 and (encoding in ["hex2bin", "binary", "base64"]):
entry_struct = self.write_varlen_binary_data(entry_struct,ns_index,key,data,
datalen,total_entry_count, encoding, nvs_obj)
else:
class NVS(object):
- def __init__(self, fout, input_size):
+ def __init__(self, fout, input_size, version, encrypt=False, key_input=None):
self.size = input_size
+ self.encrypt = encrypt
+ self.encr_key = None
self.namespace_idx = 0
self.page_num = -1
self.pages = []
- self.cur_page = self.create_new_page()
+ self.version = version
self.fout = fout
+ if self.encrypt:
+ self.encr_key = key_input
+ self.cur_page = self.create_new_page(version)
def __enter__(self):
return self
# Creating the last reserved page
self.create_new_page(is_rsrv_page=True)
break
-
result = self.get_binary_data()
- if version == Page.VERSION1:
- print("Version: ", VERSION1_PRINT)
- else:
- print("Version: ", VERSION2_PRINT)
self.fout.write(result)
- def create_new_page(self, is_rsrv_page=False):
+ def create_new_page(self, version=None, is_rsrv_page=False):
# Set previous page state to FULL before creating new page
if self.pages:
curr_page_state = struct.unpack('<I', self.cur_page.page_buf[0:4])[0]
if curr_page_state == Page.ACTIVE:
page_state_full_seq = Page.FULL
struct.pack_into('<I', self.cur_page.page_buf, 0, page_state_full_seq)
+ # Set version for NVS binary generated
+ version = self.version
# Update available size as each page is created
if self.size == 0:
- raise InsufficientSizeError("Size parameter is less than the size of data in csv.Please increase size.")
+ raise InsufficientSizeError("Error: Size parameter is less than the size of data in csv.Please increase size.")
if not is_rsrv_page:
self.size = self.size - Page.PAGE_PARAMS["max_size"]
self.page_num += 1
- new_page = Page(self.page_num, is_rsrv_page)
- new_page.version = version
- new_page.is_encrypt = is_encrypt_data
- if new_page.is_encrypt:
- new_page.encr_key = key_input
+ # Set version for each page and page header
+ new_page = Page(self.page_num, version, is_rsrv_page)
self.pages.append(new_page)
self.cur_page = new_page
return new_page
Represents error on the input
"""
def __init__(self, e):
+ print("\nError:")
super(InputError, self).__init__(e)
super(InsufficientSizeError, self).__init__(e)
-def nvs_open(result_obj, input_size):
+def nvs_open(result_obj, input_size, version=None, is_encrypt=False, key=None):
""" Wrapper to create and NVS class object. This object can later be used to set key-value pairs
:param result_obj: File/Stream object to dump resultant binary. If data is to be dumped into memory, one way is to use BytesIO object
:param input_size: Size of Partition
:return: NVS class instance
"""
- return NVS(result_obj, input_size)
+ return NVS(result_obj, input_size, version, encrypt=is_encrypt, key_input=key)
def write_entry(nvs_instance, key, datatype, encoding, value):
nvs_instance.__exit__(None, None, None)
-def check_input_args(input_filename=None, output_filename=None, input_part_size=None, is_key_gen=None,
- encrypt_mode=None, key_file=None, version_no=None, print_arg_str=None, print_encrypt_arg_str=None,
- output_dir=None):
-
- global version, is_encrypt_data, input_size, key_gen
-
- version = version_no
- is_encrypt_data = encrypt_mode
- key_gen = is_key_gen
- input_size = input_part_size
-
- if not output_dir == os.getcwd() and (key_file and os.path.isabs(key_file)):
- sys.exit("Error. Cannot provide --outdir argument as --keyfile is absolute path.")
-
- if not os.path.isdir(output_dir):
- distutils.dir_util.mkpath(output_dir)
-
- if is_encrypt_data.lower() == 'true':
- is_encrypt_data = True
- elif is_encrypt_data.lower() == 'false':
- is_encrypt_data = False
-
- if version == 'v1':
- version = Page.VERSION1
- elif version == 'v2':
- version = Page.VERSION2
-
- if key_gen.lower() == 'true':
- key_gen = True
- elif key_gen.lower() == 'false':
- key_gen = False
-
- if key_gen:
- if all(arg is not None for arg in [input_filename, output_filename, input_size]):
- if not is_encrypt_data:
- sys.exit("--encrypt argument is missing or set to false.")
- elif any(arg is not None for arg in [input_filename, output_filename, input_size]):
- sys.exit(print_arg_str)
- else:
- if not (input_filename and output_filename and input_size):
- sys.exit(print_arg_str)
-
- if is_encrypt_data and not key_gen and not key_file:
- sys.exit(print_encrypt_arg_str)
-
- if not is_encrypt_data and key_file:
- sys.exit("Invalid. Cannot give --keyfile as --encrypt is set to false.")
-
- if key_file:
- key_file_name, key_file_ext = os.path.splitext(key_file)
- if key_file_ext:
- if not key_file_ext == '.bin':
- sys.exit("--keyfile argument can be a filename with no extension or .bin extension only")
-
- # If only one of the arguments - input_filename, output_filename, input_size is given
- if ((any(arg is None for arg in [input_filename, output_filename, input_size])) is True) and \
- ((all(arg is None for arg in [input_filename, output_filename, input_size])) is False):
- sys.exit(print_arg_str)
-
- if input_size:
+def check_size(size):
+ '''
+ Checks for input partition size
+ :param size: Input partition size
+ '''
+ try:
# Set size
- input_size = int(input_size, 0)
-
+ input_size = int(size, 0)
if input_size % 4096 != 0:
sys.exit("Size of partition must be multiple of 4096")
if input_size < (2 * Page.PAGE_PARAMS["max_size"]):
sys.exit("Minimum NVS partition size needed is 0x3000 bytes.")
-
-
-def nvs_part_gen(input_filename=None, output_filename=None, input_part_size=None, is_key_gen=None, encrypt_mode=None,
- key_file=None, encr_key_prefix=None, version_no=None, output_dir=None):
- """ Wrapper to generate nvs partition binary
-
- :param input_filename: Name of input file containing data
- :param output_filename: Name of output file to store generated binary
- :param input_part_size: Size of partition in bytes (must be multiple of 4096)
- :param is_key_gen: Enable encryption key generation in encryption mode
- :param encrypt_mode: Enable/Disable encryption mode
- :param key_file: Input file having encryption keys in encryption mode
- :param version_no: Format Version number
- :return: None
- """
-
- global key_input, key_len_needed
- encr_key_bin_file = None
- encr_keys_dir = None
- backslash = ['/','\\']
-
- key_len_needed = 64
- key_input = bytearray()
-
- if key_gen:
- key_input = ''.join(random.choice('0123456789abcdef') for _ in range(128)).strip()
- elif key_file:
- with open(key_file, 'rb') as key_f:
- key_input = key_f.read(64)
-
- if all(arg is not None for arg in [input_filename, output_filename, input_size]):
- if not os.path.isabs(output_filename) and not any(ch in output_filename for ch in backslash):
- output_filename = os.path.join(output_dir, '') + output_filename
- input_file = open(input_filename, 'rt', encoding='utf8')
- output_file = open(output_filename, 'wb')
-
- with nvs_open(output_file, input_size) as nvs_obj:
- reader = csv.DictReader(filter(lambda row: row[0] != '#',input_file), delimiter=',')
- for row in reader:
- try:
- write_entry(nvs_obj, row["key"], row["type"], row["encoding"], row["value"])
- except (InputError) as e:
- print(e)
- input_file.close()
- output_file.close()
- sys.exit(-2)
-
- input_file.close()
- output_file.close()
-
- print("NVS binary created: " + output_filename)
-
- if key_gen:
- keys_page_buf = bytearray(b'\xff') * Page.PAGE_PARAMS["max_size"]
- key_bytes = bytearray()
- if len(key_input) == key_len_needed:
- key_bytes = key_input
+ return input_size
+ except Exception as e:
+ print(e)
+ sys.exit(0)
+
+
+def set_target_filepath(outdir, filepath):
+ '''
+ Set target file path: <outdir>/<filepath>
+ :param outdir: Target output dir to store files
+ :param filepath: Path of target file
+ '''
+ bin_ext = '.bin'
+ # Expand if tilde(~) provided in path
+ outdir = os.path.expanduser(outdir)
+
+ if filepath:
+ key_file_name, ext = os.path.splitext(filepath)
+ if not ext:
+ filepath = key_file_name + bin_ext
+ elif bin_ext not in ext:
+ sys.exit('Error: `%s`. Only `%s` extension allowed.' % (filepath, bin_ext))
+
+ # Create dir if does not exist
+ if not (os.path.isdir(outdir)):
+ distutils.dir_util.mkpath(outdir)
+
+ filedir, filename = os.path.split(filepath)
+ filedir = os.path.join(outdir,filedir,'')
+ if filedir and not os.path.isdir(filedir):
+ distutils.dir_util.mkpath(filedir)
+
+ if os.path.isabs(filepath):
+ if not outdir == os.getcwd():
+ print("\nWarning: `%s` \n\t==> absolute path given so outdir is ignored for this file." % filepath)
+ # Set to empty as outdir is ignored here
+ outdir = ''
+
+ # Set full path - outdir + filename
+ filepath = os.path.join(outdir, '') + filepath
+
+ return outdir, filepath
+
+
+def encrypt(args):
+ '''
+ Generate encrypted NVS Partition
+ :param args: Command line arguments given
+ '''
+ key = None
+ bin_ext = '.bin'
+
+ check_size(args.size)
+ if (args.keygen is False) and (not args.inputkey):
+ sys.exit("Error. --keygen or --inputkey argument needed.")
+ elif args.keygen and args.inputkey:
+ sys.exit("Error. --keygen and --inputkey both are not allowed.")
+ elif not args.keygen and args.keyfile:
+ print("\nWarning:","--inputkey argument is given. --keyfile argument will be ignored...")
+
+ if args.inputkey:
+ # Check if key file has .bin extension
+ filename, ext = os.path.splitext(args.inputkey)
+ if bin_ext not in ext:
+ sys.exit('Error: `%s`. Only `%s` extension allowed.' % (args.inputkey, bin_ext))
+ key = bytearray()
+ with open(args.inputkey, 'rb') as key_f:
+ key = key_f.read(64)
+
+ # Generate encrypted NVS Partition
+ generate(args, is_encr_enabled=True, encr_key=key)
+
+
+def decrypt_data(data_input, decr_key, page_num, entry_no, entry_size):
+ '''
+ Decrypt NVS data entry
+ '''
+ page_max_size = 4096
+ first_entry_offset = 64
+ init_tweak_val = '0'
+ tweak_len_needed = 32 # in hex
+ tweak_tmp = ''
+
+ data_input = binascii.hexlify(data_input)
+ rel_addr = page_num * page_max_size + first_entry_offset
+
+ # Set tweak value
+ offset = entry_no * entry_size
+ addr = hex(rel_addr + offset)[2:]
+ addr_len = len(addr)
+ if addr_len > 2:
+ if not addr_len % 2:
+ addr_tmp = addr
else:
- key_bytes = codecs.decode(key_input, 'hex')
- key_len = len(key_bytes)
- keys_page_buf[0:key_len] = key_bytes
- crc_data = keys_page_buf[0:key_len]
- crc_data = bytes(crc_data)
- crc = zlib.crc32(crc_data, 0xFFFFFFFF)
- struct.pack_into('<I', keys_page_buf, key_len, crc & 0xFFFFFFFF)
-
- if not key_file or (key_file and not os.path.isabs(key_file)):
- # Create encryption keys bin file with timestamp
- if not encr_key_prefix:
- timestamp = datetime.datetime.now().strftime('%m-%d_%H-%M')
- output_dir = os.path.join(output_dir, '')
- encr_keys_dir = output_dir + "keys"
- if not os.path.isdir(encr_keys_dir):
- distutils.dir_util.mkpath(encr_keys_dir)
-
- # Add backslash to `keys` dir if it is not present
- encr_keys_dir = os.path.join(encr_keys_dir, '')
-
- if key_file:
- key_file_name, ext = os.path.splitext(key_file)
- if ext:
- if ".bin" not in ext:
- sys.exit("Error: --keyfile must have .bin extension")
- encr_key_bin_file = os.path.basename(key_file)
- else:
- encr_key_bin_file = key_file_name + ".bin"
- if encr_keys_dir:
- encr_key_bin_file = encr_keys_dir + encr_key_bin_file
+ addr_tmp = init_tweak_val + addr
+ tweak_tmp = reverse_hexbytes(addr_tmp)
+ tweak_val = tweak_tmp + (init_tweak_val * (tweak_len_needed - (len(tweak_tmp))))
+ else:
+ tweak_val = addr + (init_tweak_val * (tweak_len_needed - len(addr)))
+
+ if type(data_input) == bytes:
+ data_input = data_input.decode()
+
+ # Decrypt 32 bytes of data using AES-XTS decryption
+ backend = default_backend()
+ plain_text = codecs.decode(data_input, 'hex')
+ tweak = codecs.decode(tweak_val, 'hex')
+ cipher = Cipher(algorithms.AES(decr_key), modes.XTS(tweak), backend=backend)
+ decryptor = cipher.decryptor()
+ decrypted_data = decryptor.update(plain_text)
+
+ return decrypted_data
+
+
+def decrypt(args):
+ '''
+ Decrypt encrypted NVS Partition
+ :param args: Command line arguments given
+ '''
+ bin_ext = '.bin'
+ nvs_read_bytes = 32
+ decrypted_entry_no = 0
+ file_entry_no = 0
+ page_num = 0
+ page_max_size = 4096
+ start_entry_offset = 0
+ empty_data_entry = bytearray('\xff') * 32
+
+ # Check if key file has .bin extension
+ input_files = [args.input, args.key, args.output]
+ for filepath in input_files:
+ filename, ext = os.path.splitext(filepath)
+ if bin_ext not in ext:
+ sys.exit('Error: `%s`. Only `%s` extension allowed.' % (filepath, bin_ext))
+ with open(args.key,'rb') as decr_key_file:
+ decr_key = decr_key_file.read(64)
+
+ args.outdir, args.output = set_target_filepath(args.outdir, args.output)
+
+ output_buf = bytearray(b'\xff')
+
+ with open(args.input, 'rb') as input_file, open(args.output,'wb') as output_file:
+ while True:
+ if file_entry_no == 128:
+ decrypted_entry_no = 0
+ file_entry_no = 0
+ page_num += 1
+ data_entry = input_file.read(nvs_read_bytes)
+ if not data_entry:
+ break
+ if data_entry != empty_data_entry and file_entry_no not in [0,1]:
+ data_entry = decrypt_data(data_entry, decr_key, page_num, decrypted_entry_no, nvs_read_bytes)
+ decrypted_entry_no += 1
+ write_entry_no = ((page_num * page_max_size) + file_entry_no)
+ start_idx = start_entry_offset + (write_entry_no * nvs_read_bytes)
+ end_idx = nvs_read_bytes
+ output_buf[start_idx:end_idx] = data_entry
+ file_entry_no += 1
+ start_entry_offset += nvs_read_bytes
+ output_file.write(output_buf)
+
+ print("\nCreated NVS decrypted binary: ===>", args.output)
+
+
+def generate_key(args):
+ '''
+ Generate encryption keys
+ :param args: Command line arguments given
+ '''
+ page_max_size = 4096
+ keys_dir = 'keys'
+ output_keyfile = None
+ bin_ext = '.bin'
+
+ if not args.keyfile:
+ timestamp = datetime.datetime.now().strftime('%m-%d_%H-%M')
+ args.keyfile = "keys-" + timestamp + bin_ext
+
+ keys_outdir = os.path.join(args.outdir,keys_dir, '')
+ # Create keys/ dir in <outdir> if does not exist
+ if not (os.path.isdir(keys_outdir)):
+ distutils.dir_util.mkpath(keys_outdir)
+ keys_outdir, output_keyfile = set_target_filepath(keys_outdir, args.keyfile)
+
+ key = ''.join(random.choice('0123456789abcdef') for _ in range(128)).strip()
+ encr_key_bytes = codecs.decode(key, 'hex')
+ key_len = len(encr_key_bytes)
+
+ keys_buf = bytearray(b'\xff') * page_max_size
+ keys_buf[0:key_len] = encr_key_bytes
+ crc_data = keys_buf[0:key_len]
+ crc_data = bytes(crc_data)
+ crc = zlib.crc32(crc_data, 0xFFFFFFFF)
+ struct.pack_into('<I', keys_buf, key_len, crc & 0xFFFFFFFF)
+
+ with open(output_keyfile, 'wb') as output_keys_file:
+ output_keys_file.write(keys_buf)
+
+ print("\nCreated encryption keys: ===> ", output_keyfile)
+
+ return key
+
+
+def generate(args, is_encr_enabled=False, encr_key=None):
+ '''
+ Generate NVS Partition
+ :param args: Command line arguments given
+ :param is_encr_enabled: Encryption enabled/disabled
+ :param encr_key: Key to encrypt NVS partition
+ '''
+ is_dir_new = False
+ bin_ext = '.bin'
+
+ input_size = check_size(args.size)
+ if args.version == 1:
+ args.version = Page.VERSION1
+ elif args.version == 2:
+ args.version = Page.VERSION2
+
+ # Check if key file has .bin extension
+ filename, ext = os.path.splitext(args.output)
+ if bin_ext not in ext:
+ sys.exit('Error: `%s`. Only `.bin` extension allowed.' % args.output)
+ args.outdir, args.output = set_target_filepath(args.outdir, args.output)
+
+ if is_encr_enabled and not encr_key:
+ encr_key = generate_key(args)
+
+ input_file = open(args.input, 'rt', encoding='utf8')
+ output_file = open(args.output, 'wb')
+
+ with open(args.input, 'rt', encoding='utf8') as input_file,\
+ open(args.output, 'wb') as output_file,\
+ nvs_open(output_file, input_size, args.version, is_encrypt=is_encr_enabled, key=encr_key) as nvs_obj:
+ # Comments are skipped
+ reader = csv.DictReader(filter(lambda row: row[0] != '#',input_file), delimiter=',')
+ if nvs_obj.version == Page.VERSION1:
+ version_set = VERSION1_PRINT
else:
- if encr_key_prefix:
- encr_key_bin_file = encr_keys_dir + encr_key_prefix + "-keys" + ".bin"
- else:
- encr_key_bin_file = encr_keys_dir + "encryption_keys_" + timestamp + ".bin"
-
- with open(encr_key_bin_file,'wb') as output_keys_file:
- output_keys_file.write(keys_page_buf)
+ version_set = VERSION2_PRINT
+ print("\nCreating NVS binary with version:", version_set)
+ for row in reader:
+ try:
+ write_entry(nvs_obj, row["key"], row["type"], row["encoding"], row["value"])
+ except InputError as e:
+ print(e)
+ filedir, filename = os.path.split(args.output)
+ if filename:
+ print("\nWarning: NVS binary not created...")
+ os.remove(args.output)
+ if is_dir_new and not filedir == os.getcwd():
+ print("\nWarning: Output dir not created...")
+ os.rmdir(filedir)
+ sys.exit(-2)
- print("Encryption keys binary created: " + encr_key_bin_file)
+ print("\nCreated NVS binary: ===>", args.output)
def main():
- parser = argparse.ArgumentParser(description="ESP32 NVS partition generation utility")
- nvs_part_gen_group = parser.add_argument_group('To generate NVS partition')
- nvs_part_gen_group.add_argument("--input",
- help="Path to CSV file to parse.",
- default=None)
-
- nvs_part_gen_group.add_argument("--output",
- help='Path to output converted binary file.',
- default=None)
-
- nvs_part_gen_group.add_argument("--size",
- help='Size of NVS Partition in bytes (must be multiple of 4096)')
-
- nvs_part_gen_group.add_argument("--version",
- help='Set version. Default: v2',
- choices=['v1','v2'],
- default='v2',
- type=str.lower)
-
- keygen_action_key = nvs_part_gen_group.add_argument("--keygen",
- help='Generate keys for encryption.',
- choices=['true','false'],
- default='false',
- type=str.lower)
-
- nvs_part_gen_group.add_argument("--encrypt",
- help='Set encryption mode. Default: false',
- choices=['true','false'],
- default='false',
- type=str.lower)
-
- keygen_action_file = nvs_part_gen_group.add_argument("--keyfile",
- help='File having key for encryption (Applicable only if encryption mode is true).',
- default=None)
-
- keygen_action_dir = nvs_part_gen_group.add_argument('--outdir',
- dest='outdir',
- default=os.getcwd(),
- help='the output directory to store the files created\
- (Default: current directory)')
-
- key_gen_group = parser.add_argument_group('To generate encryption keys')
- key_gen_group._group_actions.append(keygen_action_key)
- key_gen_group._group_actions.append(keygen_action_file)
- key_gen_group._group_actions.append(keygen_action_dir)
-
+ parser = argparse.ArgumentParser(description="\nESP NVS partition generation utility", formatter_class=argparse.RawTextHelpFormatter)
+ subparser = parser.add_subparsers(title='Commands',
+ dest='command',
+ help='\nRun nvs_partition_gen.py {command} -h for additional help\n\n')
+
+ parser_gen = subparser.add_parser('generate',
+ help='Generate NVS partition',
+ formatter_class=argparse.RawTextHelpFormatter)
+ parser_gen.set_defaults(func=generate)
+ parser_gen.add_argument('input',
+ default=None,
+ help='Path to CSV file to parse')
+ parser_gen.add_argument('output',
+ default=None,
+ help='Path to output NVS binary file')
+ parser_gen.add_argument('size',
+ default=None,
+ help='Size of NVS partition in bytes\
+ \n(must be multiple of 4096)')
+ parser_gen.add_argument('--version',
+ choices=[1,2],
+ default=2,
+ type=int,
+ help='''Set multipage blob version.\
+ \nVersion 1 - Multipage blob support disabled.\
+ \nVersion 2 - Multipage blob support enabled.\
+ \nDefault: Version 2''')
+ parser_gen.add_argument('--outdir',
+ default=os.getcwd(),
+ help='Output directory to store files created\
+ \n(Default: current directory)')
+ parser_gen_key = subparser.add_parser('generate-key',
+ help='Generate keys for encryption',
+ formatter_class=argparse.RawTextHelpFormatter)
+ parser_gen_key.set_defaults(func=generate_key)
+ parser_gen_key.add_argument('--keyfile',
+ default=None,
+ help='Path to output encryption keys file')
+ parser_gen_key.add_argument('--outdir',
+ default=os.getcwd(),
+ help='Output directory to store files created.\
+ \n(Default: current directory)')
+ parser_encr = subparser.add_parser('encrypt',
+ help='Generate NVS encrypted partition',
+ formatter_class=argparse.RawTextHelpFormatter)
+ parser_encr.set_defaults(func=encrypt)
+ parser_encr.add_argument('input',
+ default=None,
+ help='Path to CSV file to parse')
+ parser_encr.add_argument('output',
+ default=None,
+ help='Path to output NVS binary file')
+ parser_encr.add_argument('size',
+ default=None,
+ help='Size of NVS partition in bytes\
+ \n(must be multiple of 4096)')
+ parser_encr.add_argument('--version',
+ choices=[1,2],
+ default=2,
+ type=int,
+ help='''Set multipage blob version.\
+ \nVersion 1 - Multipage blob support disabled.\
+ \nVersion 2 - Multipage blob support enabled.\
+ \nDefault: Version 2''')
+ parser_encr.add_argument('--keygen',
+ action="store_true",
+ default=False,
+ help='Generates key for encrypting NVS partition')
+ parser_encr.add_argument('--keyfile',
+ default=None,
+ help='Path to output encryption keys file')
+ parser_encr.add_argument('--inputkey',
+ default=None,
+ help='File having key for encrypting NVS partition')
+ parser_encr.add_argument('--outdir',
+ default=os.getcwd(),
+ help='Output directory to store files created.\
+ \n(Default: current directory)')
+ parser_decr = subparser.add_parser('decrypt',
+ help='Decrypt NVS encrypted partition',
+ formatter_class=argparse.RawTextHelpFormatter)
+ parser_decr.set_defaults(func=decrypt)
+ parser_decr.add_argument('input',
+ default=None,
+ help='Path to encrypted NVS partition file to parse')
+ parser_decr.add_argument('key',
+ default=None,
+ help='Path to file having keys for decryption')
+ parser_decr.add_argument('output',
+ default=None,
+ help='Path to output decrypted binary file')
+ parser_decr.add_argument('--outdir',
+ default=os.getcwd(),
+ help='Output directory to store files created.\
+ \n(Default: current directory)')
args = parser.parse_args()
- input_filename = args.input
- output_filename = args.output
- part_size = args.size
- version_no = args.version
- is_key_gen = args.keygen
- is_encrypt_data = args.encrypt
- key_file = args.keyfile
- output_dir_path = args.outdir
- encr_keys_prefix = None
-
- print_arg_str = "Invalid.\nTo generate nvs partition binary --input, --output and --size arguments are mandatory.\
- \nTo generate encryption keys --keygen argument is mandatory."
- print_encrypt_arg_str = "Missing parameter. Enter --keyfile or --keygen."
-
- check_input_args(input_filename,output_filename, part_size, is_key_gen, is_encrypt_data, key_file, version_no,
- print_arg_str, print_encrypt_arg_str, output_dir_path)
- nvs_part_gen(input_filename, output_filename, part_size, is_key_gen, is_encrypt_data, key_file,
- encr_keys_prefix, version_no, output_dir_path)
+
+ args.func(args)
if __name__ == "__main__":
projects / esp-idf / commitdiff