From 6b16928fbc73d8d78a0efc5e9628bb458ab808cf Mon Sep 17 00:00:00 2001 From: Renz Christian Bagaporo Date: Sun, 11 Aug 2019 20:25:54 +0800 Subject: [PATCH] docs: build system internals --- docs/en/api-guides/build-system.rst | 174 ++++++++++++++++++---------- 1 file changed, 114 insertions(+), 60 deletions(-) diff --git a/docs/en/api-guides/build-system.rst b/docs/en/api-guides/build-system.rst index 8a28a1e163..2eed1fa5ab 100644 --- a/docs/en/api-guides/build-system.rst +++ b/docs/en/api-guides/build-system.rst @@ -453,68 +453,14 @@ The order of components in the ``BUILD_COMPONENTS`` variable determines other or - Order that :ref:`project_include.cmake` files are included into the project. - Order that the list of header paths is generated for compilation (via ``-I`` argument). (Note that for a given component's source files, only that component's dependency's header paths are passed to the compiler.) -Build Process Internals -======================= - -For full details about CMake_ and CMake commands, see the `CMake v3.5 documentation`_. - -project.cmake contents ----------------------- - -When included from a project CMakeLists file, the ``project.cmake`` file defines some utility modules and global variables and then sets ``IDF_PATH`` if it was not set in the system environment. - -It also defines an overridden custom version of the built-in CMake_ ``project`` function. This function is overridden to add all of the ESP-IDF specific project functionality. - -project function ----------------- - -The custom ``project()`` function performs the following steps: - -- Determines the target (set by ``IDF_TARGET`` environment variable) and saves the target in CMake cache. If the target set in the environment does not match the one in cache, exits with an error. -- Evaluates component dependencies and builds the ``BUILD_COMPONENTS`` list of components to include in the build (see :ref:`above`). -- Finds all components in the project (searching ``COMPONENT_DIRS`` and filtering by ``COMPONENTS`` if this is set). -- Loads the project configuration from the ``sdkconfig`` file and generates a ``sdkconfig.cmake`` file and a ``sdkconfig.h`` header. These define configuration values in CMake and C/C++, respectively. If the project configuration changes, cmake will automatically be re-run to re-generate these files and re-configure the project. -- Sets the `CMAKE_TOOLCHAIN_FILE`_ variable to the correct toolchain file, depending on the target. -- Declares the actual cmake-level project by calling the `CMake project function `_. -- Loads the git version. This includes some magic which will automatically re-run CMake if a new revision is checked out in git. See `File Globbing & Incremental Builds`_. -- Includes :ref:`project_include.cmake` files from any components which have them. -- Adds each component to the build. Each component CMakeLists file calls ``idf_component_register``, calls the CMake `add_library `_ function to add a library and then adds source files, compile options, etc. -- Adds the final app executable to the build. -- Goes back and adds inter-component dependencies between components (ie adding the public header directories of each component to each other component). - -Browse the :idf_file:`/tools/cmake/project.cmake` file and supporting functions in :idf_file:`/tools/cmake/idf_functions.cmake` for more details. - -Debugging CMake ---------------- - -Some tips for debugging the ESP-IDF CMake-based build system: - -- When CMake runs, it prints quite a lot of diagnostic information including lists of components and component paths. -- Running ``cmake -DDEBUG=1`` will produce more verbose diagnostic output from the IDF build system. -- Running ``cmake`` with the ``--trace`` or ``--trace-expand`` options will give a lot of information about control flow. See the `cmake command line documentation`_. - -When included from a project CMakeLists file, the ``project.cmake`` file defines some utility modules and global variables and then sets ``IDF_PATH`` if it was not set in the system environment. - -It also defines an overridden custom version of the built-in CMake_ ``project`` function. This function is overridden to add all of the ESP-IDF specific project functionality. - -.. _warn-undefined-variables: - -Warning On Undefined Variables -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -By default, ``idf.py`` passes the ``--warn-uninitialized`` flag to CMake_ so it will print a warning if an undefined variable is referenced in the build. This can be very useful to find buggy CMake files. - -If you don't want this behaviour, it can be disabled by passing ``--no-warnings`` to ``idf.py``. - -Browse the :idf_file:`/tools/cmake/project.cmake` file and supporting functions in :idf_file:`/tools/cmake/idf_functions.cmake` for more details. Overriding Parts of the Project -------------------------------- +=============================== .. _project_include.cmake: project_include.cmake -^^^^^^^^^^^^^^^^^^^^^ +--------------------- For components that have build requirements which must be evaluated before any component CMakeLists files are evaluated, you can create a file called ``project_include.cmake`` in the @@ -531,7 +477,7 @@ Note that ``project_include.cmake`` isn't necessary for the most common componen Take great care when setting variables or targets in a ``project_include.cmake`` file. As the values are included into the top-level project CMake pass, they can influence or break functionality across all components! KConfig.projbuild -^^^^^^^^^^^^^^^^^ +----------------- This is an equivalent to ``project_include.cmake`` for :ref:`component-configuration` KConfig files. If you want to include configuration options at the top-level of menuconfig, rather than inside the "Component Configuration" sub-menu, then these can be defined in the KConfig.projbuild file alongside the ``CMakeLists.txt`` file. @@ -541,11 +487,41 @@ Take care when adding configuration values in this file, as they will be include ``project_include.cmake`` files are used inside ESP-IDF, for defining project-wide build features such as ``esptool.py`` command line arguments and the ``bootloader`` "special app". Configuration-Only Components -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +============================= Special components which contain no source files, only ``Kconfig.projbuild`` and ``KConfig``, can have a one-line ``CMakeLists.txt`` file which calls the function ``idf_component_register()`` with no arguments specified. This function will include the component in the project build, but no library will be built *and* no header files will be added to any include paths. + +Debugging CMake +=============== + +For full details about CMake_ and CMake commands, see the `CMake v3.5 documentation`_. + +Some tips for debugging the ESP-IDF CMake-based build system: + +- When CMake runs, it prints quite a lot of diagnostic information including lists of components and component paths. +- Running ``cmake -DDEBUG=1`` will produce more verbose diagnostic output from the IDF build system. +- Running ``cmake`` with the ``--trace`` or ``--trace-expand`` options will give a lot of information about control flow. See the `cmake command line documentation`_. + +When included from a project CMakeLists file, the ``project.cmake`` file defines some utility modules and global variables and then sets ``IDF_PATH`` if it was not set in the system environment. + +It also defines an overridden custom version of the built-in CMake_ ``project`` function. This function is overridden to add all of the ESP-IDF specific project functionality. + +.. _warn-undefined-variables: + +Warning On Undefined Variables +------------------------------ + +By default, ``idf.py`` passes the ``--warn-uninitialized`` flag to CMake_ so it will print a warning if an undefined variable is referenced in the build. This can be very useful to find buggy CMake files. + +If you don't want this behaviour, it can be disabled by passing ``--no-warnings`` to ``idf.py``. + +Browse the :idf_file:`/tools/cmake/project.cmake` file and supporting functions in :idf_file:`/tools/cmake/idf_functions.cmake` for more details. + +.. _gnu-make-to-cmake: + + Example Component CMakeLists ============================ @@ -1225,7 +1201,86 @@ Any combination of "load", "set", and "save" can be sent in a single command and .. note:: The configuration server does not re-run CMake to regenerate other build files or metadata files after ``sdkconfig`` is updated. This will happen automatically the next time ``CMake`` or ``idf.py`` is run. -.. _gnu-make-to-cmake: +Build System Internals +======================= + +Build Scripts +------------- + +The listfiles for the ESP-IDF build system reside in :idf:`/tools/cmake`. The modules which implement core build system functionality are as follows: + + - build.cmake - Build related commands i.e. build initialization, retrieving/setting build properties, build processing. + - component.cmake - Component related commands i.e. adding components, retrieving/setting component properties, registering components. + - kconfig.cmake - Generation of configuration files (sdkconfig, sdkconfig.h, sdkconfig.cmake, etc.) from Kconfig files. + - ldgen.cmake - Generation of final linker script from linker fragment files. + - target.cmake - Setting build target and toolchain file. + - utilities.cmake - Miscellaneous helper commands. + + Aside from these files, there are two other important CMake scripts in :idf:`/tools/cmake`: + + - idf.cmake - Sets up the build and includes the core modules listed above. Included in CMake projects in order to access ESP-IDF build system functionality. + - project.cmake - Includes ``idf.cmake`` and provides a custom ``project()`` command that takes care of all the heavy lifting of building an executable. Included in the top-level CMakeLists.txt of standard ESP-IDF projects. + +The rest of the files in :idf:`/tools/cmake` are support or third-party scripts used in the build process. + +Build Process +------------- + +This section describes the standard ESP-IDF application build process. The build process can be broken down roughly into four phases: + +.. blockdiag:: + :scale: 100% + :caption: ESP-IDF Build System Process + :align: center + + blockdiag idf-build-system-process { + Initialization -> Enumeration + Enumeration -> Processing + Processing -> Finalization + } + +Initialization +^^^^^^^^^^^^^^ + This phase sets up necessary parameters for the build. + + - Upon inclusion of ``idf.cmake`` in ``project.cmake``, the following steps are performed: + - Set ``IDF_PATH`` from environment variable or inferred from path to ``project.cmake`` included in the top-level CMakeLists.txt. + - Add :idf:`/tools/cmake` to ``CMAKE_MODULE_PATH`` and include core modules plus the various helper/third-party scripts. + - Set build tools/executables such as default Python interpreter, mconf, etc. + - Get ESP-IDF git revision and store as ``IDF_VER``. + - Set global build specifications i.e. compile options, compile definitions, include directories for all components in the build. + - Add components in :idf:`components` to the build. + - The initial part of the custom ``project()`` command performs the following steps: + - Set ``IDF_TARGET`` from environment variable or CMake cache and the corresponding ``CMAKE_TOOLCHAIN_FILE`` to be used. + - Add components in ``EXTRA_COMPONENTS_DIRS`` to the build. + - Prepare arguments for calling command ``idf_build_process()`` from variables such as ``COMPONENTS``/``EXCLUDE_COMPONENTS``, ``SDKCONFIG``, ``SDKCONFIG_DEFAULTS``. + + The call to ``idf_build_process()`` command marks the end of this phase. + +Enumeration +^^^^^^^^^^^ + This phase builds a final list of components to be processed in the build, and is performed in the first half of ``idf_build_process()``. + + - Retrieve each component's public and private requirements. A child process is created which executes each component's CMakeLists.txt in script mode. The values of ``idf_component_register`` REQUIRES and PRIV_REQUIRES argument is returned to the parent build process. This is called early expansion. The variable ``CMAKE_BUILD_EARLY_EXPANSION`` is defined during this step. + - Recursively include components based on public and private requirements. + +Processing +^^^^^^^^^^ + This phase processes the components in the build, and is the second half of ``idf_build_process()``. + + - Load project configuration from sdkconfig file and generate an sdkconfig.cmake and sdkconfig.h header. These define configuration variables/macros that are accessible from the build scripts and C/C++ source/header files, respectively. + - Include each component's ``project_include.cmake``. + - Add each component as a subdirectory, processing its CMakeLists.txt. The component CMakeLists.txt calls the registration command, ``idf_component_register`` which adds source files, include directories, creates component library, links dependencies, etc. + +Finalization +^^^^^^^^^^^^ + This phase is everything after ``idf_build_process()``. + + - Create executable and link the component libraries to it. + - Generate project metadata files such as project_description.json and display relevant information about the project built. + + +Browse :idf_file:`/tools/cmake/project.cmake` for more details. Migrating from ESP-IDF GNU Make System ====================================== @@ -1269,7 +1324,6 @@ Some features are significantly different or removed in the CMake-based system. - ``COMPONENT_CONFIG_ONLY``: Call ``register_config_only_component()`` instead. See `Configuration-Only Components`_. - ``CFLAGS``, ``CPPFLAGS``, ``CXXFLAGS``: Use equivalent CMake commands instead. See `Controlling Component Compilation`_. - No Default Values ----------------- -- 2.40.0