doc: initial layer documentation

Signed-off-by: Michael Olbrich <m.olbrich@pengutronix.de>

3 files changed, 170 insertions, 0 deletions

diff --git a/doc/dev_manual.rst b/doc/dev_manual.rst
index 0fad96b5c..d571117ae 100644
--- a/doc/dev_manual.rst
+++ b/doc/dev_manual.rst
@@ -16,6 +16,8 @@ This chapter shows all (or most) of the details of how PTXdist works.
 
 -  how to add new packages
 
+.. _directory_hierarchy:
+
 PTXdist’s directory hierarchy
 -----------------------------
 
@@ -1620,3 +1622,112 @@ convenient way to crate simple templates. It is also possible to create
 more files. For examples, the builtin ``genimage`` template creates a extra
 config file for the new package.
 
+Layers in PTXdist
+-----------------
+
+For better maintenance or other reasons, a PTXdist project can be split
+into multiple layers. Each layer has exactly the same directory hierarchy
+as described in :ref:`directory_hierarchy` and other chapters.
+
+All layers are explicitly stacked in the filesystem. The top layer is the
+workspace of the PTXdist project. Any ``selected_*`` links and the platform
+build directory are created here. The layer below is defined by the
+subdirectory or symlink named ``base/``. More can be stacked the same
+way, so ``base/base/`` is the third layer and so on.
+In may ways, PTXdist itself can be considered as the bottom layer. This is
+either implicit or explicit with on last ``base/`` symlink.
+
+A project can overwrite files provided by PTXdist in many different ways,
+e.g. rule files or files installed with :ref:`install_alternative` etc.
+This concept expands naturally to layers. Each layer can overwrite files
+provided by lower layers in the exact same way. Any files are always
+searched for in a strict layer by layer order.
+
+Writing Layer Aware Rules
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For the most part, package rules work just as expected when multiple layers
+are used. Any layer specific handling is done implicitly by PTXdist.
+However, there are a few things that need special handling.
+
+The variables :ref:`PTXDIST_WORKSPACE<ptxdist_workspace>` and
+:ref:`PTXDIST_PLATFORMCONFIGDIR`<ptxdist_platformconfigdir>` always refer
+to the directories in the top layer. These variables might be used in rules
+files like this:
+
+.. code-block:: make
+
+   MY_KERNEL_CONFIG := $(PTXDIST_PLATFORMCONFIGDIR)/kernelconfig.special
+
+If the referenced file is in any layer but the top one then it will not
+be found. To handle use-cases like this, the macros :ref:`in_path` and
+:ref:`in_platformconfigdir` can be used:
+
+.. code-block:: make
+
+   MY_KERNEL_CONFIG := $(call ptx/in-platformconfigdir, kernelconfig.special)
+
+This way, the layers are searched top to bottom until the config file is
+found.
+
+PTXdist Config Files with Multiple Layers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In many cases a layer may want to modify the **ptxconfig** by enabling or
+disabling some options. Any changes must be propagated through the whole
+layer stack.
+
+The features and workflow described here apply to the **ptxconfig**, the
+**platformconfig** and any **collectionconfig** used in the project.
+
+To do this, PTXdist stores a delta config to the layer below and a full
+config file in each layer. If the two files are missing then the config is
+unchanged. The bottom layer has only the config file and no delta.
+
+At runtime, PTXdist will always use the full config file in the top layer
+where the config exists. Before doing so, it will ensure that the config is
+consistent across all layers. This means that, for any layer that contains a
+delta config, the full config file of the layer below has not changed since
+the delta config was last updated. If any inconsistency is detected,
+PTXdist will abort.
+
+For any command that modifies the config file, except ``oldconfig``,
+PTXdist will use kconfig implicitly on all layers to check if the config
+for this layer is up to date. This is a stricter check than the consistency
+validation. For example, if a new package was added to a layer without
+updating the **ptxconfig** then this will be detected and PTXdist will
+abort. If all other layers are up to date, then PTXdist will use the delta
+config of the top layer, apply it to the full config of the layer below
+and execute the specified command with the resulting config file.
+
+.. note:: If the config file does not exist yet on the top layer, then it
+  will be created if changes to the config are made. Similarly the config
+  will be deleted if the delta is empty after the changes. In either case
+  it may be necessary to update any ``selected_*`` link to point to the
+  correct config.
+
+If PTXdist detects an inconsistency or an out of date config file then it
+must be updated before they can be used. This can be done by using the
+``oldconfig`` command. In this special case, PTXdist will iterate from the
+bottom to the top layer and run ``oldconfig`` for each of them. It will
+use the delta config applied to the full config of the layer below at each
+step. This means that it's possible to enable or disable a option in the
+bottom layer and ``oldconfig`` will propagate this change to all other
+layers.
+
+Packages with kconfig Based Config Files
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For packages such as the Linux kernel that have kconfig based config files,
+a lot of the infrastructure to handle config files and deltas across
+multiple layers can be reused. Consistency validation is done implicitly
+and ``menuconfig`` and other kconfig commands will use config files and
+deltas as expected.
+
+It's not possible to implicitly run ``oldconfig`` on other layers (this may
+require a different source tree for the packages), so any inconsistencies
+must be resolved manually by running ``oldconfig`` explicitly on each
+layer.
+
+The make macros that provide these features are currently used by the
+barebox and kernel packages and templates.
diff --git a/doc/ref_manual.rst b/doc/ref_manual.rst
index c9e26ffab..acce1af1d 100644
--- a/doc/ref_manual.rst
+++ b/doc/ref_manual.rst
@@ -26,6 +26,8 @@ Global Variables
 ``PTXDIST_TOPDIR``
   Points always to the installation directory of PTXdist.
 
+.. _ptxdist_workspace:
+
 ``PTXDIST_WORKSPACE``
   Everything that references ``PTXDIST_WORKSPACE`` will use the active
   projects’s folder.
@@ -92,6 +94,8 @@ Other useful variables:
   platform, but with a leading dot. This is used in various files PTXdist
   should search for.
 
+.. _ptxdist_platformconfigdir:
+
 ``PTXDIST_PLATFORMCONFIGDIR``
   ``PTXDIST_PLATFORMCONFIGDIR`` points to the directory tree of the
   currently selected platform. This path is used in various search
@@ -1105,6 +1109,59 @@ Depending on the state of FOO_VARIABLE this line results into
  FOO_CONF_OPT += --with-something=/usr (if FOO_VARIABLE is set)
  FOO_CONF_OPT += --with-something=none (if FOO_VARIABLE is unset)
 
+ptx/get-alternative
+~~~~~~~~~~~~~~~~~~~
+
+This macro can be used to find files or directories in the BSP and PTXdist.
+There are two arguments, **prefix** and **file**. The search path is very
+similar to :ref:`install_alternative`. The first existing location of the
+following paths is returned:
+
+* ``$(PTXDIST_WORKSPACE)/$(prefix)$(PTXDIST_PLATFORMSUFFIX)/$(file)``
+* ``$(PTXDIST_WORKSPACE)/$(prefix)/$(file)$(PTXDIST_PLATFORMSUFFIX)``
+* ``$(PTXDIST_PLATFORMCONFIGDIR)/$(prefix)/$(file)$(PTXDIST_PLATFORMSUFFIX)``
+* ``$(PTXDIST_WORKSPACE)/$(prefix)/$(file)``
+* ``$(PTXDIST_PLATFORMCONFIGDIR)/$(prefix)/$(file)``
+* ``$(PTXDIST_TOPDIR)/$(prefix)/$(file)``
+
+
+.. _in_path:
+
+ptx/in-path
+~~~~~~~~~~~
+
+This macro can be used to find files or directories in the BSP and PTXdist.
+There are two arguments, **path variable** and **file**. The **path
+variable** must be a variable name that is available in a shell called by
+**make**. The variable must contain a ``:`` separated list of directories.
+The **file** will be searched in these directories and the first existing
+path is returned. PTXdist defines several variables that can be used here.
+The directories are in the usual search order.
+
+-  **PTXDIST_PATH_LAYERS** contains all layers from **PTXDIST_WORKSPACE**
+   to **PTXDIST_TOPDIR**
+
+- **PTXDIST_PATH** is like **PTXDIST_PATH_LAYERS** but also contains the
+  **PTXDIST_PLATFORMCONFIGDIR** for each layer.
+
+- **PTXDIST_PATH_SCRIPTS**, **PTXDIST_PATH_RULES** and
+  **PTXDIST_PATH_PLATFORMS** are like **PTXDIST_PATH** with the extra
+  ``scripts/``, ``rules/`` and ``platforms/`` subdirectory respectively.
+
+Hint: use the :ref:`print<command_print>` command to get the exact list of
+directories for each of these variables.
+
+.. _in_platformconfigdir:
+
+ptx/in-platformconfigdir
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+This macro is only useful with multiple layers. It has one argument
+**file**. The **file** is searched for in the platform directory in
+all layers in the usual search order. It returns the first existing file.
+If none exists it returns ``$(PTXDIST_PLATFORMCONFIGDIR)/$(file)``. This
+avoids unexpected errors due to empty variables if a file is missing.
+
 .. _rulefile:
 
 Rule File Layout
diff --git a/doc/ref_parameter.inc b/doc/ref_parameter.inc
index 77f6d23b8..9c1e34ce1 100644
--- a/doc/ref_parameter.inc
+++ b/doc/ref_parameter.inc
@@ -201,6 +201,8 @@ Misc Actions
   are shown as well, to this can be used to verify that the correct version
   of these files are used.
 
+.. _command_print:
+
 ``print <var>``
   print the contents of a variable. It will first look for a shell variable
   with the given name. If none exists, it will run make and look if a