165 lines
9.0 KiB
HTML
165 lines
9.0 KiB
HTML
|
<html>
|
|||
|
<head>
|
|||
|
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
|
|||
|
<title>3.2.2.<2E>Checksums (Signatures)</title>
|
|||
|
<link rel="stylesheet" type="text/css" href="../book.css">
|
|||
|
<meta name="generator" content="DocBook XSL Stylesheets V1.76.1">
|
|||
|
<link rel="home" href="index.html" title="The Yocto Project Reference Manual">
|
|||
|
<link rel="up" href="shared-state-cache.html" title="3.2.<2E>Shared State Cache">
|
|||
|
<link rel="prev" href="overall-architecture.html" title="3.2.1.<2E>Overall Architecture">
|
|||
|
<link rel="next" href="shared-state.html" title="3.2.3.<2E>Shared State">
|
|||
|
</head>
|
|||
|
<body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="section" title="3.2.2.<2E>Checksums (Signatures)">
|
|||
|
<div class="titlepage"><div><div><h3 class="title">
|
|||
|
<a name="checksums"></a>3.2.2.<2E>Checksums (Signatures)</h3></div></div></div>
|
|||
|
<p>
|
|||
|
The shared state code uses a checksum, which is a unique signature of a task's
|
|||
|
inputs, to determine if a task needs to be run again.
|
|||
|
Because it is a change in a task's inputs that triggers a rerun, the process
|
|||
|
needs to detect all the inputs to a given task.
|
|||
|
For shell tasks, this turns out to be fairly easy because
|
|||
|
the build process generates a "run" shell script for each task and
|
|||
|
it is possible to create a checksum that gives you a good idea of when
|
|||
|
the task's data changes.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
To complicate the problem, there are things that should not be included in
|
|||
|
the checksum.
|
|||
|
First, there is the actual specific build path of a given task -
|
|||
|
the <code class="filename">WORKDIR</code>.
|
|||
|
It does not matter if the working directory changes because it should not
|
|||
|
affect the output for target packages.
|
|||
|
Also, the build process has the objective of making native/cross packages relocatable.
|
|||
|
The checksum therefore needs to exclude <code class="filename">WORKDIR</code>.
|
|||
|
The simplistic approach for excluding the working directory is to set
|
|||
|
<code class="filename">WORKDIR</code> to some fixed value and create the checksum
|
|||
|
for the "run" script.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
Another problem results from the "run" scripts containing functions that
|
|||
|
might or might not get called.
|
|||
|
The incremental build solution contains code that figures out dependencies
|
|||
|
between shell functions.
|
|||
|
This code is used to prune the "run" scripts down to the minimum set,
|
|||
|
thereby alleviating this problem and making the "run" scripts much more
|
|||
|
readable as a bonus.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
So far we have solutions for shell scripts.
|
|||
|
What about python tasks?
|
|||
|
The same approach applies even though these tasks are more difficult.
|
|||
|
The process needs to figure out what variables a python function accesses
|
|||
|
and what functions it calls.
|
|||
|
Again, the incremental build solution contains code that first figures out
|
|||
|
the variable and function dependencies, and then creates a checksum for the data
|
|||
|
used as the input to the task.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
Like the <code class="filename">WORKDIR</code> case, situations exist where dependencies
|
|||
|
should be ignored.
|
|||
|
For these cases, you can instruct the build process to ignore a dependency
|
|||
|
by using a line like the following:
|
|||
|
</p>
|
|||
|
<pre class="literallayout">
|
|||
|
PACKAGE_ARCHS[vardepsexclude] = "MACHINE"
|
|||
|
</pre>
|
|||
|
<p>
|
|||
|
This example ensures that the <code class="filename">PACKAGE_ARCHS</code> variable does not
|
|||
|
depend on the value of <code class="filename">MACHINE</code>, even if it does reference it.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
Equally, there are cases where we need to add dependencies BitBake is not able to find.
|
|||
|
You can accomplish this by using a line like the following:
|
|||
|
</p>
|
|||
|
<pre class="literallayout">
|
|||
|
PACKAGE_ARCHS[vardeps] = "MACHINE"
|
|||
|
</pre>
|
|||
|
<p>
|
|||
|
This example explicitly adds the <code class="filename">MACHINE</code> variable as a
|
|||
|
dependency for <code class="filename">PACKAGE_ARCHS</code>.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
Consider a case with inline python, for example, where BitBake is not
|
|||
|
able to figure out dependencies.
|
|||
|
When running in debug mode (i.e. using <code class="filename">-DDD</code>), BitBake
|
|||
|
produces output when it discovers something for which it cannot figure out
|
|||
|
dependencies.
|
|||
|
The Yocto Project team has currently not managed to cover those dependencies
|
|||
|
in detail and is aware of the need to fix this situation.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
Thus far, this section has limited discussion to the direct inputs into a task.
|
|||
|
Information based on direct inputs is referred to as the "basehash" in the
|
|||
|
code.
|
|||
|
However, there is still the question of a task's indirect inputs - the
|
|||
|
things that were already built and present in the Build Directory.
|
|||
|
The checksum (or signature) for a particular task needs to add the hashes
|
|||
|
of all the tasks on which the particular task depends.
|
|||
|
Choosing which dependencies to add is a policy decision.
|
|||
|
However, the effect is to generate a master checksum that combines the basehash
|
|||
|
and the hashes of the task's dependencies.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
At the code level, there are a variety of ways both the basehash and the
|
|||
|
dependent task hashes can be influenced.
|
|||
|
Within the BitBake configuration file, we can give BitBake some extra information
|
|||
|
to help it construct the basehash.
|
|||
|
The following statements effectively result in a list of global variable
|
|||
|
dependency excludes - variables never included in any checksum:
|
|||
|
</p>
|
|||
|
<pre class="literallayout">
|
|||
|
BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH"
|
|||
|
BB_HASHBASE_WHITELIST += "DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS"
|
|||
|
BB_HASHBASE_WHITELIST += "FILE_DIRNAME HOME LOGNAME SHELL TERM USER"
|
|||
|
BB_HASHBASE_WHITELIST += "FILESPATH USERNAME STAGING_DIR_HOST STAGING_DIR_TARGET"
|
|||
|
</pre>
|
|||
|
<p>
|
|||
|
The previous example actually excludes
|
|||
|
<a class="link" href="ref-variables-glos.html#var-WORKDIR" title="WORKDIR"><code class="filename">WORKDIR</code></a>
|
|||
|
since it is actually constructed as a path within
|
|||
|
<a class="link" href="ref-variables-glos.html#var-TMPDIR" title="TMPDIR"><code class="filename">TMPDIR</code></a>, which is on
|
|||
|
the whitelist.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
The rules for deciding which hashes of dependent tasks to include through
|
|||
|
dependency chains are more complex and are generally accomplished with a
|
|||
|
python function.
|
|||
|
The code in <code class="filename">meta/lib/oe/sstatesig.py</code> shows two examples
|
|||
|
of this and also illustrates how you can insert your own policy into the system
|
|||
|
if so desired.
|
|||
|
This file defines the two basic signature generators <code class="filename">OE-Core</code>
|
|||
|
uses: "OEBasic" and "OEBasicHash".
|
|||
|
By default, there is a dummy "noop" signature handler enabled in BitBake.
|
|||
|
This means that behavior is unchanged from previous versions.
|
|||
|
<code class="filename">OE-Core</code> uses the "OEBasic" signature handler by default
|
|||
|
through this setting in the <code class="filename">bitbake.conf</code> file:
|
|||
|
</p>
|
|||
|
<pre class="literallayout">
|
|||
|
BB_SIGNATURE_HANDLER ?= "OEBasic"
|
|||
|
</pre>
|
|||
|
<p>
|
|||
|
The "OEBasicHash" <code class="filename">BB_SIGNATURE_HANDLER</code> is the same as the
|
|||
|
"OEBasic" version but adds the task hash to the stamp files.
|
|||
|
This results in any metadata change that changes the task hash, automatically
|
|||
|
causing the task to be run again.
|
|||
|
This removes the need to bump <a class="link" href="ref-variables-glos.html#var-PR" title="PR"><code class="filename">PR</code></a>
|
|||
|
values and changes to metadata automatically ripple across the build.
|
|||
|
Currently, this behavior is not the default behavior for <code class="filename">OE-Core</code>
|
|||
|
but is the default in <code class="filename">poky</code>.
|
|||
|
</p>
|
|||
|
<p>
|
|||
|
It is also worth noting that the end result of these signature generators is to
|
|||
|
make some dependency and hash information available to the build.
|
|||
|
This information includes:
|
|||
|
</p>
|
|||
|
<pre class="literallayout">
|
|||
|
BB_BASEHASH_task-<taskname> - the base hashes for each task in the recipe
|
|||
|
BB_BASEHASH_<filename:taskname> - the base hashes for each dependent task
|
|||
|
BBHASHDEPS_<filename:taskname> - The task dependencies for each task
|
|||
|
BB_TASKHASH - the hash of the currently running task
|
|||
|
</pre>
|
|||
|
<p>
|
|||
|
</p>
|
|||
|
</div></body>
|
|||
|
</html>
|