documentation/poky-ref-manual: New chapter introduced
Long-term strategy for the YP Reference Manual is that it contains reference material and not "how-to-information". A step in this direction is to isolate any discussions on components and other areas of YP that need talked about. So to start with, I have created a new chapter for now named "Technical Details" that so far has a discussion of some components and shared state. This is a step in the direction of making this manual a reference manual and not a "how to" manual. Changes included removing redundant material from the 'usingpoky.xml' chapter and also adding the new chapter 'technical-details' into the 'poky-ref-manual.xml' file used for the make. (From yocto-docs rev: a01477f787768230bc25da2d094326922be23dd4) Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
parent
e7cfb3b469
commit
5dd34a717e
|
@ -92,6 +92,8 @@
|
|||
|
||||
<xi:include href="extendpoky.xml"/>
|
||||
|
||||
<xi:include href="technical-details.xml"/>
|
||||
|
||||
<xi:include href="../bsp-guide/bsp.xml"/>
|
||||
|
||||
<xi:include href="development.xml"/>
|
||||
|
|
|
@ -0,0 +1,229 @@
|
|||
<!DOCTYPE chapter PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
|
||||
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
|
||||
<chapter id='technical-details'>
|
||||
<title>Technical Details</title>
|
||||
|
||||
<para>
|
||||
This chapter provides technical details for various parts of the Yocto Project.
|
||||
Currently, topics include Yocto Project components and shared state (sstate) cache.
|
||||
</para>
|
||||
|
||||
<section id='usingpoky-components'>
|
||||
<title>Yocto Project Components</title>
|
||||
|
||||
<para>
|
||||
The BitBake task executor together with various types of configuration files form the
|
||||
Yocto Project core.
|
||||
This section overviews the BitBake task executor and the
|
||||
configuration files by describing what they are used for and how they interact.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
BitBake handles the parsing and execution of the data files.
|
||||
The data itself is of various types:
|
||||
<itemizedlist>
|
||||
<listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
|
||||
pieces of software</para></listitem>
|
||||
<listitem><para><emphasis>Class Data:</emphasis> An abstraction of common build
|
||||
information (e.g. how to build a Linux kernel).</para></listitem>
|
||||
<listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
|
||||
policy decisions, etc.
|
||||
Configuration data acts as the glue to bind everything together.</para></listitem>
|
||||
</itemizedlist>
|
||||
For more information on data, see the
|
||||
<ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html#yocto-project-terms'>
|
||||
Yocto Project Terms</ulink> section in
|
||||
<ulink url='http://www.yoctoproject.org/docs/latest/dev-manual/dev-manual.html'>
|
||||
The Yocto Project Development Manual</ulink>.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
BitBake knows how to combine multiple data sources together and refers to each data source
|
||||
as a <link linkend='usingpoky-changes-layers'>'layer'</link>.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Following are some brief details on these core components.
|
||||
For more detailed information on these components see the
|
||||
<link linkend='ref-structure'>'Reference: Directory Structure'</link>
|
||||
appendix.
|
||||
</para>
|
||||
|
||||
<section id='usingpoky-components-bitbake'>
|
||||
<title>BitBake</title>
|
||||
|
||||
<para>
|
||||
BitBake is the tool at the heart of the Yocto Project and is responsible
|
||||
for parsing the metadata, generating a list of tasks from it,
|
||||
and then executing those tasks.
|
||||
To see a list of the options BitBake supports, use the following help command:
|
||||
<literallayout class='monospaced'>
|
||||
$ bitbake --help
|
||||
</literallayout>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The most common usage for BitBake is <filename>bitbake <packagename></filename>, where
|
||||
<filename>packagename</filename> is the name of the package you want to build
|
||||
(referred to as the "target" in this manual).
|
||||
The target often equates to the first part of a <filename>.bb</filename> filename.
|
||||
So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
|
||||
might type the following:
|
||||
<literallayout class='monospaced'>
|
||||
$ bitbake matchbox-desktop
|
||||
</literallayout>
|
||||
Several different versions of <filename>matchbox-desktop</filename> might exist.
|
||||
BitBake chooses the one selected by the distribution configuration.
|
||||
You can get more details about how BitBake chooses between different
|
||||
target versions and providers in the
|
||||
<link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
BitBake also tries to execute any dependent tasks first.
|
||||
So for example, before building <filename>matchbox-desktop</filename>, BitBake
|
||||
would build a cross compiler and <filename>eglibc</filename> if they had not already
|
||||
been built.
|
||||
<note>This release of the Yocto Project does not support the <filename>glibc</filename>
|
||||
GNU version of the Unix standard C library. By default, the Yocto Project builds with
|
||||
<filename>eglibc</filename>.</note>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
A useful BitBake option to consider is the <filename>-k</filename> or
|
||||
<filename>--continue</filename> option.
|
||||
This option instructs BitBake to try and continue processing the job as much
|
||||
as possible even after encountering an error.
|
||||
When an error occurs, the target that
|
||||
failed and those that depend on it cannot be remade.
|
||||
However, when you use this option other dependencies can still be processed.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='usingpoky-components-metadata'>
|
||||
<title>Metadata (Recipes)</title>
|
||||
|
||||
<para>
|
||||
The <filename>.bb</filename> files are usually referred to as "recipes."
|
||||
In general, a recipe contains information about a single piece of software.
|
||||
The information includes the location from which to download the source patches
|
||||
(if any are needed), which special configuration options to apply,
|
||||
how to compile the source files, and how to package the compiled output.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The term "package" can also be used to describe recipes.
|
||||
However, since the same word is used for the packaged output from the Yocto
|
||||
Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
|
||||
this document avoids using the term "package" to refer to recipes.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='usingpoky-components-classes'>
|
||||
<title>Classes</title>
|
||||
|
||||
<para>
|
||||
Class files (<filename>.bbclass</filename>) contain information that is useful to share
|
||||
between metadata files.
|
||||
An example is the Autotools class, which contains
|
||||
common settings for any application that Autotools uses.
|
||||
The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
|
||||
about common classes and how to use them.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='usingpoky-components-configuration'>
|
||||
<title>Configuration</title>
|
||||
|
||||
<para>
|
||||
The configuration files (<filename>.conf</filename>) define various configuration variables
|
||||
that govern the Yocto Project build process.
|
||||
These files fall into several areas that define machine configuration options,
|
||||
distribution configuration options, compiler tuning options, general common configuration
|
||||
options and user configuration options (<filename>local.conf</filename>, which is found
|
||||
in the Yocto Project files build directory).
|
||||
</para>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section id="considering-shared-state-cache">
|
||||
<title>Considering Shared State Cache</title>
|
||||
|
||||
<para>
|
||||
By design, the Yocto Project builds everything from scratch unless it can determine that
|
||||
a given task's inputs have not changed.
|
||||
While building from scratch ensures that everything is current, it does also
|
||||
mean that a lot of time could be spent rebuiding things that don't necessarily need built.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The Yocto Project build process uses a shared state caching scheme to avoid having to
|
||||
rebuild software when it is not necessary.
|
||||
Because the build time for a Yocto image can be significant, it is helpful to try and
|
||||
determine what really needs built and what can be skipped given a particular project's
|
||||
build process.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The scheme that the Yocto Project uses involves checksum generation and comparison for
|
||||
a task's inputs.
|
||||
The scheme also employs an area of memory called the shared state cache that is
|
||||
pointed to by the <filename>SSTATE_DIR</filename> variable.
|
||||
This area contains task output generated from a previous build.
|
||||
If a given task's checksum matches the checksum of a previous build for the same
|
||||
task, the build process uses the state of the cache rather than rerunning that
|
||||
task.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The previous paragraph is a simplistic explanation of how the build process
|
||||
uses checksums and shared state memory cache to avoide building tasks that
|
||||
don't need built.
|
||||
If you want a bit more explanation on the topic,
|
||||
see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
|
||||
State - What does it mean and why should I care?</ulink>" from the Yocto
|
||||
Project discussion archives.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
As with all schemes, this one has some drawbacks.
|
||||
It is possible that you could make implicit changes that are not factored into the checksum
|
||||
calculation, but do affect a task's output.
|
||||
A good example is perhaps when a tool changes its output.
|
||||
Let's say that the output of <filename>rpmdeps</filename> needed to change.
|
||||
The result of the change should be that all the "package", "package_write_rpm",
|
||||
and "package_deploy-rpm" sstate-cache items would become invalid.
|
||||
But, because this is a change that is external to the code and therefore implicit,
|
||||
the associated sstate-cache items do not become invalidated.
|
||||
In this case, the build process would use the cache items rather than running the
|
||||
task again.
|
||||
Obviously, these types of implicit changes can cause problems.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
To avoid these problems during the build, you need to understand the effects of any
|
||||
change you make.
|
||||
Note that any changes you make directly to a function automatically are factored into
|
||||
the checksum calculation and thus, will invalidate the associated area of sstate cache.
|
||||
You need to be aware of any implicit changes that are not obvious changes to the
|
||||
code and could affect the output of a given task.
|
||||
Once you are aware of such a change, you can take steps to invalidate the cache
|
||||
and force the task to run.
|
||||
The step to take is as simple as changing a function's comments in the source code.
|
||||
For example, to invalidate package sstate files, change the comment statments
|
||||
of <filename>do_package</filename> or one of the functions it calls.
|
||||
The change is purely cosmetic, but it causes the checksum to be recalculated and
|
||||
forces the task to be run again.
|
||||
</para>
|
||||
|
||||
<note>
|
||||
For an example of a commit that makes a cosmetic change to invalidate an sstate,
|
||||
see this
|
||||
<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
|
||||
</note>
|
||||
</section>
|
||||
|
||||
</chapter>
|
||||
<!--
|
||||
vim: expandtab tw=80 ts=4
|
||||
-->
|
|
@ -4,150 +4,9 @@
|
|||
<title>Using the Yocto Project</title>
|
||||
|
||||
<para>
|
||||
This section gives an overview of the components that make up the Yocto Project
|
||||
followed by information about Yocto Project builds and dealing with any
|
||||
problems that might arise.
|
||||
This section describes the common usage for the Yocto Project.
|
||||
</para>
|
||||
|
||||
<section id='usingpoky-components'>
|
||||
<title>Yocto Project Components</title>
|
||||
|
||||
<para>
|
||||
The BitBake task executor together with various types of configuration files form the
|
||||
Yocto Project core.
|
||||
This section overviews the BitBake task executor and the
|
||||
configuration files by describing what they are used for and they they interact.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
BitBake handles the parsing and execution of the data files.
|
||||
The data itself is of various types:
|
||||
<itemizedlist>
|
||||
<listitem><para><emphasis>Recipes:</emphasis> Provides details about particular
|
||||
pieces of software</para></listitem>
|
||||
<listitem><para><emphasis>Class Data:</emphasis> An abstraction of common build
|
||||
information (e.g. how to build a Linux kernel).</para></listitem>
|
||||
<listitem><para><emphasis>Configuration Data:</emphasis> Defines machine-specific settings,
|
||||
policy decisions, etc.
|
||||
Configuration data acts a the glue to bind everything together.</para></listitem>
|
||||
</itemizedlist>
|
||||
For more information on data, see the
|
||||
<ulink url='http://www.yoctoproject.org/docs/1.1/dev-manual/dev-manual.html#yocto-project-terms'>
|
||||
Yocto Project Terms</ulink> section in
|
||||
<ulink url='http://www.yoctoproject.org/docs/1.1/dev-manual/dev-manual.html'>
|
||||
The Yocto Project Development Manual</ulink>.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
BitBake knows how to combine multiple data sources together and refers to each data source
|
||||
as a <link linkend='usingpoky-changes-layers'>'layer'</link>.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
Following are some brief details on these core components.
|
||||
For more detailed information on these components see the
|
||||
<link linkend='ref-structure'>'Reference: Directory Structure'</link>
|
||||
appendix.
|
||||
</para>
|
||||
|
||||
<section id='usingpoky-components-bitbake'>
|
||||
<title>BitBake</title>
|
||||
|
||||
<para>
|
||||
BitBake is the tool at the heart of the Yocto Project and is responsible
|
||||
for parsing the metadata, generating a list of tasks from it,
|
||||
and then executing those tasks.
|
||||
To see a list of the options BitBake supports, use the following help command:
|
||||
<literallayout class='monospaced'>
|
||||
$ bitbake --help
|
||||
</literallayout>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The most common usage for BitBake is <filename>bitbake <packagename></filename>, where
|
||||
<filename>packagename</filename> is the name of the package you want to build
|
||||
(referred to as the "target" in this manual).
|
||||
The target often equates to the first part of a <filename>.bb</filename> filename.
|
||||
So, to run the <filename>matchbox-desktop_1.2.3.bb</filename> file, you
|
||||
might type the following:
|
||||
<literallayout class='monospaced'>
|
||||
$ bitbake matchbox-desktop
|
||||
</literallayout>
|
||||
Several different versions of <filename>matchbox-desktop</filename> might exist.
|
||||
BitBake chooses the one selected by the distribution configuration.
|
||||
You can get more details about how BitBake chooses between different
|
||||
target versions and providers in the
|
||||
<link linkend='ref-bitbake-providers'>Preferences and Providers</link> section.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
BitBake also tries to execute any dependent tasks first.
|
||||
So for example, before building <filename>matchbox-desktop</filename>, BitBake
|
||||
would build a cross compiler and <filename>eglibc</filename> if they had not already
|
||||
been built.
|
||||
<note>This release of the Yocto Project does not support the <filename>glibc</filename>
|
||||
GNU version of the Unix standard C library. By default, the Yocto Project builds with
|
||||
<filename>eglibc</filename>.</note>
|
||||
</para>
|
||||
|
||||
<para>
|
||||
A useful BitBake option to consider is the <filename>-k</filename> or
|
||||
<filename>--continue</filename> option.
|
||||
This option instructs BitBake to try and continue processing the job as much
|
||||
as possible even after encountering an error.
|
||||
When an error occurs, the target that
|
||||
failed and those that depend on it cannot be remade.
|
||||
However, when you use this option other dependencies can still be processed.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='usingpoky-components-metadata'>
|
||||
<title>Metadata (Recipes)</title>
|
||||
|
||||
<para>
|
||||
The <filename>.bb</filename> files are usually referred to as "recipes."
|
||||
In general, a recipe contains information about a single piece of software.
|
||||
The information includes the location from which to download the source patches
|
||||
(if any are needed), which special configuration options to apply,
|
||||
how to compile the source files, and how to package the compiled output.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The term "package" can also be used to describe recipes.
|
||||
However, since the same word is used for the packaged output from the Yocto
|
||||
Project (i.e. <filename>.ipk</filename> or <filename>.deb</filename> files),
|
||||
this document avoids using the term "package" to refer to recipes.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='usingpoky-components-classes'>
|
||||
<title>Classes</title>
|
||||
|
||||
<para>
|
||||
Class files (<filename>.bbclass</filename>) contain information that is useful to share
|
||||
between metadata files.
|
||||
An example is the Autotools class, which contains
|
||||
common settings for any application that Autotools uses.
|
||||
The <link linkend='ref-classes'>Reference: Classes</link> appendix provides details
|
||||
about common classes and how to use them.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id='usingpoky-components-configuration'>
|
||||
<title>Configuration</title>
|
||||
|
||||
<para>
|
||||
The configuration files (<filename>.conf</filename>) define various configuration variables
|
||||
that govern the Yocto Project build process.
|
||||
These files fall into several areas that define machine configuration options,
|
||||
distribution configuration options, compiler tuning options, general common configuration
|
||||
options and user configuration options (<filename>local.conf</filename>, which is found
|
||||
in the Yocto Project files build directory).
|
||||
</para>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
|
||||
<section id='usingpoky-build'>
|
||||
<title>Running a Build</title>
|
||||
|
||||
|
@ -221,83 +80,6 @@
|
|||
due to some components not being rebuilt.
|
||||
</para>
|
||||
</section>
|
||||
|
||||
<section id="considering-shared-state-cache">
|
||||
<title>Considering Shared State Cache</title>
|
||||
|
||||
<para>
|
||||
By design, the Yocto Project builds everything from scratch unless it can determine that
|
||||
a given task's inputs have not changed.
|
||||
While building from scratch ensures that everything is current, it does also
|
||||
mean that a lot of time could be spent rebuiding things that don't necessarily need built.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The Yocto Project build process uses a shared state caching scheme to avoid having to
|
||||
rebuild software when it is not necessary.
|
||||
Because the build time for a Yocto image can be significant, it is helpful to try and
|
||||
determine what really needs built and what can be skipped given a particular project's
|
||||
build process.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The scheme that the Yocto Project uses involves checksum generation and comparison for
|
||||
a task's inputs.
|
||||
The scheme also employs an area of memory called the shared state cache that is
|
||||
pointed to by the <filename>SSTATE_DIR</filename> variable.
|
||||
This area contains task output generated from a previous build.
|
||||
If a given task's checksum matches the checksum of a previous build for the same
|
||||
task, the build process uses the state of the cache rather than rerunning that
|
||||
task.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
The previous paragraph is a simplistic explanation of how the build process
|
||||
uses checksums and shared state memory cache to avoide building tasks that
|
||||
don't need built.
|
||||
If you want a bit more explanation on the topic,
|
||||
see "<ulink url='https://lists.yoctoproject.org/pipermail/yocto/2011-March/003366.html'>Shared
|
||||
State - What does it mean and why should I care?</ulink>" from the Yocto
|
||||
Project discussion archives.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
As with all schemes, this one has some drawbacks.
|
||||
It is possible that you could make implicit changes that are not factored into the checksum
|
||||
calculation, but do affect a task's output.
|
||||
A good example is perhaps when a tool changes its output.
|
||||
Let's say that the output of <filename>rpmdeps</filename> needed to change.
|
||||
The result of the change should be that all the "package", "package_write_rpm",
|
||||
and "package_deploy-rpm" sstate-cache items would become invalid.
|
||||
But, because this is a change that is external to the code and therefore implicit,
|
||||
the associated sstate-cache items do not become invalidated.
|
||||
In this case, the build process would use the cache items rather than running the
|
||||
task again.
|
||||
Obviously, these types of implicit changes can cause problems.
|
||||
</para>
|
||||
|
||||
<para>
|
||||
To avoid these problems during the build, you need to understand the effects of any
|
||||
change you make.
|
||||
Note that any changes you make directly to a function automatically are factored into
|
||||
the checksum calculation and thus, will invalidate the associated area of sstate cache.
|
||||
You need to be aware of any implicit changes that are not obvious changes to the
|
||||
code and could affect the output of a given task.
|
||||
Once you are aware of such a change, you can take steps to invalidate the cache
|
||||
and force the task to run.
|
||||
The step to take is as simple as changing a function's comments in the source code.
|
||||
For example, to invalidate package sstate files, change the comment statments
|
||||
of <filename>do_package</filename> or one of the functions it calls.
|
||||
The change is purely cosmetic, but it causes the checksum to be recalculated and
|
||||
forces the task to be run again.
|
||||
</para>
|
||||
|
||||
<note>
|
||||
For an example of a commit that makes a cosmetic change to invalidate an sstate,
|
||||
see this
|
||||
<ulink url='http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/package.bbclass?id=737f8bbb4f27b4837047cb9b4fbfe01dfde36d54'>commit</ulink>.
|
||||
</note>
|
||||
</section>
|
||||
</section>
|
||||
|
||||
<section id='usingpoky-install'>
|
||||
|
|
Loading…
Reference in New Issue