ref-manual, dev-manual: Applied review comments for test section

The section and variables for performing automated runtime tests
on images was reviewed by Paul Eggleton.  The suggested changes
were made.

(From yocto-docs rev: c2f84ea3c162892e4da3df30fb833f88bab3d3cc)

Signed-off-by: Scott Rifenbark <scott.m.rifenbark@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This commit is contained in:
Scott Rifenbark 2013-09-23 09:19:53 -07:00 committed by Richard Purdie
parent 56c4de93b9
commit e44538f4e4
3 changed files with 109 additions and 81 deletions

View File

@ -4252,24 +4252,22 @@
</section>
</section>
<section id="running-and-writing-tests-for-a-qemu-image">
<title>Running and Writing Tests for a QEMU Image</title>
<section id="performing-automated-runtime-testing">
<title>Performing Automated Runtime Testing</title>
<para>
The OpenEmbedded build system makes available a series of automated
tests for QEMU images.
These tests are commands that run on the target system over
<filename>ssh</filename> and are written in Python, thus making
use of the <filename>unittest</filename> module.
</para>
<para>
tests for images.
<note>
The current release of Yocto Project supports these tests
for QEMU images only.
</note>
These tests are written in Python making use of the
<filename>unittest</filename> module, and the majority of them
run commands on the target system over
<filename>ssh</filename>.
This section describes how you set up the environment to use these
tests, run available tests, and write and add your own tests.
You can the same information in a different form and with example
log files of test runs on the
<ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
Wiki page.
</para>
<section id="qemu-image-enabling-tests">
@ -4278,9 +4276,8 @@
<para>
In order to run tests, you need to do the following:
<itemizedlist>
<listitem><para><emphasis>Ensure you run the test
automatically with no interaction and under
<filename>sudo</filename>:</emphasis>
<listitem><para><emphasis>Set up to avoid interaction
with <filename>sudo</filename> for networking:</emphasis>
To accomplish this, you must do one of the
following:
<itemizedlist>
@ -4300,12 +4297,13 @@
<listitem><para>Manually configure a tap interface
for your system.</para></listitem>
<listitem><para>Run as root the script in
<filename>scripts/runqemu-gen-tapdev</filename>,
<filename>scripts/runqemu-gen-tapdevs</filename>,
which should generate a list of tap devices.
List generation is usually done in environments
similar to AutoBuilder.</para></listitem>
This is the option typically chosen for
Autobuilder-type environments.
</para></listitem>
</itemizedlist></para></listitem>
<listitem><para><emphasis>Set up the
<listitem><para><emphasis>Set the
<filename>DISPLAY</filename> variable:</emphasis>
You need to set this variable so that you have an X
server available (e.g. start
@ -4321,11 +4319,10 @@
<filename>${DEPLOY_DIR}/rpm</filename> so it can run
smart channel commands. That means your host's firewall
must accept incoming connections from 192.168.7.0/24,
which is the default class used for
<filename>tap0</filename> devices by
<filename>runqemu</filename>.</para></listitem>
<listitem><para><emphasis>Have your QEMU image built:</emphasis>
The QEMU image on which you want to run tests needs to
which is the default IP range used for tap devices
by <filename>runqemu</filename>.</para></listitem>
<listitem><para><emphasis>Have your image built:</emphasis>
The image on which you want to run tests needs to
be built.
For example, the following command would build the
<filename>core-image-sato</filename> image when
@ -4333,18 +4330,34 @@
<literallayout class='monospaced'>
bitbake core-image-sato
</literallayout></para></listitem>
<listitem><para><emphasis>Globally inherit
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink></emphasis>
Edit your <filename>local.conf</filename> file as
<listitem><para><emphasis>Run the tests:</emphasis>
You can start the tests automatically or manually.
To have the tests start automatically after the
OpenEmbedded build system successfully creates a QEMU
image, set the
<ulink url='&YOCTO_DOCS_REF_URL;#var-TEST_IMAGE'><filename>TEST_IMAGE</filename></ulink>
variable to "1" in your
<filename>local.conf</filename> file in the
<ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
To manually run the tests, globally inherit
<ulink url='&YOCTO_DOCS_REF_URL;#ref-classes-testimage'><filename>testimage.class</filename>:</ulink>
by editing your <filename>local.conf</filename> file as
follows:
<literallayout class='monospaced'>
INHERIT += "testimage"
</literallayout>
After editing your
<filename>local.conf</filename> file, use BitBake to
run the tests:
<literallayout class='monospaced'>
bitbake -c testimage &lt;qemu-image&gt;
</literallayout></para></listitem>
</itemizedlist>
</para>
<note>
If you built your QEMU image using <filename>rm_work</filename>,
Regardless of how you initiate the tests, if you built your
image using <filename>rm_work</filename>,
most of the tests will fail with errors because they rely on
<filename>${WORKDIR}/installed_pkgs.txt</filename>.
</note>
@ -4355,14 +4368,16 @@
<para>
After setting up to run the tests, use BitBake to start
the standard suite of tests:
the standard suite of tests manually or, if you are running
them automatically, build your image:
<literallayout class='monospaced'>
bitbake core-image-sato -c testimage
bitbake core-image-sato
</literallayout>
</para>
<para>
Once you start the tests, the following happens:
Once the tests start, the following happens:
<itemizedlist>
<listitem><para>A copy of the root filesystem is written
to <filename>${WORKDIR}/testimage</filename>.
@ -4400,10 +4415,10 @@
All test files reside in
<filename>meta/lib/oeqa/runtime</filename> in the
<link linkend='source-directory'>Source Directory</link>.
Tests (also referred to as modules) have a one-to-one
relationship with the filenames in <filename>runtime</filename>.
Modules can have multiple classes and test methods and are
usually grouped together by the area tested (e.g tests for
A test name (sometimes referred to as a module) maps to a
method name within a class within a module.
Tests can be used within multiple classes and the tests
are usually grouped together by the area tested (e.g tests for
<filename>systemd</filename> reside in
<filename>meta/lib/oeqa/runtime/systemd.py</filename>).
</para>
@ -4412,12 +4427,13 @@
You can add tests to any layer provided you place them in the
proper area and you extend
<ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
in the <filename>local.conf</filename> file.
in the <filename>local.conf</filename> file as normal.
Be sure that tests reside in
<filename>&lt;layer&gt;/lib/oeqa/runtime</filename>.
<note>
Be sure that module names do not collide with module names
provided by the Yocto Project.
used in the default set of test modules in
<filename>meta/lib/oeqa/runtime</filename>.
</note>
</para>
@ -4458,12 +4474,12 @@
<listitem><para>Run the default set of tests simply by
running the following:
<literallayout class='monospaced'>
bitbake &lt;qemu_image&gt; -c testimage
bitbake &lt;image&gt; -c testimage
</literallayout></para></listitem>
<listitem><para>The default tests for the image are defined
as:
<literallayout class='monospaced'>
DEFAULT_TEST_SUITES_pn-&lt;qemu_image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"
DEFAULT_TEST_SUITES_pn-&lt;image&gt; = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg"
</literallayout></para></listitem>
<listitem><para>Add your own test to the list of the
by using the following:
@ -4487,15 +4503,16 @@
<para>
As mentioned previously, all new test files need to be in the
proper place for the build system to find them.
New tests can go in the
<link linkend='source-directory'>Source Directory</link> at
<filename>meta/lib/oeqa/runtime</filename>.
Alternatively, a layer can add its own tests in
<filename>&lt;layer&gt;/lib/oeqa/runtime</filename> as long
as you extend <filename>BBPATH</filename>.
Just remember that filenames need to map directory to test
(module) names and you not use module names that collide with
existing core tests.
New tests for additional functionality outside of the core
should be added to the layer that adds the functionality, in
<filename>&lt;layer&gt;/lib/oeqa/runtime</filename> (as
long as
<ulink url='&YOCTO_DOCS_REF_URL;#var-BBPATH'><filename>BBPATH</filename></ulink>
is extended in the layer's
<filename>layer.conf</filename> file as normal.
Just remember that filenames need to map directly to test
(module) names and that you do not use module names that
collide with existing core tests.
</para>
<para>

View File

@ -940,8 +940,12 @@
<para>
You can use this class to enable running a series of automated tests
for QEMU images.
The class handles loading the tests and starting the QEMU image.
for images.
The class handles loading the tests and starting the image.
<note>
The current release of Yocto Project supports these tests
for QEMU images only.
</note>
</para>
<para>
@ -950,15 +954,13 @@
The tests are commands that run on the target system over
<filename>ssh</filename>.
they are written in Python and make use of the
<filename>unittest modulebuild</filename> software.
<filename>unittest</filename> module.
</para>
<para>
For information on how to enable, run, and create new tests, see the
"<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
section and the
<ulink url='https://wiki.yoctoproject.org/wiki/Image_tests'>Image Tests</ulink>
Wiki page.
"<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section.
</para>
</section>

View File

@ -5220,21 +5220,27 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<glossentry id='var-TEST_IMAGE'><glossterm>TEST_IMAGE</glossterm>
<glossdef>
<para>
Automatically runs the series of automated tests for QEMU
images when a QEMU image is successfully built.
These tests are commands that run on the target system over
<filename>ssh</filename> and are written in Python.
Automatically runs the series of automated tests for
images when an image is successfully built.
<note>
The current release of Yocto Project supports these tests
for QEMU images only.
</note>
These tests are written in Python making use of the
<filename>unittest</filename> module, and the majority of
them run commands on the target system over
<filename>ssh</filename>.
You can set this variable to "1" in your
<filename>local.conf</filename> file in the
<ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>
to have the OpenEmbedded build system automatically run
these tests after a QEMU image successfully builds:
these tests after an image successfully builds:
<literallayout class='monospaced'>
TEST_IMAGE = "1"
</literallayout>
For more information on enabling, running, and writing
these tests, see the
"<ulink url='&YOCTO_DOCS_DEV_URL;#running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
"<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section in the Yocto Project Development Manual and the
"<link linkend='ref-classes-testimage'><filename>testimage.bbclass</filename></link>"
section.
@ -5245,24 +5251,18 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<glossentry id='var-TEST_QEMUBOOT_TIMEOUT'><glossterm>TEST_QEMUBOOT_TIMEOUT</glossterm>
<glossdef>
<para>
The time in seconds allowed for a QEMU image to boot before
tests begin to run against the image using the following
command:
<literallayout class='monospaced'>
bitbake &lt;qemu_image&gt; -c testimage
</literallayout>
</para>
<para>
The time in seconds allowed for an image to boot before
automated runtime tests begin to run against an
image.
The default timeout period to allow the boot process to
reach the login prompt is 500 seconds.
You can edit this default value in the
You can specify a different value in the
<filename>local.conf</filename> file.
</para>
<para>
For more information on testing QEMU images, see the
"<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
For more information on testing images, see the
"<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section in the Yocto Project Development Manual.
</para>
</glossdef>
@ -5272,15 +5272,16 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<glossdef>
<para>
An ordered list of tests (modules) to run against
a QEMU image using the following command:
<literallayout class='monospaced'>
bitbake &lt;qemu_image&gt; -c testimage
</literallayout>
an image when performing automated runtime testing.
</para>
<para>
The Yocto Project provides a core set of tests that can
be used against QEMU images.
The OpenEmbedded build system provides a core set of tests
that can be used against images.
<note>
The current release of Yocto Project supports these
tests for QEMU images only.
</note>
Tests include <filename>ping</filename>,
<filename>ssh</filename>, <filename>df</filename> among
others.
@ -5289,6 +5290,14 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
<literallayout class='monospaced'>
TEST_SUITES_append = " mytest"
</literallayout>
Alternatively, you can provide the "auto" option to
have all applicable tests run against the image.
<literallayout class='monospaced'>
TEST_SUITES_append = " auto"
</literallayout>
Using this option causes the build system to automatically
run tests that are applicable to the image.
Tests that are not applicable are skipped.
</para>
<para>
@ -5307,8 +5316,8 @@ recipes-graphics/xorg-font/font-alias_1.0.3.bb:PR = "${INC_PR}.3"
</para>
<para>
For more information on testing QEMU images, see the
"<ulink url='running-and-writing-tests-for-a-qemu-image'>Running and Writing Tests for a QEMU Image</ulink>"
For more information on testing images, see the
"<ulink url='&YOCTO_DOCS_DEV_URL;#performing-automated-runtime-testing'>Performing Automated Runtime Testing</ulink>"
section in the Yocto Project Development Manual.
</para>
</glossdef>