From 35ec4d2a93af01521dd7f769ceb3a8105f7a243b Mon Sep 17 00:00:00 2001 From: Scott Rifenbark Date: Fri, 20 Sep 2013 12:00:56 -0700 Subject: [PATCH] dev-manual: New section on running QEMU tests. Added a new section called "Running and Writing Tests for a QEMU Image" to the "Common Tasks" chapter. This information is based on Stefan Stanacar's wiki page information. The section provides the same information as the wiki sans the log files. (From yocto-docs rev: ca2294821e34bbcd0afe7dd27421e8c6a828d700) Signed-off-by: Scott Rifenbark Signed-off-by: Richard Purdie --- .../dev-manual/dev-manual-common-tasks.xml | 414 ++++++++++++++++++ 1 file changed, 414 insertions(+) diff --git a/documentation/dev-manual/dev-manual-common-tasks.xml b/documentation/dev-manual/dev-manual-common-tasks.xml index 7f59c39f2d..2dbc3d14b8 100644 --- a/documentation/dev-manual/dev-manual-common-tasks.xml +++ b/documentation/dev-manual/dev-manual-common-tasks.xml @@ -4252,6 +4252,420 @@ +
+ Running and Writing Tests for a QEMU Image + + + The OpenEmbedded build system makes available a series of automated + tests for QEMU images. + These tests are commands that run on the target system over + ssh and are written in Python, thus making + use of the unittest module. + + + + This section describes how you set up the environment to use these + tests, run available tests, and write and add your own tests. + You can the same information in a different form and with example + log files of test runs on the + Image Tests + Wiki page. + + +
+ Enabling Tests + + + In order to run tests, you need to do the following: + + Ensure you run the test + automatically with no interaction and under + sudo: + To accomplish this, you must do one of the + following: + + Add + NOPASSWD for your user + in /etc/sudoers either for + ALL commands or just for + runqemu-ifup. + You must provide the full path as that can + change if you are using multiple clones of the + source repository. + + On some distributions, you also need to + comment out "Defaults requiretty" in + /etc/sudoers. + + Manually configure a tap interface + for your system. + Run as root the script in + scripts/runqemu-gen-tapdev, + which should generate a list of tap devices. + List generation is usually done in environments + similar to AutoBuilder. + + Set up the + DISPLAY variable: + You need to set this variable so that you have an X + server available (e.g. start + vncserver for a headless machine). + + Be sure your host's firewall + accepts incoming connections from + 192.168.7.0/24: + Some of the tests (in particular smart tests) start a + HTTP server on a random high number port, which is + used to serve files to the target. + The smart module serves + ${DEPLOY_DIR}/rpm so it can run + smart channel commands. That means your host's firewall + must accept incoming connections from 192.168.7.0/24, + which is the default class used for + tap0 devices by + runqemu. + Have your QEMU image built: + The QEMU image on which you want to run tests needs to + be built. + For example, the following command would build the + core-image-sato image when + MACHINE is set to "qemu". + + bitbake core-image-sato + + Globally inherit + testimage.class: + Edit your local.conf file as + follows: + + INHERIT += "testimage" + + + + + + If you built your QEMU image using rm_work, + most of the tests will fail with errors because they rely on + ${WORKDIR}/installed_pkgs.txt. + +
+ +
+ Running Tests + + + After setting up to run the tests, use BitBake to start + the standard suite of tests: + + bitbake core-image-sato -c testimage + + + + + Once you start the tests, the following happens: + + A copy of the root filesystem is written + to ${WORKDIR}/testimage. + + The image is booted under QEMU using the + standard runqemu script. + + A default timeout of 500 seconds occurs + to allow for the boot process to reach the login prompt. + You can change the timeout period by setting + TEST_QEMUBOOT_TIMEOUT + in the local.conf file. + + Once the boot process is reached and the + login prompt appears, the tests run. + The full boot log is written to + ${WORKDIR}/testimage/qemu_boot_log. + + Each test module loads in the order found + in TEST_SUITES. + You can find the full output of the commands run over + ssh in + ${WORKDIR}/testimgage/ssh_target_log. + + If no failures occur, the task running the + tests ends successfully. + You can find the output from the + unittest in the task log at + ${WORKDIR}/temp/log.do_testimage. + + + + + + All test files reside in + meta/lib/oeqa/runtime in the + Source Directory. + Tests (also referred to as modules) have a one-to-one + relationship with the filenames in runtime. + Modules can have multiple classes and test methods and are + usually grouped together by the area tested (e.g tests for + systemd reside in + meta/lib/oeqa/runtime/systemd.py). + + + + You can add tests to any layer provided you place them in the + proper area and you extend + BBPATH + in the local.conf file. + Be sure that tests reside in + <layer>/lib/oeqa/runtime. + + Be sure that module names do not collide with module names + provided by the Yocto Project. + + + + + You can change the set of tests run by appending or overriding + TEST_SUITES + variable in local.conf. + Each name in TEST_SUITES represents a + required test for the image. + Module skipping is not allowed even if a test is not suitable + for an image (e.g. running the rpm tests on + an image without rpm). + Appending "auto" to TEST_SUITES causes the + build system to try to run all tests that are suitable for the + image (i.e. the build system decides whether to run each test). + + + + The order you list tests in TEST_SUITES + is important. + The order influences test dependencies. + Consequently, tests that depend on other tests should be added + after the test on which they depend. + For example, since ssh depends on the + ping test, ssh + needs to come after ping in the list. + The test class provides no re-ordering or dependency handling. + + Each module can have multiple classes with multiple test + methods. + And, Python unittest rules apply. + + + + + The following list summarizes how to run the tests: + + Run the default set of tests simply by + running the following: + + bitbake <qemu_image> -c testimage + + The default tests for the image are defined + as: + + DEFAULT_TEST_SUITES_pn-<qemu_image> = "ping ssh df connman syslog xorg scp vnc date rpm smart dmesg" + + Add your own test to the list of the + by using the following: + + TEST_SUITES_append = " mytest" + + Run a specific list of tests as follows: + + TEST_SUITES = "test1 test2 test3" + + Remember, order is important. + Be sure to place a test that is dependent on another test + later in the order. + + +
+ +
+ Writing New Tests + + + As mentioned previously, all new test files need to be in the + proper place for the build system to find them. + New tests can go in the + Source Directory at + meta/lib/oeqa/runtime. + Alternatively, a layer can add its own tests in + <layer>/lib/oeqa/runtime as long + as you extend BBPATH. + Just remember that filenames need to map directory to test + (module) names and you not use module names that collide with + existing core tests. + + + + To create a new test, start by copying an existing module + (e.g. syslog.py or + gcc.py are good ones to use). + Test modules can use code from + meta/lib/oeqa/utils, which are helper + classes. + + + + Structure shell commands such that you rely on them and they + return a single code for success. + Be aware that sometimes you will need to parse the output. + See the df.py and + date.py modules for examples. + + + + You will notice that all test classes inherit + oeRuntimeTest, which is found in + meta/lib/oetest.py. + This base class offers some helper attributes, which are + described in the following sections: + + +
+ Class Methods + + + Class methods are as follows: + + hasPackage(pkg): + Returns "True" if pkg is in the + installed package list of the image, which is based + on + ${WORKDIR}/installed_pkgs.txt + that is generated during the + do.rootfs task. + + hasFeature(feature): + Returns "True" if the feature is in + IMAGE_FEATURES + or + DISTRO_FEATURES. + + restartTarget(params): + Restarts the QEMU image optionally passing + params to the + runqemu script's + qemuparams list (e.g "-m 1024" for + more memory). + + +
+ +
+ Class Attributes + + + Class attributes are as follows: + + pscmd: + Equals "ps -ef" if procps is + installed in the image. + Otherwise, pscmd equals + "ps" (busybox). + + tc: + The called text context, which gives access to the + following attributes: + + d: + The BitBake data store, which allows you to + use stuff such as + oeRuntimeTest.tc.d.getVar("VIRTUAL-RUNTIME_init_manager"). + + testslist and testsrequired: + Used internally. + The tests do not need these. + + filesdir: + The absolute path to + meta/lib/oeqa/runtime/files, + which contains helper files for tests meant + for copying on the target such as small + files written in C for compilation. + + qemu: + Provides access to the + QemuRunner object, + which is the class that boots the image. + The qemu attribute + provides the following useful attributes: + + ip: + The machine's IP address. + + host_ip: + The host IP address, which is only + used by smart tests. + + + target: + The SSHControl object, + which is used for running the following + commands on the image: + + host: + Used internally. + The tests do not use this command. + + timeout: + A global timeout for commands run on + the target for the instance of a + test. + The default is 300 seconds. + + run(cmd, timeout=None): + The single, most used method. + This command is a wrapper for: + ssh root@host "cmd". + The command returns a tuple: + (status, output), which are what + their names imply - the return code + of 'cmd' and whatever output + it produces. + The optional timeout argument + represents the number of seconds the + test should wait for 'cmd' to + return. + If the argument is "None", the + test uses the default instance's + timeout period, which is 300 + seconds. + If the argument is "0", the test + runs until the command returns. + + copy_to(localpath, remotepath): + scp localpath root@ip:remotepath. + + copy_from(remotepath, localpath): + scp root@host:remotepath localpath. + + + + + +
+ +
+ Instance Attributes + + + A single instance attribute exists, which is + target. + The target instance attribute is + identical to the class attribute of the same name, which + is described in the previous section. + This attribute exists as both an instance and class + attribute so tests can use + self.target.run(cmd) in instance + methods instead of + oeRuntimeTest.tc.target.run(cmd). + +
+
+
+
Debugging With the GNU Project Debugger (GDB) Remotely