Commit Graph

2537 Commits

Author SHA1 Message Date
Patrick Ohly 1fb309447f runqemu: support virtio drive type
Setting QB_DRIVE_TYPE=/dev/vd selects virtio without triggering any
warnings. Previously, that was only possible by setting an unknown
value and relying on the fallback to virtio, which caused some
warnings to be printed.

(From OE-Core rev: 5cbf102662dde6f706a19fa133cfd1e7475eb8c1)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-29 11:17:22 +01:00
Kristian Amlie e3e78df1b2 wic: Fix typo in help screen.
This was overlooked when f6a064d969 was merged.

(From OE-Core rev: 29a209822488ab687abdb1ceffdd9c7af5b3db68)

Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-29 11:17:22 +01:00
Robert Yang 52493384aa oe-find-native-sysroot: don't clear OECORE_NATIVE_SYSROOT
The OECORE_NATIVE_SYSROOT may come from environment vars, so don't clear
it, otherwise it broke runqemu-extract-sdk on sdk.

[YOCTO #11409]

(From OE-Core rev: 91faa9132821db4e7010b79943b2cfb804d6f4c5)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-29 11:17:22 +01:00
Martin Jansa 9784ad79a4 runqemu-gen-tapdevs: fix runqemu-ifup script call
The tunctl binary is here:

OE @ /OE/openembedded-core # find /OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/
/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/
/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/sysroot-providers
/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/sysroot-providers/qemu-helper-native
/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/usr
/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/usr/bin
/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/usr/bin/tunctl

But the script still complains that it cannot find tunctl:

OE @ /OE/openembedded-core # ./scripts/runqemu-gen-tapdevs 1026 1026 4 /OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/
Note: Destroying pre-existing tap interface tap0...
TUNSETIFF: Device or resource busy
Creating 4 tap devices for UID: 1026 GID: 1026...
Creating tap0
Error running tunctl: Error: Unable to find tunctl binary in '/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/', please bitbake qemu-helper-native

The message is actually from runqemu-ifup, which is called from runqemu-gen-tapdevs as:
++ ./scripts/runqemu-ifup 1026 1026 /OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/

But runqemu-ifup expects 3rd parameter to be STAGING_BINDIR_NATIVE directly not just SYSROOT dir
STAGING_BINDIR_NATIVE=$3
because tunctl is then used as:
TUNCTL=$STAGING_BINDIR_NATIVE/tunctl

It looks like it got broken by:
commit cc5513bf7a6114e14bb307acb88a44e9cf0aed8a
Author: Ed Bartosh <ed.bartosh@linux.intel.com>
Date:   Wed Apr 12 23:40:59 2017 +0300

    runqemu: use bindir_native property to run ifup/down scripts

    Used self.bindir_native to point out to the native sysroot
    when running runqemu-ifup and runqemu-ifdown scripts.

    [YOCTO #11266]
    [YOCTO #11193]

    Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Seemingly obvious fix would be to call runqemu-gen-tapdevs with path to STAGING_BINDIR_NATIVE in 4th parameter as well, but that won't work, because runqemu-gen-tapdevs checks for TUNCTL=$SYSROOT/usr/bin/tunctl

OE @ /OE/openembedded-core # ./scripts/runqemu-gen-tapdevs 1026 1026 4 /OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/usr/bin/
Error: /OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/usr/bin//usr/bin/tunctl is not an executable

I've tested that with this change it can call tunctl:
OE @ /OE/openembedded-core # ./scripts/runqemu-gen-tapdevs 1026 1026 4
/OE/build/oe-core/tmp-glibc/sysroots-components/x86_64/qemu-helper-native/usr/bin
Note: Destroying pre-existing tap interface tap0...
TUNSETIFF: Device or resource busy
Creating 4 tap devices for UID: 1026 GID: 1026...
Creating tap0
Creating tap1
Creating tap2
Creating tap3
Note: For systems running NetworkManager, it's recommended
Note: that the tap devices be set as unmanaged in the
Note: NetworkManager.conf file. Add the following lines to
Note: /etc/NetworkManager/NetworkManager.conf
[keyfile]
unmanaged-devices=interface-name:tap*

but runqemu itself still doesn't work for me:
OE qemux86@ ~/build/oe-core $ runqemu
runqemu - INFO - Running MACHINE=qemux86 bitbake -e...
runqemu - INFO - Running ls -t /OE/build/oe-core/tmp-glibc/deploy/images/qemux86/*.qemuboot.conf...
runqemu - INFO - CONFFILE: /OE/build/oe-core/tmp-glibc/deploy/images/qemux86/core-image-sato-qemux86-20170427174052.qemuboot.conf
runqemu - INFO - Overriding conf file setting of STAGING_DIR_NATIVE to /OE/build/oe-core/tmp-glibc/work/i586-oe-linux/defaultpkgname/1.0-r0/recipe-sysroot-native from Bitbake environment
runqemu - INFO - Continuing with the following parameters:

KERNEL: [tmp-glibc/deploy/images/qemux86/bzImage--4.10.9+git0+ad2e885015_fe0fb8da3d-r0.2-qemux86-20170427085800.bin]
MACHINE: [qemux86]
FSTYPE: [ext4]
ROOTFS: [tmp-glibc/deploy/images/qemux86/core-image-sato-qemux86-20170427174052.rootfs.ext4]
CONFFILE: [/OE/build/oe-core/tmp-glibc/deploy/images/qemux86/core-image-sato-qemux86-20170427174052.qemuboot.conf]

runqemu - INFO - Running /bin/ip link...
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap0.lock...
runqemu - INFO - Using preconfigured tap device tap0
runqemu - INFO - If this is not intended, touch /tmp/qemu-tap-locks/tap0.skip to make runqemu skip tap0.
runqemu - INFO - Network configuration: 192.168.7.2::192.168.7.1:255.255.255.0
runqemu - INFO - Running ldd tmp-glibc/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin//qemu-system-i386...
runqemu - INFO - Running tmp-glibc/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin//qemu-system-i386 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:02 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -drive file=tmp-glibc/deploy/images/qemux86/core-image-sato-qemux86-20170427174052.rootfs.ext4,if=virtio,format=raw -vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci   -cpu qemu32 -m 256 -serial mon:vc -serial null -kernel tmp-glibc/deploy/images/qemux86/bzImage--4.10.9+git0+ad2e885015_fe0fb8da3d-r0.2-qemux86-20170427085800.bin -append 'root=/dev/vda rw highres=off  mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1 '
qemu-system-i386: -netdev tap,id=net0,ifname=tap0,script=no,downscript=no: could not configure /dev/net/tun (tap0): Device or resource busy
runqemu - INFO - Releasing lockfile for tap device 'tap0'
Traceback (most recent call last):
  File "/OE/build/oe-core/openembedded-core/scripts/runqemu", line 1235, in <module>
    ret = main()
  File "/OE/build/oe-core/openembedded-core/scripts/runqemu", line 1228, in main
    config.start_qemu()
  File "/OE/build/oe-core/openembedded-core/scripts/runqemu", line 1139, in start_qemu
    raise Exception('Failed to run %s' % cmd)
Exception: Failed to run tmp-glibc/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin//qemu-system-i386 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:02 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -drive file=tmp-glibc/deploy/images/qemux86/core-image-sato-qemux86-20170427174052.rootfs.ext4,if=virtio,format=raw -vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci   -cpu qemu32 -m 256 -serial mon:vc -serial null -kernel tmp-glibc/deploy/images/qemux86/bzImage--4.10.9+git0+ad2e885015_fe0fb8da3d-r0.2-qemux86-20170427085800.bin -append 'root=/dev/vda rw highres=off  mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1 '

(From OE-Core rev: a31b1434c5f1edbd4e8faca813b4f084297c061d)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-29 11:17:22 +01:00
Paul Eggleton 6ffec971e8 runqemu: fix incorrect calls to get variable values
We were specifying a default parameter; the get() function defined here
does not take such a parameter. I appears this code had not been tested.
This fixes runqemu erroring out immediately when used within the eSDK.

(From OE-Core rev: e4548531112c824653ae42b9bcc335a7ca8588e0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-19 10:38:36 +01:00
Robert Yang 1fa1a7f174 oe-run-native: print more error messages
Fixed:
$ bitbake bmap-tools-native -ccleansstate && bitbake bmap-tools-native && oe-run-native bmap-tools-native bmaptool --help
[snip]
Error: Unable to find '' in <PATH>
[snip]

Note the blank '' word, it was because "tools" was overrided, now fix it.

And also check whether the recipe is a native one or not.

(From OE-Core rev: ba2884f6ad3a4e746fc80cbd707f83fa8abd4210)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-14 23:25:54 +01:00
Paul Eggleton bce0b509bf devtool: extract: drop erroneous bb.event.TaskStarted
This is a non-existent event - we already have the actual
bb.build.TaskSucceeded further down in the list hence why it wasn't
noticed earlier.

(From OE-Core rev: 4e059a5ceb6f44401154e89e37f56de1d664a7cb)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-14 09:47:08 +01:00
Paul Eggleton 02c39d3fad devtool: extract: fix handling of failed tasks
If a task such as do_fetch fails when we're extracting source for a
recipe (within devtool modify / upgrade / extract / sync) then we should
naturally stop processing instead of blundering on; in order to do that
we need to be listening for the TaskFailed event. Thanks to Richard
Purdie for noticing and fixing this.

(From OE-Core rev: 9174b845bf6a6be7753bf6b921959b1f3f2dcbc0)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-14 09:47:08 +01:00
Paul Eggleton 2338032d0b devtool: modify: add --keep-temp option for debugging
Most of the other extract-based commands have this option but oddly I
left it out for modify - I guess because if I was debugging an issue here
I just used devtool extract to do so, but there's no reason why we can't
have it here and it is useful.

(From OE-Core rev: 98fbc46e1a51237213bd7825a922389d3ab2ad9b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-14 09:47:08 +01:00
brian avery 314df06b81 oe-run-native: explicitly use bash
This script sources another script (oe-find-native-sysroot) with
arguments.  It was using /bin/sh. Sourcing with arguments works only in
bash so it was failing in dash.  This commit makes it dash proof.

(From OE-Core rev: 334020a800434d20e7c3312890a2baca295c41c7)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:11 +01:00
brian avery c8b67a463f oe-find-native-sysroot: add appopriate suggestion
right now, if it fails, the script tells the user to run bitbake foo
-caddto_recipe_sysroot. This works for native recipes but not things
like meta-ide. This patch checks whether the recipe is native and gives
out the appopriate warning.

(From OE-Core rev: fc61211efd57d1858954e5cd241fce58dee9d01b)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:11 +01:00
Robert Yang 4a0b6af446 oe-find-native-sysroot: work with RSS
The generic STAGING_DIR_NATIVE is gone since RSS, so when find
OECORE_NATIVE_SYSROOT, the user has to specify which recipe's
STAGING_DIR_NATIVE will be used as OECORE_NATIVE_SYSROOT.

* The usage is changed from ". oe-find-native-sysroot" to
  ". oe-find-native-sysroot <recipe>".
* The oe-run-native's usage has changed from
  "oe-run-native tool" to "oe-run-native native-recipe tool".

(From OE-Core rev: e2f6d937bd897083779507ecb9ecd15513b35f1f)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Ed Bartosh 5897c5edc4 Revert "scripts: change way we find native tools (pseudo)"
This reverts commit f200f37699.

This reverts the patch that fixed runqemu-extract-sdk. It failed
to fix other issues in the script/tools that were introduced by RSS. The
following patch from Robert Yang fixes both.  Therefore, reverting this
patch in favor of his.

(From OE-Core rev: 33263b69e6b470b4f02172dabe6815df8ab0cd0a)

Signed-off-by: brian avery <brian.avery@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Ed Bartosh 4393cc550b runqemu: use bindir_native property to run ifup/down scripts
Used self.bindir_native to point out to the native sysroot
when running runqemu-ifup and runqemu-ifdown scripts.

[YOCTO #11266]
[YOCTO #11193]

(From OE-Core rev: cc5513bf7a6114e14bb307acb88a44e9cf0aed8a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Ed Bartosh 4d3bececf6 runqemu: add bindir_native property
Isolated logic of getting path to native bin directory in
new bindir_native property method.

This property is going to be used to obtain location of
qemu-sytem and tunctl.

(From OE-Core rev: 26e97f7ebb7e3302e3d3c6646fb58baf395d62be)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Ed Bartosh 7ffcdd0a07 runqemu: get qemu from qemu-helper-native sysroot
If rm_work is enabled image native sysroot can be removed.
This makes runqemu to fail trying to find qemu binary.

Used native sysroot of qemu-helper-native to find system qemu
binary.

(From OE-Core rev: d42c02caaa4d6fb47681aa7ffe8b27fa38141e6a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Markus Lehtonen 11747280b5 oe-build-perf-report-email.py: use pwd for getting user name
Use pwd data instead of os.getlogin() to get the username for fallback
email address. os.getlogin() basically returns the name of the user
logged in on the controlling terminal of the process and raises an
exception on some systems if the process does not have a controlling
terminal - when executed from a cron script, for example.

(From OE-Core rev: 66b7f495bb000c043ae24176dcdec0cb087eeecf)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Richard Purdie ffabe5d1b9 oe-selftest: Error if known problem variables are set
Setting SANITY_TESTED_DISTROS or PRSERV_HOST are known to break oe-selftest.
Rather than have the user experience this, refuse to execute unless the
environment is correct.

Ideally we'd try and unset these but that is a more invasive change and this
at least makes people aware of the problem.

[YOCTO #11292]

(From OE-Core rev: 0c9b981e88c76da316e76f17e6da3a03b87c5008)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Patrick Ohly 68079d0f87 yocto-compat-layer: better handling of per-machine world build breakage
It is fairly common that BSP layers enable recipes when choosing
machines from that layer without checking whether the recipe actually
builds in the current distro. That breaks "bitbake world", retrieving
signatures and thus the test_machine_signatures test.

It's better to let that test continue with the signatures that can be
retrieved and report the broken world build separately. Right now, the
new test_machine_world iterates over all machines. More elegant and
useful in combination with a (currently missing) selection of which
tests to run would be to generate one test instance per machine. But that
is not straightforward and has to wait.

The "-k" argument alone was not enough to proceed despite failures,
because bitbake then still returns a non-zero exit code. The existance
of the output file is taken as sign that the bitbake execution managed
was not fatally broken.

(From OE-Core rev: 02f5d7836b726e40fef82b50b8145acc839b360b)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Patrick Ohly de76d1cc94 yocto-compat-layer: test signature differences when setting MACHINE
Selecting a machine is only allowed to affect the signature of tasks
that are specific to that machine. In other words, when MACHINE=A and
MACHINE=B share a recipe foo and the output of foo, then both machine
configurations must build foo in exactly the same way. Otherwise it is
not possible to use both machines in the same distribution.

This criteria can only be tested by testing different machines in combination,
i.e. one main layer, potentially several additional BSP layers and an explicit
choice of machines:
yocto-compat-layer --additional-layers .../meta-intel --machines intel-corei7-64 imx6slevk -- .../meta-freescale

To simplify the analysis and limit the amount of output, mismatches
are sorted by task order such that tasks that run first are also
reported first. Following tasks for the same recipe and set of
machines then get pruned, because they are likely to be different
because of the underlying task (same approach as in
test_signatures). The difference here is that we get information about
all machines. The task order in the base configuration serves as
heuristic for sorting that merged list.

The test has already found issues in go-cross (depended on
tune-specific libgcc) and gdb-cross (had a tune-specific path
unnecessarily), so it is also useful to uncover issues that are not
caused by the BSP layer itself.

(From OE-Core rev: cb0d3de4540e412cfcb7804b4b1689141c80e3a1)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Paul Eggleton 40d1771944 recipetool: create: hide missing npm error when called from devtool
If devtool is called with a URL to a source repository containing a
node.js module, we don't know that until recipetool has fetched it, and
due to the structure of the code we have to exit with a special code in
order to let devtool know it needs to build nodejs-native. We also want
to suppress the error message that recipetool would normally print under
these circumstances; there is already a mechanism for this but it wasn't
operative in the case where we're pointed to a source repository rather
than an npm:// URL, so create some plumbing so that we know to hide the
message.

(From OE-Core rev: 0c2d0fbb1c6c5b82183799eb7ef80074f86bcfc4)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Paul Eggleton 48f8d1201a devtool: add: prevent repeatedly running recipetool
If recipetool returns with exit code 14 this means devtool needs to
build nodejs-native and then call it again. If recipetool returns exit
code 14 again then clearly something has gone wrong and we should just
quit with an error.

(From OE-Core rev: 8d7cced6e06d7c2037f5ab75ac859f501129532e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Paul Eggleton bb8f141d0c devtool: add: fix node.js/npm handling with recipe specific sysroots
The change over to recipe specific sysroots means that we can no longer
get a known location simply from configuration for the npm binary - we
need to get the recipe sysroot for nodejs-native, look there for npm if
we need to check it's present, and add that to PATH when calling out to
npm. Unfortunately this means anywhere we need to get that path we have
to have parsed all recipes, otherwise we have no reliable way of
resolving nodejs-native. Thus we have to change recipetool create to
always parse all recipes (the structure of the code does not allow us to
do this conditionally).

In the worst case, if npm hasn't already been added to its own sysroot
and we are fetching from a source repository rather than an npm
registry, this gets a bit ugly because we end up parsing recipes three
times:
1) recipetool startup, which then fetches the code and determines it's
   a node.js module, finds that npm isn't available and then exits with
   a specific error to tell devtool it needs to build npm
2) when we invoke bitbake -c addto_recipe_sysroot nodejs-native
3) when we re-invoke recipetool

This code is badly in need of refactoring, but now is unfortunately not
the time to do that, so we're going to have to live with this ugliness
for now.

Fixes [YOCTO #10992].

(From OE-Core rev: acfdbd796c99882b8586023c8c6b848716105c8d)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:10 +01:00
Paul Eggleton 0440cbccda recipetool: create: fix for regression in npm license handling
OE-Core commit c0cfd9b1d54b05ad048f444d6fe248aa0500159e added handling
for AND / OR in license strings coming from npm, but made the assumption
that an & would always be present in the license value. Check if it's
there first so we don't fail if it isn't.

(From OE-Core rev: abe2955df2dc558de6068d9373dfcb47d690704b)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-13 10:54:09 +01:00
Patrick Ohly 31e53430f1 yocto-compat-layer: add --additional-layers
The new --addditional-layers parameter takes a list of layer
directories and adds them to the build configuration before starting
testing. The resulting base configuration then more closely matches
a full distro.

This is relevant in two cases:
1. some layers like meta-freescale dynamically enable more recipes
   in their layer.conf depending on which other layers are active,
   so testing only against OE-core might miss problems which occur
   only when also some other layers are active
2. BSP layers might be fine in combination with machines from
   OE-core, but might break in combination with some other machines

As before, test_signatures only warns about signature changes
introduced by the layer which is under testing, and not those changes
introduced by the additional layers.

(From OE-Core rev: 0e8528f7c6201e8a5d2799123241c0e1b85081ce)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:58 +01:00
Patrick Ohly e93a2ab3e3 yocto-compat-layer: also determine tune flags for each task
locked-sigs.inc groups tasks according to their tune flags (allarch,
i586, etc.). Also retrieve that information while getting signatures,
it will be needed to determine when setting a machine changes tasks
that aren't machine-specific.

(From OE-Core rev: 67f9a8759f47680dbf349797801b2a1e8d149377)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:58 +01:00
Patrick Ohly 937c1ea974 yocto-compat-layer: include bitbake-diffsigs output
After filtering out potential false positives, it becomes feasible to
include the output of bitbake-diffsigs for those tasks which
definitely have a change.

Depends on bitbake-diffsigs with the "--signature" parameter.

Enhanced output now is:

   AssertionError: False is not true : Layer meta-xxxx changed 120 signatures, initial differences (first hash without, second with layer):
      gstreamer1.0-plugins-base:do_fetch: 76973f19f2e30d282152bdd7e4efe5bb -> e6e7c6fa9f2bd59d7d8d107f7c6ca1ac
         Task dependencies changed from:
         ['PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
         to:
         ['GST_IMX_PATCHES_TO_APPEND', 'PV', 'SRCREV', 'SRC_URI', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]', 'base_do_fetch']
         basehash changed from d679d30bd1ea41c56e57419b57587f3c to 090a79b45f5fa26d10f9d34e2ed7a1e6
            List of dependencies for variable SRC_URI changed from '{'PV', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]'}' to '{'GST_IMX_PATCHES_TO_APPEND', 'PV', 'SRC_URI[md5sum]', 'SRC_URI[sha256sum]'}'
         changed items: {'GST_IMX_PATCHES_TO_APPEND'}
         Dependency on variable GST_IMX_PATCHES_TO_APPEND was added
         Variable SRC_URI value changed:
         "     http://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-${PV}.tar.xz     file://get-caps-from-src-pad-when-query-caps.patch     file://0003-ssaparse-enhance-SSA-text-lines-parsing.patch     file://0004-subparse-set-need_segment-after-sink-pad-received-GS.patch     file://encodebin-Need-more-buffers-in-output-queue-for-bett.patch     file://make-gio_unix_2_0-dependency-configurable.patch     file://0001-introspection.m4-prefix-pkgconfig-paths-with-PKG_CON.patch     file://0001-Makefile.am-don-t-hardcode-libtool-name-when-running.patch     file://0002-Makefile.am-prefix-calls-to-pkg-config-with-PKG_CONF.patch     file://0003-riff-add-missing-include-directories-when-calling-in.patch     file://0004-rtsp-drop-incorrect-reference-to-gstreamer-sdp-in-Ma.patch [--] {+${GST_IMX_PATCHES_TO_APPEND}+}"

      pulseaudio:do_install: 6bb6fe23e11a6d5fef9c3a25e73e4f9c -> 3f54ea75673a792e307197cfa6ef2694
         basehash changed from ac4efcfa783bd04a5a98a2c38719aedd to 37679d99623a37c8df955da3a01415a5
         Variable do_install value changed:
         @@ -1,3 +1,7 @@
              autotools_do_install
           	install -d ${D}${sysconfdir}/default/volatiles
          	install -m 0644 ${WORKDIR}/volatiles.04_pulse  ${D}${sysconfdir}/default/volatiles/volatiles.04_pulse
         +    if [ -e "${WORKDIR}/daemon.conf" ] && [ -e "${WORKDIR}/default.pa" ]; then
         +        install -m 0644 ${WORKDIR}/daemon.conf ${D}${sysconfdir}/pulse/daemon.conf
         +        install -m 0644 ${WORKDIR}/default.pa ${D}${sysconfdir}/pulse/default.pa
         +    fi

[YOCTO #11161]

(From OE-Core rev: 312edd42b6cc553de4d476c76e8e36a882e11cdd)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:58 +01:00
Ed Bartosh 8838dd2dbd oe-selftest: test wic sparse_copy API
Added new parameter 'api' to sparse_copy function to specify
underlying filemap API to use. By default sparse_copy will
try both available APIs.

Added test case for sparse_copy to wic test suite.

(From OE-Core rev: 88701cef6ba399e82f96ed1b0eef9a44ed8c1687)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:57 +01:00
Robert Yang 7792688ac2 qemux86-directdisk.wks: vda -> sda
Previously, runqemu grep root=/dev/sd or root=/dev/hd on the image, and
would use vda if no grep result, now we have set QB_DRIVE_TYPE to
"/dev/sd" by default, and the device will be /dev/sda, so use sda to
replace vda in the test case.

(From OE-Core rev: 3f1f42984021d87ac43aaa16f38b706c2c965e02)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:57 +01:00
Robert Yang be72eaf86f runqemu: use self.rootfs to replace self.nfs_dir
We can use self.rootfs as self.nfs_dir when self.fstype is nfs, this can
reduce the code's complexity and we can re-use the code of checking
ROOTFS conflictions.

(From OE-Core rev: 1aafa13ae6faf620acac7338c42a8838e75da6b9)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:57 +01:00
Robert Yang f57393b386 runqemu: do not rely on grepping images
Fixed when the image is large and not enough memory:
  grep: memory exhausted
  Aborted

[YOCTO #11073]

(From OE-Core rev: a99deb30a0138594147ae28aab016fe4b74b8959)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:57 +01:00
Robert Yang f2c09f7867 runqemu: run without arguments
Since we can get MACHINE and others from env vars and "bitbake -e",
"runqemu" can work without any arguments.

(From OE-Core rev: 9ebcb2b6f41420ae3686afad03bb26a68cfacf95)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:57 +01:00
Robert Yang f6aaba31cf runqemu: support env vars explicitly
Use self.env_vars to support get vars from environment explicity. The
MACHINE, ROOTFS and KERNEL was supported by shell based runqemu, and
the help text says support them from env vars, so add them back.

[YOCTO #11141]

(From OE-Core rev: 20008d0bfe2cacecba77e11b0a0faf3d959eaf1e)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-12 15:09:57 +01:00
Li Zhou a2d5c0b602 python2/3: Move config/Makefile from core package to dev package
Move config/Makefile in libdir from core package to dev package for
python, because it is only needed in development process.

(From OE-Core rev: 8b55d055f046677c18eeaefe3ca18869eedeb14d)

Signed-off-by: Li Zhou <li.zhou@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:18 +01:00
Paul Eggleton 1f19d9dfe5 buildhistory-diff: add option to compare actual signature differences
Use the code underpinning bitbake-diffsigs to add an option to
buildhistory-diff to determine and display the differences between the
actual signature inputs, with a twist - we collapse identical
changes across different tasks, showing only the most recent task to
have that difference, meaning that there's less noise to wade through
when you just want to know what changed in order to cause some
rebuilding you're seeing.

(From OE-Core rev: 86cb4b01f2020553902554e512c02147eb4e0f51)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Paul Eggleton 9049c09793 buildhistory-diff: add option to compare task signature list
Having added writing out of the task signature list to buildhistory
(when BUILDHISTORY_FEATURES includes "task"), we now need a way to
compare the list. This just shows which tasks have been added / changed
signature / removed.

(From OE-Core rev: 63bd7e9f780a98dda458d612877495756bcc5463)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Paul Eggleton 313e551b55 buildhistory-diff: operate from buildhistory directory
If the cwd is named "buildhistory" and the user hasn't specified an
alternative path on the command line, then assume that the current
directory is the buildhistory directory. This makes it easier to run
buildhistory-diff and also interact with the buildhistory git repository
as you no longer have to jump into the buildhistory directory and up to
the parent again when doing so.

(From OE-Core rev: e4ccec2e4c2f521a2bb473083b42aefd494eea23)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-11 18:10:17 +01:00
Richard Purdie 684eefd7a3 scripts: Drop cleanup-workdir
This script appears broken and is actively breaking build directories.
For example, binutils-cross-i586 gets run do_fetch to do_populate_sysroot by:

$ oe-selftest -r devtool.DevtoolTests.test_devtool_virtual_kernel_modify

then:

$ oe-selftest -r oescripts.BuildhistoryDiffTests.test_buildhistory_diff

wipes out the contents of tmp/work/x86_64-linux/binutils-cross-i586/2.28-r0/
but does not wipe out the corresponding stamps, then:

$ oe-selftest -r runtime-test.Postinst.test_postinst_rootfs_and_boot

needs binutils-cross-i586:do_populate_lic and if and only if this is
unavailable from sstate, it fails since it thinks the source is already
unpacked when it isn't resulting in:

WARNING: binutils-cross-i586-2.28-r0 do_populate_lic: Could not copy license file /home/pokybuild/yocto-autobuilder/yocto-worker/nigh
tly-oe-selftest/build/build/tmp/work/x86_64-linux/binutils-cross-i586/2.28-r0/git/COPYING3.LIB to /home/pokybuild/yocto-autobuilder/yocto
-worker/nightly-oe-selftest/build/build/tmp/work/x86_64-linux/binutils-cross-i586/2.28-r0/license-destdir/binutils-cross-i586/COPYING3.LI
B: [Errno 2] No such file or directory: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/x86_64-l
inux/binutils-cross-i586/2.28-r0/git/COPYING3.LIB'
ERROR: binutils-cross-i586-2.28-r0 do_populate_lic: QA Issue: binutils-cross-i586: LIC_FILES_CHKSUM points to an invalid file: /home/
pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/x86_64-linux/binutils-cross-i586/2.28-r0/git/COPYING3 [license-checksum]

and similar errors.

Its safer for users to wipe tmp than try and maintain scripts which try
to remove pieces of tmp and get it wrong so remove the script.

(From OE-Core rev: 809b23c829f352c0eae455ea89f53e2a9ee87f06)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Markus Lehtonen dff701ffd4 scripts/oe-build-perf-report: improve guessing of args
Search remote branches, too, when finding the latest commit.

(From OE-Core rev: 7b502702be5d484c6f1f903426ba63cf04f8ca4a)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Markus Lehtonen 182dd45f1f oe-build-perf-report-email.py: use proper fallback email address
Use properly formatted fallback email address instead of just the
username.

(From OE-Core rev: 5e07504d7a74f0641e2a374b9d12590ce9c9cc89)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Markus Lehtonen 60658499c2 build-perf-test-wrapper.sh: support extra args for email script
Make it possible to provide (extra) command line arguments to the
oe-build-perf-test-email script via a new environment variable
OE_BUILD_PERF_REPORT_EMAIL_EXTRA_ARGS.

(From OE-Core rev: 9cdd4ea5e006fe326bdf39ea437b9ba61a66b778)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:44 +01:00
Dmitry Rozhkov e9cf698d09 python3: fix run-time deps for core python3 libraries
The http.server module from python3-netclient imports the html module
which is in python3-html. Also xmlrpc.server imports pydoc which is a
part of python3-pydoc. But those run-time dependencies are missing
from python3-netclient and python3-xmlrpc respectively.

Add the missing run-time dependencies.

(From OE-Core rev: 8e30b726c44f873e5fd9d3f36c3464a29b97abd8)

Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:43 +01:00
Patrick Ohly 7d4fa58c5c yocto-compat-layer: limit report of signature changes
Typically a single change cascades through the entire task dependency
chain. Developers had to figure that out themselves, based on hard to
read and interpret output (not sorted, no indention, no explanations):

   $ yocto-compat-layer.py -n meta-xxxx
   ...
   AssertionError: True is not false : Layer meta-xxxx changed signatures.
   webkitgtk:do_install changed fe2edc9082bc0da98f9cb1391c52f565 -> b3a44684c5cd9aacd3f7c6ed88eefab5
   gstreamer1.0-plugins-good:do_configure changed 3b2f8211be3fe08422bf6087f3af16d1 -> 7d80e42fa1f4f01ff4dfe2ea4477d382
   pulseaudio:do_package_qa changed 5d0a58ada66ff17f5576555302ac319a -> 0e13bcb96143d1ae54c451bc3de0aa30
   epiphany:do_prepare_recipe_sysroot changed 29e1b277dbcb005bd54950594c50d91b -> d3c45527b37677a0668ce483c6db3052
   ...
   gst-player:do_packagedata changed 9ce6efdd357dd74919bc4957458b1e95 -> d0c083ce629f37adfc9c4ba9eff81f83
   gstreamer1.0-plugins-base:do_install changed 1161cd867d15bea63e5dd5d9abf0519c -> 5bf2b652a2d77fee3eedb35af2f201a0
   gstreamer1.0-rtsp-server:do_packagedata changed 6781dc3070f80b843ed1970d74dd323e -> 454620c2e3b9fea87e525d14b6ed0344
   alsa-plugins:do_packagedata changed 1808c3f737cb805b169d004e948ea19c -> 480124b7fa5eab1f73bf96440d725231

Now the tool automates the problem analysis: it retrieves the depgraph
using the tinfoil API and only reports those tasks with modified
signatures whose dependencies have not changed, i.e. those tasks which
definitely introduce a change.

>From the previous example, that just leaves two tasks that need to be
checked:

   AssertionError: False is not true : Layer meta-xxxx changed 120 signatures, initial differences (first hash without, second with layer):
      gstreamer1.0-plugins-base:do_fetch: 76973f19f2e30d282152bdd7e4efe5bb -> e6e7c6fa9f2bd59d7d8d107f7c6ca1ac
      pulseaudio:do_install: 668eb1e30af129df9806b0aa0d7c10cd -> 1196bdb88eef56eeee4613bb06b9387e

This pruning might be a bit too aggressive in the sense that tasks
which inherit a change and then add more changes themselves won't be
reported initially. They will be found when fixing the reported tasks
and re-running the check.

For a developer it seems better to have something listed which
definitely is a problem and needs fixing instead of everything,
including the tasks which don't need fixes.

(From OE-Core rev: 7ab0e09de75bfd7e7498bfa72d1f2f5d02a96747)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:42 +01:00
Patrick Ohly efd3b0ee30 yocto-compat-layer: fix also other command invocations
In commit 5b9ac62ab535d, one place was fixed where a command was
invoked such that failures caused double stack traces and stderr was
lost. The same problem also occurs elsewhere, triggered for example by
a layer with parsing problems.

Now a new utility method is used instead of repeating the code.

(From OE-Core rev: b6c72c0d169473e2626938be2ee59f850624612e)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:42 +01:00
Markus Lehtonen f81a532dd6 oe-build-perf-report-email.py: fix one file path
Sending report email was not working correctly if the script was given
an html report path that contained directory components.

(From OE-Core rev: 1da641661bb5963fcbd7ac2c20bc997c3eae6f18)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:42 +01:00
Robert Yang 74d7d12b37 runqemu: use realpath for imgdir
The DEPLOY_DIR_IMAGE maybe relative or absolute path since it can be
read from env vars, so use realpath for both imgdir and
DEPLOY_DIR_IMAGE when compare.

(From OE-Core rev: dad9f27278850d0d3818344fea877835632576cb)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:42 +01:00
Robert Yang c70a74402f runqemu: fix 2 typos
* "is it" -> "it is"
* Remove "<image>.qemuboot.conf =" in the error message which looked strange.

(From OE-Core rev: a6152dd9f6f4e17855548ceffa8d864855a67f5c)

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-10 23:00:42 +01:00
Chen Qi 9cacf8488c wic: improve error message
When using `wic create mkefidisk -e core-image-minimal', the following
error message appeared.

  Please bake it with 'bitbake parted-native' and try again.

However, following this command doesn't do any help. The same problem
still appeared.

The problem is that when we 'bitbake parted-native', it doesn't have
anything to do with core-image-minimal. And the required tool 'parted'
is not under core-image-minimal's recipe-sysroot-native directory.

Improve the error message so that following it could get things done.

(From OE-Core rev: f0425c0a0f1c98f65bf61fd9aa7e023ed41a35fa)

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:22:13 +01:00
Ed Bartosh b5bc885ba7 Revert "filemap: remove FilemapSeek class"
FIEMAP API is not supported by tmpfs file system, but
SEEK_HOLE/SEEK_DATA is supported.

Returned back FilemapSeek class that implements support
of SEEK_HOLE/SEEK_DATA API to make sparse_copy API working
on tmpfs again.

This reverts commit 6b80c13f7a.

(From OE-Core rev: e75bd6a7dd5c1b4bad039c35cf4a2ffc2f77c60a)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:22:13 +01:00
Markus Lehtonen 86eaa6d83f build-perf-test-wrapper.sh: support sending email reports
Add new '-E' command line option for sending an email report to
specified recipient.

(From OE-Core rev: 46e76ffd460933ab35da4cfd3509f7c5de5ecd93)

Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-04-05 23:22:13 +01:00