Currently we have a hierarchy of pkgdata directories and the code has to put together
a search path and look through each in turn until it finds the data it needs.
This has lead to a number of hardcoded paths and file globing which
is unpredictable and undesirable. Worse, certain tricks that should be
easy like a GL specific package architecture become problematic with the
curretn search paths.
With the modern sstate code, we can do better and construct a single pkgdata
directory for each machine in just the same way as we do for the sysroot. This
is already tried and well tested. With such a single directory, all the code that
iterated through multiple pkgdata directories and simply be removed and give
a significant simplification of the code. Even existing build directories adapt
to the change well since the package contents doesn't change, just the location
they're installed to and the stamp for them.
The only complication is the we need a different shlibs directory for each
multilib. These are only used by package.bbclass and the simple fix is to
add MLPREFIX to the shlib directory name. This means the multilib packages will
repackage and the sstate checksum will change but an existing build directory
will adapt to the changes safely.
It is close to release however I believe the benefits this patch give us
are worth consideration for inclusion and give us more options for dealing
with problems like the GL one. It also sets the ground work well for
shlibs improvements in 1.6.
(From OE-Core rev: 1b8e4abd2d9c0901d38d89d0f944fe1ffd019379)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some packages like grub have already stripped binaries e.g.
ERROR: QA Issue: File '/boot/grub/kernel.img' from grub was already
stripped, this will prevent future debugging!
ERROR: QA run found fatal errors. Please consider fixing them.
We would like to have a possibility to skip it using something like
INSANE_SKIP_${PN} = "already-stripped"
This adds the logic to do so
it acts at PN level and not at package level. so something like
INSANE_SKIP_${PN}-misc = "already-stripped" wont work.
(From OE-Core rev: 765982f4c050d9cd3eb608d630312da482c737c7)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
shlibs dependency calculations on darwin we not functioning correctly, we
need to process the filename without the complete path. If we don't,
"." characters in the path cause problems.
(From OE-Core rev: 07e697d651178a84007123181fca38e4d98ae0e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When copying the sources for the debug source package we use cpio -Ll
which means to copy files as hardlinks and to dereference symlinks.
It appears there is a bug in cpio since -Ll will copy symlinks and
not dereference them. We therefore do a second pass over copied symlinks
resolving them into files. Ideally we would copy these as hardlinks as well
however it doesn't seem worth the extra code and effort for what amounts
to a corner case for a minor space improvement.
This means that the -dbg packages no longer contain broken symlinks.
[YOCTO #5020]
(From OE-Core rev: 2ca2c4747f645a0d478c2171fff4c65752188285)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The error case was gcc-dbg packing files from libgcc-dbg reported in [1]
[1] http://lists.openembedded.org/pipermail/openembedded-core/2013-July/080728.html
(From OE-Core rev: fd2c0af422d67da870a5a11720d06a871e660215)
Signed-off-by: Andreas Müller <schnitzeltony@googlemail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When external commands such as debugedit fail, it can be useful to see
their output, so use oe.utils.getstatusoutput() instead of
subprocess.call() to capture this and print it on failure.
(From OE-Core rev: 34179cc78b730ecb8ff3f4feb4beee2d17498ce3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is the possibility that if we don't iterate through the multiprocessing
pool objects we might not catch return codes and this could lead to hung/zombie
processes either temproarily or on a wider scale. Adding this certainly doesn't
hurt anything and is better practise so we might as well do it.
Its not 100% clear if this fixes some issues or not.
(From OE-Core rev: 89c8493d4d85044cd72af2756569d15e87cd5947)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refactor packaging sanity tests from package.bbclass to insane.bbclass
so that the message can respect WARN_QA (print the warning message and
go on the task) and ERROR_QA (print the error message and fail the
task).
- For the bb.warn(), give it a message name and add it to WARN_QA, then
use package_qa_handle_error() to handle it.
- For the bb.error(), give it a message name and add it to ERROR_QA,
then use package_qa_handle_error() to handle it.
- All the bb.warn() and bb.error() have been replaced in
package.bbclass.
- A few bb.warn() and bb.error() in insane.bbclass have been kept since
they can not be replaced or doesn't have to, for example the
bb.error() in package_qa_check_license(), it will print the error
message and then invoke bb.fatal() to fail the task, I think that we
don't have to replace it with package_qa_handle_error().
- Put all the WARN_QA and ERROR_QA in one line, so that they can be
redefined by the user easily.
[YOCTO #3190]
[YOCTO #4396]
(From OE-Core rev: 2f117ee615b703db07d3274ac592e2bd653743dd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
RP's comment:
"What we're trying to do is move everything to use a standard mechanism
for reporting issues of this type (do_package). With insane.bbclass, you
can elect whether a given type of error is a warning or error and fails
the task."
* The package.bbclass had used package_qa_handle_error() which is from
insane.bbclass, and we will use it for handling other warnings and
errors, so let package.bbclass inherit insane.bbclass, this change will
make the insane as a requirement (always included).
* Change the "PACKAGEFUNCS ?=" to "+=", otherwise there would be an
error like:
Exception: variable SUMMARY references itself!
This is because we let package.bbclass inherit insane.bbclass, and
PACKAGEFUNCS has been set in insane.bbclass, so the "PACKAGEFUNCS ?="
will set nothing, then the "emit_pkgdata" doesn't run which will
cause this error.
* Add a QA_SANE variable in insane.bbclass, once the error type
is ERROR_QA, it will fail the task and stop the build.
[YOCTO #3190]
[YOCTO #4396]
(From OE-Core rev: 852dead71387c66ec0cba7c71e3814a74e409560)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bb.fatal() is defined as:
def fatal(*args):
logger.critical(''.join(args))
sys.exit(1)
So anything after bb.fatal() in the same code block doesn't have any
effect, e.g.:
bb.fatal("%s_%s: %s" % (var, pkg, e))
raise e
The "raise e" should be removed.
I searched all the files which use bb.fatal(), only the following 3
classes have this issues:
insane.bbclass
package.bbclass
package_rpm.bbclass
[YOCTO #4461]
(From OE-Core rev: 4c47525c5a171b1282615c9fbc7d84addef85f92)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=c376f1f49cea182a2887945840ab97a20970a373
fixed a valid issue where the sources file was accumulating information
and subsequent task runs of do_packacge were not cleaning it. The
fix is wrong however since we're removing the file within a loop.
This fix removes the file outside the loop ensuring it is not truncated
and contains the correct information.
(From OE-Core rev: a015881f2207aded601459ba3eebbefb0002b3c5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are various bits of cruft that have built up around our file accesses. This patch
cleans some of them up, specifically:
* Remove pointless "from __builtin__ import file"
* Use open(), not file()
* Wrap file usage in a with container to ensure files are closed
* Add missing .close() calls in some cases
(From OE-Core rev: a43e0a8ecd0441131e929daf998c3cd454d9c8f3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I mistakenly thought subprocess had getcmdstatus in python 2. It doesn't so lets
add a wrapper and have this work in both worlds.
(From OE-Core rev: 2253e9f12734c6e6aa489942b5e4628eca1fa29d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The commands module is removed in python3. Use the subprocess module instead
and the pipes module to replace the mkargs usage.
(From OE-Core rev: e2e1dcd74bc45381baccf507c0309dd792229afe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update older code to use modern exception handling syntax which
is the form accepted by python 3.
(From OE-Core rev: b010501cd089e649a68f683be0cf4d0aac90fbe3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An excluded package left its files behind, which meant they could end up in
another package instead, meaning we could ship GPLv3 binaries even with GPLv3
in INCOMPATIBLE_LICENSE. Skip the files belonging to the excluded packages to
prevent this from occurring.
(From OE-Core rev: c045bfe6b991006ac80f0e2d06a8917ae58d9262)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
THe USERADD_PARAMS and similar group variables are package specific variables
which should get added to the vardeps of the packaging process. This
change also ensures they get remapped correctly by the multilib code.
(From OE-Core rev: 805add18e7b917aaef61c82c0b5f2e9682af1d0d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we put a valid glob like "*/foo/*" into FILES, populate_packages
breaks with a "file exists" message. This is because the glob expansion
does not have "./" prefix however there may already be an entry in
the seen list which does have such a prefix. The easiest/simplest fix
right now is to add the prefix if it doesn't exist which only happens
for certain globs.
(From OE-Core rev: 138c5f3af892e33f576fc7dd268e122b179f82a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
allarch multilib recipes are meant to provide a list of different multilib variants.
Unfortunately since the pkgdata also has mappings for these, they get mapped back to
the original package name which means the effect is undone at package creation time
when the remapping code is called.
This patch adds in a conditional to break that chain meaning the packages get
the correct RPROVIDES and image builds work correctly with opkg.
[YOCTO #3453]
(From OE-Core rev: 1a1927f8a04fe0a2b3b853ebdd33ccb807f00b59)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Removed the package files parsing routine from the
packageinfo.bbclass file and added it to the
package.bbclass file.
(From OE-Core rev: 225e7826b0d082f43db82201e826b98b3a95cd57)
Signed-off-by: Andrei Dinu <andrei.adrianx.dinu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After the recent cached path operation improvements, we need to tell
the cache about directory changes. /usr/src/debug was was cached as
not created and this was leading to bugs in file ownership. This
change updates the cache and avoids this issue.
(From OE-Core rev: 553a8010e81448629a94ab04fd989e6e49066f2b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The command for generating debug sources appends to the file, therefore we need
to ensure it does not exist before we run the command else we end up doing
more work that we should have to.
(From OE-Core rev: 48dfd61901ce07491ef913cf04dbd9de9a52759f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an error occurs in subprocess.call() we currently don't catch it. In particular
we have issues where debugedit is segfaulting unnoticed. This fixes up
various code paths to catch the errors.
[YOCTO #4089]
(From OE-Core rev: 262a69ffd33e9d001a7a15fc73671a015e3b5dd1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, various standard library operations like os.walk(),
os.path.isdir() and os.path.islink() each call stat or lstat which
involves a syscall into the kernel. There is no caching since they could
conceivably have changed on disk. The result is that for something like
the do_package task of the kernel we're spending over two minutes making
868,000 individual stat calls for 23,000 files. This is suboptimal.
This patch adds lib/oe/cachedpath.py which are a set of replacement
functions for these operations which use cached stat data rather than
hitting the kernel each time. It gives a nice performance improvement
halving the build time of the kernel do_package.
(From OE-Core rev: 556dee0c4d6d8a87c0cddbd2f60fe5917d009f18)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* same as original and default version, but does not package source files in PN-dbg
(From OE-Core rev: 3c8452c3abae74a42989c0fbd5ba303788528750)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was realised that .so files which were not marked as executable were not
gettings stripped. This was wasting space in images. This patch ensures
they do get processed by the code correctly.
[YOCTO #3973]
(From OE-Core rev: 725354886ae3650a7a4875d4c0bffcfab7e8cc40)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the do_package function was split into different sections, the vardeps
were not updated to match meaning some function changes weren't reflected
in the sstate checksums. Fix this.
[YOCTO #3980]
(From OE-Core rev: 2e7ddbcca3f613c3b5725dcc3d457985d41e5f16)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In pakcage.bbclass,function replace('-dev', '') is
used to cut '-dev' suffix. but it will cause the
name "lib32-device-dev" to "lib32ice",
so change it to [:-4].
(From OE-Core rev: 91a3a48798029ca88a46a528200f4c5252f245e9)
Signed-off-by: Song.Li <Song.Li@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit ec2aab09769f4b6817d74d2175afa2b7c7598750 introduced a regression
on packages which contain symlinks with unresolvable path components
(e.g. lsof-dbg). While assigning a variable, an exception was raised
and the exception handler accessed this variable.
Patch deals both with the dangling path components by assuming them as
valid directories and by avoiding the broken assignment.
(From OE-Core rev: 579369b0fb27fad6d628746a50b9b798078500f6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oe.path.realpath() provides are common and more correct implementation
for resolving symlinks within sysroot. Use it.
Old implementation suffered from lot of problems; e.g.
* redundant code
* calls 'os.stat()' which references files on host; this can give wrong
results about existing/non-existing and can cause EPERM (instead of
the catched ENONENT) exceptions
* does not deal with special cases like '..' leaving the sysroot.
(From OE-Core rev: ec2aab09769f4b6817d74d2175afa2b7c7598750)
Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
After rewrite of split_and_strip_files src var is no longer
available here where it is used so we should replace it with
a valid value
(From OE-Core rev: b07e79ea049d341a5a0a7e4bb72151f49336c5bc)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This needs to run before PKGR and other variables using PRAUTO are used,
including the expansion optimisations.
(From OE-Core rev: eb09ebc9680eb50e23778dcc1c2697d13cf073ae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* sometimes it's useful to do something with packages created
by do_split_packages later in do_package_prepend, e.g. in:
http://lists.linuxtogo.org/pipermail/openembedded-devel/2013-February/043824.html
I have .bbclass which adds some postinst, postrm and RDEPENDS,
but ttf-mplus is using do_split_packages to create those packages
(From OE-Core rev: 5aa52c6882d2929b2e530c4fa297c6d3e97d4f9f)
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add headers to document the different phases of do_package and
make the steps clearer.
(From OE-Core rev: b6438c94035a014902ec89af63ff3787cd8c67f6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Prepending to populate_packages is rather ugly and means its hard to trace
errors and also profiling informaiton is summed together in one function.
This patch starts to split out the prepends to become separate functions
to avoid these issues. This is generally a neater way to write functions
than prepending to where there can sometimes be variable scope issues
and we've been bitten by whitespace issues in the past.
(From OE-Core rev: 4f9963d1d82ee896fe9491d6a8b32be42cd06f14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than an exec() per directory, we might as well exec one command and
be done with it.
(From OE-Core rev: 82ae9cfb09ee5c0aa6402c972d71e2b64d1ce8bc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Call getVar outside the loop
* Drop unneeded PATH export (bitbake does this already)
* Drop unused variable
* Simplify if statement nesting
* Simplify variable expandion to a getVar call (expand would just call getVar)
(From OE-Core rev: 52b506145bcddc133ca93a8c9f7343de69d10907)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can use the cleandirs and dirs flags for the fuctions to handle
directory cleaning and creation at the bitbake level rather than
using these calls within the functions
(From OE-Core rev: 4b31d6f6f0a2a6b9e504ffae0d3b2099cbd7dddc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The split_and_strip_files funciton was hard to follow and its usage of prefixes
to strings was unusual. This rewrites it to use a list of hardlinks, symlinks and
elffiles where each list is iterated over at the correct point.
This means we can avoid creating dandling symlinks for example so we can simply
delete the cleanup code for this.
The isfile() check is also removed which gives a significant improvement in speed.
Its uneeded since os.walk will have already checked things in files are files.
(From OE-Core rev: 0cd295d8cdc8cc39d6b6c7d26ea8a2a10a979d7c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
These are in bb.utils so lets the correct function and avoid the overhead
of the fixup/warning for the deprecated usage.
(From OE-Core rev: d17329db4842c50af1a3d7f5f20e692c89913fba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PATH is already exported, we don't need to do this each time we run
something, its just noise and overhead.
(From OE-Core rev: 060f617cea4ea0a5af28d31ea19c0387e9773fce)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing check for SKIP_FILEDEPS can be overridden per recipe
using SKIP_FILEDEPS_pn-${PN}. However, there's no mechanism for
letting a single package within a recipe use SKIP_FILEDEPS.
This patch adds SKIP_FILEDEPS_<pkg>, by analogy to FILES_<pkg>.
Note that it only works one way; if the recipe has SKIP_FILEDEPS = 1,
the checks for individual packages will never be reached.
(From OE-Core rev: 94557b500ad38a49aec40629015ed0b24e167f76)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We may as well expand the RDEPENDS when reading and writing as this function does.
if we don't do this, we could accidentally duplicate data and it also turns out
to be much less efficient.
(From OE-Core rev: d62a3151e7b252911c172a605b3089de355663c5)
(From OE-Core rev: 8f602e84e6ae4e346acef7cf5473343039fa352a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we do a signficant amount of tree traversal in many different places
which in inefficient. We can assume that the files don't change and cache the
file list which gives an efficiency improvement which this patch does using
a global variable.
(From OE-Core rev: 2d7608842d2dab07065e60aab729a5c8fd6b7907)
(From OE-Core rev: 5c7c8347eb1bc25d194be6f4be142ba0924e2600)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Khem Raj
Andreas Müller
Paul Eggleton
Robert Yang
Christopher Larson
Andrei Dinu
Martin Jansa
Song.Li
Enrico Scholz
Peter Seebach