Currently bitbake only adds files to its dependency list if they exist.
If you add 'include foo.inc' to your recipe and the file doesn't exist,
then later you add the file, the cache will not be invalidated.
This leads to another bug which is that if files don't exist and then
you add them and they should be found first due to BBPATH, again the
cache won't invalidate.
This patch adds in tracking of files we check for the existence of so
that if they are added later, the cache correctly invalidates. This
necessitated a new version of bb.utils.which which returns a list of
files tested for.
The patch also adds in checks for duplicate file includes and for now
prints a warning about this. That will likely become a fatal error at
some point since its never usually desired to include a file twice.
The same issue is also fixed for class inheritance. Now when a class
is added which would be found in the usual search path, it will cause
the cache to be invalidated.
Unfortunately this is old code in bitbake and the patch isn't the
neatest since we have to work within that framework.
[YOCTO #5611]
[YOCTO #4425]
(Bitbake rev: 22e6b1c4c4afb27057689bbc94cbdf1f19f93e3d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
print disappears into the ether, so use logger.warn and clean up the messages.
(Bitbake rev: 90f91f7402ff69f3fe9fba5f94a53d371303ce34)
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_PRESERVE_ENV wasn't working since data.inheritFromOS wasn't getting a
correct list of keys to import into the data store. This fixes
things so it does add all environment variables into the data store
when BB_PRESERVE_ENV is used.
(Bitbake rev: 843e9339c5ee3c99657a40a0e2c7dbd777b6ef06)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Interestingly the previous version comparision was broken and we were always using
the compatibility code, masking this bug. Oops.
(Bitbake rev: d48e8bcb24e8fa5d4fd60fd2c9927a95976d8d8b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we've moved to require python 2.7.3, we can jettison the compatibility
workarounds/hacks for older python versions.
(Bitbake rev: a51c402304f2080a76720f9b31d6dfdbed393bba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current execution context management for bitbake is ugly and the
use of a global variable is nasty. Fixing that is hard, however we
can improve things to start to establish an API for accessing
and changing that context.
This patch also adds in an explicit reset of the context when we reparse
the configuration data which starts to improve the lifecycle of the data
in setups like hob.
(Bitbake rev: 6c3281a140125337fc75783973485e16785d05a1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move various random imports to the start of the modules as cleanup
and avoid an import issue with bb.process on python 2.6.
(Bitbake rev: aed4adfbe3a591ca4f8e41fb763c9f961bf2e6d5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 3 is stricter about how files are accessed. Specficially:
* Use open(), not file()
* Use binary mode for binary files (when checksumming)
* Use with statements to ensure files get closed
* Add missing file close statements
(Bitbake rev: 9f08b901375ba640f47596f1bcf43f98a931550f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Similarly to the better_exec improvements, improve the compile failure
messages to be more user readable.
(Bitbake rev: 9bc92d0210e13e4cc98727f6c9ec2f47c2221e77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current bitbake tracebacks are hard to read/confusing and sometimes
incomplete. This patch attempts to do better by:
* Moving the note about the exact exception to the end to make things
read in sequence
* Merged the initial stack trace to become part of the code dump
* Added handling for "/xxxx" file paths since we can load these files
and include the data as part of the trace
* Dropped the ERROR: prefix to every line, allowing the error messages to
be spacially accosicated in the UIs
* Moved the "From:" line to the top of each code block and ensured its present
consistently
With the complexity now in this funciton, I've added try/except wrapping around
it to ensure we catch exceptions in the exception handler too.
Example before:
"""
ERROR: Error executing a python function in /media/build1/poky/meta/recipes-core/eglibc/eglibc-initial_2.17.bb:
TypeError: 'filter' object is not subscriptable
ERROR: The stack trace of python calls that resulted in this exception/failure was:
ERROR: File "do_populate_lic", line 13, in <module>
ERROR:
ERROR: File "do_populate_lic", line 6, in do_populate_lic
ERROR:
ERROR: File "license.bbclass", line 99, in find_license_files
ERROR:
ERROR: File "/media/build1/poky/meta/lib/oe/license.py", line 38, in visit_string
ERROR: if pos > 0 and license_pattern.match(elements[pos-1]):
ERROR:
ERROR: The code that was being executed was:
ERROR: 0009: destdir = os.path.join(d.getVar('LICSSTATEDIR', True), d.getVar('PN', True))
ERROR: 0010: copy_license_files(lic_files_paths, destdir)
ERROR: 0011:
ERROR: 0012:
ERROR: *** 0013:do_populate_lic(d)
ERROR: 0014:
ERROR: [From file: 'do_populate_lic', lineno: 13, function: <module>]
ERROR: 0002:def do_populate_lic(d):
ERROR: 0003: """
ERROR: 0004: Populate LICENSE_DIRECTORY with licenses.
ERROR: 0005: """
ERROR: *** 0006: lic_files_paths = find_license_files(d)
ERROR: 0007:
ERROR: 0008: # The base directory we wrangle licenses to
ERROR: 0009: destdir = os.path.join(d.getVar('LICSSTATEDIR', True), d.getVar('PN', True))
ERROR: 0010: copy_license_files(lic_files_paths, destdir)
ERROR: [From file: 'do_populate_lic', lineno: 6, function: do_populate_lic]
ERROR: 0095: lic_files_paths.append((os.path.basename(path), srclicfile))
ERROR: 0096:
ERROR: 0097: v = FindVisitor()
ERROR: 0098: try:
ERROR: *** 0099: v.visit_string(license_types)
ERROR: 0100: except oe.license.InvalidLicense as exc:
ERROR: 0101: bb.fatal('%s: %s' % (d.getVar('PF', True), exc))
ERROR: 0102: except SyntaxError:
ERROR: 0103: bb.warn("%s: Failed to parse it's LICENSE field." % (d.getVar('PF', True)))
ERROR: [From file: 'license.bbclass', lineno: 99, function: find_license_files]
ERROR: Function failed: do_populate_lic
ERROR: Logfile of failure stored in: /media/build1/poky/build/tmp/work/i586-poky-linux/eglibc-initial/2.17-r3/temp/log.do_populate_lic.17442
"""
Example after:
"""
ERROR: Error executing a python function in /media/build1/poky/meta/recipes-core/eglibc/eglibc-initial_2.17.bb:
The stack trace of python calls that resulted in this exception/failure was:
File: 'do_populate_lic', lineno: 13, function: <module>
0009: destdir = os.path.join(d.getVar('LICSSTATEDIR', True), d.getVar('PN', True))
0010: copy_license_files(lic_files_paths, destdir)
0011:
0012:
*** 0013:do_populate_lic(d)
0014:
File: 'do_populate_lic', lineno: 6, function: do_populate_lic
0002:def do_populate_lic(d):
0003: """
0004: Populate LICENSE_DIRECTORY with licenses.
0005: """
*** 0006: lic_files_paths = find_license_files(d)
0007:
0008: # The base directory we wrangle licenses to
0009: destdir = os.path.join(d.getVar('LICSSTATEDIR', True), d.getVar('PN', True))
0010: copy_license_files(lic_files_paths, destdir)
File: 'license.bbclass', lineno: 99, function: find_license_files
0095: lic_files_paths.append((os.path.basename(path), srclicfile))
0096:
0097: v = FindVisitor()
0098: try:
*** 0099: v.visit_string(license_types)
0100: except oe.license.InvalidLicense as exc:
0101: bb.fatal('%s: %s' % (d.getVar('PF', True), exc))
0102: except SyntaxError:
0103: bb.warn("%s: Failed to parse it's LICENSE field." % (d.getVar('PF', True)))
File: '/media/build1/poky/meta/lib/oe/license.py', lineno: 38, function: visit_string
0034: new_elements = []
0035: elements = filter(lambda x: x.strip(), license_operator.split(licensestr))
0036: for pos, element in enumerate(elements):
0037: if license_pattern.match(element):
*** 0038: if pos > 0 and license_pattern.match(elements[pos-1]):
0039: new_elements.append('&')
0040: element = '"' + element + '"'
0041: elif not license_operator.match(element):
0042: raise InvalidLicense(element)
Exception: TypeError: 'filter' object is not subscriptable
ERROR: Function failed: do_populate_lic
ERROR: Logfile of failure stored in: /media/build1/poky/build/tmp/work/i586-poky-linux/eglibc-initial/2.17-r3/temp/log.do_populate_lic.3275
ERROR: Task 9 (/media/build1/poky/meta/recipes-core/eglibc/eglibc-initial_2.17.bb, do_populate_lic) failed with exit code '1
"""
(Bitbake rev: c5de66b870406d9bd1161a9b7e2b04fe6eb065fe)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The BB_ENV_WHITELIST doesn't work well and flushes BB_ENV_EXTRAWHITE,
here is an example:
$ export BB_ENV_WHITELIST
$ export BB_NUMBER_THREADS=10 (or other value)
Edit conf/local.conf, change "BB_NUMBER_THREADS =" to "BB_NUMBER_THREADS ?="
$ bitbake -e | grep '^BB_NUMBER_THREADS ='
we will notice that BB_NUMBER_THREADS' value doesn't change, though
BB_NUMBER_THREADS in both BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE.
This is because the "approved" inside the function approved_variables
doesn't include BB_ENV_WHITELIST or BB_ENV_EXTRAWHITE when
BB_ENV_WHITELIST is set (they are incuded by preserved_envvars()), so
the BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE will be removed from the env
in the first call from bin/bitbake, and when it is called again by
cooker.py, their value will be None, then the vars inside them will be
removed from the env.
Add BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE to the "approved" would fix
the problem.
[YOCTO #4031]
(Bitbake rev: d2b07e6516dd308d0045a7fdb72b588af9d676ad)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Improve the remove function to better handle cases where path contains
special shell chars like whitespaces, '$', ';' or'\'.
Thanks to Enrico Scholz for the fix.
(Bitbake rev: 617511c9f86cc4ef52457653c8adff582d94bce3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Whilst shutils.rmtree() is pythonic, its also slow. Its faster to
use rm -rf which makes optimial use of the right syscalls.
(Bitbake rev: 96088ebdec08e49ba9e8dbcac437bfcdc21f5983)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to empty out the environment whilst we build the cooker but
we need the environment for the UIs since hob uses DISPLAY and other
session variables.
This patch adapts the utils functions to return removed environment
components so we can reinject them for use by the UI, allowing hob
to work again.
(Bitbake rev: fc330d810099c57fefd4e706159a73ad8401d97c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Maintining hardcoded lists of user environmental variables is hacky,
replace these with the savedenv datastore. Allow access to that
through the BB_ORIGENV variable.
(Bitbake rev: 0a99563a4ea270594fd9a61da46f9387fb79dc66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python 2.7 shows hangs with issues in its pool implmenetation. Rather than
try and hack around these, add a copy of the working pool implementation
to the compat module from 2.7.3.
(Bitbake rev: c9eb742637131e8dbd526d2ad9b458abea0a2d87)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The API for explode_dep_versions is flawed since there can only be one version
constraint against any given dependency. This adds a new function with an API
without this limitation. explode_dep_versions() is maintained with a warning
printed when its used in a situation where information is lost.
This should allow a simple transition to the new API to fix the lost dependency
information.
join_deps() is updated to deal with data in either format.
(Bitbake rev: babeeded21827d8d3e7c7b785a62332ee9d45d4f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Refactor the explode_dep_versions to be more lenient on whitespace values.
The required format is:
foo (= 1.10)
foo (=1.10)
foo ( = 1.10)
foo ( =1.10)
foo ( = 1.10 )
foo ( =1.10 )
(Bitbake rev: 39c1c12c58fadd854098cf14ebe92f4d307a36dd)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
explode_dep_versions is not able to have duplicate entries. Previously
duplicate entries ended up with the last item being the one returned to
the caller.
We now detect a collision. We do allow an empty item to have a comparison
added to it, or a duplicate with the same comparison without error.
When a collision is detected a ValueError exception is thrown.
Allowed:
foo foo (= 1.12) foo
Invalid:
foo (= 1.12) foo (= 1.13)
(Bitbake rev: d40448f0483a2959e9dcaac9b6dd35839f396a6e)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adding a generic backtrace to better_compile and better_exec is pointless,
therefore reraise the exception as a bb.BBHandledException so the
generic code doesn't confuse the user even more.
(Bitbake rev: b3d97130e1e70fe969399277dcd7cccd888103d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently as soon as execution passes outside the code fragment being
executed by better_exec, we don't get any good traceback information,
just a likely obscure reference to some function name which may
or may not be identifiable.
This patch adds code to query the datastore if present, allowing a more
meaningful back trace to be displayed in many cases.
[YOCTO #2981]
(Bitbake rev: 0edf8431f9ff52581afe0d3ef525c59909af02ba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This improves the stacktraces dumped by bitbake when for example anonymous
python functions fail.
Also default to passing code strings to better_exec to match the behaviour of
simple_exec to aid the transition.
(Bitbake rev: 7e8205929ae953731a6854ea80b197847cff5771)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
_ and LANG no longer need to be preserved from the external environment.
The value of _ changes between non-pseudo-wrapped and pseudo-wrapped
invocations (e.g. between "bitbake -p" and "bitbake target") and this
will currently trigger a full reparse in the absence of a whitelist
entry in BB_HASHCONFIG_WHITELIST, which is not ideal.
LANG used to be preserved in order to ensure the C locale was being
used for tools invoked by bitbake, however we now set LC_ALL in
bitbake.conf to take care of this.
Second part of the fix for [YOCTO #2600].
Acked-by: Jason Wessel <jason.wessel@windriver.com>
(Bitbake rev: 1c531dff2fb055ecab2d462027eecec3fabc2a44)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the path passed to which contains empty elements, it will search
the current working directory for the file which is correct baheviour.
Various pieces of code assume the path returned is a full path though.
This commit ensures we don't return relative paths.
(Bitbake rev: 4de24ccc10e40cc088b8515095df59f69b12715d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Having two different version comparision algorithms in bitbake has never seemed
like a sensible idea. Worryingly, they also return different results to each other.
The vercmp_string API is relatively unused with no users in OE-Core or BitBake
itself for example. This patch converts it to use vercmp internalls, bringing
consitency to the comparisions which is easy now we have other recently added
functions. Yes, this changes behaviour but in this case I'd prefer we were
consistent than having two different comparisions.
(Bitbake rev: a569c816e016447d60624c59a750709d59a0f455)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The version compare function vercmp() was not exatcly conforming to
Debian rules, e.g. it reported 'r1' > 'r1.1' but the Debian rules says
'r1' < 'r1.1'; it didn't support the "~" either.
Modified the vercmp() to meet Debian rules, so that it's compatible to
the rules used in opkg.
This part of the buf fixing of [YOCTO #2233].
(Bitbake rev: 97b610c54c60b5a40fa7f6a09fa23ce17b38f93a)
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a new API in command.py to get the cpu count in order to set the appropriate default BB_NUMBER_THREADS and PARALLEL_MAKE variables.
(Bitbake rev: 335047b2e440e65713e88fabb24b47a9c82f939b)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
USER is the correct variable to use, also this can affect sstate
cache as well.
(Bitbake rev: d7f9edda65dae2e046871afa275c5a51dff48fc4)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The lockfile retry parameter is expected to return immediately after
attempting to take the lock. There was a bug in the logic which this
patch fixed to ensure it does that.
(Bitbake rev: f421ef819f00ac659504d9af41bcc8323422ff8c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change which modified inheritFromOS to use the intial
environment, rather than the current environment, introduced a bug such
that variables which had been cleaned from the environment where still set
in the data store.
This patch changes things such that a list of approved environment
variables is saved after the environment is cleaned and only the variables
in this list are inherited in inheritFromOS.
CC: James Limbouris <james.limbouris@gmail.com>
CC: Chris Larson <clarson@kergoth.com>
(Bitbake rev: cb6c07054e8baf94614713ec257c643b22266d75)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
At the moment it bugs me a lot that we only have one effective logging
level for bitbake, despite the logging module having provision to do
more advanced things. This patch:
* Changes the core log level to the lowest level we have messages of
(DEBUG-2) so messages always flow through the core logger
* Allows build.py's task logging code to log all the output regardless
of what output is on the console and sets this so log files now
always contain debug level messages even if these don't appear
on the console
* Moves the verbose/debug/debug-domains code to be a UI side setting
* Adds a filter to the UI to only print the user requested output.
The result is more complete logfiles on disk but the usual output to the
console.
There are some behaviour changes intentionally made by this patch:
a) the -v option now controls whether output is tee'd to the console.
Ultimately, we likely want to output a message to the user about where the
log file is and avoid placing output directly onto the console for every
executing task.
b) The functions get_debug_levels, the debug_levels variable, the
set_debug_levels, the set_verbosity and set_debug_domains functions are
removed from bb.msg.
c) The "logging" init function changes format.
d) All messages get fired to all handlers all the time leading to an
increase in inter-process traffic. This could likely be hacked around
short term with a function for a UI to only request events greater than
level X. Longer term, having masks for event handlers would be better.
e) logger.getEffectiveLevel() is no longer a reliable guide to what
will/won't get logged so for now we look at the default log levels instead.
[YOCTO #304]
(Bitbake rev: 45aad2f9647df14bcfa5e755b57e1ddab377939a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bb.utils.contains(X, "A", true, false) will return true for substring matches,
e.g. if X = "ABC". This is not what most users expect from the function.
In the common OE use of this function there is the case of "touchscreen" and "screen" being
used as independent variables. Whilst it could be argued there isn't a problem in that
specific case (touchscreens are usually on screens), there is no substring usage of this
function is OE-Core so this patch changes the behaviour to match only full strings.
It also fixes a bug where duplicate entries would confuse multiple matches, e.g.
contains(X, ["A", "B"], ...) would match X = "A A" which is clearly wrong.
(Bitbake rev: 3d8647b68a8e66c7b240ed5fed7406e1b78fabf6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This function is needed by some of the early .conf setup we need
to improve the machine/tune files in Openembedded. We need to add
it here since the location in oe.utils can't be accessed until after
base.bbclass parses which is too late for our needs.
(Bitbake rev: abc67ed6921c98ed581f101ec1acc589fd9ce7e9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are usecases where we don't want to block waiting for a lockfile
so enhance the lockfile handling functions to support this.
(Bitbake rev: 97e8adf03e5fab1fd40c3d53c48f7b333bc2e145)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create the lockfile directory if it doesn't exist, rather than erroring out if
it doesn't exist (was also racy).
Also improve the wording of the error message shown when the lockfile's
directory is not writable.
Note for the future, this function should be improved, particularly with
regard to its exception handling. It should be catching the *exact*
exception(s) it will encounter when the file is locked, and continuing in that
case only. If it did that, there'd be no need for the proactive directory
writability check, as bb.utils.lockfile() would raise an appropriate IOError
for that case.
(Bitbake rev: 238151441c74db53d6e4d4753f4f96c32f6f13b6)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A file we're copying might be on a readonly filesystem so if we can already read
it, don't try and add read permission.
Fixes BUGID #771 in Yocto.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This allows fetching git repositories using Kerberos authentication.
(Bitbake rev: d761cf98284b02eb3d3a1f879782c501c284b698)
Signed-off-by: Javier Martin <javier.martin@vista-silicon.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Ross Burton
Robert Yang
Mark Hatle
Paul Eggleton
Lianhao Lu
Shane Wang
Matthew McClintock
Joshua Lock
Chris Larson
Javier Martin
Saul Wold
Bernhard Reutner-Fischer