VAL = "" (not shown)
VAL = " " (shown as "")
VAL = " x" (shown as "x")
would all show up rather differently to what would be expected in the
bitbake -e output. This fixes things so they appear consistently.
The output for running some shell functions may also change slightly
but shouldn't change in a way that is likely to cause problems.
[YOCTO #5507]
(Bitbake rev: 9f37afff200d748beddc2a70f55a72c2714e3120)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Move the variable lookup to the outer loop for performance, replacing
a now unneeded parameter (after the previous changes).
(Bitbake rev: 8a59a8707cecbde257fca169775ce8ff7709928b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When looking up flag variable dependencies, large chunks of the function
aren't needed. Optimise the function flow accordingly for speed.
(Bitbake rev: 1bf3aee698ad35f6815ea2c75471a96511a29d55)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of multiple calls to getVarFlag, make one call to getVarFlags, only expanding
the flags that need to be expanded. This improves performance.
(Bitbake rev: eba1e9545cc933820d40de96f023b2307b3c4d0b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Compute a cache of the list of potential export variables so
that we don't have to compute the list from scratch.
(Bitbake rev: f41f46f7eaa6889edeb3a4e4ddedc07084686c60)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Profiling shows the creation of keys() has overhead and we're better using
an iterator rather than the memory associated with the huge list of keys
when iterating the whoe datastore. We minimise the number of times
we do this to twice only per recipe.
(Bitbake rev: e63448d9ee331b0f45fb9a0197d0dbee49eb2fa0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The if statement current causes the size of parent to be calcuated which
is like a len() operation on a datastore. Since we're only interested
whether the value is none, checking explictly for this gives a
small performance gain.
(Bitbake rev: 43a245bde318545ea75ca4ce7894395c1cf9b32a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was necessary to append ${TOPDIR}/recipes/images to BBFILES.
Implemented the mechanism to append a value to a variable: a command and
the method in cooker.
[YOCTO #4193]
(Bitbake rev: 4aedbee90bd92395c2460a68702e6ede00e256c9)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently we account of the top level function's vardeps but not
those of any subfunction. This would imply we'd have to manually
write the dependencies of all parent functions which would be crazy.
This patch adds the dependencies to fix the issue.
(Bitbake rev: 658008cc6a8acd5c1f26877c9444c96002db01e7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BitBake commit 7c568132c54a21161de28907159f902462f1e2bb resulted in a
fairly serious performance regression during parsing, almost doubling
the time taken to do a full parse and almost certainly impacting
performance during building. The expandKeys function is called
frequently, and if we avoid using keys() and instead just use the normal
variable lookup mechanism, performance is restored.
(Bitbake rev: 034b392e9877309f15940b258fc2c16f16fb40b5)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When two variables are defined as:
${var} = "bar"
foo = "foobar"
The value of 'foo' when ${var} == foo becomes indeterminate. We
want to warn a user when this situation has been encountered so they
can take corrective actions.
In the above example usually foo == bar, unless multilibs are enabled.
Then ml-foo = "ml-foobar".
(Bitbake rev: 7c568132c54a21161de28907159f902462f1e2bb)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds tracking of the history of variable assignments.
The changes are predominantly localized to data_smart.py and
parse/ast.py. cooker.py and data.py are altered to display the
recorded data, and turn tracking on for the bitbake -e case.
The data.py update_data() function warns DataSmart.finalize()
to report the caller one further back up the tree.
In general, d.setVar() does what it used to do. Optionally,
arguments describing an operation may be appended; if none
are present, the operation is implicitly ignored. If it's
not ignored, it will attempt to infer missing information
(name of variable, value assigned, file and line) by examining
the traceback. This slightly elaborate process eliminates a
category of problems in which the 'var' member of the keyword
arguments dict is set, and a positional argument corresponding
to 'var' is also set. It also makes calling much simpler for
the common cases.
The resulting output gives you a pretty good picture of what
values got set, and how they got set.
RP Modifications:
a) Split from IncludeHistory to separate VariableHistory
b) Add dedicated copy function instead of deepcopy
c) Use COW for variables dict
d) Remove 'value' loginfo value and just use 'details'
e) Desensitise code for calling order (set 'op' before/after
infer_caller_details was error prone)
f) Fix bug where ?= "" wasn't shown correctly
g) Log more set operations as some variables mysteriously acquired
values previously
h) Standardise infer_caller_details to be triggered from .record()
where at all possible to reduce overhead in non-enabled cases
i) Rename variable parameter names to match inference code
j) Add VariableHistory emit() function to match IncludeHistory
k) Fix handling of appendVar, prependVar and matching flag ops
l) Use ignored=True to stop logging further events where appropriate
(Bitbake rev: f00524a3729000cbcb3317fee933ac448fae5e2d)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, calling a shell function using exec_func may result in
dependent functions being missing, if the shell parser can't detect
the dependency, even if it was specified with the vardeps flag.
This patch ensures the function looks at the flag and considers it
when deciding which other functions need to be output.
[YOCTO #3561]
(Bitbake rev: a906c94328b040d5076571bd593b57be7ee403a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The inherits cache contains duplicate entries, some with the full patch, some
just starting classes/xxx. This is a waste of parse time and potentially
error prone. This patch fixes various pieces of code so the absolute paths are
always preferred and work correctly. The inherits_class function did not work
with full paths so the patch fixes this.
(Bitbake rev: f3a71e509af196391ec126d079cf1bd178e62ad5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of logging the function/variable separately as a NOTE when
failing to expand, re-raise ExpansionError with more contextual
information. This means that the full details are reported in Hob as
well as actually reporting the original error message in any UI where
we previously did not. For example, we used to get this with tab/space
indentation issues in a python function:
NOTE: Error expanding variable populate_packages
ERROR: Unable to parse /path/to/recipename.bb
Now, we will get this:
ERROR: ExpansionError during parsing /path/to/recipename.bb: Failure
expanding variable populate_packages: IndentationError: unindent does
not match any outer indentation level (<string>, line 4)
Fixes [YOCTO #3162].
(Bitbake rev: ce5c7a95a359cdaecab7c4a519ad4f9df029da82)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Mixed spacing in python functions can cause subtle issues and
generally confuses users. We've standardised on 4 space indentation,
adding this warning helps ensure consistency and avoid bugs. It
also makes _prepend and _append operations on python functions slightly
less risky.
(Bitbake rev: c51cfaf48d3b12a19b01e824b6ba4230376bcad4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a dependency to each variable on a filtered list of its varflags.
This is intended to catch things such as SRC_URI checksums, varflags
controlling extra functionality from classes (e.g. the recently updated
update-alternatives class in OE-Core), etc. and ensure their values
influence the sstate checksums.
There is an exclusion list which needs to be set via bitbake.conf
(BB_SIGNATURE_EXCLUDE_FLAGS), if this is not set then the functionality
is disabled. The existing vardepsexclude mechanism can also be used to
exclude undesired varflags, but they must be fully specified, e.g.:
do_patch[vardepsexclude] += "do_patch[someflag]"
Implements [YOCTO #2517].
(Bitbake rev: 56c1ab18fcaf4ac245dcb412ed55e8e0af07883b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have a problem if we want to inject specific information into the variable
dependency code. There are cases for example where we want a dependency
on the value of X but it doesn't matter how X was constructed or what
dependencies it might have had, we only care about the absolute value.
With the current code, its near enough impossible to do this.
This patch adds such a mechanism so the user can trigger this with code like:
baselib[vardepvalue] = "${baselib}"
It also refactors some of the code so we do variable lookups once
instead of doing this in two different functions.
[YOCTO #1583]
(Bitbake rev: 6c879b44ccf42dc73fe4467076e114700d7ba81b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the vardeps flag is not None, we now silence the warnings about
non-literal usage for that variable.
(Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- If a name is passed to the parser, prepend the messages with "while
parsing <name>:". This gives a bit more context.
- Tweak the warning messages slightly (they had to be altered anyway to
inject the variable being parsed).
Before:
DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \
not a literal
After:
DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \
''%s' % var' is not a string literal
(Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent change which modified inheritFromOS to use the intial
environment, rather than the current environment, introduced a bug such
that variables which had been cleaned from the environment where still set
in the data store.
This patch changes things such that a list of approved environment
variables is saved after the environment is cleaned and only the variables
in this list are inherited in inheritFromOS.
CC: James Limbouris <james.limbouris@gmail.com>
CC: Chris Larson <clarson@kergoth.com>
(Bitbake rev: cb6c07054e8baf94614713ec257c643b22266d75)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now that we have a pristine copy of the variables available at launch time
we can use them when looking to inherit the OS's environment.
(Bitbake rev: 21c21fcc5871e81d8d497b6baed605cdd74c4571)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update exception handling syntax to use the modern style:
except ExcType as localvar
(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Even when a variable was whitelisted, any dependencies of that variable
could still creep into the task hash due to the way the whitelisting
code worked. This patch changes thing to ensure that when whitelisted,
that whitelisting applies to the variable and any dependencies it has.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[BUGID# 645], modify the emit_var()
1. Added "#" to the beginning of each line if the comment contains
multiple lines.
2. Added "\" to the end of each line if the shell variable value
contains multiple lines.
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Previously we emitted two newlines for export and unexport.
One newline for export and unexport is enough (and makes the scripts
look better and a tad smaller).
(Bitbake rev: ba060160fdf1278a273fb2b77d36b8c681807ecf)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adds a vardepsexclude flag that can be used to exclude a dependency of
a variable (the opposite of vardeps). This will allow the exclusion of
variables from the hash generation much more selectively than blanket
whitelisting using BB_HASHBASE_WHITELIST.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.
Currently, this code only finalises the environment in he worker as doing so
in the server means variables are unavailable in the worker. If we switch
back to fork() calls instead of exec() this code will need revisting.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
These functions allow generation of dependency data between funcitons and
variables allowing moves to be made towards generating checksums and allowing
use of the dependency information in other parts of bitbake.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
types.IntType -> int
types.StringType -> basestring
...
Also moves our ImmutableTypes tuple into our own namespace.
(Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Richard Purdie
Cristiana Voicu
Paul Eggleton
Mark Hatle
Peter Seebach
Christopher Larson
Joshua Lock
Scott Garman
Lianhao Lu
Bernhard Reutner-Fischer
Richard Purdie