If SkipParse is raised from something which isn't anonymous python, it wasn't
being handled correctly. This improves the handling for example from within inline
python.
(Bitbake rev: 7467d7d66b24cc8f43ab168e65895e7c4aee6092)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
An issue was uncovered where changing:
IMAGE_INSTALL_append = "X"
to
IMAGE_INSTALL_append = "X Y"
in local.conf would not get noticed by bitbake. The issue is that
the configuration hash doesn't account for overrides or key expansion.
This patch improves get_hash to account for these. This means the hash
does account for changes like the above.
[YOCTO #3503]
(Bitbake rev: 86bf1f5603e8f98019544e45f51bd0db9a48112a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Instead of logging the function/variable separately as a NOTE when
failing to expand, re-raise ExpansionError with more contextual
information. This means that the full details are reported in Hob as
well as actually reporting the original error message in any UI where
we previously did not. For example, we used to get this with tab/space
indentation issues in a python function:
NOTE: Error expanding variable populate_packages
ERROR: Unable to parse /path/to/recipename.bb
Now, we will get this:
ERROR: ExpansionError during parsing /path/to/recipename.bb: Failure
expanding variable populate_packages: IndentationError: unindent does
not match any outer indentation level (<string>, line 4)
Fixes [YOCTO #3162].
(Bitbake rev: ce5c7a95a359cdaecab7c4a519ad4f9df029da82)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If this regular expression is unanchored, it would accept strings like:
do_install_append1
do_install_appendsomelongstring
and treat them like they were do_install_append. Clearly this isn't desirable.
Only one instance of this type of issue was found in OE-Core and has been fixed
so correcting the regexp should be safe to do.
(Bitbake rev: 23bd5300b4a99218a15f4f6b0ab4091d63a602a5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Variables which used multiple overrides and the append/prepend operators were
not functioning correctly. This change fixes that.
This fixes the testcase:
OVERRIDES = "linux:x86"
TESTVAR = "original"
TESTVAR_append_x86 = " x86"
TESTVAR_append_x86_linux = " x86+linux"
TESTVAR_append_linux_x86 = " linux+x86"
[YOCTO #2672]
(Bitbake rev: dc35a2e506e15fb7ddbf74c3b3280e9e83ab33bb)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the variable name is not specified then don't confuse the error message
by starting off with "Failure expanding variable None...".
(Bitbake rev: 9cb16f3c73751e7cf6d495586a6193f06eb97b1f)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For config hash, we put the keys in structure of "set()", which is not
order sensitive. Therefore when calculating the md5 value for config
hash, we need to identify the order of the keys.
(Bitbake rev: 0f1b142a3f6b8125bf023c2e5ec269618869abf7)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Adopt the BB_HASHCONFIG_WHITELIST as a exclusion list for variables that are
not needed in cache hash calculation.
(Bitbake rev: ae8cf138b5eb8f1f28a7143b8d67ad06cbe43061)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
CC: Christopher Larson <kergoth@gmail.com>
CC: Martin Jansa <martin.jansa@gmail.com>
CC: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously we use the file time stamp to judge if a cache is valid.
Here this commit introduce a new method, which calculates the total
hash value for a certain configuration's key/value paris, and tag
it into cache filename, for example, bb_cache.dat.xxxyyyzzz.
This mechanism also ensures the cache's correctness if user
dynamically setting variables from some frontend GUI, like HOB.
(Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds appendVar and prependVar functions to the data store
meaning python code would no longer have to do the getVar, append and
the setVar dance that much of the current python code does.
It also adds corresponding variants for flags.
Currently there is no spacing added by these functions. That could be
added as a parameter if desired.
If these functions turn out to be hotspots in the code, there are tricks
that could potentially be used to increase the speed of these specific
operations within the datastore.
(Bitbake rev: 4a4046268f84b85559eea2c4b6a6004ad8cccb77)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the vardeps flag is not None, we now silence the warnings about
non-literal usage for that variable.
(Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- If a name is passed to the parser, prepend the messages with "while
parsing <name>:". This gives a bit more context.
- Tweak the warning messages slightly (they had to be altered anyway to
inject the variable being parsed).
Before:
DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \
not a literal
After:
DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \
''%s' % var' is not a string literal
(Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if passing expand=True to getVar() function, it will pass the
handling to getVarFlag(), which doesn't get any benefit from the expand
cache.
Call the expand() function separately in getVar() to make use of the
expand cache, which can decrease the parsing time by 40%.
(from current 49s to 27s)
(Bitbake rev: 6555a77c199f41bf35460138764e03e30c56d29f)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As the code stands, setting a variable with ??= could result in a ?=
variable not overriding it. This patch fixes the issue by allowing
the ast to make lookups that ignore any ??= set variables.
(Bitbake rev: 32fee2e650dfdd3aa9a7572dad1251e0c24ca34b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we delete a variable we no longer expect it to override other
variables.
To do this we remove it from the list of active overrides at deletion
time. It turns out we already had to do this at override expansion time
so this cleans up the code to be more consistent as an added bonus.
(Bitbake rev: d924ff9ede57c3dea6e1c738ba3633f18d460b14)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you d.delVar(), you expect the variable to be gone. Even empty
variables continue to exist in the datastore and are still user visible
unfortunately. The COW siutation means you can't just remove it
since it might unmask a variable from an inner copy.
This patch therefore stops empty variables from appearing in key lists
exposed to the external world making empty variables an internal
implementation detail only.
(Bitbake rev: 2b5548c591d4cfde9238d2cc0959c42cfc08f09c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we're going to creat the seen set() anyway, we might as well use
it directly. If we don't do this, we see thousands of function calls
with associated overhead on profiles.
(Bitbake rev: 9d43e3279895639ee4899df635f2546c7ee13737)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you do:
OVERRIDES = "z"
DEPENDS_prepend = "a "
DEPENDS = "b"
DEPENDS_z = "c"
d.update_data()
d.getVar("DEPENDS")
gives "a c"
d.update_data()
d.getVar("DEPENDS")
then gives "c"
This patch changes the behaviour such that at the time bitbake expands the DEPENDS_z
override, it removes "DEPENDS_z" from the data store. In the above example this would
mean that it wouldn't matter how often you call d.update_data(), you'd always get
"a c" back.
See the bitbake-devel mailing list for further discussion and analysis of the
potential impact of this change.
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If expanding a variable triggers an exception the caller currently has no
way to supress the error message or otherwise handle the siutation. An
example of where this is a problem is "bitbake -e" showing tracebacks and
errors for variables like SRCPV in OE/Poky.
Secondly in a chained expansion fails, log mesages are recorded for
every step of the expansion, not just the innermost error which is
where the real failure occured.
To fix this we introduce a new exception ExpansionError which callers
can handle as appropriate.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if a variable has been set with ??= and the code looks it up
before the data finalisation phase, no value is found. This is causes
serious problems for anonymous python functions which manipulate data, or
for the fetcher revision handling code where revisions can be set with
??=.
There is also a significant performance implication for processing lazy
assignment in finalise.
Moving the check for a default value into getVarFlag addresses both
the timing issue and the performace. This change gives a 7% real time
performance improvement to parsing the Poky metadata. The cost of the
check at this point is minimal since we have all the data flags available.
This should also fix Yocto bug 752.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 53740977521bc81ffa37adfa7bbeb8f2a80ea165)
build: write logfiles per task, not per function
Based on d14f9bf6 from poky, reworked for master and other cleanup.
(Bitbake rev: beadff2eca1eb95f0411115dd72ddb4c3c44c604)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Ensure it raises KeyError for a missing key, this is required to use this as a
mapping in various places, e.g. as locals in an eval.
(Bitbake rev: 8d661ce0c303e8d69f17c1d095545d5ed086d1d5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We use a custom Logger subclass for our loggers
This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug
(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Provide __len__, __iter__, and the getitem/setitem/delitem methods, and its
mixed in versions of keys(), values(), items(), etc will automatically behave,
making the DataSmart act more like a real mapping.
(Bitbake rev: 89b5351c656d263b0ce513cee043bc046d20a01e)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Where a variable name consisted of an append/prepend combined with an override
and there was also an append/prepend to the variable, the override could be lost
if the override was not in OVERRIDES.
For example:
FOO = "A"
FOO_append = "B"
FOO_append_virtclass-native = "C"
could result in "AB" even though virtclass-native was in OVERRIDES.
With this patch applied, the result is "ABC" as would be expected.
The problem was the deletion of the _append/_prepend flag was happening
if *any* append/prepend was procesed, the result should really be that
it should contain any unprocessed append/prepend.
Kevin Tian deserves credit for looking into this and working out the
problem here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This commit is derived from Chris Larson's checksum work, turned into a
standalone piece of code for parsing python and shell functions.
The deindent code has been replaced with code to work around indentation
for speed. The original NodeVisitor in the ast was replaced with a faster
class walk call.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The root cause is, I was testing the use of renameVar() from finalize, in
order to get flags copied over when an override is applied, but renameVar
removes the original, whereas the old code did not do so. Going back to the
old method, will revisit the override/flags later on.
(Bitbake rev: 2f7c498abcf675e5b8de197d8056a0581670c2bd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
types.IntType -> int
types.StringType -> basestring
...
Also moves our ImmutableTypes tuple into our own namespace.
(Bitbake rev: 83674a3a5564ecb1f9d2c9b2d5b1eeb3c31272ab)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The methodpool, ${@} expansions, anonymous python functions, event handlers
now all run with the same global context, ensuring a consistent environment
for them. Added a bb.utils.better_eval function which does an eval() with the
same globals as better_exec.
(Bitbake rev: 424d7e267b009cc19b8503eadab782736d9597d0)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Richard Purdie
Paul Eggleton
Ross Burton
Dongxiao Xu
Christopher Larson
Richard Purdie