The bb.utils.contains function usage is getting increasingly used in the metadata
but isn't handled automatically by the python dependency tracking code. This patch
changes that and also adds the "OE" names for the functions.
Whilst there are reasons this is a bad idea, its likely outweighed by the
shear number of these references and the current holes in dependency information
which we're now relying heavily upon.
(Bitbake rev: 0b9d117631ce909312d53b93289e61defc6be01c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to track appendVar and prependVar calls just as we do for getVar in order
to ensure we're not missing variable dependencies.
(Bitbake rev: 767b4751232f4ee3979deb4d3f733fcf9ee2bd44)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want to be able to reuse most this functionality for the file
checksum cache.
(Bitbake rev: 0fe3cb1438d297f90dd0fc6b26362ecbff75c76d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
See the comment in the code in the commit for more information.
(Bitbake rev: 2d56dc7b1f0d186e14c4c8a949b280b6b3fc31de)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the initial implementation there was a relcutance on my part to
generate incremental cache components on the fly since it would lead
to some duplicate code.
We are now seeing problems where each thread reading in the saved cache
file causes significant overhead and can make the process appear to hang
on a many core build, particularly when the cache file is large.
This patch changes the code to maintain the delta in a separate dict
right from the start. The code duplication isn't too bad and could be
mitigated in other ways if it becomes an issue.
[YOCTO #2039 partial]
(Bitbake rev: cdd5d0dee6ab12326b252b6b505a316a52638cac)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the vardeps flag is not None, we now silence the warnings about
non-literal usage for that variable.
(Bitbake rev: e724b9f417d1baf898f5afc6376c73c1a2ad8db9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two usual cases involving bb.data.expand:
- Calling it with a string literal -- "bb.data.expand('${FOO}/${BAZ}/bleh', d)".
- Calling it on getVar results (legacy) -- "bb.data.expand(bb.data.getVar('FOO', d), d)"
Nothing in any of the usual layers uses it in any other way, and I'm
having trouble coming up with any real use cases beyond this. The first
of the above cases is already tracked, via the expandWithRefs called
on the python code string. The second didn't emit a warning anyway,
since the getVar was already handled.
Given this, I see no reason for us to maintain explicit expansion
tracking. Further, we weren't using its results anyway (the var_expands
member).
(Bitbake rev: 405dfe69e6a608826e599ebf2f83ef8cf5083b96)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- If a name is passed to the parser, prepend the messages with "while
parsing <name>:". This gives a bit more context.
- Tweak the warning messages slightly (they had to be altered anyway to
inject the variable being parsed).
Before:
DEBUG: Warning: in call to 'bb.data.getVar': argument ''%s' % var' is \
not a literal
After:
DEBUG: while parsing emit_pkgdata, in call of bb.data.getVar, argument \
''%s' % var' is not a string literal
(Bitbake rev: 1060193ae4d54e667735dbff5d1d2be49a3f95c9)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the previous method, using the compare_name methods, we split the
requested match name by '.', reversed it, then compared them piecemeal
during the node traversal. The new method walks the nodes and hands back
the name of what's being called, and then we check that. This also
consolidates the two different implementations of traversal of the
attribute/name nodes (one in compare_name, one for the execs).
(Bitbake rev: 84e535b5165c7e936c5b1486bdf4626ed3649f5f)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The split is even less necessary now that we use ast.walk rather than an
actual NodeVisitor subclass.
(Bitbake rev: d6c44fac184abae8395bfa7078f06675218aa534)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously, it was calling var_expands.update() rather than add(), with
a string argument, resulting in adding each character of that string to
the var_expands set, rather than the string itself.
(Bitbake rev: 8e4e75383e43d6da2c16ec5286186a0d0569b0f8)
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current codeparser cache handling hurts performance badly even
over a couple of cores and certainly on many core systems, it can
spent huge amounts of time in the codeparser cache save functions.
This patch reworks the cache handling so that each parsing thread
saves out its own "differences" file compared to any existing core
cache and then the main bitbake thread picks these up and merges
things back together.
This was tested on systems with small and large numbers of cores
and was found to perform orders of magnitude better in all cases
despite the more complex code.
(Bitbake rev: 9f27563d66523f5af1028f173d53ee75e0877d46)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Commit 036cf3cd11b3a6836b77f5ffa760ceee6b71b1ef missed the needed
brackets to handle more then a type of exception.
(Bitbake rev: abecbb4c0af83c6b4ee248b0f03b779f84b13390)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Before this change, the codeparser cache was only being saved for the main
server process. This is suboptimal as it leaves code being re-evaluated at
task execution time and increases parse time.
We use the multiprocess Finalize() functionality to ensure each process
saves out its cache. We need to update the cache save function to be multiprocess
friendly with locking.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The pysh we're using is modified, and we don't want to risk it conflicting
with one from elsewhere.
(Bitbake rev: 1cbf8a9403b4b60d59bfd90a51c3e4246ab834d6)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
For a given input to this code, the output doesn't change to implement a persistent
cache of the data to speed up parsing.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This commit is derived from Chris Larson's checksum work, turned into a
standalone piece of code for parsing python and shell functions.
The deindent code has been replaced with code to work around indentation
for speed. The original NodeVisitor in the ast was replaced with a faster
class walk call.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Richard Purdie
Paul Eggleton
Christopher Larson
Christopher Larson
Otavio Salvador
Joshua Lock
Bernhard Reutner-Fischer
Richard Purdie