Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some tweaks to use python 3 syntax in a python 2 compatible way.
(Bitbake rev: 322949c77dbaa4db01b5a43d85b39a2af67ba7b2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possible to prevent a recipe to be cached, and thus,
parsed every time.
Use with care.
[YOCTO #8853]
(Bitbake rev: 78335c1fbe5266116700c2413aac28b00423a75b)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If no cache file name is given a default from class variable is used,
like before.
(Bitbake rev: 2602a312818f564961de7dfa63c429d45ff9e5ac)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Drop unused 'd' argument from the cache save methods, simplifying the
API.
(Bitbake rev: 81bc1f20662c39ee8db1da45b1e8c7eb64abacf3)
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In certain circumstances it can be useful to get access to the world
targets list from a recipe in order to add dependencies on some or all
of the items in it. If a special function, 'calculate_extra_depends' is
defined in the recipe, and the recipe is to be built, then call it at
the right point before we calculate which tasks should be run. The
function can append items to the "deps" list in order to add
dependencies. This is not as tidy a solution as I would have liked, but
it does at least do the job.
As part of this change, the buildWorldTargets function was moved to
bb.providers to make it possible to call from taskdata.
Part of the implementation of [YOCTO #8600].
(Bitbake rev: aba0dce57c889495ec5c13919991a060aeff65d2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The rprovides maybe contain duplicated lines when parse again, we need
check it before add to cachedata.rproviders, similar to what we had done
to cachedata.providers.
(Bitbake rev: 6c488afb0fe30a9655ec62a1d22f9f388365f012)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'stamp-base' and 'stamp-base-clean' related codes are no longer useful,
clean them up.
[YOCTO #8468]
(Bitbake rev: 7b4c42b315d4a26dd8f2ceb874a94737bf9f183e)
Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Trying to expand a variable which isn't a string doesn't make sense.
(Bitbake rev: 54d0ddd166a6707b4f8c8535639e3055b783363b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a space in a directory name containing a file in file-checksums
(e.g. from a file:// url), you currently get tracebacks from bitbake. This
improves the code to handle colons and spaces in the file-checksums names
since it possible to figure out the correct names.
[YOCTO #8267]
(Bitbake rev: 87282b283921a58426f24fb21151db457c5bca66)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Memory resident bitbake has one current flaw, changes in the base configuration
are not noticed by bitbake. The parsing cache is also refreshed on each invocation
of bitbake (although the mtime cache is not cleared so its pointless).
This change adds in pyinotify support and adds two different watchers, one
for the base configuration and one for the parsed recipes.
Changes in the latter will trigger a reparse (and an update of the mtime cache).
The former will trigger a complete reload of the configuration.
Note that this code will also correctly handle creation of new configuration files
since the __depends and __base_depends variables already track these for cache
correctness purposes.
We could be a little more clever about parsing cache invalidation, right now we just
invalidate the whole thing and recheck. For now, its better than what we have and doesn't
seem to perform that badly though.
For education and QA purposes I can document a workflow that illustrates this:
$ source oe-init-build-env-memres
$ time bitbake bash
[base configuration is loaded, recipes are parsed, bash builds]
$ time bitbake bash
[command returns quickly since all caches are valid]
$ touch ../meta/classes/gettext.bbclass
$ time bitbake bash
[reparse is triggered, time is longer than above]
$ echo 'FOO = "1"' >> conf/local.conf
$ time bitbake bash
[reparse is triggered, but with a base configuration reload too]
As far as changes go, I like this one a lot, it makes memory resident bitbake
truly usable and may be the tweak we need to make it the default.
The new pyinotify dependency is covered in the previous commit.
(Bitbake rev: 0557d03c170fba8d7efe82be1b9641d0eb229213)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if you reference a file url, its checksum is included in the
task hash, however if you change to a different file at a different
location, perhaps taking advantage of the FILESPATH functionality, the
system will not reparse the file in question and change its checksum to
match the new file.
To correctly handle this, the system not only needs to know if the
existing file still exists or not, but also check the existance
of every file it would have looked at when computing the original file.
We already do this in the bitbake parsing code for class inclusion. This
change uses the same technique to log the file list we looked at and
if files in these locations exist when they previously did not, to
invalidate and reparse the file.
Since data stored in the cache is flattened text, we have to use a string
form of the data and split on the ":" character which is ugly, but is
an internal detail we can improve later if a better method is found.
The cache version changes to trigger a reparse since the previous
cache data is now incompatible.
[YOCTO #7019]
(Bitbake rev: 6c0706a28d72c591f1b75b6e3f3b645859387c7e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It turns out the codeparser cache is the bottleneck I've been observing when running
bitbake commands, particularly as it grows. There are some things we can do about
this:
* We were processing the cache with "intern()" at save time. Its actually much
more memory efficient to do this at creation time.
* Use hashable objects such as frozenset rather than set so that we can
compare objects
* De-duplicate the cache objects, link duplicates to the same object saving
memory and disk usage and improving speed
* Using custom setstate/getstate to avoid the overhead of object attribute names
in the cache file
To make this work, a global cache was needed for the list of set objects as
this was the only way I could find to get the data in at setstate object creation
time :(.
Parsing shows a modest improvement with these changes, cache load time is
significantly better, cache save time is reduced since there is now no need
to reprocess the data and cache is much smaller.
We can drop the compress_keys() code and internSet code from the shared cache
core since its no longer used and replaced by codeparser specific pieces.
(Bitbake rev: 4aaf56bfbad4aa626be8a2f7a5f70834c3311dd3)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If there is a corrupt/invalid cache file, we'd keep trying to reopen
it. This is pointless, simplify the code paths and delete the dead
file.
(Bitbake rev: c22441f7025be012ad2e62a51ccb993c3a0e16c9)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If we're writing out merged data to disk, its safe to assume that either
we loaded the data or couldn't. Loading it again is relatively pointless
and time consuming.
(Bitbake rev: 14c31e18e4a4a52ee54056aeefb09ff8c295b475)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This gives us an easy way to find out which recipes have been excluded
from world when there are many layers.
(Bitbake rev: cf33d4bfc53ae8dc6353fa295e1acd789d4fc2d2)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.
(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since we now get wildcards in the file checksum list in the cache, we
need to ignore them when checking to see if they still exist. This
fixes connman-gnome reparsing on every bitbake execution in OE-Core.
(Bitbake rev: 340c250fc664414ab2715a454bedbd19e8efe103)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The change to use the expansion cache in VariableParse was incorrect as
it was adding in references it shouldn't have been. This patch corrects
the codepaths and ensures the references are correct.
The cache version is bumped since the previous bug could have leave
to invalid checksum calculations and a clean cache is therefore desireable.
The impact of the bug was that sstate was not getting reused when it should
and some tasks were also being rerun when they should not have been.
(Bitbake rev: 8a42d082315bd6ce091d006bf83476db257fa48b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.
This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.
Known issues:
* Hob is broken with this patch since it writes to the configuration
and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
see if we can streamline it.
These are issues which will be followed up in subsequent patches.
This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.
(Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AUTOINC was meant to appear once at the start of the version string.
The list of names may not be sorted meaning it could get inserted in
the middle. This patch simplifies the code and ensures it appears at
the start.
Include cache version bump to ensure the cache picks up these changes.
(Bitbake rev: ad8bf10d873abb94d987860a3f6d06b134fb8a99)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Go through the cached list of file checksums and check if any of the
files no longer exist; if any are missing then invalidate the cached
recipe, which will force it to be reparsed and thus force the list of
files to be collected again. This prevents a warning when moving a file
to a different location that is still picked up by the recipe, e.g.
moving a file from a "files" subdirectory to one named with the recipe
name (${BPN}).
Fixes [YOCTO #4474].
(Bitbake rev: e1aa103593c46612c4d9000524174983b5994dc9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some apparently debugging was left in in a previous commit. This caused
bitbake to return a list of bbappends when things changed from the cache.
Make this a proper debug message.
(Bitbake rev: 1965e5cbdfede18d7b7cb0218e0a5147c3f1c884)
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Python 3 is stricter about how files are accessed. Specficially:
* Use open(), not file()
* Use binary mode for binary files (when checksumming)
* Use with statements to ensure files get closed
* Add missing file close statements
(Bitbake rev: 9f08b901375ba640f47596f1bcf43f98a931550f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
All it now does is function in a similar way to EXCLUDE_FROM_WORLD and
since we have a better named variable for this, lets just drop the
usage of BROKEN at the bitbake level.
(Bitbake rev: 8f4dcf794545273417d78ba18f51aa2b81606ae4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There are two "self.file_depends =" lines in cache.py::CoreRecipeInfo:
class CoreRecipeInfo(RecipeInfoCommon):
__slots__ = ()
cachefile = "bb_cache.dat"
def __init__(self, filename, metadata):
self.file_depends = metadata.getVar('__depends', False)
[snip]
self.file_depends = metadata.getVar('__depends', False)
They are duplicated, remove the last one.
[YOCTO #3795]
(Bitbake rev: 409a4aa0e166c25a8cfcd91e54f1b61cced74e87)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The data returned by get_file_depends() may me used in contexts like
checksums where order is important. The current usage of sets means
that some of the checksums can change in circumstances they should not.
This patch changes to use lists, thereby removing the problem.
(Bitbake rev: a44285fc4109236ab89f7aad0a1fc9220eec19b6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently when we execute a task, we don't remove other potentially stale
stamps. This can mean if you switch between two different versions of a
recipe without a clean, the build can get very confused.
This patch adds in functionality to allow a wildcard expression of stamp
files to be removed when creating a new stamp file. This patch adds in
the core of the code to enable this but it also requires metadata support
to enable it.
When writing this improvement I went through several different options but
this was the only way I could find to allow things like noexec tasks to
function correctly (where stamps need to be created without the data store).
[YOCTO #2961]
(Bitbake rev: e026469b307522e5b6a680e0ae5587749d33dcae)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Gathers a list of paths to have checksums calculated at parse time, and
processes these when calculating task hashes. Checksums are cached with
the file's current mtime. Thus, changing any local file in SRC_URI will
now cause the do_fetch taskhash to change, thus forcing a rebuild.
This change adds very roughly about an 8% increase in parse time (a few
seconds) and maybe a few seconds during runqueue generation, so a fairly
moderate performance hit.
Note that since paths are resolved at parse time, this will not force
a rebuild when files are introduced which would cause that resolved path
to be different - for example, where a machine-specific version of a file
was added without otherwise changing the recipe. This will need to be
handled in a future update.
Code to hook this into the signature generator was courtesy of
Richard Purdie <richard.purdie@linuxfoundation.org>.
Implements [YOCTO #2044].
(Bitbake rev: c993b7c457f8b7776e8a5dff253bfa0724bc2cae)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We want to be able to reuse most this functionality for the file
checksum cache.
(Bitbake rev: 0fe3cb1438d297f90dd0fc6b26362ecbff75c76d)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The changes include:
- Clean some events in event.py
- Fire essential events for Hob2 to handle with more information.
- knotty changes
(Bitbake rev: 9ede881620c501574f014e600cea6947ea908ac2)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously we use the file time stamp to judge if a cache is valid.
Here this commit introduce a new method, which calculates the total
hash value for a certain configuration's key/value paris, and tag
it into cache filename, for example, bb_cache.dat.xxxyyyzzz.
This mechanism also ensures the cache's correctness if user
dynamically setting variables from some frontend GUI, like HOB.
(Bitbake rev: 1c1df03a6c4717bfd5faab144c4f8bbfcbae0b57)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the addition of new cache domains, the summary, license and section
information is no longer requred in the core cache since its only used by
the hob UI. This patch removes the duplicated entries.
It also adds the DESCRIPTION field to the cache for the benefit of hob2.
(Bitbake rev: 33ffb2e99825cb643b148b3462c2d4cf33ff5f58)
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add a FAKEROOTNOENV which does the opposite of the FAKEROOTENV variable
and is data loaded into the environment for tasks without the fakeroot
flag.
The intent here is to provide a way to control the environment when we
aren't needing fakeroot context which allows us to unload the preload
from memory entirely and gain a valuable speedup.
I'm not 100% happy with needing this at the bitbake level, particularly
with the cache hit but it does give a valuable speedup.
(Bitbake rev: 05c29ab0b2ad3c521414cabb6a92bca15c6e919c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The bNeedUpdate variable doesn't reflect its purpose, and doesn't match
coding style (type encoded in variable name, camel case) - rename to
cache_ok.
(Bitbake rev: cc4158db215493ac270e27f92d977bcbe337f3db)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
"Nothing PROVIDES" errors often come up when a recipe has been skipped
for some reason, and therefore it is useful to print out that reason
information when showing the error so that the user understands why the
error has occurred.
Given that we already feed the reason information into the skiplist for
various situations (COMMERCIAL_LICENSE, COMPATIBLE_MACHINE etc.) this
should now output a useful error message for skipped recipes.
Fixes [YOCTO #846], [YOCTO #1127]
(Bitbake rev: 6765218430e31c165888f26fbc75023c89a6eab2)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures we understand e.g. "virtual:multilib:lib64:pn" for multilib.
(Bitbake rev: 0b9dee799cfe2229ec62c257dda36bd6c15d34a9)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if a user adds a new .bbappend file to the system, the cache still
thinks the cached data is valid. This code fixes that to ensure additions and
changed in append application order are accounted for.
[YOCTO #1091]
(Bitbake rev: 54fe91fe96aaae47c40077c5f441c79da71da777)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch, based on proof of concept code from Richard adds code to
bitbake to allow individual tasks to optionally specify their stamp
file using the stamp-base flag. This takes the same form as the STAMP
variable but can be specified on a per task basis.
Code is also added to runqueue to ensure that if two tasks share the
same stamp file, only one will be executed at once.
A significant usecase for this code is to share source code (${S})
between recipes where separate build directories (${B}) are used.
(Bitbake rev: 41bef02bef8379590ba012319aebe05068a8081e)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This uses the correct index of self.pn when setting up cachedata's
pkg_pn, fixing the output of bitbake -s.
This fixes bug [YOCTO #1149].
(Bitbake rev: d000493c09ac5c1dcbab22d3a91296a9cb194ac0)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add skiplist to cooker that allows access to the list of packages skipped
via SkipPackage (this includes COMPATIBLE_MACHINE, INCOMPATIBLE_LICENSE,
etc.) This can be used to enhance error reporting.
(From Poky rev: 6c12b7b1099c77b87d4431d55e949cf7c5f52ded)
(Bitbake rev: 7d2363f35350be27a33f568c23eb07fcd3d27e53)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is to support extra cache. If user needs to request
extra cache fields besides CoreRecipeInfo fields, just add a new
XXXRecipeInfo class definition as Hob Does.
Currently supported Extra RecipeInfo name is an array. We can
support multiple extra cache fields at the same time besides
CoreRecipeInfo which is needed by all clients.
(Bitbake rev: 077657e50ad032c0fa876bf54e9802af2686e0fb)
Signed-off-by: Liping Ke <liping.ke@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using hob ui interface, we need extra cache fields.
We will save ui required extra cache fields into a separate
cache file. This patch introduce this caches_array parameter.
It will be used in the extra cache implementation (following patch).
Caches_array at least contains CoreRecipeInfo. If users need extra
cache fields support, such as 'hob', caches_array will contain
more relevant elements such as HobRecipeInfo.
(Bitbake rev: d50389ae692377c957afec7c846fc2ce2c070a09)
Signed-off-by: Liping Ke <liping.ke@intel.com>
This patch is for refactorying current cache implementation, the
main reason is for introducing extra cache fields requests for
image creator as well as other users. The refactory parts include:
Move cache data retrieve methods out of Cache Data Fields
Definition. Since this retrieve methods will be shared for
both CoreRecipeInfo as well as the new introduced extra RecipeInfo
in the following patches.
(Bitbake rev: f0f53506926a3f79181796dde177f11f0a396b75)
Signed-off-by: Liping Ke <liping.ke@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Markus Lehtonen
Paul Eggleton
Robert Yang
Chen Qi
Robert P. J. Day
Mark Hatle
Shane Wang
Dongxiao Xu
Joshua Lock
Christopher Larson
Scott Garman
Liping Ke