The recent change to Queue up events before the UI is spawned (in
26eda93337) broke the xmlrpc server because the
uievent implementation of BBUIEventQueue expects pickled strings for its
queue_event() method.
This is because the RPC exposed event.send() method must accept pickled
strings, but for xmlrpc event.send() is just mapped to queue_event().
Work around this by adding a send_event method which unpickles strings and
hands them off to queue_event() which can then be used for the remapping.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
It turns out that while log filters added with addFilter are only associated
with that logger, and not its children, handlers are inherited, and handlers
can be filters. So, let's add filtering to our existing LogHandler class
which dispatches our log records as bitbake events.
(Bitbake rev: 0153ace246e7c88366f45c8f035a2b4505a1c115)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
we should cache SRCREV whenever possible, the only exception is
when SREREV is auto rev. so change the logic to only set __BB_DONT_CACHE
at SRCREV = "${AUTOREV}" case
Signed-off-by: Yu Ke <ke.yu@intel.com>
Current fetcher has annoying "SRCREVINACTION" deadlock,
which occurs when SRCREV=${AUTOREV}=@bb.fetch.get_srcrev():
get_srcrev()->setup_localpath()->srcrev_internal_helper()
->evaluate SRCREV->get_srcrev()
current fetcher resolve the deadlock by introducing a
"SRCREVINACTION" condition check. Althoguh it works, it is
indeed not clean.
This patch use antoehr idea to break the deadlock: break
the dependency among SRCREV and get_srcrev(), i.e. assign
a specific keyword "AUTOINC" to AUTOREV. when Fetcher meet
this keyword, it will check and set the latest revision to
urldata.revision. get_srcrev later can use the urldata.revision
for value evaluation(SRCPV etc). In this case, SRCREV no longer
depends on get_srcrev, and there is not deadlock anymore.
Signed-off-by: Yu Ke <ke.yu@intel.com>
FetchData has some fetch method specific data, and only fetch method knows how
to initialize it. originally it is mostly initialized in Fetch.localpath().
But now there is requirement to call Fetch.latest_revision() before
Fetch.localpath(), thus require another earlier place for initialization. so
urldata_init is introduced for this purpose. it will be called in FetchData:__init__
and make all the Fetch functions useable after that.
Signed-off-by: Yu Ke <ke.yu@intel.com>
bb.fetch2 is copied from bb.fetch, and has many bb.fetch referrence.
Fix these referrence with bb.fetch2 referrence
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We will be needing this information to improve the tracebacks of python code
from the metadata, as well as to give the user information about where
variables were defined, so they know how it ended up the way it is.
(Bitbake rev: 9615c538b894f71a2d1a0ba6b3f260db91e75786)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Errors can result from these expansions, but for skipped recipes, we
shouldn't care about those failures. This fixes the same issue which
Richard Purdie fixed in poky, commit 847b717.
(Bitbake rev: 96ee6840010c1ae1080e6bf7ff0f4eb2d361e84b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This cleans up the knotty console messages to be a lot quieter and cleaning,
in keeping with the expectations of most users.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
[BUGID# 645], modify the emit_var()
1. Added "#" to the beginning of each line if the comment contains
multiple lines.
2. Added "\" to the end of each line if the shell variable value
contains multiple lines.
Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
The current parameters are not useful to the stampfile generator function
as they can't uniquely define a task. This updated things so the
parameters can identify unique tasks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Move stamp file deletion out of the internal stamp helper function
* Add a new function to return the path to a stamp for a given task
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We can use the string split method for this instead.
(Bitbake rev: aa9646717b3ee1006628246a7c495f601e62391c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This ensures that when a failure occurs very early on in bitbake startup, the
message formatting ematches that used by the UIs.
(Bitbake rev: c8ff0fd3e9f050a668f1a069cf37ee37db3664fa)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I think this is less confusing, and avoids needing to know about the *range*
of logging levels, instead simply asking what we really want to know.
(Bitbake rev: dc2264387617586b5c0a61e126c75edde5e7abcd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The various alternative UIs have been updated to once again be functional
with the latest bitbake internals. Each of the UIs still have much room for
functional improvement.
In particular, they have been updated to:
- interact with the new process based server
- handle the current set of events and notifications fired from the server
and its associated subsystems
(Bitbake rev: b947e7aa405966262c0614cae02e7978ec637095)
Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
python tasks calling shell functions using exec_func() would show the log
file as /dev/null. It makes most sense for all the task logging to be setup
centrally by exec_task(), at least with the current code base in Poky.
This commit will need discussion in relation to upstream bitbake and the
IO redirection could be better handled using a context manager (although
task contexts shouldn't ever nest).
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Think this got inadvertantly dropped when switching to the new API.
(Bitbake rev: 628c5159d1151b89f2b7210c8819489e8dc9a84d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Rather than having to run .addDomain() and then .getValue(domain, key),
.setValue(domain, key), etc, now it just works as mappings.
As an example:
setValue(domain, key) -> persist[domain][key] = value
It also arranges things so we can construct objects of this type using any
arbitrary filename/path for the sqlite3 database, rather than being so
tightly bound to the metadata.
(Bitbake rev: d9e8b8af308ae871efdc8ef0782be30af8c1f894)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Caching the database connection can cause serious issues if it results in
multiple processes (e.g. multiple tasks) simultaneously using the same
connection.
This reverts commit 8a6876752b90efd81d92f0947bfc9527d8260969.
(Bitbake rev: 60b9b18eafad5ac46c7cf1048d749d673c2ee0ad)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Only mark fn as clean if it is clean.
This saves us from removing (prematurely added) fn from our clean set
and saves me a few percent of runtime (and misleading debugging output
from remove()).
(Bitbake rev: 884365228fcaac07421ac1440d4946693fb628c5)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Previously we emitted two newlines for export and unexport.
One newline for export and unexport is enough (and makes the scripts
look better and a tad smaller).
(Bitbake rev: ba060160fdf1278a273fb2b77d36b8c681807ecf)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
When using a logfile, we weren't sending input to the child process.
(Bitbake rev: 5ec4ca7e45bdf6d259503fc67155395e89ba6329)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Without sorting, it's very difficult to find the information you're
looking for. Now, the lists are all sorted alphabetically for easy
viewing.
(Bitbake rev: 80e3d3a130b9dee72c11c6946bb5ff7705111d7c)
Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: bdd7813d8eecf7b6b636322e748ca6bf69118513)
Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: f42e6d33b986183eee2c74aa7b1bbc9b339276a9)
Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* Allows generating version information from SCMs during build.
* Note that tar doesn't need to use --exclude '.git', because
git checkout-index doesn't clone the repository.
(Bitbake rev: 05cbc1d1a01c667c77688f36fbc5b61c5f452a3a)
Signed-off-by: Andreas Oberritter <obi@opendreambox.org>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The previous attempt was incorrect. The issue isn't that subprocess fails to
set PWD, it's that PWD is in the metadata, inherited from the environment, and
is re-exported, overwriting the actual accurate one in the shell environment
with the old one from the metadata. So, ensure that PWD in the metadata is
not exported.
We can ditch this when the environment handling is reworked (e.g. poky's
commit to do so).
(Bitbake rev: 2c8683234acf514706b2b69f5b29405485e664dd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Moved the logic for comparing revisions from cooker into command
- Removed 'Cooker' from the event names
- Renamed the 'ExitCode' event into CommandExit, and changed CommandFailed to
be a subclass of CommandExit
(Bitbake rev: c51ed5d7a9971fad6019dac6c35a71b8a54ab16a)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We now wrap the main() in a try/except, ensuring that both the main portion of
bin/bitbake and the UI raising an exception will be shown to the user. For
the server and workers, we can ensure in the server itself that exceptions are
handled correctly.
(Bitbake rev: 240d4a7ae80a6636c302ae84266ddfed7a7fcedd)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Also don't bother passing logfile to exec_func_python, at least until we start
adding the logfile as a file handler to the bitbake logger.
(Bitbake rev: f99ee4680c9f67b7ed13fc06044ba2382f9a782c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: 53740977521bc81ffa37adfa7bbeb8f2a80ea165)
build: write logfiles per task, not per function
Based on d14f9bf6 from poky, reworked for master and other cleanup.
(Bitbake rev: beadff2eca1eb95f0411115dd72ddb4c3c44c604)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
As these may run the UI in a blocking fashion and then return the exit code,
'init' was an inappropriate name, and 'main' is more appropriate.
(Bitbake rev: 4d081a0ed759bd526ab01849d650bd9e8d80ddd1)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We want to match the requested pattern at the beginning of the string,
otherwise things behave in an unintuitive manner wrt ASSUME_PROVIDED (e.g.
ASSUME_PROVIDED += "gtk+" will also assume foo-gtk+ is provided), and the user
can always use '.*gtk+' to get the old behavior.
(Bitbake rev: 5670134ab2eb573d39df3c3231677cdb1a1dfc72)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Rather than updating the progress bar based on the recipe being processed
(whether cached or parsed), consider only parsed recipes. This reduces the
instability in progress rate introduced by the cached entries, and allows the
ETA to be resurrected and be a bit more useful.
(Bitbake rev: 618480f7739f6ae846f67a57bee5a78efb37839d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Currently, the progress bar is an indication of the processing of our recipes,
which includes loading the cache file, then for each recipe, either adding the
existing cached information to the CacheData or parsing the recipe from disk.
These tasks clearly take different amounts of time, so the ETA is unreliable
today. We'll resurrect this functionality after we revamp the progress
handling, fully incorporating the load of the cache file.
(Bitbake rev: 80867372dcbef91ebaf7d77a77ca871741dd3f74)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Without explicitly joining the thread, it's possible for the process to end
(e.g. after a bitbake -p) and kill off the thread without waiting for it to
exit cleanly. So, register the thread join with atexit.
(Bitbake rev: 97ce57e6f860d3e6f34cc7a603ed1eeac4f423d3)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This ensures that the time spent loading the cache from disk occurs with the
progress bar up. Though the progress bar stays at 0% during this period, I
think this is an improvement over the multi-second stall which occurred
previously before the progress bar came up. Ideally, we'd integrate cache
loading from disk into the progress display, but this is a first step.
(Bitbake rev: f6d0a5c219f9deb84f702450d30d868ba6271f77)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Previously, the cache was actually being loaded from disk twice whenever using
-b or -e -b. This also moves the bb_cache instance into the CookerParser, as
it's not needed by the cooker itself at all.
(Bitbake rev: dd0ec2f7b18e2a9ab06c499b775670516bd06ac8)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This version uses a thread rather than a process, to avoid problems with
waitpid handling. This gives slightly less overall build time reduction than
the separate process for it did (this reduces a -c compile coreutils-native by
about 3 seconds, while the process reduced it by 7 seconds), however this time
is quite insignificant relative to a typical build.
The biggest issue with non-backgrounded syncing is the perceived delay before
work begins, and this resolves that without breaking anything, or so it seems.
(Bitbake rev: 5ab6c5c7b007b8c77c751582141afc07c183d672)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Rather than adding nocache items to the cache, then copying the cache and
removing them to sync it, don't add them in the first place. Also use 'with'
for the cachefile.
(Bitbake rev: 343b6f6255ad020c39e30742175a241f0859a5a6)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This utilizes python's multiprocessing module. The default number of threads
to be used is the same as the number of available processor cores, however,
you can manually set this with the BB_NUMBER_PARSE_THREADS variable.
(Bitbake rev: c7b3ec819549e51e438d293969e205883fee725f)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
If the only recipes's we reparsed this run were those flagged as not to be
cached, there's no point in re-saving the cache, as those items won't be
included anyway.
(Bitbake rev: 1e0c4dbcbec886a30b89f8b4bb365c3c927ef609)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This class holds the particular pieces of information about a recipe which are
needed for runqueue to do its job.
By using it, I think we improve code clarity, reduce method sizes, reduce
overuse of primitive types, and prepare for parallel parsing. In addition,
this ditches the leaky abstraction whereby bb.cache attempted to hide the
difference between cached data and a full recipe parse. This was a remnant
from the way things used to be done, and the code using it had to know the
difference anyway. If we choose to reimplement caching of the full recipes,
we can do it in bb.parse, in a completely transparent way.
(Bitbake rev: 992cc252452221f5f23575e50eb67528b2838fdb)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
range() allocates an actual list when called. xrange() is just an iterator
and creates the next range item on demand. This provides a slight
performance increase.
In python 3, range will do what xrange does currently, but the upgrade will
be handled by the 2to3 tool.
(Bitbake rev: 73b40f06444cb877a5960b2aa66abf7dacbd88f0)
Signed-off-by: Bob Foerster <robert@erafx.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
1) too spammy
2) can be implemented in the metadata instead
This reverts commit 8da9744fcdf856abebcfbe9e3bc1b8cf07bc317b.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
as noted by rp in ac00ca89a4e43cd4f38ba86455079d31be78e644
(Bitbake rev: 8da9744fcdf856abebcfbe9e3bc1b8cf07bc317b)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
no functional changes
(Bitbake rev: e88834fb7c6821cc29c12d296f2edd51f6eb3746)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Several fetcher need a way to strip leading slashes off a local path.
This helper-function consolidates all such occurances.
(Bitbake rev: 823a02185ed109054c6c1ae366221aaed0353f24)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
osc had it already spelled correctly?!
(Bitbake rev: b8bb4433de7a981c6826173e926ca34705c4ac70)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
If a recipe depends on a file, and that file is out of date, we show a
message, but if that file was removed, we do not, until now.
(Bitbake rev: 67984ba0ac2db79874541bc031f2e3e9ff7a6c32)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Parallel processes interacting with the persist_data db can quite easily
explode without this.
(Bitbake rev: b3d5432cff0ff28f4c8a5bcf10efa3e383b4fd4d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This avoids alot of misleading log-messages like "Removing FOO from cache"
if FOO was not in the cache and as such is not a removal candidate.
(Bitbake rev: de34a403e206867e09410ad4925c7b9cff04fee6)
Signed-off-by: Bernhard Reutner-Fischer <rep.dot.nop@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
The pysh we're using is modified, and we don't want to risk it conflicting
with one from elsewhere.
(Bitbake rev: 1cbf8a9403b4b60d59bfd90a51c3e4246ab834d6)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
There are occasions when developing when I want a package always to
grab the latest copy of a package. Witht eh CVS fetcher you can do
this by setting the `date' tag to `now'. This patch adds similar
functionality to the mercurial fetcher: if the revision to fetch is
`tip' then always grab from the server, and don't use the cached
tarball.
Oh, and I fixed a typo in the Class comment.
(Bitbake rev: 01b85608d8a37f8af66dfd80133e950120679079)
Signed-off-by: Peter Chubb <peter.chubb@nicta.com.au>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Use bb.utils.explode_deps to break up the rdepends and rrecommends strings.
This fixes the same issue which was fixed by a number of patches floating
around, but uses explode_deps rather than regular expressions.
(Bitbake rev: 83cdb23f8b89453a3527a276bd0b4deb85d63deb)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
* without this fix, we get :
updating working directory
74 files updated, 0 files merged, 0 files removed, 0 files unresolved
abort: There is no Mercurial repository here (.hg not found)!
(Bitbake rev: 75ea005ac8fc05b2b3afca803d77a6b5f558efee)
Signed-off-by: Eric Bénard <eric@eukrea.com>
Tested-by: Frans Meulenbroeks <fransmeulenbroeks@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This was inadvertantly removed when trying to reduce the amount of duplicated
information the user sees when a failure occurs.
(Bitbake rev: 850d6158ea9daa58e896fd6b258d586df797dcf4)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Ensure it raises KeyError for a missing key, this is required to use this as a
mapping in various places, e.g. as locals in an eval.
(Bitbake rev: 8d661ce0c303e8d69f17c1d095545d5ed086d1d5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This option will exclude the SCM metadata from tar files.
Tested with gcc where svn tar which used to be 156M for gcc 4.5
is now 77M
(Bitbake rev: f264cb6d43472525ad787b0887764ea696ec52ba)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Queue up any events fired to the UI before the UI exists
- At exit, check if UIs exist, and if not, flush the queue of LogRecords to
the console directly.
- When establishing a connection from the UI to the server, flush the queue of
events to the queue in the server connection, so the UI will receive them
when it begins its event loop.
(Bitbake rev: 73488aeb317ed306f2ecf99cc9d3708526a5933c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Don't store key/value pairs when the value is None
- Delete the depends_cache when we're done with it
This reduces the memory usage after sync on initial parse by roughly 11.5% on
this machine.
(Bitbake rev: c7eb4c989459d182fdf9c81a627d32b7ef11626b)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
A SystemExit from a python function wasn't being raised as a FuncFailed, which
resulted in it not being caught by the exception handlers in the runqueue for
the worker process, which resulted in a SystemExit exit, rather than os._exit,
which causes all manner of problems when used in a forked process. This fixes
it by ensuring we raise a FuncFailed when seeing exceptions which aren't
instances of Exception.
(Bitbake rev: dafe92fe9f387450d9f9e9ff41c99388998b7495)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Per the python documentation, os.waitpid returns the exitcode shifted up by 8
bits, and we weren't compensating, resulting in a display of 'failed with 256'
when a worker process exits with a code of 1.
(Bitbake rev: 90c2b6cb24dc9c82f0a9aa9d23f2d1ed2e6ff301)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
- Drop EventException
- Use FuncFailed as the primary function failure exception, using TaskFailed
for the event (leaving it up to the process running exec_{func,task} to
display the more detailed information available in the exception).
- Switch InvalidTask to an exception rather than an event, as that's a
critical issue.
- Reduce the number of messages shown to the user when a task fails -- they
don't need to be told it fails 12 times. Work remains in this area though.
(Bitbake rev: 06b742aae2b8013cbb269cc30554cff89e3a5667)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
We use a custom Logger subclass for our loggers
This logger provides:
- 'debug' method which accepts a debug level
- 'plain' method which bypasses log formatting
- 'verbose' method which is more detail than info, but less than debug
(Bitbake rev: 3b2c1fe5ca56daebb24073a9dd45723d3efd2a8d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This kills firing of Msg* events in favor of just passing along LogRecord
objects. These objects hold more than just level and message, but can also
have exception information, so the UI can decide what to do with that.
As an aside, when using the 'none' server, this results in the log messages in
the server being displayed directly via the logging module and the UI's
handler, rather than going through the server's event queue. As a result of
doing it this way, we have to override the event handlers of the base logger
when spawning a worker process, to ensure they log via events rather than
directly.
(Bitbake rev: c23c015cf8af1868faf293b19b80a5faf7e736a5)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
(Bitbake rev: f7c181a0f6ab0b4d33bf80a0e24a788de441f82b)
Signed-off-by: C Michael Sundius <msundius@sundius.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Only shows warnings that come from bb, oe, or <string>
(Bitbake rev: 57018687f60b222ab220dd904c4bf870780171e9)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Provide __len__, __iter__, and the getitem/setitem/delitem methods, and its
mixed in versions of keys(), values(), items(), etc will automatically behave,
making the DataSmart act more like a real mapping.
(Bitbake rev: 89b5351c656d263b0ce513cee043bc046d20a01e)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
It needs to be a generator, so scheduler subclasses have the option to skip
buildable tasks and return a later one.
(Bitbake rev: a8c61e41bc6277222e4cde667ad0b24bd1597aa0)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
If you create a runqueue scheduler class in a python module, available in the
usual python search path, you can now make it available to bitbake via the
BB_SCHEDULERS variable, and the user can then select it as they select any
other scheduler.
Example usage:
In a test.py I placed appropriately:
import bb.runqueue
class TestScheduler(bb.runqueue.RunQueueScheduler):
name = "myscheduler"
In local.conf, to make it available and select it:
BB_SCHEDULERS = "test.TestScheduler"
BB_SCHEDULER = "myscheduler"
(Bitbake rev: 4dd38d5cfb80f9bb72bc41a629c3320b38f7314d)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
SIGINT should be from the user, not a script. It also doesn't work as
reliably to shut down processes, as it's not always interpreted as a
termination request. In addition, it causes KeyboardInterrupt exceptions in
the worker processes, which can interfere with our exception handling.
(Bitbake rev: e5f6e0e9de4c6d1dfdd269d2bf7f83c00c415a27)
Signed-off-by: Chris Larson <clarson@kergoth.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Bug 606 report that if $DL_DIR is read-only, do_fetch will
simply hang without any error message.
The root cause is that: bb.fetch.go()->bb.utils.lockfile()
will try to lock file ${DL_DIR}/xxxxx.lock. Since ${DL_DIR}
is read-only, it will cause IOError exception. Although
lockfile() can catch the exception, currently code simply
ignore all the exception and continue the loop. it make
sense if the exception is caused by locking contention,
but in the read-only $DL_DIR case, it cause endless waiting
unfortunately.
So this patch add read-only check for lockfile to avoid the
silent hang.
Fix [BUGID #606]
Signed-off-by: Yu Ke <ke.yu@intel.com>
Stupdi typo/thinko from me had depexp exiting once recipes had parsed
as I'd used a return the while loop where I'd meant a continue...
Signed-off-by: Joshua Lock <josh@linux.intel.com>
In previous exec() model, cooker is re-initialized from scratch with environmental
variable exported accordingly. Now in fork() model, environmental variables are
not exported again, and thus original method to export BB_TASKHASH doesn't apply
now which breaks all sstate packages. Now we can set data variable directly instead.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
If the checksum check failed, the .md5 stamp file would still have been created
meaning subsequent builds would proceed with the corrupt file. Reorder the calls
to avoid this. Also raise a specific error for the checksum not specified error
case.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
This patch add the per-recipe SRC_URI checksum verification.
- SRC_URI format
The format of SRC_URI checksum follow OE definition:
1. SRC_URI has single src
SRC_URI = "http://some.domain/file.tar.gz"
SRC_URI[md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
2. SRC_URI has multiple src, every src need specify name
SRC_URI = "http://some.domain/file1.tar.gz;name=name1 \
http://some.domain/file2.tar.gz;name=name2 "
SRC_URI[name1.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name1.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
SRC_URI[name2.md5sum] = "xxxxxxxxxxxxxxx"
SRC_URI[name2.sha256sum] = "xxxxxxxxxxxxxxxxxxxxxx"
- SRC_URI checking invocation:
the checksum checking is invoked in do_fetch phase,
so it can be invoked manually by
# bitbake -f -c fetch <recipe_name>
if recipes has no SRC_URI checksum item, bitbake will show warning:
"
WARNING: Missing SRC_URI checksum for xxxx.tar.gz, consider to add
SRC_URI[md5sum] = "5c69f16d452b0bb3d44bc3c10556c072"
SRC_URI[sha256sum] = "f4e0ada8d4d516bbb8600a3ee7d9046c9c79e38cd781df9ffc46d8f16acd1768"
"
thus recipe author can add it to recpie file after SRC_URI
- control variable BB_STRICT_CHECKSUM
when SRC_URI checksum is missing, this variable decide pass or not
if BB_STRICT_CHECKSUM = "1", bitbake should fatal in this case, otherwise bitbake just pass
Signed-off-by: Yu Ke <ke.yu@intel.com>
sstate hash validation is done at initialization of RunQueueExecuteScenequeue.
However the index of 'valid' list returned from the validation doesn't
correspond to setscene task ID. It's just an intermediate namespace between
runqueue and sstate hash func. Use it as setscene task ID fully mess the flow.
Previously this doesn't cause trouble because all setscene tasks are passed. Commit
58396a5d24 add 'noexec' concept to setscene
tasks which grabs some tasks out of the list and thus trigger this problem
Without this fix there're ~50 recipes (gzip-native, glib, ...) rebuilt weirdly
with a minimal build, even though existing sstate packages could accelerate them.
there's another typo using wrong task ID in a debug message which further hide
this issue
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Add a parameter to getVarFlag() to auto-expand the value of the flag. This
makes getVarFlag() more consistent with getVar(), and allows expansion of
vardeps and vardepsexclude (which has been done in this commit).
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Adds a vardepsexclude flag that can be used to exclude a dependency of
a variable (the opposite of vardeps). This will allow the exclusion of
variables from the hash generation much more selectively than blanket
whitelisting using BB_HASHBASE_WHITELIST.
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Currently, anything whitelisted in the environment makes it into the worker
processes. This is undesireable and the worker environment should be as
clean as possible. This patch adapts bitbake sosme variables are loaded into
bitbake's datastore but not exported by default. Any variable can be exported
by setting its export flag.
Currently, this code only finalises the environment in he worker as doing so
in the server means variables are unavailable in the worker. If we switch
back to fork() calls instead of exec() this code will need revisting.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
idle command processing in each of the servers does not handle an explicit
None return value, which means the goggle UI ends up repeatedly adding
"Tasks Summary:" rows to the list.
This patch modifies BBCooker.buildTargets.buildTargetsIdle to return False
when BuildCompleted is fired, as is done in BBCooker.buildFile.buildFileIdle.
It may be that the correct way to fix this is to change the idle command
processing in the servers.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Construct a ProgressBar and pass it to the RunningBuild.handle_event() so
that goggle users are notified of metadata parsing progress.
UI's with status make users less nervous
Signed-off-by: Joshua Lock <josh@linux.intel.com>
It seems likely that the user would want to view the most recently emitted
messages so this patch sets the message dislay treeview to scroll to any
newly added rows.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Python 2.7's library changes some of xmlrpclib's internal implementation such
that interacting with a proxy to BitBakes SimpleXMLRPCServer would cause
BitBake to crash.
The issue was traced to changes in the xmlrpclib.Transport implementation and
Python bug #8194 (http://bugs.python.org/issue8194).
This patch introduces a workaround by create a subclass of
xmlrpclib.Transport, which overrides the offending methods with the Python
2.6.6 implementation copy and pasted from the Python 2.6.6 xmlrpclib, and
using this BBTransport implementation for both xmlrpclib.Server objects we
create.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
ProgressBar will be useful in other UI elements so make it it's own class.
Make ProgressBar a subclass of gtk.Dialog, rather than gtk.Window, so that we
can suggest the window manager parent the ProgressBar to the widget passed at
as parent.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Due to some recent change *somewhere* we need to explicitly look at the
name attribute on the instances class, rather than the name attribute of
the instance.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Move the test for ignored messages to the start of the message handling loop to
avoid doing work for messages which are only going to be ignored.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Some of the events we where trying to look for have had their class names
changed, fix these references.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
BB_TASKHASH_WHITELIST is expected to filter out native tasks from the
dependency list for target recipe's checksum. However current code
actually implements the opposite. All native sstate packages end up
to have empty task dependency while target sstate packages still have
native tasks counted into the checksum.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Where a variable name consisted of an append/prepend combined with an override
and there was also an append/prepend to the variable, the override could be lost
if the override was not in OVERRIDES.
For example:
FOO = "A"
FOO_append = "B"
FOO_append_virtclass-native = "C"
could result in "AB" even though virtclass-native was in OVERRIDES.
With this patch applied, the result is "ABC" as would be expected.
The problem was the deletion of the _append/_prepend flag was happening
if *any* append/prepend was procesed, the result should really be that
it should contain any unprocessed append/prepend.
Kevin Tian deserves credit for looking into this and working out the
problem here.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Now 'runtaskdeps' is assigned before taskhash whitelist is filtered out, and thus
always contains original task dependencies. This causes problem to diagnose sstate
related problem. So fix it by appending filtered dep to 'runtaskdeps' in the fly.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
This means the noexec messages are only shown once as the stamp files are now
correctly created.
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
two dep lists should be sorted before the comparison, or else it just
causes false-positive confusion.
Also fix 'taskdeps' to 'runtaskdeps' when printing out the difference.
Signed-off-by: Kevin Tian <kevin.tian@intel.com>
Joshua Lock
Richard Purdie
Chris Larson
Yu Ke
Lianhao Lu
Richard Purdie
Bob Foerster
Bernhard Reutner-Fischer
Bob Foerster
Andreas Oberritter
Peter Chubb
Eric BENARD
Khem Raj
C Michael Sundius
Holger Hans Peter Freyther
Chris Larson
Kevin Tian
Paul Eggleton