Commit Graph

106 Commits

Author SHA1 Message Date
Richard Purdie 631229df72 bitbake: event/ast: Add RecipeTaskPreProcess event before task finalisation
There are various pieces of code which need to run after the tasks are
finalised but before bitbake locks in on the task dependencies. This
adds such an event so dependency changes in anonymous python can
be accounted for and acted upon by these specific event handlers.

(Bitbake rev: 4dcd0e53f5ff4bf4f2d6cbdc51ff33a5f5f206af)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-20 11:52:52 +00:00
Richard Purdie 840e221ea7 bitbake: event: Replace slightly elderly bbmake references!
bbmake hasn't existed for a while (a decade+) so we should
probably remove the references.

(Bitbake rev: 4170cf874850b950f31d2e36c895c110c0b096c3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2017-01-09 13:39:13 +00:00
Patrick Ohly 268bf22263 bitbake: monitordisk: add event
The current disk usage is interesting and may be worth logging over
time as part of the build statistics. Instead of re-implementing the
code and the configuration option (BB_DISKMON_DIRS), the information
gathered by monitordisk.py is made available to buildstats.bbclass via
a new event.

This has pros and cons:
- there is already a useful default configuration for "interesting" directories
- no code duplication
- on the other hand, users cannot configure recording separately from
  monitoring (probably not that important)

(Bitbake rev: f065ac17d0031dca6309ddbff18c8792630de865)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Patrick Ohly 083365143e bitbake: cooker process: fire heartbeat event at regular time intervals
The intended usage is for recording current system statistics from
/proc in buildstats.bbclass during a build and for improving the
BB_DISKMON_DIRS implementation.

All other existing hooks are less suitable because they trigger at
unpredictable rates: too often can be handled by doing rate-limiting
in the event handler, but not often enough (for example, when there is
only one long-running task) cannot because the handler does not get
called at all.

The implementation of the new heartbeat event hooks into the cooker
process event queue. The process already wakes up every 0.1s, which is
often enough for the intentionally coarse 1s delay between
heartbeats. That value was chosen to keep the overhead low while still
being frequent enough for the intended usage.

If necessary, BB_HEARTBEAT_EVENT can be set to a float specifying
the delay in seconds between these heartbeat events.

(Bitbake rev: 7cf22ea057d28c54bd98dc1ab7a43402a29ff1f5)

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-12-07 10:42:22 +00:00
Aníbal Limón 924ca1037c bitbake: bb.event: fix infinite loop on print_ui_queue
If bitbake ends before _uiready and bb.event.LogHandler was add
to the bitbake logger it causes an infinite loop when logging
something.

The scenario is print_ui_queue is called at exit and executes
the log handlers [2] one of them is bb.event.LogHandler this handler
appends the same entry to ui_queue causing the inifine loop [3].

In order to fix a new copy of the ui_queue list is created when iterate
ui_queue.

[YOCTO #10399]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=10399#c0
[2] http://git.openembedded.org/bitbake/tree/lib/bb/event.py?id=41d9cd41d40b04746c82b4a940dca47df02514fc#n156
[3]
http://git.openembedded.org/bitbake/tree/lib/bb/event.py?id=41d9cd41d40b04746c82b4a940dca47df02514fc#n164

(Bitbake rev: 46fecca9d531a07788b5cac8b2dc6a8267d8b6d0)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-15 10:01:44 +01:00
Joshua Lock 1110dde73a bitbake: event: prevent unclosed file warning in print_ui_queue
Use logger.addHandler(), rather than assigning an array of Handlers
to the loggers handlers property directly, to avoid a warning from
Python 3 about unclosed files:

$ bitbake
Nothing to do.  Use 'bitbake world' to build everything, or run 'bitbake --help' for usage information.
WARNING: /home/joshuagl/Projects/poky/bitbake/lib/bb/event.py:143: ResourceWarning: unclosed file <_io.TextIOWrapper name='/home/joshuagl/Projects/poky/build/tmp/log/cooker/qemux86/20161004094928.log' mode='a' encoding='UTF-8'>
  logger.handlers = [stdout]

(Bitbake rev: 1e23b1f1a80066223b98e18b163840051ac74944)

Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:25:34 +01:00
Aníbal Limón 79012f943d bitbake: bb/event.py: fire_ui_handlers enable threading lock support
In some cases there is a need to fire bb events into multiple
python threads so locking is needed (writing to a fd/socket).

Adding a helper functions for disable/enable by request to avoid
overhead.

[YOCTO #10330]

(Bitbake rev: a583dc0b296415ec904c081c4de96ceef46732a8)

Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-10-05 10:10:11 +01:00
Ed Bartosh d649c64beb bitbake: event.py: output errors and warnings to stderr
All logging messages are printed on stdout when processing
UI event queue. This makes it impossible to distinguish between
errors and normal bitbake output. Output to stderror or stdout
depending on log level should fix this.

(Bitbake rev: 56ac0d4c7a5f47aeb707b15a0c305d9f73aae945)

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-09-15 12:15:07 +01:00
Elliot Smith c471740f5b bitbake: cooker: add BuildInit event
In situations where a bitbake run fails before the build
properly starts and BuildStarted is fired, a UI has no way
to get at the targets passed to the build. This makes it
difficult for the UI to report on the targets which failed.

Fire a BuildInit event before running buildTargets() or
buildFile(). This enables a UI to capture targets passed to
buildTargets(), even if the build fails (e.g. the targets
themselves are invalid).

[YOCTO #8440]

(Bitbake rev: ac02fda870965bf7d44ff5688eda54d2d11ab9c7)

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-08-11 00:09:26 +01:00
Paul Eggleton 8d56d596bb bitbake: knotty: add code to support showing progress for sstate object querying
Add support code on the BitBake side to allow sstate.bbclass in
OpenEmbedded to report progress when it is checking for availability of
artifacts from shared state mirrors.

Part of the implementation for [YOCTO #5853].

(Bitbake rev: 070ae856da0715dbaf4c560c837ea796ffc29f00)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-07-08 09:57:26 +01:00
Richard Purdie ff6d458f9a bitbake: parse/ast, event: Ensure we reset registered handlers during parsing
When parsing, we should reset the event handlers we registered when
done. If we don't do this, parse order may change the build, depending
on what the parse handlers do to the metadata.

This issue showed up as a basehash change:

ERROR: Bitbake's cached basehash does not match the one we just generated (
/media/build1/poky/meta/recipes-core/meta/nativesdk-buildtools-perl-dummy.bb.do_unpack)!

This is due to the eventhandler in nativesdk.bbclass being run, despite
this .bb file not inheriting nativesdk.bbclass. The parse order was
different between the signature generation and the main multithreaded
parse.

Diffsigs showed:

bitbake-diffsigs 1.0-r2.do_unpack.sigbasedata.*
basehash changed from 887d1c25962156cae859c1542e69a8d7 to cb84fcfafe15fc92fb7ab8c6d97014ca
Variable PN value changed from 'nativesdk-buildtools-perl-dummy' to '${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}'

with PN being set by the event handler.

(Bitbake rev: 0219271d4130c1f4cf071c7577a4101c54c04921)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-15 08:35:07 +01:00
Richard Purdie 0f2c59367a bitbake: bitbake: Convert to python 3
Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.

(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-06-02 08:24:02 +01:00
Richard Purdie a46d485856 bitbake: event/msg: Pass formatted exceptions
python3 can't cope with the previous approach we were using to pass
exceptions through the RPC. Avoid this by creating a formatted exception
on the sender side.

(Bitbake rev: d7db75020ed727677afbad07a90fb3eac0bf2c45)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-13 13:41:32 +01:00
Christopher Larson cd20dd057b bitbake: bb.event: handle __builtins__ as a module
Fixes pypy support.

(Bitbake rev: a3e5d9337f5400aab13df63f261e750178f8a661)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2016-05-06 10:18:12 +01:00
Richard Purdie 0381b78aa4 bitbake: event/utils/methodpool: Add a cache of compiled code objects
With the addition of function line number handling, the overhead of
the compile functions is no longer negligible. We tend to compile
the same pieces of code over and over again so wrapping a cache around
this is beneficial and removes the overhead of line numbered functions.

Life cycle of a cache using a global like this is in theory problematic
although in reality unlikely to be an issue. It can be dealt with
if/as/when we deal with the other global caches.

(Bitbake rev: 98d7002d1dca4b62042e1589fd5b9b3805d57f7a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-22 00:02:05 +00:00
Richard Purdie 0019edc818 bitbake: ast/event/utils: Improve tracebacks to include file and line numbers more correctly
Currently bitbake tracebacks can have places where the line numbers are
inaccurate and filenames may be missing. These changes start to try and
correct this.

The only way I could find to correct line numbers was to compile as a
python ast, tweak the line numbers then compile to bytecode. I'm open
to better ways of doing this if anyone knows of any.

This does mean passing a few more parameters into functions, and putting
more data into the data store about functions (i.e. their filenames
and line numbers) but the improvement in debugging is more than worthwhile).

Before:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "run_buildstats(e)", line 43, in run_buildstats(e=<bb.build.TaskStarted object at 0x7f7b7c57a590>)
NameError: global name 'notexist' is not defined

ERROR: Build of do_patch failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "run_buildstats(e)", line 43, in run_buildstats
NameError: global name 'notexist' is not defined

----------------

After:
----------------
ERROR: Execution of event handler 'run_buildstats' failed
Traceback (most recent call last):
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats(e=<bb.build.TaskStarted object at 0x7efe89284e10>):
         if isinstance(e, bb.build.TaskStarted):
    >        trigger = notexist
             pn = d.getVar("PN", True)
NameError: global name 'notexist' is not defined

ERROR: Build of do_package failed
ERROR: Traceback (most recent call last):
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 560, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/media/build1/poky/bitbake/lib/bb/build.py", line 497, in _exec_task
    event.fire(TaskStarted(task, logfn, flags, localdata), localdata)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 170, in fire
    fire_class_handlers(event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 109, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/media/build1/poky/bitbake/lib/bb/event.py", line 81, in execute_handler
    ret = handler(event)
  File "/media/build1/poky/meta/classes/buildstats.bbclass", line 143, in run_buildstats
    trigger = notexist
NameError: global name 'notexist' is not defined
----------------

(Bitbake rev: 1ff860960919ff6f8097138bc68de85bcb5f88b0)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-18 12:18:19 +00:00
Richard Purdie 5de3800c30 bitbake: event: Fix subprocess event error traceback failures
If subprocess raises a CalledProcessError() error, e.g. from a call
like subprocess.check_call("false"), bitbake would try and pass the
object over IPC and fail, leading to an unusual error:

('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())%

To avoid this, we turn the value into a string which prevents the
issues the IPC has trying to deal with the object (for the same reason
we deal with tracebacks here too).

[YOCTO #8752]

(Bitbake rev: 05695424b918fc81b16cbac70d79d8271a0b6045)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-12-14 23:13:06 +00:00
Richard Purdie 4b9de767f0 bitbake: event/server: Add _uiready flag to handle missing error messages
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.

The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.

This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.

[YOCTO #8239]

(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-09-04 16:23:56 +01:00
Paul Eggleton 294948c732 bitbake: lib/bb/cooker: add interrupted flag to BuildCompleted event
Allow any listeners for this event (such as buildhistory.bbclass in
OpenEmbedded) to find out if the build was interrupted rather than
completing normally. The value will be 0 if not interrupted, 1 if
interrupted waiting for remaining tasks to complete, or 2 if force
interrupted (stopping any running tasks immediately).

(Bitbake rev: df2b778efd2ecc48f6c5a3ed446f6459f2250035)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-29 14:03:49 +01:00
Paul Eggleton 715d857174 bitbake: Fix default function parameter assignment to a list
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:

http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments

(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-08-19 18:05:44 +01:00
Richard Purdie 8c3dd2d9eb bitbake: event: Handle recursive events and the data store better
Events can call each other recursively, e.g. an event handler can call
bb.note which in turn generates another event. If these loop, it
can lead to multiple deletions of 'd' from __builtins__ which
can fail since __builtins__ is global scope.

Add handling to only remove 'd' when we added it and it wasn't already
present.

(Bitbake rev: b45952650ce8f470f124df36185b79e0d3a1783a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-26 09:27:34 +01:00
Richard Purdie df123ddab4 bitbake: event: Inject 'd' into event handlers
To quote Chris Larson:

"""
e.data.getVar() gets a bit old in a large event handler, and it means a
simple handler has to be modified if switching between an event handler
(e.g. RecipeParsed) and anonymous python. I think it would make sense
to restore the 'd' convention here to align with python elsewhere.
It'd just be a convenience, d==e.data, to avoid the common pattern of
setting it at the top of the event handler.
"""

I couldn't find a way to inject 'd' via locals/globals due to the use
of a function parameter so this left __builtins__ as the only way
I could find to make this work.

[YOCTO #7668]

(Bitbake rev: 44ac81e5281fb62ad00e2f79a9d754118ea62526)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-23 11:57:57 +01:00
Richard Purdie 4bedfeffa2 bitbake: cooker/event: Add an event which lists all stamps reachable after parsing
The metadata can potentially use such an event to clean up any
"unreachable" data, solving several problems we currently have
where obsolete data may continue to exist in the shared areas.

(Bitbake rev: c5e6f929f3d5eeb7954660dea62611c58b795ff8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2015-06-09 13:30:09 +01:00
Paul Eggleton 9c02ad9fc9 bitbake: event: fix resetting class handlers object
If you don't explicitly specify to use a global variable when doing an
assignment, you will be setting a local variable instead, which means
this function wasn't working at all. It explains some odd behaviour we
have seen in the layer index where event handlers were sometimes
bleeding into other contexts where they should not have been.

(Bitbake rev: ac45ea848901b0f6cd23087b662dde8ce9cd807e)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-09 22:15:49 +00:00
Paul Eggleton c1ef10afb6 bitbake: event: add a means of filtering events internally
When using external tinfoil-based utilities, it is useful to be able to
turn off most of the event handlers; for example sstate_eventhandler
doesn't like being sent events for any recipe which has been skipped.

(Bitbake rev: 41236c28985a3b66b3737382a94e39dbf6480160)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-12-09 22:15:48 +00:00
Robert P. J. Day 6d08e5bb09 bitbake: lib/bb/*.py: Typo fixes/grammar/comment fixes, nothing functional.
(Bitbake rev: 587b144ee409d444494d8d7f2d1c53ede8f7c953)

Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-08-25 08:57:45 +01:00
Alexandru DAMIAN f4456df240 bitbake: toasterui: fixing event.data clash
This patch fixes a name collision on the event.data
in the MetadataEvent class. event.data is used in the
event system as a copy of "d" in certain situations,
and this collision triggered a bug leading to data loss.

[YOCTO #6332]

(Bitbake rev: 3f191b7cfe95aea4d4e96babf001d62d45dd3aaa)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-07-23 20:07:48 +01:00
Richard Purdie 91a164cb87 bitbake: event: Add SkipRecipe event to replace SkipPackage
In the depths of time we were rather confused about naming. bb files
are recipes, the event to skip parsing them should be SkipRecipe,
not SkipPackage. This changes bitbake to use the better name but
leaves the other around for now. We can therefore start removing
references to it from the metadata.

(Bitbake rev: 98d9e6e0f514a7cb7da1d99bf4bd5602b89426d6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Richard Purdie 044296ce4c bitbake: event.py: Allow passthrough of BBHandledException events
We need BBHandledException events to be passed through to the higher
layers, they don't need addition of any traceback since they've already
been reported to the user.

(Bitbake rev: f63b61f8997862439519f474cc09f3e05e95288c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Richard Purdie a26667054b bitbake: event: Fix event handlers to raise SkipPackage
If an event handler triggers a SkipPackage event, we really want that
event to be received and processed by the higher code levels. Currently
it was getting caught and ignored which was leading to recipes
being present when they clearly shouldn't have been.

In general this exception catching looks to be doing the wrong thing. It
was introduced in
http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/bitbake/lib/bb/event.py?id=37cb4cc02b2e2b6c338c5943747e0a1ef15176b3
but we likely want exceptions to pass through to the higher layers.

This patch therefore removes that code.

(Bitbake rev: 79211b3434855942f9fad4a1db69ce7be911327c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-06-01 14:29:31 +01:00
Richard Purdie 8c1e43ca58 bitbake: cooker/event: Overhaul sanity test mechanism
Sanity tests are currently a pain as its hard to control when they run. This results
in issues where for example the bitbake -e output is not useful as the sanity tests
prevent it from executing. The sanity tests should run later than the base configuration.

This patch changes the sanity tests to always be event triggered with the option of
returning either events on the status, or raising errors. A new cooker feature is used
to change the behaviour depending on the controlling UI.

This does need a change to sanity.bbclass in the OE metadata but its worth the pain
for the increased flexibility and control this offers UIs and the improvement to the
user experience.

(Bitbake rev: 32e171bcc92c6e27fefee971e8714ddf8e1a8ac1)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2014-03-27 09:42:08 +00:00
Alexandru DAMIAN 24ae636fde bitbake: bitbake: event: adding generic event for metadata usage
Adding the generic bb.event.MetadataEvent that is
targeted specifically at metadata usage. This is
needed in order to let the metadata code send and receive
events during asynchrous execution without having
to define each event specifically in Bitbake.

Metadata code should subscribe to and fire the MetadataEvent
in order to communicate asynchronously, and identify
the object using event.type field, and parse the
data in the event.data field.

Knotty UI will ignore these event by default.

This deprecates RequestPackageInfo/PackageInfo, and that
event pair will be removed in the future.

(Bitbake rev: ae1ea51aaab73e010d1c3db39df058bebebc11dd)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-09-22 12:19:44 +01:00
Cristiana Voicu 8d134e999e bitbake: bitbake/event.py: UIhandler filter should work without a mask
The default for the mask will be * (all the handlers)

(Bitbake rev: 4c95e5f46cf2a656100bbf5a0e5a09d506abf9b9)

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-27 11:16:58 +01:00
Richard Purdie 69aab78dd8 bitbake: bitbake: Add ui event handlers filtering
Add functionality to allow UIs to update and change the types of events they
recieve. To do this we need to add a new command and also need to be able
to obtain the current event hander ID. In the case of xmlrpc, this is
straightforward, in the case of the process server we need to save the result
in a multiprocessing.Value() so we can retrive it. An excplit command
was added to the server API to facilitate this.

The same function can also be used to mask or unmask specific log messages,
allowing the UI to optionally differ from the standard set of message
filtering.

Based upon work by Cristiana Voicu <cristiana.voicu@intel.com>

(Bitbake rev: ba5a6c88785d9889d4172ec79937ac2a5555327e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-26 11:29:46 +01:00
Richard Purdie 25e410b743 bitbake: event/msg: Add primitive server side UI log record filtering
Currently one of the bigger bottlenecks in bitbake is passing all the
log messages over IPC to the UI. This is worthwhile if the UI is going
to use them, pointless otherwise. The memory resident bitbake suffers
from this performance issue particularly badly.

This patch filters the log events on the server side with the global
log levels and hence reduces the traffic. This speeds up parsing
(18.5s down to 17s) and bitbake general command overhead is reduced
(7.3s for a NOP to 6.2s).

What isn't added here is general event filtering or the ability to
change the log levels once set. Provision is made for adding this
in a follow up patch though.

(Bitbake rev: 1bf0e88f57ba0bca62532e81d0d62cf88e2abcbb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-23 17:29:54 +01:00
Paul Eggleton 840a427ed2 bitbake: taskdata: report close matches with NoProvider errors
Assuming there is no known reason why an item is not provided, show
close matches on the assumption that it might have been a typo or
other mistake.

(Bitbake rev: ed81b0856b4a3892b53d39871eaaa6273390ea75)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-08-16 11:21:02 +01:00
Bogdan Marinescu 5aa5fc470e bitbake: bitbake: Add event mask flag support
Add a flag to event handlers which lists the events a given handler wishes to
process. By default event handlers recieve all events but this means
we can stop running code in many cases if we know it doesn't want the event.

This is part of the fix for YOCTO #3812, but implements filtering only
for class event handlers; the other part (events filter for UIs) will be
the subject of a different patch.

(Bitbake rev: 074003a4e7530a72863b9c685fc5c31b0f08c039)

Signed-off-by: Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-14 12:52:58 +01:00
Richard Purdie d0f0e5d9e6 bitbake: runqueue: Split runqueue to use bitbake-worker
This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.

This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.

Known issues:

* Hob is broken with this patch since it writes to the configuration
  and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
  execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
  see if we can streamline it.

These are issues which will be followed up in subsequent patches.

This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.

(Bitbake rev: b2e26f1db28d74f2dd9df8ab4ed3b472503b9a5c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-06-14 12:52:56 +01:00
Richard Purdie 507d5cc2cc bitbake: cooker/cookerdata/event: Improve class handlers management
Similarly to the execution context changes, establish better lifetime
management API of the class event handlers.

(Bitbake rev: 54e35a6cceead9521f8b1dacd48e55064e85c8bd)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-24 10:34:53 +01:00
Richard Purdie b9bd05b672 bitbake: event: Drop Handled/NotHandled status return values
These have long since been deprecated, lets remove them.

(Bitbake rev: 3dc83bbb1bf387bb7ecea2e17f0f72cfccecba92)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2013-05-24 10:34:52 +01:00
Bogdan Marinescu 6cc2f06d43 bitbake: event/hob: Add a button for network tests in the proxy settings
This patch allows the user to check the network connectivity in
the "Proxy" page ("Settings" dialog) by adding a button which provides
this functionality. It also disables retrigerring sanity checks if the
proxy values are changed, since now the proxy checks are explicit.
Note that this patch depends on a patch in oe-core
("sanity.bbclass: trigger network tests explicitly"). It will
not work properly if the patch in oe-core is not merged.

[YOCTO #3026]

(Bitbake rev: cb1354d29c0be27aee57b9783c724457ef6725fb)

Signed-off-by: Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-10-24 21:13:06 +01:00
Bogdan Marinescu 7414ba713d bitbake: Add sanity check progress screen
This patch adds a sanity check progress screen to hob. The screen
is displayed when Hob executes the sanity check procedure. The screen
is displayed for at least 5 seconds. If a network error is detected,
a special dialog is displayed which lets the user open the proxy
configuration page directly.
Note that currently bitbake triggers the network tests only when
the value of its TMPDIR variable changes, which happens fairly rare
on my system. This is the subject of another bug (#3026).
Version 2 of the patch splits the changes in two parts (sanity.bbclass
belongs to oe-core).

[YOCTO #3025]

(Bitbake rev: b48f1351271cc066ffe919db112b14834a6d8f8f)

Signed-off-by: Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-27 16:45:26 +01:00
Jason Wessel 98ac5e4e62 bitbake: event.py, knotty.py, ncurses.py, runningbuild.py: Add support for LogExecTTY event
The LogExecTTY even is intended to provide the ability to spawn a task
on a the controlling tty, if a tty is availble.  When a controlling
tty is not availble the previous behavior is preserved where a warning
is issued about the action an end user must execute.

All the available UI's were tested against the new event type.

This feature is primarily intended for hooking up a screen client
session automatically on the controlling tty to allow for a more
streamlined end user experience when using a pure command line driven
environment.  The changes that send the LogExecTTY event are in the
oe-core side.

(Bitbake rev: cffe80d82a46aaf52ff4a7b6409435754043553f)

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-24 15:35:32 +01:00
Christopher Larson 749f4f6744 bitbake: compat, event: use OrderedDict from py2.7 for the event handlers
This ensures that our event handlers get run in registration order, making the
behavior more deterministic. I pulled in the python2.7 OrderedDict to avoid
essentially reimplementing a version of it ourselves, figuring we can drop it
when we bump our required python version next.

(Bitbake rev: 44aa0b0537d3fbd1272015e7677948f84d8c0607)

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-20 10:29:33 +01:00
Paul Eggleton 5b53671f27 bitbake: lib/bb/event: improve handling of event queue on exit
If BitBake exits before a UI handler (server) has been registered, we
print the event queue; if there are any errors or other non-debug
messages just print these and suppress the rest of the message queue.

This improves the output when sanity check failures occur with OE-Core
by avoiding printing a long stream of uninformative debug messages.

(Bitbake rev: 8668a94cb1841798636b68fe123400d6b81f6574)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-09-14 09:50:34 +01:00
Richard Purdie 21773a997a bitbake: event/ast: Use better_exec instead of simple_exec
This improves the stacktraces dumped by bitbake when for example anonymous
python functions fail.

Also default to passing code strings to better_exec to match the behaviour of
simple_exec to aid the transition.

(Bitbake rev: 7e8205929ae953731a6854ea80b197847cff5771)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-22 13:59:57 +01:00
Kang Kai b2a959a0ce bitbake: monitordisk: fire event DISKFULL when terminate build
Part of [Yocto #2168]

Add a event DiskFull to descript the termination by disk monitor.

Update check() to fire the event DiskFull when terminates the build.
This could help UIs to deal this scenario and show more information to
end user.

(Bitbake rev: 0a0fca3d94a1db6458ae21501a66461a334410ed)

Signed-off-by: Kang Kai <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-08-02 15:28:39 +01:00
Paul Eggleton 5a975239ea hob: handle sanity check failures as a separate event
In order to show a friendlier error message that does not bury the
actual sanity error in our typical preamble about disabling sanity
checks, use a separate event to indicate that sanity checks failed.

This change is intended to work together with the related change to
sanity.bbclass in OE-Core.

Fixes [YOCTO #2336].

(Bitbake rev: 24b631acdaa143a4de39c6e1328849660c66f219)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-05-30 17:23:31 +01:00
Dongxiao Xu 86abb3699a event.py: Add SanityCheck and SanityCheckPassed events
(Bitbake rev: 4d7bf9d813229b78b1cd87d06f7042e7923b7db4)

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-04-24 10:14:47 +01:00
Shane Wang 8fa33800ef bitbake: change for adding progress bar in Hob2.
The changes include:
- Clean some events in event.py
- Fire essential events for Hob2 to handle with more information.
- knotty changes

(Bitbake rev: 9ede881620c501574f014e600cea6947ea908ac2)

Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
2012-02-23 22:52:17 +00:00