Various misc changes to convert bitbake to python3 which don't warrant
separation into separate commits.
(Bitbake rev: d0f904d407f57998419bd9c305ce53e5eaa36b24)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently xmlrpc server implicitly sets itself into single use mode
when bitbake server is started with anonymous port (0) or no port is
provided in command line. In this mode bitbake shuts down xmlrpc server
after build is done. This assumption is incorrect in some cases.
For example Toaster uses bitbake in this mode and expects xmlrpc server
to stay in memory.
Till recent changes single use mode was always unset due to the bug.
When the bug was fixed it broke toaster builds as Toaster couldn't
communicate with bitbake server in single use mode.
Reimplemented logic of setting single use mode. The mode is explicity
set when --server-only command line parameter is not provided to bitbake.
It doesn't depend on the port number anymore.
[YOCTO #9275]
[YOCTO #9240]
[YOCTO #9252]
(Bitbake rev: afc0dd5c532684f6201b1e12bbf4c226ea19062d)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
XMLRPCServer.single_use attribute was always set to False.
This caused xmlrpc server to keep running after build is done as
BitBakeServerCommands.removeClient only shuts down server if its
single_use attribute is set to True.
(Bitbake rev: 0a60b0928a0a746a60d2c2f294ff1903963c7086)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current code in connect method sets up event queue, which requires
registering UI handler. This functionality may not be needed for
some operations, e.g. for server termination.
Moved functionality of setting up event queue in from 'connect'
method to 'setupEventQueue' in BitBakeXMLRPCServerConnection class.
(Bitbake rev: 4429871da76d6bd29e023ff42740fe7daa6b40fa)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Current code throws Exception("Could not register UI event handler")
if event handler can't be registered. The real reason of this is that
cooker is in busy state. Error message lacks information about this.
Added error message to the return value of registerEventHandler.
Included returned error message into the log message and exception
text.
(Bitbake rev: 07de1ca7d57dcd0cc37406feae2949da12a3fa7a)
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you start and suspend a bitbake execution so the bitbake lock is held,
then try and run "bitbake -w '' X", you will see bitbake return an error exit
code but print no message about what happened at all.
The reason is that the -w option creates a "UI" which swallows the messages. The
code which handles this exit failure mode thinks a UI has printed the messages
and therefore doesn't do so.
This adds in an extra parameter to the UI registration code so that we
can figure out whether its a primary UI or not and base decisions on whether
to display information on that instead. This fixes the error shown above and
some bizarre failures on the Yocto Project Autobuilder.
[YOCTO #8239]
(Bitbake rev: d1d60a68c2de40c2984d5040d14251c1be121b0b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With python you should not assign a list as the default value of a
function parameter - because a list is mutable, the result will be that
the first time a value is passed it will actually modify the default.
Reference:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
(Bitbake rev: 7859f7388f2e3f675d0e1527cfde18625f36f637)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The recent inotify changes are causing a 100% cpu usage issue in the
idle handlers. To avoid this, we update the idle functions to optionally
report a float value which is the delay before the function needs to be
called again. 1 second is fine for the inotify handler, in reality its
more like 0.1s due to the default idle function sleep.
This reverts performance regressions of 1.5 minutes on a kernel build
and ~5-6 minutes on a image from scratch.
(Bitbake rev: 0e0ba408c2dce14a0fabd3fdf61d8465a031495b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes sure to delete an idle function that
raises an exception for the xmlrpc server.
The counterpart functionality in the process server was
added with:
commit db50630948.
duthor: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Wed Aug 20 22:31:06 2014 +0000
bitbake: process: Deal with infinite looping of the server
This patch fixes
[YOCTO #7316]
(Bitbake rev: e7c9a6788d969c901fd6394416ac3936e62c4c72)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We roll back the client connection if some error happens,
like during setFeatures, as to leave the server accessible
to other clients.
(Bitbake rev: 4e4a2ee2f05f8741b2e09263e328420363975b02)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We add support to specify a connection token in the command line
and in the environment variable BBTOKEN.
When a client registers to a bitbake server, that client
will have exclusive access to the server. The client is identified
by a server-supplied token. If a client terminates, we cannot
reconnect to the server as the token is lost.
This patch adds the capability to specify the connection token
in the command line for xmlrpc clients. This allows us
to have bitbake work as an already-authenticated
client with the server and resume sending commands to a server.
(Bitbake rev: db5390940c0afbcdc9fbcf1225761968ae51d4a7)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When we use the XMLRPC client API to connect to a bitbake server,
we want to receive errors from the API instead of having the
API exiting without warning.
Thus the "bb.fatal" calls have been replaced with "bb.warn" calls,
and we re-raise the original exception for handling by the
original caller.
The bitbake starting script has been modified to properly test
for failures in calling the client API and handle them.
Additional error handling added in the client, as to prevent
fatal crashes.
(Bitbake rev: eb63f08c33644f64752aaae2146a000956ce894a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The configuration reading code should live in the
main bitbake entry point, and the server modules should
be supplied with correct configuration instead of attempting
to parse from configuration files.
This patch moves the endpoint address reading from XMLRPC
to the bitbake main script.
(Bitbake rev: ac5753274ff932e1d6f073ab4dab7bd6fe5355a1)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch makes sure that the setFeatures command is marked
as read-only and that it can only run if the cooker is in
the initial state.
Additionally, remove logging from the XMLRPC module in favor
of sending the exception to the client for easy processing.
[YOCTO #6089]
(Bitbake rev: f0a1a3e24757f7658d272035620465f92a3e4c3c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rather than passing featureset around various places where the data doesn't
really belong, run a command at connection time to set the appropriate
features. This is similar to what the process server does.
(Bitbake rev: c3b5cc5691291c74dd315c4439c80e0e4b2b5c1d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is to build on the --status-only option for bitbake and
expose a mechanism where the oe init scripts can easily switch between
memres server and the non-memres server.
In the case of the standard oe init script the following
can shut down the server:
if [ -z "$BBSERVER" ] && [ -f bitbake.lock ] ; then
grep ":" bitbake.lock > /dev/null && BBSERVER=`cat bitbake.lock` bitbake --status-only
if [ $? = 0 ] ; then
echo "Shutting down bitbake memory resident server with bitbake -m"
BBSERVER=`cat bitbake.lock` bitbake -m
fi
fi
A similar function can be used to automatically detect if the server
is already running for the oe memres init script. This new
functionality allows for the memres init script to be started in a new
shell and connect up to an alaready running server without seeing the
error of trying to start the server multiple times.
(Bitbake rev: b1803958de8d7c3c3279841e38604a08dc2316cc)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the ability to dynamically select a port for the
bitbake memory resident server when the BBSERVER port is set to -1.
This allows for running multiple instances of the bitbake memory
resident server on the same system in different build directories.
The client portion of the bitbake instance can also request that the
server automatically start when using the auto port feature. This is
to deal with a bitbake instance that eventually times out and exits or
that has died for some unknown reason.
The new functionality allows for lazy startup of the server after
sourcing the init script for the memory resident functionality.
(Bitbake rev: d6abc07ff385357d312d8435b89e0a9c1f965433)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Showing a traceback when unable to connect to a bitbake server is
rather ugly. This change allows us to show a sensible error message.
(Bitbake rev: 26913202f83fbbecdce95da59515af102bcde4a7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implementing feature set selection that allows a client
to enable specific features in the server at connection time.
Only enabling of features is supported, as there is
no way to safely remove data loaded into the cooker.
Once enabled, a feature will remain enabled for the
life of the cooker.
Client-server connection now supports specifying the feature
set required by the client. This is implemented in the Process
server using a managed proxy list, so the server cooker
will now load dynamically needed features based on what client
connects to it.
In the XMLRPC server the feature set is requested by
using a parameter for registerUIHandler function.
This allows observer-only clients to also specify features
for the server.
The server code configuration now is completly separated
from the client code. All hardcoding of client knowledge is
removed from the server.
The extra_caches is removed as the client can now specify
the caches it needs using the feature. The UI modules
now need to specify the desired featureSet. HOB is modified
to conform to the featureSet specification.
The only feature available is CookerFeatures.HOB_EXTRA_CACHES
which forces loading the bb.cache_extra:HobRecipeInfo class.
(Bitbake rev: 98e594837aab89ea042cfa9f3740d20a661b14e2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BitBakeUIEventServer is an unused class that pushes UI
events over a separate thread.
The current version of XMLRPC server works just fine with
the classic UI event handlers, so this class is not needed.
(Bitbake rev: 8e8e17631d790271b1be747c4b45059ec38ab606)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the server only expects a single client to connect (i.e.
no bind parameter set, so there is no way for the
clients to get the server port), stop the server after
the first client exits.
(Bitbake rev: eb6bae56f62082bf147045311154cbae4bca0f4c)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The existing backend server implementations were inefficient since they
were sleeping for the full length of the timeouts rather than being woken when
there was data ready for them. It was assumed they would wake and perhaps did
when we forked processes directory but that is no longer the case.
This updates both the process and xmlrpc backends to wait using select(). This
does mean we need to pass the file descriptors to wait on from the internals
who know which these file descriptors are but this is a logical improvement.
Tests of a pathaolgical load on the process server of ~420 rapid tasks
executed on a server with BB_NUMBER_THREAD=48 went from a wall clock
measurement of the overall command execution time of 75s to a much more
reasonable 24s.
(Bitbake rev: 9bee497960889d9baa0a4284d79a384b18a8e826)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This is a better value that the earlier infinite timeout yet still
allows for servers with high loads. It does mean the bitbake process
can hang at exit for the timeout period but that should never happen
and only happened for me in some test cases which wouldn't happen
in normal use.
(Bitbake rev: ab8d926b9bc27c58011e7db9327e031ac76ba34b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Add functionality to allow UIs to update and change the types of events they
recieve. To do this we need to add a new command and also need to be able
to obtain the current event hander ID. In the case of xmlrpc, this is
straightforward, in the case of the process server we need to save the result
in a multiprocessing.Value() so we can retrive it. An excplit command
was added to the server API to facilitate this.
The same function can also be used to mask or unmask specific log messages,
allowing the UI to optionally differ from the standard set of message
filtering.
Based upon work by Cristiana Voicu <cristiana.voicu@intel.com>
(Bitbake rev: ba5a6c88785d9889d4172ec79937ac2a5555327e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The standard python socket connect has long timouts which make sense for remote
connections but not local things like the PR Service. This adds a timeout
parameter to the common xmlrpc server creation function and sets it to a more
reasonable 5 seconds.
Making the PR server instantly exit is a good way to test the effect of this
on bitbake.
We can remove the bodged timeout in the PRServer terminate function which
has the side effect of affecting global scope.
(Bitbake rev: 8c01cff94787abbb64fbdf0c16cd63f8f97a7e03)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I add an observer only mode for the knotty UI and
the XMLRPC server that will allow the UI to register
a callback with a server in order to receive events.
The observer-UI is able to send read-only commands to the
server, and also is able to register as an event handler.
Read-only commands are the commands that do not change
the state of the server and have been marked as such in
the command module.
The observer can switch to a full client if it calls addClient
at any time, and the server has no other client running.
(Bitbake rev: 4de9ee21f1fa4d04937cc7430fb1fc8b7a8f61e2)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Now we've moved to require python 2.7.3, we can jettison the compatibility
workarounds/hacks for older python versions.
(Bitbake rev: a51c402304f2080a76720f9b31d6dfdbed393bba)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
XMLRPC API ran a XMLRPC server at port+2 in order
to provide endpoints to register an event server
in which to dump the events.
This is no longer used, so we remove it.
(Bitbake rev: e171a363913a86e56266f4c9d107110c7f5221e3)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch enables the XMLRPC server to remain
resident in memory after a task is run, and to accept
a new controlling client. To check the server after
task completion, do
lsof bitbake.lock
in the build directory. Kill the server with kill.
(Bitbake rev: e823e1f0675ff3794eb39ef0b4df2d7a220f4013)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In an attempt to minimize code duplication, create
clear interfaces, and maximize code reuse through OOP,
bb.server adds base classes for the BitBakeServer,
BitBakeServerConnection and actual server implementations
instructed in particular server types.
These classes document the minimum interfaces that the
derived classes must implement, and provide boilerplate code.
Changes to None, Process and XMLRPC servers as to use
the common server infrastructure.
(Bitbake rev: 6db4a64cef20f8d0aba804db4c4e1eec7b112b46)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Added code in XMLRPC server that creates a stub local server
for a client-only connection and is able to connect to
a remote server, and receive events from the remote server.
Added the option to start a client with a remote server in
bitbake.
Original code by Bogdan Marinescu <bogdan.a.marinescu@intel.com>
(Bitbake rev: 25b2af76104d5aaf6435de8c158e0407512f97ce)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to be able to identify different clients over a
stateless XMLRPC connection, we add a custom header named
Bitbake-token, which identifies each client.
We refactor the rest of the code to use the new transport.
Based on a patch by Bogdan Marinescu <bogdan.a.marinescu@intel.com>
(Bitbake rev: a00c2186bffe848a7cedf31969b904f8f7322ae6)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the removal of the none server type, we can remove the launchUI
method and simplify the code slightly.
(Bitbake rev: 9bef2f2dd0bcaa59528ebcb3c1ce053b7dff1ec6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I appreciate a fixed value is suboptimal but this is better than not
working at all!
(Bitbake rev: d338a1ad14680af41d218772decd03e98eb2cad5)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This code is dead and doesn't do anything so lets remove it.
(Bitbake rev: 8d45739f49618757a5d7d79782deda355e3981ec)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In server/client split model, the client will bind to a specific address
and port. We need to pass the values to BitBakeServerConnection().
(Bitbake rev: c8e19c5c389efc06696084c6f9439ba75472c5b7)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When start bitbake as a server only process, we need to assign certain
interface to it.
(Bitbake rev: 95b97d2dc6466ea3d99371f5b5bd68f6f3c99074)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass host and port to BitbakeServerInfo class instead of the "server"
instance. With this change, GUI can connect with server individually
by host address and port.
(Bitbake rev: 5124351d6c287185723c98e6e4400c5402a5c8a8)
Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Create a standard format server class instance with method calls
for each step in the server setup. There should be enough hooks
for each of the different server types.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Upstream have fixed the xmlrpclip.Transport() bug from Python #8194 for
the Python 2.7.2 release, therefore as we know which versions of the
standard library are affected we can only use our copy/paste class when
it's needed.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
- Queue up any events fired to the UI before the UI exists
- At exit, check if UIs exist, and if not, flush the queue of LogRecords to
the console directly.
- When establishing a connection from the UI to the server, flush the queue of
events to the queue in the server connection, so the UI will receive them
when it begins its event loop.
(Bitbake rev: 73488aeb317ed306f2ecf99cc9d3708526a5933c)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <rpurdie@linux.intel.com>
Python 2.7's library changes some of xmlrpclib's internal implementation such
that interacting with a proxy to BitBakes SimpleXMLRPCServer would cause
BitBake to crash.
The issue was traced to changes in the xmlrpclib.Transport implementation and
Python bug #8194 (http://bugs.python.org/issue8194).
This patch introduces a workaround by create a subclass of
xmlrpclib.Transport, which overrides the offending methods with the Python
2.6.6 implementation copy and pasted from the Python 2.6.6 xmlrpclib, and
using this BBTransport implementation for both xmlrpclib.Server objects we
create.
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Richard Purdie
Ed Bartosh
Paul Eggleton
Alexandru DAMIAN
Jason Wessel
Dongxiao Xu
Joshua Lock
Chris Larson
Richard Purdie