If we don't sort the data, the values can reorder changing the
signatures meaning we get confused builds and significant cache
misses.
(Bitbake rev: 8f453bb11d72afc90a986ac604b3477d97eaf9a8)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
That has no impact on the builds themselves, so we should just
remove that line from the variable history.
[YOCTO #5561]
(Bitbake rev: ae0ed55e80b7bd30c775b128b4114b306a50ff69)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Setting INHERIT var means to remove other operations made on INHERIT.
This is too intrusive, so we decided to use append for this case.
[YOCTO #5448]
(Bitbake rev: a2d0122c198ee50325e6f0e2f5d1c2284475fc7b)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher made the rather bold assumption that if it fetched from the upstream,
the revisions were present and correct. These checks are fast and ensure that
really is the case. The avoids accidental network accessed and missing
branch configuration problems.
(Bitbake rev: a9112a102a89049cda597dad449e922c9e957a5d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If warnings come from recipes parsing
and not from package build, 'parent' object
will be None; so don't update the color for it.
[YOCTO #5621]
(Bitbake rev: f9d24f55a5ffa9257e7ba3257be1210687513733)
Signed-off-by: Valentin Popa <valentin.popa@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The FILES_INFO structure is now much simpler, so remove all of the
horrible mangling we had to do here in order to read it.
(Bitbake rev: 11a664292064dbf76850bf21ba386f78a43a56b6)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This treeview code was obviously copy-pasted from one of the config
dialogs and the variables were never renamed. Rename them now to improve
readability.
(Bitbake rev: 93b3ddad4396d757d1d0c199ac168f8b1a205fe3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
PKGSIZE is now in bytes in pkgdata, so we need to treat it as such in
the UI code for Hob / Toaster.
(Bitbake rev: 3b5ff814cd4a3efa4b17c6b343ec39c9acca5c9e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
initConfigurationData and loadConfigurationData are similar functions, the only
reason for them appears to be to be able to reset the pre/post configuration
files. The current code is confusing and unmaintainable.
Instead this patch creates a new Sync command which allows these to be explicitly
set. The init and load functions can then be merged into one. There is then no
need for a parseConfiguration command, we can simply reset the server to have the
settings take effect.
The reset fuction is not an instant value return and triggers an event so it should
be an Async command, not a sync one.
The number of calls for the set pre/post command is probably higher than it
need be but someone with more familiarity with the hob code base can probably
figure out the right places its needed (maybe just init_cooker?).
(Bitbake rev: bae5210d7e048022f083361964ebec7daf1608f7)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Might as well use this functionality now we have it available.
(Bitbake rev: cd7f4d85e3f187140d1bb0aecf82f657a8f8701a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The previous "contains" changes caused a ~3% parsing speed impact.
Looking at the cause of those changes was interesting:
* Use of defaultdict was slower than just checking for missing entries
and setting them when needed.
* Even the "import collections" adversely affects parsing speed
* There was a missing intern function for the contains cache data
* Setting up a log object for each variable has noticeable overhead
due to the changes in the code paths uses, we can avoid this.
* We can call getVarFlag on "_content" directly within VariableParse
for a noticeable speed gain since its a seriously hot code path.
This patch therefore tweaks the code based on the above observations to
get some of the speed back.
(Bitbake rev: fca802187a2a30686a8a07d2b6b16a3e5716e293)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
I've always found it somewhat annoying that this icon's rotation
animation wasn't quite correct; this was because it was looping around
to the second position instead of the first, which made it appear to
stutter.
(Bitbake rev: 78ad15b669b9c7cde41f7bd1ab884c1d2e0db91b)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you had more than 15 layers the system would crash since one more
value is added to one array than the other. This fixes the code
so equal numbers of values are added to the arrays and hence
doesn't crash when many layers are enabled.
(Bitbake rev: 4e65463886a2ef245b2f8974e82e9cb942af224b)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The idea is to build on the --status-only option for bitbake and
expose a mechanism where the oe init scripts can easily switch between
memres server and the non-memres server.
In the case of the standard oe init script the following
can shut down the server:
if [ -z "$BBSERVER" ] && [ -f bitbake.lock ] ; then
grep ":" bitbake.lock > /dev/null && BBSERVER=`cat bitbake.lock` bitbake --status-only
if [ $? = 0 ] ; then
echo "Shutting down bitbake memory resident server with bitbake -m"
BBSERVER=`cat bitbake.lock` bitbake -m
fi
fi
A similar function can be used to automatically detect if the server
is already running for the oe memres init script. This new
functionality allows for the memres init script to be started in a new
shell and connect up to an alaready running server without seeing the
error of trying to start the server multiple times.
(Bitbake rev: b1803958de8d7c3c3279841e38604a08dc2316cc)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch adds the ability to dynamically select a port for the
bitbake memory resident server when the BBSERVER port is set to -1.
This allows for running multiple instances of the bitbake memory
resident server on the same system in different build directories.
The client portion of the bitbake instance can also request that the
server automatically start when using the auto port feature. This is
to deal with a bitbake instance that eventually times out and exits or
that has died for some unknown reason.
The new functionality allows for lazy startup of the server after
sourcing the init script for the memory resident functionality.
(Bitbake rev: d6abc07ff385357d312d8435b89e0a9c1f965433)
Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently bitbake only adds files to its dependency list if they exist.
If you add 'include foo.inc' to your recipe and the file doesn't exist,
then later you add the file, the cache will not be invalidated.
This leads to another bug which is that if files don't exist and then
you add them and they should be found first due to BBPATH, again the
cache won't invalidate.
This patch adds in tracking of files we check for the existence of so
that if they are added later, the cache correctly invalidates. This
necessitated a new version of bb.utils.which which returns a list of
files tested for.
The patch also adds in checks for duplicate file includes and for now
prints a warning about this. That will likely become a fatal error at
some point since its never usually desired to include a file twice.
The same issue is also fixed for class inheritance. Now when a class
is added which would be found in the usual search path, it will cause
the cache to be invalidated.
Unfortunately this is old code in bitbake and the patch isn't the
neatest since we have to work within that framework.
[YOCTO #5611]
[YOCTO #4425]
(Bitbake rev: 78d285871e4b8c54ccc4602d571e85f922e37ccd)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The 'bbappend in f' incorrectly compares the current recipe with the avaliable
bbappends recipes. This comparsion causes unrequested bbappend files to be
appended, e.g. in the case of 'libgcc_4.8.bb', the bbappends for 'libgcc_4.8.bb'
and 'gcc_4.8.bb' are added to the filelist (because 'gcc_4.8.bb' is contained in
the 'libgcc_4.8.bb' string) which in turn causes the gcc_4.8.bbappend files to
be appended to the libgcc_4.8 recipe.
This should be a 'bbappend == f' to match the previous implementation of this
function, such that if no wildcard is present the recipe names must match
exactly.
This issue was introduced by commit 31bc9af9cd56e7b318924869970e850993fafc5f,
which it related to [YOCTO #5411].
(Bitbake rev: 991cbeedbde8bd25ce08c669b1bfac8b99e33149)
Signed-off-by: Nathan Rossi <nathan.rossi@xilinx.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Redraw the entire notebook widget after
number-of-issues drawable is incremented
(because for this case, the notebook widget
doesn't refresh it's children automatically).
[YOCTO #5596]
(Bitbake rev: 347b2ead091f00ee60703f6f3d17cfdd9075ac07)
Signed-off-by: Valentin Popa <valentin.popa@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
One of the current frustrations with the sstate checksums is that
code like base_contains('X', 'y',...) adds a full dependency on X
and varies depend even on whitespace changes in X.
This patch adds special handling of the contains functions to expand
the first parameter and check for the flag specified by the second
parameter (assuming its a string).
The result is then appended to the value of the variable with a "Set"
or "Unset" status. If the flag is added/removed, the stored variable
value changes and hence the checksum changes. No dependency on X
is added so it is free to change with regard to other flags or
whitespace.
This code is far from ideal, ideally we'd have properly typed variables
however it fixes a major annoyance of the current checksums and
is of enough value its worth adding in a stopgap solution. It shouldn't
significantly restrict any propely typed variable implementation in
future.
(Bitbake rev: ed2d0a22a80299de0cfd377999950cf4b26c512e)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently tasks have no knowledge of which other tasks they depend
upon. This makes it impossible to do at least two things which would be
desirable/interesting:
a) Have the ability to create per recipe sysroots
b) Allow the aclocal files to be present only for the entries in
DEPENDS (directly and indirectly)
By exporting task data through this new variable, tasks can inspect
their dependencies and then take actions based upon this.
(Bitbake rev: 84f1dde717dac22435005b79d03ee0b80a3e8e62)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This unlikely looking function was found to be eating a lot of CPU time
since it gets called once per trip through the idle loop if we're not
running a maximum number of processes. This was particularly true in
world builds of 13,000 tasks.
Calling the computation code is pretty pointless because until some
other task finishes nothing is going to become available to build.
We can know when things become available so this patch teaches the
scheduler this knowledge.
It also:
* skips any coputation when nothing can be built
* if there is only one available item to build, ignore the priority map
* precomputes the stamp filenames, rather than doing it every time
* saves the length of the array rather than calculating it each time
(the extra function overhead is significant)
Timing wise, initially, 5000 iterations through here was 20s, with
the patch 200000 calls takes the same time. The end result is that
builds get up and running faster.
(Bitbake rev: 4841c1d37c503a366f99e3a134dca7440e3a08ea)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no good reason to separately track var_references and
references so merge them and remove the unneeded variable.
(Bitbake rev: 64d4cbd6360c96574cece70205ea3aecc3f8bae6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In 6a48474de9505a3700863f31839a7c53c5e18a8d the url parameter to a
number of functions was removed. However, not all calls to
latest_revision() were fixed...
(Bitbake rev: 7c94ca56b2fd85a989089f58b3dcce3172a778f2)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
VAL = "" (not shown)
VAL = " " (shown as "")
VAL = " x" (shown as "x")
would all show up rather differently to what would be expected in the
bitbake -e output. This fixes things so they appear consistently.
The output for running some shell functions may also change slightly
but shouldn't change in a way that is likely to cause problems.
[YOCTO #5507]
(Bitbake rev: fcba5ef0053dc0ef5360e4912609e5d52f5046b0)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With a SRC_URI = " \
p4://depot/folder/...;module=localfolder/localsubfolder;changeslist=${P4CHANGELIST} \
"
the subfolders of //depot/folder/... get renamed when mapped to the
local folder structure. They lose the first 3 letters. This
patch fixes that.
Issue reported by and patch sent from katutxakurra@gmail.com
[YOCTO #5380]
(Bitbake rev: 40e06dc459d9c0b5d42d65b2d2c846196fd36b1f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There has been a continuing call for supporting wildcard in bbappend
filenames. The wildcard is actually allow matching of the name and
version up to the point of encountering the %. This approach will
allow for matching of the major or major.minor.
Exampes:
busybox_1.21.1.bb
busybox_1.21.%.bbappend will match
busybox_1.2%.bbappend will also match
if we update to busybox_1.3.0.bb the above won't match, but a busybox_1.%.bb
will.
[YOCTO #5411]
(Bitbake rev: 31bc9af9cd56e7b318924869970e850993fafc5f)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using the dry run option (-n), bitbake would still try and fire
a specific fakeroot worker. This is doomed to failure since it might
well not have been built.
Add in some checks to prevent the failures.
[YOCTO #5367]
(Bitbake rev: f34d0606f87ce9dacadeb78bac35879b74f10559)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is no good reason to keep passing around the url parameter when
its contained within urldata (ud). This is left around due to
legacy reasons, some functions take it, some don't and its time
to cleanup.
This is fetcher internal API, there are a tiny number of external users
of the internal API (buildhistory and distrodata) which can be fixed up
after this change.
(Bitbake rev: 6a48474de9505a3700863f31839a7c53c5e18a8d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
urldata contains the url so we might as well stop passing around
pointless function parameters. This was done for legacy reasons but
its time to clean this mess up.
This is a first step in cleanup and is a standalone patch but there is
more to be done in a second patch.
(Bitbake rev: 06590cfebbcf6565a17b80cc298e3ecdfaba4656)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A typo in buildinfo helper leads to a bug where
information about tasks is not correctly stored.
This patch fixes the typo.
(Bitbake rev: 67b752993a2c64cba9ccc4fa662f0bddf081e74a)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch changes bitbake to log an error to the
command line when the build is interrupted via Ctrl-C.
This is needed to inform the user that not all tasks
required for the build have been executed, and
the build is not complete.
Internally, the Bitbake server will return a CommandFailed
event that will be logged by Toaster as build failure.
(Bitbake rev: 9a658e8b1511f1b9f91663f546f748fdfbc8965f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the computed name of a directory contains an undefined variable
reference, bitbake dutifully creates a directory with a name that has
${...} in it. However, the actual task script created then tries to cd
to that directory, and the cd command fails, because no such directory
exists -- because the shell has helpfully removed the ${...} which did
not match any actual variables.
Since we want the name to be used exactly-as-is, add single quotes around
the name so this doesn't cause strange failures running tasks, which
allows us to progress past such failures and get to a point where they
can be diagnosed.
(Bitbake rev: 2809c2e6f2f35f9b08058950be896947ab5a0284)
Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We have do_bundle_initramfs which is a task inserted after compile and
before build. It is not covered by sstate.
If we run a build with a valid sstate cache present, the setsceneverify
function realises it will rerun the do_compile step (due to the
bundle_initramfs task) and hence marks do_populate_sysroot to rerun.
do_install, a dependency of do_populate_sysroot is left as marked as
covered by sstate.
What we need to do is traverse the dependency tree for any setsceneverify
invalided task and ensure any dependencies are also invalidated. We can
stop at any point we reach another setscene task though.
This means the do_populate_sysroot task has the data from do_install
available and doesn't crash.
(Bitbake rev: f21910157d873c030b149c4cdc5b57c5062ab5a6)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If an SCM mirror is in PREMIRRORS, the tarball is downloaded and then found
by the "upstream" check and handled correctly.
If an SCM mirror is in MIRRORS, the tarball is downloaded but not used
since there is no "upstream" run after MIRRORS completes. It therefore
sits there useless and unused. This code change forces the upstream to
run after a mirror tarball is found and fixes the usage of SCM mirrors
in MIRRORS.
(Bitbake rev: a66ee0994645aa5658b2f5ea134ed17d89f8751a)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Using git merge-base for checking for ancestors is nice but required git 1.8.0
which is not in many distrbutions yet. We therefore revert to a more ugly
check using git branch --contains until such times as we can upgrade.
(Bitbake rev: 31467c0afe0346502fcd18bd376f23ea76a27d61)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The default value of 0.5 seconds before sending the pr-server a
SIGTERM is not enough to guarantee that sqlite has committed all
the pr data to the database. By polling the pid to see if it is
still running, this allows the pr-server process to shutdown
cleanly and finish the final pr data commit.
(Bitbake rev: 22eec978e70794923c85689928c6be0cfe71cdcd)
Signed-off-by: Konrad Scherer <Konrad.Scherer@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The current use of git log to check if a given revision is present can be
a little fragile.
For example if revision X was on branch A, and then later added to branch
B, the update checks would not notice this since they just check for X
being in the repository.
We also had some autobuilder corruption where an older packed-refs file
was copied over a new repository containing newer pack files. There
was no update to the refs file since the revision was present but
not accessible in any branch.
The correct fix is to check that the required revisions are present
on the specific branches. This patch does this using merge-base.
(Bitbake rev: 89abfbc1953e3711d6c90aff793ee622c22609b1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to track the file where a configuration
variable was defined, this patch bring these changes:
* a new feature is defined in CookerFeatures, named
BASEDATASTORE_TRACKING. When a UI requests BASEDATASTORE_TRACKING,
the base variable definition are tracked when configuration
is parsed.
* getAllKeysWithFlags now includes variable history in the
data dump
* toaster_ui.py will record the operation, file path
and line number where the variable was changes
* toaster Simple UI will display the file path
and line number for Configuration page
There is a change in the models to accomodate the recording
of variable change history.
[YOCTO #5227]
(Bitbake rev: 78e58fed82f2a71f052485de0052d7b9cca53ffd)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
0 (not applicable) is not a valid sstate_result for tasks with
outcome 2 (sstate), which should return 3 (restored), 2
(failed) or 1 (missed).
Sstate_result for tasks with outcome 2 is equal to the outcome
of _setscene corespondent task.
[YOCTO #5220]
(Bitbake rev: 8ff8d75318ea88ba80c744b471e486901ef6749a)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the user explicitly asks to build a target that is listed in the
value of ASSUME_PROVIDED, show a warning mentioning that it will be
ignored.
(Bitbake rev: 65be09f5e2a4a41e65c9232e208d8154b822fc4e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When the build fails and I click "Edit packages", we should be returned
to the tab we launched the build from. Hob should remember the tab,
not the search state.
[YOCTO #5257]
(Bitbake rev: afdea16516084ef6c046d80e12bf244f25a9da50)
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The buildstats file path changes based on the
optional PE variable that may be defined for a
recipe.
The toasterui simply ignored the PE value, and
as such it didn't correctly reach buildstats files
for some of the tasks.
This patch fixes the issue.
[YOCTO #5073]
(Bitbake rev: 97b8ab88edc7c8dfb26b4cf305701ec96e52cc4f)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch addresses an issue where a failed sceneQueue task
entry was not updated on the Fail event. As a result, it
always showed the task as not-available.
[YOCTO #5216]
(Bitbake rev: 9b99a417f58381bac4bda412bcfd11de50403318)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The AUTHOR field in most recipes is not defined,
or it's not really consistently set in the metadata,
Also does it seem particularly useful.
This patch removes the AUTHOR variable from the
toaster system
[YOCTO #5449]
(Bitbake rev: da3ac049300be84defab7b32b0b99ab07c7d0a27)
Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Richard Purdie
Cristiana Voicu
Valentin Popa
Paul Eggleton
Jason Wessel
Nathan Rossi
Yi Zhao
Peter Kjellerstedt
Florin Sarbu
Saul Wold
Alexandru DAMIAN
Peter Seebach
Konrad Scherer