Under certain contitions such as local file:// mirrors of all files,
bitbake might not check checksums. This is not desirable and we should
always check the checksum where possible.
This only exception to this is the git mirror tarball case where the
checksum is unknown but we need to create the .done stamp for other
reasons. This patch preserves thta functionality but in a more specific
section of code where this doesn't interfere in normal checksums.
(Bitbake rev: 2592a43e422c06a6f6174ada562e9c8ba25e1ec1)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Under certain circumstances, bitbake could remove files which were
outside its control since it unconditionally removes ud.localpath.
In the file:// case this can point at external paths.
This patch converts it to use the clean() method which will remove
files when its safe to do so but not in the file:// case since the
file:// handler has an empty clean method. This means bitbake no
longer removes files outside its control and is generally much safer.
(Bitbake rev: 9ad1cebe2220b41da2141cdb3f0a403abb46d77c)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: b766630ed7e099fffe817928a2811272677a1b26)
Signed-off-by: Jeff Polk <jeff.polk@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
As UI request, in recipes selection page, if user exclude a item,
the related depends recipes will be excluded together,so the view
clearly to add it.
[YOCTO #2100]
(Bitbake rev: c9eed04c6275ef2c694f89e047f85c7de76f89b6)
Signed-off-by: Liming An <limingx.l.an@intel.com>
Signed-off-by: Shane Wang <shane.wang@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 5a1e62a8cdd4f0253b0dc59046b3b4c3186af461)
Signed-off-by: Gary Thomas <gary@mlbassoc.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were seeing duplicate endings to urls:
DEBUG: For url ['http', 'www.apache.org', '/dist/subversion/subversion-1.7.1.tar.bz2', '', '', {}] comparing ['http', 'www.apache.org', '/dist', '', '', {}] to ['http', 'archive.apache.org', '/dist', '', '', {}]
DEBUG: For url http://www.apache.org/dist/subversion/subversion-1.7.1.tar.bz2 returning http://archive.apache.org/dist/subversion/subversion-1.7.1.tar.bz2/subversion-1.7.1.tar.bz2
This patch addresses this by only performing substitutions when really needed.
(Bitbake rev: a17473d51f54a9dfb4eeaa5ad516c4851d5ac142)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
A lot of our checks in the fetcher code assume that if the downloaded
file exists, the download finished successfully; this meant that if
BitBake was interrupted in the middle of a fetch then it would not
resume the fetch the next time, but instead attempt to use the
half-fetched file and usually fail as a result.
Since we're already writing a ".done" stamp file when a fetch completes,
just check for its existence and assume the download didn't complete
successfully if it isn't present.
(Bitbake rev: 721e986624529aedef96dd06c9fe6243f2897b10)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Make the warning for the initial fetch failure a single line - we don't
need the full command and output here yet, but write it into the log in
full as a debug message. However, if fetching from mirrors fails as well
then print out the full details for the first error that occurred as an
ERROR rather than a WARNING.
Since this is logged as an ERROR, combined with an earlier patch it
suppresses the full log which does make the output much more readable
for any fetch error.
Fixes [YOCTO #1832].
(Bitbake rev: 6bbdc7d259c0cc041b62dbdb26cfc3ec6edcb6f3)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The termination signal and exit code of the fetch process were not being
decoded correctly, resulting in bitbake reporting that the process
terminated with a signal of the exit code (if it was under 255). There
are functions in the Python os module to do this decoding correctly (for
Unix at least), so let's use them.
(Bitbake rev: 50aea9a76e40cf71cc3f1462c88298e4846a031c)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* The "name" argument to FuncFailed is rarely used as a name in actual
usage within bitbake, so don't treat it as one in the output.
* Don't print URL for FetchError if it was not specified (i.e. don't
output "Fetcher failure for URL 'None'")
* Don't include URL in "unable to fetch from any source" message since
we supply it to FetchError and it will be printed anyway.
* Don't include URL in "checksum failed" message for the same reason
(Bitbake rev: 86811bd85e2e453ee92a05fe60160d9b49ac69e8)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
BB_STRICT_CHECKSUM caused a backtrace as the FetchError parameters are
incorrectly specified such that FetchError is being passed 8 params
when it's expecting 3. This fixes the parameters so we're passing a
formatted string and the url.
(Bitbake rev: b8b2f4287c9125542b18a294c0a94ed89a7e73a8)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently if you specify a mirror like:
file://.* http://linux.freescale.net/yocto/sstate-cache
it won't work as you expect whilst:
file://.* http://linux.freescale.net/yocto/sstate-cache/
will since it has the trailing slash.
This patch handles both cases correctly. It also adds some debug to
the uri_replace function since its near impossible to debug it without
some kind of output.
[YOCTO #1578]
(Bitbake rev: a0246bf09c93bb657eaf6ba61d090b247ed33640)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch improves the usability of checksums by only requiring one checksum
be defined.
Further, checksum verification will provide as much information as possible
at, rather than a bit at a time. No longer will you need to run fetch, see an
md5sum mismatch, fix it, run fetch, seen an sha256sum mismatch, fix it and
fetch again. If neither checksum is defined we now report both missing sums
at once - rather than one after the other.
Finally, if both sums are incorrect, we'll report both incorrect sums at the
same time.
Fixes part of [YOCTO #1399]
(Bitbake rev: 30ca41e596955fcbbe1ae6f4436ecf14cef96f8d)
Signed-off-by: Joshua Lock <josh@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
git could need these environment variables when working behind
a proxy
(Bitbake rev: dca46cc2e1c75b6add2c4801e2994a4812745f5b)
Signed-off-by: Matthew McClintock <msm@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently, if checksum mismatch, fetcher will try mirror,
and if mirror fetching fail, it will show error "Unable
to fetch URL xxx from from any source" which actually
hide the real reason and make user confuse.
so showing warning message will make it more clear.
Fix [YOCTO #1256]
(Bitbake rev: b3e924297d670963714343d02f7898798fec84fd)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This patch is for [YOCTO #1085] fix.
If the upstream fails a checksum, retry from the MIRROR before giving up.
This will add more robust fetching if an upstream serves a bad file or webpage.
fetching of distcc prior to the move from samba -> googlecode is a good example
of this.
(Bitbake rev: b631e922257de52bf2247c01152d9856c870e7d0)
Signed-off-by: Yu Ke <ke.yu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update exception handling syntax to use the modern style:
except ExcType as localvar
(Bitbake rev: dbf5f42b06bef81749b13aa99945cc1292a6676d)
Signed-off-by: Scott Garman <scott.a.garman@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The symptom of this problem is something like a cvs url which specifies
a username where the username is then passed through to something like
an http mirror.
This patch fixes things by ensuring empty entries are preserved in the
new URL.
(Bitbake rev: c1d978d7bd1ac8eb1e2d50029ab2384be9f72fb4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
- use os.chmod, not os.fchmod, as the latter is missing under pypy
- rearrange our imports a bit
- don't die if sqlite3 is missing shared cache support
(Bitbake rev: f229824dc9c453adf6067500e2bf6761536e4f2f)
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The ordering constrains on the urldata_init functions are not straight
forward. To avoid further problems, create a helper function to setup
the source revisions which the init functions can all at the appropriate
point.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix a bug where ud.branches were being referenced before it was set by
the git fetcher when using AUTOREV. To do this some ordering needed
to be changed. This fixes errors like:
ERROR: Error parsing /recipes-kernel/linux/rt-tests_git.bb: Failure expanding variable
SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception
AttributeError: 'FetchData' object has no attribute 'branches'
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
With the current implementation, file:// urls as used by sstate don't access the
mirror code, breaking sstate mirror support. This change enables the usual
mirror handling. To do this, we remove the localfile special case, using the basename
paramemter instead. We also ensure the downloads directory is checked for files.
The drawback of this change is that file urls containing "*" globing require special
casing in the core.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* This patch fixes a cosmetic issue currently we get with master
WARNING: /home/kraj/work/bitbake/lib/bb/fetch2/__init__.py:733:
DeprecationWarning: Call to deprecated function bb.mkdirhier: Please use bb.utils.mkdirhier instead. bb.mkdirhier("%s/%s" % (rootdir, destdir))
(Bitbake rev: 36fe59ce314c295d239b76de34c8714def2c32d5)
Signed-off-by: Khem Raj <raj.khem@gmail.com>
Signed-off-by: Chris Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This create a clean() method in each of the fetcher modules
and correctly cleans the .done stamp file and lock files
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Enable the fetcher to be able to unpack and SRPM. By default the system will
unpack the contents of the SRPM into the WORKDIR.
A new syntax "unpack=file" was developed for the SRC_URI, to allow for a
recipe to extract a specific file within an SRPM. An unpack operation will
then be executed on the extracted file.
In order to apply extracted patches (or unpack files not specified with
unpack), you must specify the path using WORKDIR, i.e.:
file://${WORKDIR}/mypatch.patch
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
Richard Purdie
Jeff Polk
Liming An
Gary Thomas
Adrien Bustany
Paul Eggleton
Joshua Lock
Matthew McClintock
Yu Ke
Scott Garman
Chris Larson
Khem Raj
Mark Hatle
Saul Wold