It looks like we're about to see a lot of changes in branch names in repos. If
we have the prune option here, those old names are lost, the changes propagate
to our source mirrors and our old releases break.
We have the force option so any replaced references should be replaced, its only
orphaned branches which will now be preserved.
I believe this behaviour will cause us fewer problems given the changes that
look likely to happen.
(Bitbake rev: 08aafc53ba911337d2d3558654b06f294ae634e2)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Since 2017-08-17 (git version 2.14.1.473.g3ec7d702a) using deprecated
git branch parameter "--set-upstream" causes a fetcher error. Replace
it by "--set-upstream-to".
https://git.kernel.org/pub/scm/git/git.git/commit/?id=52668846ea2d41ffbd87cda7cb8e492dea9f2c4d
says, it's deprecated since 2012-08-30 so hopefully all still supported
host distributions have new enough git to support "--set-upstream-to".
ERROR: PACKAGE do_unpack: Fetcher failure: ...;
git -c core.fsyncobjectfiles=0 branch --set-upstream master origin/master failed with exit code 128, output:
fatal: the '--set-upstream' option is no longer supported. Please use '--track' or '--set-upstream-to' instead.
ERROR: PACKAGE do_unpack: Function failed: base_do_unpack
(Bitbake rev: 698e36430761cd3cf81283d58ca5bb3426bd24dd)
Signed-off-by: Andre Rosa <andre.rosa@lge.com>
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We call git ls-remote to get the latest revision from a git repository,
however by calling runfetchcmd() we can end up recursively running
git ls-remote a number of times with OE e.g. if ${SRCPV} is in PV, ${PV}
is in WORKDIR, and ${WORKDIR} is in PATH (as a result of recipe-specific
sysroots), our call to runfetchcmd() exports PATH so _lsremote() will
get called again - with the end result that we run git ls-remote 30
times in quick succession (!). Prevent that from happening by using a
guard variable and returning a dummy value if it's called recursively.
Fixes [YOCTO #11185].
(Bitbake rev: ff1ccd1db5d70b3fc9ad0d3e8f3d7b804c22bf36)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
FetchError isn't defined, use bb.fetch2.FetchError in this context.
(Bitbake rev: 945fa980e027753df2c21d84eb63dcaddb2caaee)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The touch of .done explicitly specifies the path, so there's no need for
workdir=, and "os.path.join('.')" is identical to just '.'.
(Bitbake rev: 955cbfdaa2400d15ec428b65848e6835c9f44860)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
list.index() isn't a particularly efficient operation, so keep track of our
position via enumerate() instead, which is more pythonic as well.
(Bitbake rev: dec6e90a4d27ee335e9c78aeebd277098fec94d1)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Cleanup some more usage of bb.data APIs in the fetchers.
(Bitbake rev: 9752fd1c10b8fcc819822fa6eabc2c1050fcc03b)
Signed-off-by: Andre McCurdy <armccurdy@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For spelling's sake, rename Python routine "setup_revisons" to
"setup_revisions."
(Bitbake rev: 4df59b027c02ef39d72476251ccd3fd62fc20bf6)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(Bitbake rev: 05f5421b2e44cd58c5912848de43d5884d070150)
Signed-off-by: Robert P. J. Day <rpjday@crashcourse.ca>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
getVar() now defaults to expanding by default, thus remove the True
option from getVar() calls with a regex search and replace.
Search made with the following regex: getVar ?\(( ?[^,()]*), True\)
(Bitbake rev: 3b45c479de8640f92dd1d9f147b02e1eecfaadc8)
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The old style bb.data.getVar/setVar API is obsolete. Most of bitbake
doesn't use it but there were some pieces that escaped conversion. This
patch fixes the remaining users mostly in the fetchers.
(Bitbake rev: ff7892fa808116acc1ac50effa023a4cb031a5fc)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Fix the methods in all fetchers so they don't change
the current working directory of the calling process, which
could lead to "changed cwd" warnings from bitbake.
(Bitbake rev: 6aa78bf3bd1f75728209e2d01faef31cb8887333)
Signed-off-by: Matt Madison <matt@madison.systems>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Introduce a new 'usehead' url parameter for git repositories. Specifying
usehead=1 causes bitbake to use whatever commit the repository HEAD is
pointing to. Usage of usehead=1 is only allowed for local git
repositories, i.e. it must always be accompanied with protocol=file url
parameter.
[YOCTO #9351]
(Bitbake rev: 2673fac5a9d06de937101e3fb2ddf1e60ff99abf)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Signed-off-by: Markus Lehtonen <markus.lehtonen@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Implement progress reporting support specifically for the fetchers. For
fetch tasks we don't necessarily know which fetcher will be used (we
might initially be fetching a git:// URI, but if we instead download a
mirror tarball we may fetch that over http using wget). These programs
also have different abilities as far as reporting progress goes (e.g.
wget gives us percentage complete and rate, git gives this some of the
time depending on what stage it's at). Additionally we filter out the
progress output before it makes it to the logs, in order to prevent the
logs filling up with junk.
At the moment this is only implemented for the wget and git fetchers
since they are the most commonly used (and svn doesn't seem to support
any kind of progress output, at least not without doing a relatively
expensive remote file listing first).
Line changes such as the ones you get in git's output as it progresses
don't make it to the log files, you only get the final state of the line
so the logs aren't filled with progress information that's useless after
the fact.
Part of the implementation for [YOCTO #5383].
(Bitbake rev: 4027649f422ee64b1c4e1ad8d48ac295050afbff)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Previously the code used to match a reference to its SHA-1 in
_latest_revision() used the Python "in" operator, which made it match
if the reference matched the beginning of an existing tag or
branch. This test, however, must be exact. I.e., either the reference
matches a tag or branch exactly, or it does not match at all.
(Bitbake rev: e5986c78a6108fd7578989c20efcbf0b81c97e03)
Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
It was used for workaround git 1.7.9.2 which was released in 2012 which
should not be existed on nowadays host, so remove it to avoid
confusions.
(Bitbake rev: 6140d0cc9aecf1029ca16fed47071dfcc92f4269)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This reverts commit 4193e99adce8e88f12ac88d7578ad39575f7e346.
It seems the underlying issue was caused by ":" in the url which isn't
supported. The patch was therefore incorrect.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This goes undetected most of the time, but when updating a repository,
if the ud.fullmirror file is not present, you end up getting an
exception instead of carrying on because the errno module is not
loaded (specifically "if exc.errno != errno.ENOENT").
(Bitbake rev: e6fca8480731ce817df9bee61438347a5e3d3017)
Signed-off-by: Kristian Amlie <kristian.amlie@mender.io>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Some servers, e.g. bitbucket.org can't cope with ssh:// as part of
the git url syntax. git itself is happy enough with this but you
get server side errors when using it.
This changes the git fetcher to use the more common ssh url format
which also means we need a : before the path.
Seems a shame to have to do this due to broken servers however
it should be safe enough since this other form is the one most people
use on the commandline so it should be safe enough.
[YOCTO #8864]
(Bitbake rev: 4193e99adce8e88f12ac88d7578ad39575f7e346)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Rename REGEX to UPSTREAM_CHECK_REGEX, REGEX_URI to UPSTREAM_CHECK_URI, and
GITTAGREGEX to UPSTREAM_CHECK_GITTAGREGEX to better reflect their purpose
and to reflect a common namespace.
(Bitbake rev: f0a9e783f9969573fd74edfa241ef14f18ac684e)
Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Signed-off-by: Ross Burton <ross.burton@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently this module is dereferencing errno.ENOENT but the python module "errno"
is not imported, which causes a crash when fetching from a git repository.
(Bitbake rev: 93e4c9bb2393b1074f5a01e7eaaac742a59d8086)
Signed-off-by: Romain Perier <romain.perier@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We were maintaining state in the form of ud.repochanged to determine whether
we need to write the tarball in the case where the tarball already exists, but
this state is maintained only in memory. If we need to update the git repo,
set ud.repochanged to True, and then are interrupted or killed, the tarball
will then be out of date.
Rather than maintaining this state, simply remove the out of date tarball when
we update the git repo, and it will be re-created with updated content in
build_mirror_data.
[YOCTO #6366]
(Bitbake rev: eaaa81393f181432c8586b17ade623f42c9fed2e)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When using an absolute file URI, there's no host, and the path starts with
'/', the dir under ${DL_DIR}/git2/ ends up starting with '.', so is hidden.
Remove any leading '.' to fix this.
(Bitbake rev: 8dce6964d56b36a77fb113f2ad496cc992a5ff36)
Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to know the revision associated with the upstream version in
SCM like git, this change broke return convention, oe-core recipeutils
get_recipe_upstream_version need to be updated.
tests/fetch.py: Updated git latest_versionstring test for comply new
convention.
[YOCTO #7605]
(Bitbake rev: fd175dc90024c503134c11cbd83e77d88c406ac8)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Don't limit the tag search for only tags end with ^{}.
(Bitbake rev: 7006ab313766344cf33481228465082ed5977d28)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Create a branch and named as upstream branch when checkout source
* Set the branch to track remote branch.
(Bitbake rev: 1ba20e4fe9c884515b200589fe379ad5eeda10bd)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
In order to pass connection cache object to checkstatus function
add fetch parameter.
(Bitbake rev: fbb9c6f5538084e125b58118a86968908e6f895b)
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gitpkgv_revision returns a sortable revision number that can be used
in the PKGV variable for example. To mimic meta-openembedded gitpkgv
behaviour to provide a sortable revision numner, one could set the
following:
PKGV = "1.0+${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
This would yield a package version like "1.0+69+fb5eb80".
(Bitbake rev: 989c08f62aff7b707c25c692c23284f16506b7bc)
Signed-off-by: Mike Looijmans <mike.looijmans@topic.nl>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Pass proper repository url without arguments after a semicolon.
Executing checkuri on a rule with git repository in SRC_URI does
not report errors when working offline because wrong repository
url is passed to the ls-remote command. For example
"bitbake -c checkuri glibc" command executes:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git;branch=release/2.21/master"
command in a shell subprocess to determine if url is valid.
Shell subprocess executes in fact 2 commands:
"git -c core.fsyncobjectfiles=0 ls-remote git://sourceware.org/git/glibc.git"
and
"branch=release/2.21/master"
First one returns 127 or 128 depending on error but second one
returns 0 because it is just env variable setup. Therefore we're not catching
connection error.
[YOCTO #7558]
(Bitbake rev: efa44d04137977f883db4a643b0f774e91514722)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The URL is not sent when _latest_revision generates and exception.
When performing the sanity checks it is not possible to know the URI that failed.
This add the URL when latest_revision generates an exception.
[YOCTO: #7592]
(Bitbake rev: 9f2115b07a55cb14e4a74dc6fbd3707c28a234d0)
Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If you're interested in using the checked out repository for development
(e.g. in OE with devtool) then you ideally want the origin remote to
point to the repository it was fetched from, so just set that after
cloning.
(As part of this I did a minor refactor so we have one function to
generate the repository URL, which was already in two places.)
Fixes [YOCTO #7756].
(Bitbake rev: 80ecd1c54d4c748cee3a7ce0d64013a346e7671e)
Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
If the subpath parameter to the git fetcher ends with a trailing '/',
bb.utils.prunedir() will be called on '/'...
Fixes [YOCTO #7620].
(Bitbake rev: 380a3fb372c8b0a53dd7528562e6e7a222dc76ef)
Signed-off-by: Anders Darander <anders@chargestorm.se>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the code ignores lightweight tags which has caused some user
complaints. We can't put the right search list in place easily since
the results don't come back in a good order, head happens to sort
before tags.
In the end I refactored the function so we get the complete list of
remotes and then we can filter it ourselves in the order we chose,
including checking for light weight tags, preferring the proper ones.
Hopefully this resolves the issues people have been seeing.
[YOCTO #6881]
(Bitbake rev: 07ad307065bb15a48f0015b9e4a643201abdc283)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Being able to generate a version string representing the most recent git commit
given git is useful, not least for the package reporting system.
This adds in a latest_versionstring method to the git fetcher
which allows users to query the latest version using ls-remote
and filtering the responses.
The patch also adds unittests for this function so that if
improvements are made, the original test urls can be used
to evaulate the those changes.
This is based on code from Irina Patru <irina.patru@intel.com>.
(Bitbake rev: f71c8c0354e87fed80bc845db6728e6e18ce9c4d)
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Aníbal Limón <anibal.limon@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
This makes it possble to fetch Gerrit review references which are
normally stored under refs/changes.
Please disregard previous patch with the same topic.
(Bitbake rev: 268e9c0c6830e8e621c418f20c2ca12dc840e48b)
Signed-off-by: Fredrik Svensson <fredrik.svensson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We only ever clone other repositories, if there were a problem such as power
failure, we'd blow away data and rebuild. As such we don't need fsync(). With
filesystems like ext*, the fsync pushes nearly all the data out to disk
which impacts all running processes.
We therefore set a configuration parameter to disable the fsync() calls.
Also fixup a case where basecmd wasn't being used for no good reason.
(Bitbake rev: 0a26abaf3a1e34d556c9375068dd17c879568d0f)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There is other code which can want to run ls-remote style commands with
different parameters so split out the function.
(Bitbake rev: 13f1138f5504feee0ee8e8f3a0675d0bea490351)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
We need to deference tags when trying to map them to commit IDs with
ls-remote. If we don't do this, a given commit might not show up
later in a specific branch. There appears to be no good reason not
to do this.
(Bitbake rev: 8ef24f4c834298348172b96ec0b855bf09552b09)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
When specifying tags, they're searched for unanchored so foo/bar could
match:
refs/heads/abc/foo/bar
refs/heads/xyz/foo/bar
refs/heads/foo/bar
This change anchors the expressions so they are based against heads
or tags (or any other base level tree that has been created).
(Bitbake rev: df2e0972cd1db7abd5ec8b7cb295fb0c42e284a4)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
For rebased git tree, some commits doesn't exist in any branch, and such commits are
valid in tag, the change is useful for such case.
(Bitbake rev: f594cb9f5a18dd0ab2342f96ffc6dba697b35f65)
Signed-off-by: Zhenhua Luo <zhenhua.luo@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
There was a problem:
$ bitbake xf86-video-omapfb -cfetch && bitbake xf86-video-omapfb -ccleanall
The git2_git.pingu.fi.xf86-video-omapfb.tar.gz has been removed from the
DL_DIR, but the git2_git.pingu.fi.xf86-video-omapfb.tar.gz.done still exists,
this is because the "open(ud.donestamp, 'w').close()" in try_mirror_url() will
create the git2_git.xxx.tar.gz.done, but no one removes it (the clean() in
fetch2/__init__.py removes the DL_DIR/git2/pkg.done)
This only happens on the git fetcher AFAIK.
[YOCTO #5688]
(Bitbake rev: fb2dc84875eb477661f421b21bc404d4805ce379)
Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The basecmd is initialized in urldata_init; there's no need redoing that
work.
(Bitbake rev: f8df6f746fb2e27f029a5449cee6c891b1f36f4f)
Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Currently the fetcher doesn't distinguish between names that the fetcher
needs to resolve verses branch names that the user specified.
This meant that if you specify a tag and a branch, the fetcher broke. This
separates the two so that the branch name is preserved and can be used in
appropriate places.
(Bitbake rev: e85f39fe9d1b224414b5da0780da514f75c5df92)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
The fetcher made the rather bold assumption that if it fetched from the upstream,
the revisions were present and correct. These checks are fast and ensure that
really is the case. The avoids accidental network accessed and missing
branch configuration problems.
(Bitbake rev: a9112a102a89049cda597dad449e922c9e957a5d)
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Harald Welte
Richard Purdie
Andre Rosa
Paul Eggleton
Christopher Larson
Andre McCurdy
Robert P. J. Day
Markus Lehtonen
Joshua Lock
Matt Madison
Peter Kjellerstedt
Robert Yang
Kristian Amlie
Alexander Kanavin
Romain Perier
Aníbal Limón
Mike Looijmans
Marcin Smoczyński
Mariano Lopez
Anders Darander
Fredrik Svensson
Zhenhua Luo
Olof Johansson
Florin Sarbu