The fetchgit function in nixpkgs sets the leaveDotGit argument to true
if deepClone is set to true. nix-prefetch-git did behave differently. It
would not assume --leave-dotGit if --deepClone is specified. With this
change the inconsistency is addressed by assuming --leave-dotGit if
--deepClone is specified.
When maintainers override stages of `fetchgit' (e.g. `postPatch`) it
is very easy for them to accidentally leak the outpath-hash of their
current `stdenv` into `fetchgit''s output, and therefore into the
value they paste into `sha256`.
This is a problem, because the resulting expression will break
whenever any change is made to `stdenv` or when anybody attempts to
build the expression on a different platform than the one used by the
original maintainer.
Almost as much of a problem is the fact that CI **does not catch**
these problems. The `fetchgit` is run only once, then its output goes
into cachix, and all future builds (hydra, CI, ofborg) pull from
cachix.
Let's offer maintainers the option to check that they aren't making
this mistake, by passing through `allowedRequisites`. The default
value is `null`, but it might be worth changing that at some point in
the future.
It is also sometimes difficult to communicate to package maintainers
why their expression is problematic. Having `allowedRequisites`
passed through makes it easier to do this: "look, when I switch on
`allowedRequisites` your package breaks; are you sure you meant to
hardcode the hash today's `x86_64-linux.stdenv` into your expression?`
For an example use case, see https://github.com/NixOS/nixpkgs/pull/171223
The issue above is part of a larger problem with nixpkgs infra: there
large parts of cachix cannot be reproduced easily if they are lost.
Once something ends goes into cachix, we never ever again reverify the
procedure by which it was placed into cachix.
a67950f20b added `url` attribute
from `fetchurl` and therefore also from `fetchzip`.
We previously relied on `url` from fetchgit-based fetchers
to find the repo URL but now it will just return tarballs
in the case of `fetchFrom{GitHub,GitLab}`.
Let’s add an attribute to `fetch{git,FromGitHub,FromGitLab}`
to expose a repo URL consistently.
It's hugely inefficient as we can't use shallow cloning (--depth=1).
This has been tested and adapted for quite a few hosts fetchgit is used on in
Nixpkgs. For those where fetching the hash directly doesn't work (most notably
git.savannah.gnu.org), we simply fall back to the old method.
The reason for this change is simply to avoid the following messages
that are unnecessary and can be confusing (and these messages will be
repeated for each submodule):
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint:
hint: git config --global init.defaultBranch <name>
hint:
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint:
hint: git branch -m <name>
With this change the behaviour remains unchanged (apart from the
suppressed "warning" in the console output of course) and therefore this
doesn't cause any hashes to change and by default nix-prefetch-git uses
the "fetchgit" branch anyway (branchName can be set to override the
default):
Switched to a new branch 'fetchgit'
For that reason the initial branch name doesn't matter anyway and since
we're not relying on / hardcoding "master" we could simply switch to
"main" (which seems most common nowadays). See [0] for more details on
why this wouldn't break anything.
However, since the initial branch name doesn't matter and to avoid any
additional risks it was "decided" to keep using "master" (s. #113313).
[0]: https://github.com/NixOS/nixpkgs/pull/113313#issuecomment-780589516
`git repack` and `git gc` sometimes print “Nothing new to pack.”
to stdout, which breaks redirecting output to JSON file.
Let’s move the stdout of all git calls where it is not used to stderr
so that we still receive the info but it does not pollute our output.
Especially as a new user it is a much better experience to receive a
proper help response to `-h`. Currently passing `-h` will cause some
runtime error with the `git remote` error help being shown. Not very
helpful.
It doesn't hurt to be a bit more user friendly in this case.
This is useful when running tools like NixOps or nix-review
on workstations where the upload to the builder is significantly
slower then downloading the source on the builder itself.
The script would parse the output of `git submodule status` but
didn't handle paths with spaces in them. This would result in the
following error when trying to determine the URL of the submodule:
error: key does not contain a section: .url
Summary:
According to git-submodule manpage,
"git submodule status" prefixes the hash with a '-' if it is not
initialized, and other chars in other circumstances.
(this is consistent on the various git versions tested)
nix-prefetch-git runs "git submodule init" which does you'd think,
but apparently despite this earlier versions of git before 2.16
would still give the hash the '-' suffix.
In particular this is the behavior when using 2.15 and 2.14.1
from the nixos-17.09 and nixos-17.03 channels respectively.
The script then used awk to drop the first char of the first field
which does the wrong thing when there is no prefix emitted:
while there is a space character before the hash, this is not
part of the field and so we ended up eating the first character
of the hash.
To fix this in a way that also works with the previous behavior,
this commit instead uses awk to grab the hash field
and uses tr to delete any '-' chars should they be present.
This seems to work in my testing, and for example can now
successfully fetch the source for "nginxModules.brotli"
where previously it would generate an error:
fatal: '22564a95d9ab58865a096b8d9f7324ea5f2e03e' is not a commit and a branch 'fetchgit' cannot be created from it
(we dropped a '2' from the beginning of the hash)
Currently we wrap ssh so it can find the config file passed in by
<ssh-config-file>. If one however uses ProxyCommand ssh, then ssh that
is on PATH is taken (which is also unavailable when using nix-shell
--pure), which is the plain ${openssh}/bin/ssh.
This commit makes sure our wrapped ssh is available on PATH.
