3
0
Fork 0
forked from mirrors/nixpkgs
Commit graph

57 commits

Author SHA1 Message Date
Antoine Eiche 736848723e dockerTools.pullImage: Skopeo pulls images by digest
Skopeo is used to pull images from a Docker registry (instead of a
Docker deamon in a VM).

An image reference is specified with its name and its digest which is
an immutable image identifier (unlike image name and tag).

Skopeo can be used to get the digest of an image, for instance:
$ skopeo inspect docker://docker.io/nixos/nix:1.11 | jq -r '.Digest'
2018-05-02 21:32:20 +02:00
Jean-Philippe Braun 9751771c73 dockerTools.buildImage: add /nix/store with correct permissions
Fixes #38835.
2018-04-16 10:19:01 +02:00
Ryan Trinkle 1034aa8e9c
Merge pull request #25148 from obsidiansystems/docker-dirlinks
dockerTools: optionally preserve directory symlinks
2018-04-09 17:44:09 -04:00
Sarah Brofeldt 4874ce1701 dockerTools.tarsum: Fix upstream import 2018-03-26 18:47:31 +02:00
Antoine Eiche ac0c491836 dockerTools: add --sort=name options on all tar calls
This is to go to a reproducible image build.
Note without this options image are identical from the Docker point of
view but generated docker archives could have different hashes.
2018-03-13 13:46:47 +01:00
Antoine Eiche 346996ceec dockerTools: dereference hard links in tar archives
This is to improve image creation reproducibility. Since the nar
format doesn't support hard link, the tar stream of a layer can be
different if a dependency of a layer has been built locally or if it
has been fetched from a binary cache.

If the dependency has been build locally, it can contain hard links
which are encoded in the tar stream. If the dependency has been
fetched from a binary cache, the tar stream doesn't contain any hard
link. So even if the content is the same, tar streams are different.
2018-03-13 13:46:41 +01:00
Antoine Eiche e8f452f110 dockerTools: add an onTopOfPulledImage example
This allows to test if a pulled image can be updated by using our
Docker tools.
2018-03-13 11:59:22 +01:00
Antoine Eiche ce838e52b9 dockerTools.buildImage: do not add /nix/store in the tar stream
Since the /nix/store directory is not immutable, tar can fails if it
has to push it into the layer archive.

Fixes #34137.
2018-02-14 06:40:41 +01:00
John Ericson 888404f11b treewide: Fix deps in a few other fixed output derivations 2018-01-10 11:18:44 -05:00
Wei-Ming Yang 70e9b60b33
dockerTools.examples: correct a typo in comments
This commit is for correcting a typo in comments.
2018-01-01 16:13:40 +08:00
Ryan Trinkle ded1281f45
Merge branch 'master' into docker-dirlinks 2017-11-03 10:53:00 -04:00
Robin Gloster 20677fca59
dockerTools: fix hash to accomodate the pullImage revert 2017-09-28 14:09:49 +02:00
Robin Gloster 5c6dc717a6
Revert "dockerTools.pullImage: use skopeo to pull the image"
This reverts commit 01174c5f4d.

See https://github.com/NixOS/nixpkgs/pull/29302#issuecomment-332809092
for more information. This broke image format compatibility and
therefore amongst others mesos.
2017-09-28 14:09:49 +02:00
Robin Gloster dabb296c76
Revert "dockerTools.buildImage: Switch to the format image generated by Skopeo"
This reverts commit 35f205a4b6.

This does not use a standard format and by that breaks mesos
2017-09-28 13:01:34 +02:00
Antoine Eiche ff4d7f0fd2 dockerTools.examples.nix: set NIX_PAGER=cat environment variable 2017-09-25 09:39:15 +02:00
Antoine Eiche 35f205a4b6 dockerTools.buildImage: Switch to the format image generated by Skopeo
We were using 'Combined Image JSON + Filesystem Changeset Format' [1] to
unpack and pack image and this patch switches to the format used by the registry.

We used the 'repository' file which is not generated by Skopeo when it
pulls an image. Moreover, all information of this file are also in the
manifest.json file.
We then use the manifest.json file instead of 'repository' file. Note
also the manifest.json file is required to push an image with Skopeo.

Fix #29636

[1] 749d90e10f/image/spec/v1.1.md (combined-image-json--filesystem-changeset-format)
2017-09-23 13:17:07 +02:00
Antoine Eiche cb6fc52f99 dockerTools.buildImageWithNixDb: Make output paths valid and add gcroots
The database dump doesn't contain sha and size. This leads to invalid
path in the container. We have to fix the database by using
nix-store.
Note a better way to do this is available in Nix 1.12 (since the
database dump contains all required information).

We also add content output paths in the gcroots since they ca be used
by the container.
2017-09-20 20:14:29 +02:00
Antoine Eiche df589a438e dockerTools.buildImageWithNixDb: populate the Nix Db of the image Nix store
Currently, the contents closure is copied to the layer but there is no
nix database initialization. If pkgs.nix is added in the contents,
nix-store doesn't work because there is no nix database.

From the contents of the layer, this commit generates and loads the
database in the nix store of the container. This only works if there
is no parent layer that already have a nix store (to support several
nix layers, we would have to merge nix databases of parent layers).

We also add an example to play with the nix store inside the
container. Note it seems `more` is a missing dependency of the nix
package!
2017-09-20 20:14:24 +02:00
Antoine Eiche 01174c5f4d dockerTools.pullImage: use skopeo to pull the image
Before this patch, a VM was used to spawn docker that pulled the
VM. Now, the tool Skopeo does this job well so we can simplify our
dockerTools since we doesn't need Docker anymore:)

This also fixe the regression described in
https://github.com/NixOS/nixpkgs/issues/29271 : cntlm proxy doesn't
work in 17.09 while it worked in 17.03.

Note Skopeo doesn't produce the same output than docker pull so, we
have to update sha.
2017-09-17 08:26:02 +01:00
Antoine Eiche 132e790735 dockerTools.pullImage: change the docker deamon readiness mechanism
To wait for the docker deamon, curl requests are sent. However, if a
http proxy is set, it will respond instead of the docker daemon.
To avoid this, we send docker ps command instead of curl command.
2017-09-04 10:52:16 +02:00
Mathias Schreck 86d9b09c9b dockerTools: fix image json and manifest
The image json is not exactly the same as the layer json, therefore I
changed the implementation to use the `baseJson` which doesn’t include
layer specific details like `id`, `size` or the checksum of the layer.

Also the `history` entry was missing in the image json. I’m not totally
sure if this field is required, but a I got an error from a docker
registry when I’ve tried to receive the distribution manifest of an
image without those `history` entry:

GET: `http://<registry-host>/v2/<imageName>/manifests/<imageTag>`

```json
{
  "errors": [
    {
      "code": "MANIFEST_INVALID",
      "message": "manifest invalid",
      "detail": {}
    }
  ]
}
```

I’ve also used a while loop to iterate over all layers which should make
sure that the order of the layers is correct. Previously `find` was
used and I’m not sure if the order was always correct.
2017-08-03 11:52:03 +02:00
Antoine Eiche 9082b66096 dockerTools: fix permissions on base image
If the base image has been built with nixpkgs.dockerTools, the image
configuration and manifest are readonly so we first need to change
their permissions before removing them.

Fix #27632.
2017-07-31 13:52:35 +02:00
Antoine Eiche 9ee7e8b67e docker: generate the image configuration and manifest
This is required to push images to the Docker registry v2.
2017-07-26 23:05:29 +02:00
Antoine Eiche 8a431e13b5 docker: Remove ./ pattern when packing an image
Elements in images tar.gz generated by docker don't start by './'.
2017-07-26 23:05:29 +02:00
Antoine Eiche 0a4c43065c docker: do not import configuration and manifest from the base image
Fix #27632.
2017-07-26 20:01:41 +02:00
Antoine Eiche e28d817c9a docker: lowercase image name and tag
The docker loading (docker 1.12.6) of an image with uppercase in the
name fails with the following message:
invalid reference format: repository name must be lowercase
2017-07-25 10:47:51 +02:00
Daiderd Jordan 90ff6b1d03 Merge pull request #27017 from LnL7/docker-pure-layer
docker-tools: set user/group when creating a pure layer
2017-07-15 12:22:53 +02:00
Sarah Brofeldt 9e61958399 tarsum: Fix source path in docker.src 2017-07-13 23:09:04 +02:00
Daiderd Jordan 83fbc0f035
docker-tools: fixup permssions for extraCommands 2017-07-08 13:57:04 +02:00
Daiderd Jordan dde5865140
docker-tools: set group/owner when creating a pure layer 2017-07-01 17:08:48 +02:00
Michael Fellinger 54419f3016 using inherited instead 2017-06-27 22:22:08 +02:00
Michael Fellinger 704e04b108 dockerTools.buildImage: configurable timestamp
This way not all images have to be from 47 years ago, making it much easier to find the one you're looking for.
2017-06-27 22:11:17 +02:00
Matej Cotman 8d3cb0af9e docker tools: fix pull image function 2017-05-26 18:48:16 +02:00
rht 5628cebcf0 /bin/sh -> ${stdenv.shell} 2017-04-30 17:01:07 +02:00
Ryan Trinkle dce2c258ac dockerTools: optionally preserve directory symlinks
In some cases, this seems to save a lot (>40%) of space.
2017-04-23 09:45:21 -04:00
Tim Stewart 5ca1646bb0 docker: improve reproducibility of layers
This patch fixes file modification times to $SOURCE_DATE_EPOCH, and
ensures that files originating from the store are owned by root:root.
Both changes improve reproducibility, and the latter allows proper
building on a host where the store is owned by a non-root user.
2017-04-16 22:46:24 -04:00
Lazar Bodor 186ed3bf8f dockerTools: rsync contents with -keep-dirlinks 2016-11-24 12:48:26 +11:00
Daiderd Jordan f84e2efcfd Merge pull request #20018 from adnelson/use_basename_docker
docker.buildUtils: use baseNameOf to allow for slashes in name
2016-11-01 12:29:28 +01:00
Test ca5fda39dc docker.buildUtils: use baseNameOf to allow for slashes in name 2016-10-31 10:11:48 -05:00
Tom Boettcher 30b09b8620 dockerTools: Fix loop typo.
This typo was causing only a single layer to be de-duplicated,
rather than all parent layers.
2016-10-24 14:30:00 -05:00
Allen Nelson 940aafcbcc use rsync, comment out pulls 2016-10-03 15:12:29 -05:00
Allen Nelson b9007c8d83 add examples file 2016-10-03 12:07:33 -05:00
Allen Nelson 4abe579250 add docs to docker build functions
bring back ls_tar

replace goPackages with go

don't hardcode /nix/store in vmTools

more docs
2016-09-29 12:52:57 -05:00
Tom Boettcher 1e8b69c35e dockerTools: Fix layer redundancy.
When building an image with multiple layers, files
already included in an underlying layer are supposed to
be excluded from the current layer. However, some subtleties
in the way filepaths are compared seem to be blocking this.

Specifically:
* tar generates relative filepaths with directories ending in '/'
* find generates absolute filepaths with no trailing slashes on directories

That is, paths extracted from the underlying tarball look like:
    nix/store/.../foobar/
whereas the layer being generated uses paths like:
    /nix/store/.../foobar

This patch modifies the output of "tar -t" to match the latter format.
2016-09-23 16:40:59 -05:00
Profpatsch 61462c94e6 lib/fetchers.nix: factor out impure proxy vars (#18702)
Apparently everyone just copied those variables, instead of creating a
library constant for them. Some even removed the comment. -.-
2016-09-17 21:50:01 +02:00
Nathan Zadoks b9284e445b dockerTools: disable compression in tarsum.go
Previously, tarsum would compress the (discarded) tarball produced.
That's a waste of CPU, and a waste of time.
2016-07-02 02:22:36 -04:00
Nathan Zadoks 23e9e33975 dockerTools: format tarsum.go with gofmt
Nearly all Go code on this earth is formatted with gofmt, and it's
somewhat surprising to find a Go file that isn't formatted accordingly.
2016-07-02 02:22:36 -04:00
Kamil Chmielewski 7eb671ebcd no more goPackages 2016-06-09 13:08:00 +02:00
Brian McKenna 0167b61ef4 dockerTools: only add "/nix" if it exists
The /nix path in 4d200538 of the layer tar didn't exist for some
packages, such as cacert. This is because cacert just creates an /etc
directory and doesn't depend on any other /nix paths. If we tried
putting this directory in the tar and using overlayfs with it, we'd get
"Invalid argument" when trying to remove the directory.

We now check whether the closure is non-empty before telling tar to
store the /nix directory.

Fixes #14710.
2016-04-16 01:16:49 +10:00
Brian McKenna bc2f314f73 dockerTools: make tars deterministic
There were two sources of non-determinisim coming into the images. The
first was tar mtimes, the second was pigz/gzip times.

An example image now passes with the --check flag.
2016-04-15 09:29:15 +10:00