3
0
Fork 0
forked from mirrors/nixpkgs
Commit graph

92 commits

Author SHA1 Message Date
Antoine Eiche fe6860800b dockerTools.buildImage.runAsRoot: preserve layers ordering at image unpacking
This patch preserves the ordering of layers of a parent image when the
image is unpacked.

Fixes #55290
2019-03-12 12:04:22 +01:00
Antoine Eiche af23d1e2e7 nixos/test/docker-tools: fix Nix image digest 2019-03-06 09:26:33 +01:00
Wael M. Nasreddine 9b22a51712
build-support/docker: fix the build of tarsum with Go 1.12 2019-03-05 10:53:21 -08:00
Jörg Thalheim dadc7eb329
treewide: use runtimeShell instead of stdenv.shell whenever possible
Whenever we create scripts that are installed to $out, we must use runtimeShell
in order to get the shell that can be executed on the machine we create the
package for. This is relevant for cross-compiling. The only use case for
stdenv.shell are scripts that are executed as part of the build system.
Usages in checkPhase are borderline however to decrease the likelyhood
of people copying the wrong examples, I decided to use runtimeShell as well.
2019-02-26 14:10:49 +00:00
Austin Seipp c36c048c0e dockerTools: mark store-path-to-layer.sh as executable
bcf54ce5bb introduced a treewide change to
use ${stdenv.shell} where-ever possible. However, this broke a script
used by dockerTools, store-path-to-layer.sh, as it did not preserve the
+x mode bit. This meant the file got put into the store as mode 0444,
resulting in a build-time error later on that looked like:

    xargs: /nix/store/jixivxhh3c8sncp9xlkc4ls3y5f2mmxh-store-path-to-layer.sh: Permission denied

However, in a twist of fate, bcf54ce5bb
not only introduced this regression but, in this particular instance,
didn't even fix the original bug: the store-path-to-layer.sh script
*still* uses /bin/sh as its shebang line, rather than an absolute path
to stdenv. (Fixing this can be done in a separate commit.)

Signed-off-by: Austin Seipp <aseipp@pobox.com>
2019-02-21 08:30:47 -06:00
Vladimír Čunát 024407bf9a
Merge branch 'master' into staging-next
Hydra nixpkgs: ?compare=1505754
2019-02-19 12:11:04 +01:00
Antoine Eiche da7cd82ab1 dockerTools.buildImage: preserve layers ordering at image repacking
This patch preserves the ordering of layers of a parent image when the
new image is packed.

It is currently not the case: layers are stacked in the reverse order.

Fixes #55290
2019-02-16 00:50:23 +01:00
Vladimír Čunát 3fe32b675e
Merge branch 'master' into staging 2019-01-22 15:59:42 +01:00
lewo 105ffa445d
Merge pull request #54270 from xtruder/build-support/docker/layered_image_tag_passthru
dockerTools: buildLayeredImage passthru imageTag
2019-01-20 12:21:20 +01:00
Jaka Hudoklin 1787afb861
dockerTools: buildLayeredImage passthru imageTag 2019-01-18 13:52:07 +01:00
rnhmjoj bcf54ce5bb
treewide: use ${stdenv.shell} instead of /bin/sh where possible 2019-01-16 20:37:15 +01:00
lewo 7612a6add4
Merge pull request #52870 from xtruder/pkgs/dockerTools/buildLayeredImage/extraCommands
dockerTools: allow to pass extraCommands, uid and gid to buildLayered image
2019-01-10 19:00:19 +01:00
Jaka Hudoklin 954cda5c9d
dockerTools: allow to pass extraCommands, uid and gid to buildLayeredImage 2019-01-10 16:02:23 +01:00
lewo bb86d0ca24
Merge pull request #52959 from nlewo/runAsRootParentImage
dockerTools.example.runAsRootParentImage: init
2018-12-29 12:56:47 +01:00
Antoine Eiche 43cbba0274 dockerTools.examples.runAsRootParentImage: init
Example of running something as root on top of a parent image.
This is a regression test related to the PR #52109.
2018-12-27 11:15:35 +01:00
Darius Jahandarie a3c536fcb3 dockerTools.buildImage: fix two bugs introduced in c88337c9ac 2018-12-14 15:48:58 -05:00
Graham Christensen c88337c9ac
dockerTools.buildImage: support using a layered image in fromImage
Docker images used to be, essentially, a linked list of layers. Each
layer would have a tarball and a json document pointing to its parent,
and the image pointed to the top layer:

    imageA  ----> layerA
                    |
                    v
                  layerB
                    |
                    v
                  layerC

The current image spec changed this format to where the Image defined
the order and set of layers:

    imageA  ---> layerA
            |--> layerB
            `--> layerC

For backwards compatibility, docker produces images which follow both
specs: layers point to parents, and images also point to the entire
list:

    imageA  ---> layerA
            |      |
            |      v
            |--> layerB
            |      |
            |      v
            `--> layerC

This is nice for tooling which supported the older version and never
updated to support the newer format.

Our `buildImage` code only supported the old version, so in order for
`buildImage` to properly generate an image based on another image
with `fromImage`, the parent image's layers must fully support the old
mechanism.

This is not a problem in general, but is a problem with
`buildLayeredImage`.

`buildLayeredImage` creates images with newer image spec, because
individual store paths don't have a guaranteed parent layer. Including
a specific parent ID in the layer's json makes the output less likely
to cache hit when published or pulled.

This means until now, `buildLayeredImage` could not be the input to
`buildImage`.

The changes in this PR change `buildImage` to only use the layer's
manifest when locating parent IDs. This does break buildImage on
extremely old Docker images, though I do wonder how many of these
exist.

This work has been sponsored by Target.
2018-12-05 14:25:54 -05:00
Antoine Eiche c12f75649e dockerTools.buildImageWithNixDb: simplifications and switch to closureInfo
Since Nix 2 is now the stable Nix version, we can use closureInfo
which simplifies the Nix database initialisation (size and hash are
included in the "dump").
2018-11-12 18:30:53 +01:00
Sarah Brofeldt b256df4937 dockerTools: Use nix instead of nixUnstable 2018-10-01 09:51:52 +02:00
Graham Christensen fb2d153dac
dockerTools: test buildLayeredImage 2018-09-27 14:19:43 -04:00
Graham Christensen 4fe9006190 dockerTools.buildLayeredImage: init
Create a many-layered Docker Image.

Implements much less than buildImage:

 - Doesn't support specific uids/gids
 - Doesn't support runninng commands after building
 - Doesn't require qemu
 - Doesn't create mutable copies of the files in the path
 - Doesn't support parent images

If you want those feature, I recommend using buildLayeredImage as an
input to buildImage.

Notably, it does support:

 - Caching low level, common paths based on a graph traversial
   algorithm, see referencesByPopularity in
   0a80233487993256e811f566b1c80a40394c03d6
 - Configurable number of layers. If you're not using AUFS or not
   extending the image, you can specify a larger number of layers at
   build time:

       pkgs.dockerTools.buildLayeredImage {
         name = "hello";
         maxLayers = 128;
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

 - Parallelized creation of the layers, improving build speed.
 - The contents of the image includes the closure of the configuration,
   so you don't have to specify paths in contents and config.

   With buildImage, paths referred to by the config were not included
   automatically in the image. Thus, if you wanted to call Git, you
   had to specify it twice:

       pkgs.dockerTools.buildImage {
         name = "hello";
         contents = [ pkgs.gitFull ];
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

   buildLayeredImage on the other hand includes the runtime closure of
   the config when calculating the contents of the image:

       pkgs.dockerTools.buildImage {
         name = "hello";
         config.Cmd = [ "${pkgs.gitFull}/bin/git" ];
       };

Minor Problems

 - If any of the store paths change, every layer will be rebuilt in
   the nix-build. However, beacuse the layers are bit-for-bit
   reproducable, when these images are loaded in to Docker they will
   match existing layers and not be imported or uploaded twice.

Common Questions

 - Aren't Docker layers ordered?

   No. People who have used a Dockerfile before assume Docker's
   Layers are inherently ordered. However, this is not true -- Docker
   layers are content-addressable and are not explicitly layered until
   they are composed in to an Image.

 - What happens if I have more than maxLayers of store paths?

   The first (maxLayers-2) most "popular" paths will have their own
   individual layers, then layer #(maxLayers-1) will contain all the
   remaining "unpopular" paths, and finally layer #(maxLayers) will
   contain the Image configuration.
2018-09-26 17:54:14 -04:00
Graham Christensen 2bf0ee3b2b dockertools: tarsum: turn in to a buildInput 2018-09-26 15:50:04 -04:00
Graham Christensen aedc651903
dockerTools.buildImage: test that created=now makes an unstable date 2018-09-20 13:06:14 -04:00
Graham Christensen a32d7e0c74 dockerTools.buildImage: support impure dates
Because dates are an impurity, by default buildImage will use a static
date of one second past the UNIX Epoch. This can be a bit frustrating
when listing docker images in the CLI:

    $ docker image list
    REPOSITORY   TAG      IMAGE ID       CREATED        SIZE
    hello        latest   08c791c7846e   48 years ago   25.2MB

If you want to trade the purity for a better user experience, you can
set created to now.

    pkgs.dockerTools.buildImage {
      name = "hello";
      tag = "latest";
      created = "now";
      contents = pkgs.hello;

      config.Cmd = [ "/bin/hello" ];
    }

and now the Docker CLI will display a reasonable date and sort the
images as expected:

    $ docker image list
    REPOSITORY   TAG      IMAGE ID       CREATED              SIZE
    hello        latest   de2bf4786de6   About a minute ago   25.2MB
2018-09-20 18:26:02 +02:00
Jack Kelly af5eab6ea6 dockerTools.pullImage: correct default arch 2018-09-19 16:13:15 +10:00
Nick Novitski c58b11d229 dockerTools.pullImage: control OS and architecture 2018-07-27 12:29:31 -07:00
volth 52f53c69ce pkgs/*: remove unreferenced function arguments 2018-07-21 02:48:04 +00:00
Antoine Eiche d44b778d10 dockerTools.examples: explicitly set image tag to fix docker-tools tests
docker-tools tests load images without specifying any tag
value. Docker then uses the image with tag "latest" which doesn't
exist anymore since commit 39e678e24e.
2018-07-06 16:38:42 +02:00
Mathias Schreck 39e678e24e dockerTools.buildImage: add option to use nix output hash as tag 2018-07-06 15:15:09 +02:00
lewo 0644b4d948 dockerTools.pullImage: expose image* attributes (#41366)
Attributes `imageName` and `imageTag` are exposed if the image is
built by our Nix tools but not if the image is pulled. So, we expose
these attributes for convenience and homogeneity.
2018-06-03 22:58:23 +02:00
lewo 2e98e0c003
Merge pull request #40947 from samueldr/fix/34779
dockerTools: fixes extraCommands for mkRootLayer.
2018-05-24 21:22:31 +02:00
Samuel Dionne-Riel 902b0593be tests/docker-tools: Adds regression test for #34779 2018-05-24 12:23:51 -04:00
Antoine Eiche 8f71ce7e80 skopeo: 0.1.29 -> 0.1.30
Skopeo used by our docker tools was patched to work in the build
sandbox (it used /var/tmp which is not available in the sandbox).
Since this temporary directory can now be set at build time, we remove
the patch from our docker tools.
2018-05-24 15:33:52 +02:00
Samuel Dionne-Riel 60737bd319 dockerTools: fixes extraCommands for mkRootLayer.
The extraCommands was, previously, simply put in the body of the script
using nix expansion `${extraCommands}` (which looks exactly like bash
expansion!).

This causes issues like in #34779 where scripts will eventually create
invalid bash.

The solution is to use a script like `run-as-root`.

 * * *

Fixes #34779
2018-05-24 06:51:26 -04:00
aszlig 42a0b11450
dockerTools.pullImage: Fix build with sandboxing
Regression introduced in 736848723e.

This commit most certainly hasn't been tested with sandboxing enabled
and breaks not only pullImage but also the docker-tools NixOS VM test
because it doesn't find it's certificate path and also relies on
/var/tmp being there.

Fixing the certificate path is the easiest one because it can be done
via environment variable.

I've used overrideAttrs for changing the hardcoded path to /tmp (which
is available in sandboxed builds and even hardcoded in Nix), so that
whenever someone uses Skopeo from all-packages.nix the path is still
/var/tmp.

The reason why this is hardcoded to /var/tmp can be seen in a comment in
vendor/github.com/containers/image/storage/storage_image.go:

  Do not use the system default of os.TempDir(), usually /tmp, because
  with systemd it could be a tmpfs.

With sandboxed builds this isn't the case, however for using Nix without
NixOS this could turn into a problem if this indeed is the case.

So in the long term this needs to have a proper solution.

In addition to that, I cleaned up the expression a bit.

Tested by building dockerTools.examples.nixFromDockerHub and the
docker-tools NixOS VM test.

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @nlewo, @Mic92, @Profpatsch, @globin, @LnL7
2018-05-06 04:57:24 +02:00
Antoine Eiche 736848723e dockerTools.pullImage: Skopeo pulls images by digest
Skopeo is used to pull images from a Docker registry (instead of a
Docker deamon in a VM).

An image reference is specified with its name and its digest which is
an immutable image identifier (unlike image name and tag).

Skopeo can be used to get the digest of an image, for instance:
$ skopeo inspect docker://docker.io/nixos/nix:1.11 | jq -r '.Digest'
2018-05-02 21:32:20 +02:00
Jean-Philippe Braun 9751771c73 dockerTools.buildImage: add /nix/store with correct permissions
Fixes #38835.
2018-04-16 10:19:01 +02:00
Ryan Trinkle 1034aa8e9c
Merge pull request #25148 from obsidiansystems/docker-dirlinks
dockerTools: optionally preserve directory symlinks
2018-04-09 17:44:09 -04:00
Sarah Brofeldt 4874ce1701 dockerTools.tarsum: Fix upstream import 2018-03-26 18:47:31 +02:00
Antoine Eiche ac0c491836 dockerTools: add --sort=name options on all tar calls
This is to go to a reproducible image build.
Note without this options image are identical from the Docker point of
view but generated docker archives could have different hashes.
2018-03-13 13:46:47 +01:00
Antoine Eiche 346996ceec dockerTools: dereference hard links in tar archives
This is to improve image creation reproducibility. Since the nar
format doesn't support hard link, the tar stream of a layer can be
different if a dependency of a layer has been built locally or if it
has been fetched from a binary cache.

If the dependency has been build locally, it can contain hard links
which are encoded in the tar stream. If the dependency has been
fetched from a binary cache, the tar stream doesn't contain any hard
link. So even if the content is the same, tar streams are different.
2018-03-13 13:46:41 +01:00
Antoine Eiche e8f452f110 dockerTools: add an onTopOfPulledImage example
This allows to test if a pulled image can be updated by using our
Docker tools.
2018-03-13 11:59:22 +01:00
Antoine Eiche ce838e52b9 dockerTools.buildImage: do not add /nix/store in the tar stream
Since the /nix/store directory is not immutable, tar can fails if it
has to push it into the layer archive.

Fixes #34137.
2018-02-14 06:40:41 +01:00
John Ericson 888404f11b treewide: Fix deps in a few other fixed output derivations 2018-01-10 11:18:44 -05:00
Wei-Ming Yang 70e9b60b33
dockerTools.examples: correct a typo in comments
This commit is for correcting a typo in comments.
2018-01-01 16:13:40 +08:00
Ryan Trinkle ded1281f45
Merge branch 'master' into docker-dirlinks 2017-11-03 10:53:00 -04:00
Robin Gloster 20677fca59
dockerTools: fix hash to accomodate the pullImage revert 2017-09-28 14:09:49 +02:00
Robin Gloster 5c6dc717a6
Revert "dockerTools.pullImage: use skopeo to pull the image"
This reverts commit 01174c5f4d.

See https://github.com/NixOS/nixpkgs/pull/29302#issuecomment-332809092
for more information. This broke image format compatibility and
therefore amongst others mesos.
2017-09-28 14:09:49 +02:00
Robin Gloster dabb296c76
Revert "dockerTools.buildImage: Switch to the format image generated by Skopeo"
This reverts commit 35f205a4b6.

This does not use a standard format and by that breaks mesos
2017-09-28 13:01:34 +02:00
Antoine Eiche ff4d7f0fd2 dockerTools.examples.nix: set NIX_PAGER=cat environment variable 2017-09-25 09:39:15 +02:00