the nix store may contain hardlinks: derivations may output them
directly, or users may be using store optimization which automatically
hardlinks identical files in the nix store.
The presence of these links are intended to be a 'transparent'
optimization. However, when creating a squashfs image, the image
will be different depending on whether hard links were present
on the filesystem, leading to reproducibility problems.
By passing '-no-hardlinks' to mksquashfs the files are stored
as duplicates in the squashfs image. Since squashfs has support
for duplicate files this does not lead to a larger image.
For more details see
https://github.com/NixOS/nixpkgs/issues/114331
This new patch is the patch between 4.4, and commit [1], a pull request
at [2] to upsteam Mac and BSD compatibility. That pull request is
itself a rebase of an older pull request for an earlier version of
squashfs-tools. I squashed all commits between 4.4 and [1], apart from
the BSD-specific ones, and exported the new patch from that. Attached
below is the git diff --ignore-space-change between squashfs-tools 4.4
with the previous patch applied, and with the new patch applied.
[1]: 7d31beec53e6245d3405d6ef2b96e9811ae07044
[2]: https://github.com/plougher/squashfs-tools/pull/69
---
diff --git a/squashfs-tools/info.c b/squashfs-tools/info.c
index 8efefe6..5c2f835 100644
--- a/squashfs-tools/info.c
+++ b/squashfs-tools/info.c
@@ -159,7 +159,7 @@ void *info_thrd(void *arg)
case EINTR:
continue;
default:
- BAD_ERROR("sigwaitfailed "
+ BAD_ERROR("sigwait failed "
"because %s\n", strerror(errno));
}
}
diff --git a/squashfs-tools/mksquashfs.c b/squashfs-tools/mksquashfs.c
index 67d86a5..3607448 100644
--- a/squashfs-tools/mksquashfs.c
+++ b/squashfs-tools/mksquashfs.c
@@ -35,7 +35,10 @@
#include <stddef.h>
#include <sys/types.h>
#include <sys/stat.h>
-#ifdef linux
+#ifndef linux
+#include <sys/sysctl.h>
+#else
+#include <sys/sysinfo.h>
#include <sys/sysmacros.h>
#endif
#include <fcntl.h>
@@ -52,7 +55,6 @@
#include <sys/wait.h>
#include <limits.h>
#include <ctype.h>
-#include <sys/sysinfo.h>
#ifndef FNM_EXTMATCH /* glibc extension */
#define FNM_EXTMATCH 0
@@ -5191,7 +5193,17 @@ int parse_mode(char *arg, mode_t *res)
int get_physical_memory()
{
+ /*
+ * Long longs are used here because with PAE, a 32-bit
+ * machine can have more than 4GB of physical memory
+ *
+ * sysconf(_SC_PHYS_PAGES) relies on /proc being mounted.
+ * If it fails use sysinfo, if that fails return 0
+ */
+ long long num_pages = sysconf(_SC_PHYS_PAGES);
+ long long page_size = sysconf(_SC_PAGESIZE);
int phys_mem;
+
#ifndef linux
#ifdef HW_MEMSIZE
#define SYSCTL_PHYSMEM HW_MEMSIZE
@@ -5221,16 +5233,6 @@ int get_physical_memory()
}
#undef SYSCTL_PHYSMEM
#else
- /*
- * Long longs are used here because with PAE, a 32-bit
- * machine can have more than 4GB of physical memory
- *
- * sysconf(_SC_PHYS_PAGES) relies on /proc being mounted.
- * If it fails use sysinfo, if that fails return 0
- */
- long long num_pages = sysconf(_SC_PHYS_PAGES);
- long long page_size = sysconf(_SC_PAGESIZE);
-
if(num_pages == -1 || page_size == -1) {
struct sysinfo sys;
int res = sysinfo(&sys);
diff --git a/squashfs-tools/mksquashfs.h b/squashfs-tools/mksquashfs.h
index 88d0b5c..1beefef 100644
--- a/squashfs-tools/mksquashfs.h
+++ b/squashfs-tools/mksquashfs.h
@@ -24,7 +24,6 @@
* mksquashfs.h
*
*/
-#include <pthread.h>
struct dir_info {
char *pathname;
diff --git a/squashfs-tools/pseudo.c b/squashfs-tools/pseudo.c
index f8fd529..48e6b27 100644
--- a/squashfs-tools/pseudo.c
+++ b/squashfs-tools/pseudo.c
@@ -30,7 +30,6 @@
#include <errno.h>
#include <string.h>
#include <stdlib.h>
-#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/stat.h>
diff --git a/squashfs-tools/unsquashfs.c b/squashfs-tools/unsquashfs.c
index 00615ce..c1a6183 100644
--- a/squashfs-tools/unsquashfs.c
+++ b/squashfs-tools/unsquashfs.c
@@ -38,7 +38,6 @@
#include <sys/sysinfo.h>
#include <sys/sysmacros.h>
#endif
-
#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>
@@ -1085,7 +1084,7 @@ int create_inode(char *pathname, struct inode *i)
break;
case SQUASHFS_SYMLINK_TYPE:
case SQUASHFS_LSYMLINK_TYPE: {
- struct timespec times[2] = {
+ struct timeval times[2] = {
{ i->time, 0 },
{ i->time, 0 }
};
@@ -1104,8 +1103,7 @@ int create_inode(char *pathname, struct inode *i)
goto failed;
}
- res = utimensat(AT_FDCWD, pathname, times,
- AT_SYMLINK_NOFOLLOW);
+ res = lutimes(pathname, times);
if(res == -1) {
EXIT_UNSQUASH_STRICT("create_inode: failed to set time on "
"%s, because %s\n", pathname,
I took the patch, and applied it on top of the previous squashfs-tools
commit that we packaged. It applied cleanly. Then I rebased that on top
of master, and resolved the conflicts. I'm not sure I resolved them
correctly though, I don't have access to Darwin. Somebody needs to
review this.
I took the patch, and applied it on top of the previous squashfs-tools
commit that we packaged (which required editing one line in the patch,
as it assumed to be applied on top of the reproducibility patches). Then
I rebased that on top of master, resolved one conflict, and I formatted
a new patch for this.
A new release has been made upstream. Reproducibility issues were fixed
in that release, so we no longer need those patches. For a full overview
of the changes, see the 4.4-specific readme at [1].
The alignment patch no longer applies cleanly; I disabled it for now,
and I will try to restore it in a follow-up commit.
[1]: 52eb4c279c/README-4.4
The 4k-align squashfs patch was broken in the update to squashfs 4.4dev,
such that the patch was no longer actually applied in full (command line
option not even parsed).
Without these patches, the output of mksquashfs is not reproducible.
The patches are taken from https://github.com/squashfskit/squashfskit,
a fork of squashfs-tools, licensed under the GPL2 or later like
squashfs-tools itself.
This prevents failures like "libgcc_s.so.1 must be installed for
pthread_cancel to work" that occur because Glibc assumes libgcc_s.so.1
to be in Glibc's libdir.
This solution is pretty hacky, because the libgcc_s.so.1 from
bootstrap-tools might be too old. So if we update GCC, programs might
end up using an outdated libgcc_s.so.1. Ideally, we would build
libgcc_s.so.1 *before* Glibc, which might not be impossible...
Fixes #3548.
to be in the buildInputs). While at it, update to
version 4.2. "attr" is no longer needed as a dependency.
svn path=/nixpkgs/branches/stdenv-updates/; revision=31744
