As reported by @andir, the regular expressions that match the sandbox
output are no longer matching in the recent Chromium bump as of
bb03fbc2c8.
Instead of a boolean field that determines whether namespace sandboxes
are on, the namespace sandbox is now an enum within "Layer 1 Sandbox".
I've modified the regular expressions accordingly and also ran the test
for the stable branch, which now succeeds.
Signed-off-by: aszlig <aszlig@nix.build>
Issue: https://github.com/NixOS/nixpkgs/issues/49442
Cc: @bendlas, @andir
This is a very very very ugly workaround and it's because Chromium seems
to eat keystroke for a few seconds after a new window is created.
I haven't found a better solution yet, so let's at least unbreak the
test until we come up with a better way.
Thanks to @vcunat for bringing this to my attention and also doing the
initial bisect.
The change that brought up this problem was 2b29e40153,
which updated Chromium from version 65.0.3325.181 to version
66.0.3359.117. Unfortunately the upstream changelog[1] is way too large
to actually guess what the breaking change is.
[1]: https://chromium.googlesource.com/chromium/src/+log/65.0.3325.181..66.0.3359.117?pretty=fuller&n=10000
Signed-off-by: aszlig <aszlig@nix.build>
Cc: @bendlas, @vcunat
The tests have failed because Chromium has started up displaying the
following error message in a dialog window:
Chromium can not be run as root.
Please start Chromium as a normal user. If you need to run as root for
development, rerun with the --no-sandbox flag.
So let's run as user "alice" and pass all commands using the small
helper function "ru" (to keep it short, it's for "Run as User").
Tested it by running the "stable" test on x86_64-linux.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Reported-by: @globin
Sometimes it happens that the "Type to search or enter a URL to
navigate" popup doesn't show, but all we need to know at this time is
whether Chromium has finished starting up.
So checking for the "startup done" page is a better option here.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Assigning the channelMap by the function attrset argument at the
top-level of the test expression file may reference a different
architecture than we need for the tests.
So if we get the pkgs attribute by auto-calling, this will lead to test
failure because we have a different architecture for the test than for
the browser.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This has been the case before e45c211, but it turns out that it's very
useful to override the channel packages so we can run tests with
different Chromium build options.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This makes it easier to test just a specific channel rather than to
force testing all builds down the users/testers throat. Especially this
makes it easier to test NixOS channel upgrades only against the Chromium
stable channel instead of just removing the beta/dev channels from the
tests entirely (as done in 69ec09f38a).
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Generally we shouldn't ship pre-release versions anyway, and we
certainly don't want them to be release blockers. Also, chromium
builds are just too slow to have them blocking the channel (see
https://github.com/NixOS/nixpkgs/issues/12794).
We no longer need have "SUID sandbox" enabled in the chrome://sandbox
status page and we now also check for "You are adequately sandboxed." to
be absolutely sure that we're running with proper sandboxing.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This will make the test a lot more reliable, because we no longer need
to press ESC multiple times hoping that it will close the popup.
Unfortunately in order to run this test I needed to locally revert the
gyp update from a305e6855d.
With the old gyp version however the test runs fine and it's able to
properly detect the popup.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
It's not nice to send the escape key over and over again just to ensure
the popup is closed, because even *if* it fails to close the popup 4
times in a row it's just very unlikely that it will be closed. But in
order to make really sure, we might need to do a screenshot and detect
visual changes.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Since Chromium version 42, we have a new user namespaces sandbox in the
upstream project. It's more integrated so the chrome://sandbox page
reports it as "Namespace Sandbox" instead of SUID sandbox, which we were
re-using (or abusing?) in our patch.
So if either "SUID Sandbox" or "Namespace Sandbox" reports with "Yes",
it's fine on our side.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Chromium is quite memory hungry and we frequently get random crashes in
the tests, so let's set it to 1024 MB because new releases of Chromium
most probably won't consume *less* memory.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Of course, this could be done via packageOverrides, but this is more
explicit and makes it possible to run the tests with various Chromium
overrides.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Currently, the test is only for testing the user namespace sandbox and
even that isn't very representative, because we're running the tests as
root.
But apart from that, we should have functionality for opening/closing
windows and the main goal here is to get them as deterministic as
possible, because Chromium usually isn't very nice to chained xdotool
keystrokes.
And of course, the most important "test" we have here: We know at least
whether Chromium works _at_all_.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
