Quick look at Rosetta on Linux

Yesterday, Apple did release RosettaLinux as part of the macOS Ventura preview. Let’s take a first quick look…

Location

RosettaLinux is located at /Library/Apple/usr/libexec/oah/RosettaLinux. A rosetta ELF executable is present on that directory.

Does it do AoT?

No. Rosetta on Linux exclusively operates in JIT mode, at least for the time being.

Mapping to the VM

That directory is mapped through virtiofs to the virtual machine. binfmt_misc is then used to make execution attempts of x86_64 binaries run through Rosetta.

When the Rosetta directory is mapped to the virtual machine, all of the VM runs in Total Store Ordering mode. This provides the guarantees needed by Rosetta to provide x86-compatible memory model semantics.

If TSO mode is not present or not active on a given Arm platform, Rosetta explicitly does not provide those guarantees. It does not try to handle that use case. As such, to provide proper semantics on those environments, you can use taskset to bind applications using Rosetta to a single core.

Not-really-DSMOS*

Rosetta has a mechanism to restrict it to Virtualization.framework virtual machines. The ioctl is passed through virtiofs to the host.

A re-creation of that mechanism is pasted below:

  char* AppleRosettaKey = "Our hard work\nby these words guarded\nplease don't steal\n Apple Inc";

  int fd = openat(AT_FDCWD ,"/proc/self/exe", 0);

  if (fd < 0) {
    rosetta_error("Unable to open /proc/self/exe: %lu", fd);
  }

  char key[0x45];
  int ioctl_result = ioctl(fd, _IOC(_IOC_READ, 0x61, 0x22, 0x45), key);

  if (ioctl_result < 0 || memcmp(key,AppleRosettaKey,0x45) != 0) {
    rosetta_error(
                 "Rosetta is only intended to run on Apple Silicon with a macOS host \
                 using Virtualization.framework with Rosetta mode enabled"
                 );
  }

As we can see, the mechanism is quite trivial to bypass. However, please rely on another solution if you’re running on non-Apple hardware.

Running it on non-Apple CPUs?

Running it on a Mac is fun, but what if we try to run it on other processors?

Tegra Xavier testing

My first thought was the Tegra Xavier processor, which provides sequential consistency in hardware. As such, it satisfies the programmer-visible memory ordering constraints required for Rosetta to be functional.

However, Rosetta relies on FEAT_FlagM flag manipulation instructions. Those are not implemented on Carmel. As such, I skipped over to another potential testing target.

And the CNTFRQ_EL0 timer frequency of 31.25MHz on Xavier doesn’t match with what Rosetta expects either.

Graviton3 testing

Which Arm platforms are easily accessible with flag manipulation instructions? There are some, such as the AWS Graviton3 processor, which uses Arm’s Neoverse V1 core.

rosetta error: Unsupported counter frequency, check value of CNTFRQ_EL0
Trace/breakpoint trap

Rosetta supports CNTFRQ_EL0 timer frequencies of 24 MHz and 1.00GHz, which is unlike the 1.05 GHz value used by the Graviton3 platform.

Let’s override the value reported to Rosetta with 1.00 GHz. This is close enough for things to work fine for testing purposes.

This allows to make a Geekbench 5 run, showing ~70% of native performance for Rosetta 2 on a c7g.large dual-core instance. Anecdotally, the HDR subtest of Geekbench anomalously gives higher performance inside of the Rosetta environment than outside for this configuration.

However, a reminder that x86 memory ordering semantics aren’t provided is required to put that score into perspective.

* Don’t Steal Mac OS X, the system used to restrict x86_64 macOS from running on non-Apple hardware.

edit: fixing the buffer size for the sample DSMOS code.