Experiments with Manjaro and Arch Linux

Prelude

I discovered Juju and thought this was a brilliant way to deploy applications.  I'm not quite sure what Ubuntu are doing here because they also have snappy which has some similar traits except IMO even better as the core is kept simple.  What I wanted to aim for was a way to keep my important stuff configured and deployable from the main PC, which would also simplify backups.  One of those systems is a Raspberry Pi 2.

Manjaro, take 1

So I was wandering down a twisty path of system images for the RPi2 and came across Manjaro-ARM.  This led to the main PC version.  I'd never heard of it so don't have any of the layer 8 problems Arch users seem to have with the distro.

I grabbed the network installer ISO, booted up a VM, and started the install.  I ran with the partitioning defaults and booted up a working system.

I then went to pull updates and it whined about missing keys; Dr Google had to fix that one:

sudo pacman -Syy

sudo pacman -S manjaro-keyring

sudo pacman -Suu

Hooray! A working system!  Oh, wait, I left it with the VM defaults of an 8Gb disk, and ... oops... forgot to enable lvm in the partitioning!

Argh.  So I was going to have to reinstall, and in the mean time came across all the Manjaro love from Arch users (kinda like the minority of Debian users back in the day wrt. Ubuntu)

Arch Linux, take 0

So I pull the Arch network installer, and 9 hours later still can't get the @#%@#$%$%^$#@ thing to boot.

To cut a long story short(er);

• Arch has no setup tool.  Like the Slackware of old (1993), you have to manually use CLI tools to partition, create lvm phys/vg/lg, format, mkdir the core mountpoints, mount, then run the base install tool (pacstrap).  There is no excuse for a distribution to be pulling this nonsense on end users in the 21st century.  For a one-off install I'm never going to want to know the inner workings of lvm, luks, etc.  If I'm going to make 1000 installs I'm still only going to do it once then copy the working system 1000 times.  Don't show me a graphical console then refuse to use the GUI version of parted.  Just because I can drive the cli version doesn't mean I should be forced to.
• When you run pacstrap, despite having a 600+Mb ISO image of Arch at its disposal, it insists on pulling every package (223Mb for the base install) from the network.  This is awesome fun when you have to blow it away and do it again.  And again.  And again.  And again....
• VMWare Fusion doesn't use UEFI by default, but you can enable it by shutting down the VM, editing the .vmx file in a text editor, and adding "firmware=efi".  This only really mattered because I setup the partitions with GPT like all sensible people, then realised the VMWare guest defaulted to old BIOS.
• Because there is no setup tool, nothing is setup.  All the config files are defaults.  You will need to manually tell your bootloader to pull in all the required system modules to boot (e.g. lvm2).  You will need to tell it to tell the kernel your root fs lives off in lvm-land.  You will need to ensure your filesystems are compatible with whatever lives on them, e.g. if grub can't use f2fs then, well, you shouldn't either.  Nothing is going to stop you from making an unbootable system.  Yes, you can excuse that away with blame-the-monkey-at-the-wheel.  No, its not mentioned anywhere in the installation guide.
• Because it has to pull gobs of packages every install, it takes so long the console screensaver activates.  For some reason they chose to run with "garble the left third of the screen".  What, haven't heard of just blanking it?
• Arch has no versions.  This makes it utter nonsense for systems deployment, because by necessity pacman -Syu will just grab whatever is current, not whatever you built that worked last week.  If you are a hobbyist this is probably great (until it breaks, just like running Debian sid or similar).  If you need to do serious work, you should run a serious distro like Debian, Ubuntu, or $DEITY forbid, RHEL/Centos.  You know something is horrid when even RHEL gets it right over that thing.
• The undocumented pacstrap tool doens't like it if initrd / vmlinuz already exist in /boot (from the hundreds of other times you've tried to get Arch running).  You need to rm these files or it fails.  There is no "force overwrite" option (like mkfs has -f)
• pacstrap doesn't load packages in dependency order.  This means when it runs mkinitcpio to create the initrd, it will fail.  You'll need to be paying attention to the copious output and hope your scrollback buffer goes back far enough to capture it (ok on VMware but bad luck on raw metal) or you'll wonder why the system didn't boot.
• Well, seems that's not the only thing to fail.  Apparently it doesn't know where the heck it put init.  I'd love to scroll back and look but I've had to boot twice since that pacstrap run & of course there's no logs.  Its probably worth hooking up a serial console and having syslogd drop everything there.
• Oh, LOLLOCOPTERS.  Of course, it breaks standard Unix / POSIX compliance.  /bin and /sbin are symlinks to /usr/bin and /usr/sbin.  Guess what happens when you try to boot /sbin/init and /usr isn't mounted, as totally expected on a Unix system at boot time?  Fail, Arch Linux, epic fail.  The fix is to add yet more modules into mkinitcpio.conf (usr, fsck and shutdown).

Manjaro, take 2

Well, at least this one boots.  So I took the advanced partitioning option and setup lvm and reboot.  I took care to leave /boot with ext4 for maximum bootloader compatibility.  Everything else is either btrfs or (for its special case) /var is xfs

It whines about not being able to mount the root filesystem.

Darn, looks like you actually do have to edit the mkinitcpio.conf and enable lvm2 in the HOOKS (autodetect doesn't pick it up, despite the disk partition being clearly flagged lvm).  Also, for good measure, I forced it to load the dm_mod kernel module so there'd be no mistake what I'm after.

So I reboot (VM is still set to boot from the CD, whose boot loader offers a boot from hard drive option so its actually kinda great as a backup fixit tool).

mount /dev/mapper/VolGroup00-lv_root /mnt

[... and so on for all the other partitions, /usr, /var]

mount -t proc proc /mnt/proc

mount --rbind /sys /mnt/sys

mount --rbind /dev /mnt/dev

mount --rbind /run /mnt/run

chroot /mnt /bin/bash

Turns out not only is lvm2 not in the initrd, but none of the tools are installed.  Not for it, not for any of the filesystems, not even vim for crying out loud.  I have to "nano" (ugh the pain) the initrd config, but luckily pacman works so:

pacman -S xfsprogs btrfs-progs lvm2

Reboot, all's well.

I've taken a snapshot so next one I need to pull up is all ready to go, as pacman had 260Mb of packages to pull on update.