Thank you come again.

Mozilla have claimed that their 57.x beta of Firefox (aka 'Quantum') uses 30% less memory than Google Chrome / other webkit browsers.

Try ~10x more (6th column is RSS):

darth  72483   0.0  0.6   5363724  77964  ?? S  4:30 PM  0:0.82 /Applications/Opera.app/ ...

darth   68443  1.1  0.3  5186124   81456  ?? S 4:25 PM  0:00.43 /Applications/Google Chrome.app/...

darth  92101  0.0  4.2  7095508  708936  ?? S 4:57 PM  11:31.62  /Applications/Firefox Beta.app/...

(All browsers open on news.google.com)

Experiments with Manjaro and Arch Linux

Prelude

I discovered Juju and thought this was a brilliant way to deploy applications.  I'm not quite sure what Ubuntu are doing here because they also have snappy which has some similar traits except IMO even better as the core is kept simple.  What I wanted to aim for was a way to keep my important stuff configured and deployable from the main PC, which would also simplify backups.  One of those systems is a Raspberry Pi 2.

Manjaro, take 1

So I was wandering down a twisty path of system images for the RPi2 and came across Manjaro-ARM.  This led to the main PC version.  I'd never heard of it so don't have any of the layer 8 problems Arch users seem to have with the distro.

I grabbed the network installer ISO, booted up a VM, and started the install.  I ran with the partitioning defaults and booted up a working system.

I then went to pull updates and it whined about missing keys; Dr Google had to fix that one:

sudo pacman -Syy

sudo pacman -S manjaro-keyring

sudo pacman -Suu

Hooray! A working system!  Oh, wait, I left it with the VM defaults of an 8Gb disk, and ... oops... forgot to enable lvm in the partitioning!

Argh.  So I was going to have to reinstall, and in the mean time came across all the Manjaro love from Arch users (kinda like the minority of Debian users back in the day wrt. Ubuntu)

Arch Linux, take 0

So I pull the Arch network installer, and 9 hours later still can't get the @#%@#$%$%^$#@ thing to boot.

To cut a long story short(er);

• Arch has no setup tool.  Like the Slackware of old (1993), you have to manually use CLI tools to partition, create lvm phys/vg/lg, format, mkdir the core mountpoints, mount, then run the base install tool (pacstrap).  There is no excuse for a distribution to be pulling this nonsense on end users in the 21st century.  For a one-off install I'm never going to want to know the inner workings of lvm, luks, etc.  If I'm going to make 1000 installs I'm still only going to do it once then copy the working system 1000 times.  Don't show me a graphical console then refuse to use the GUI version of parted.  Just because I can drive the cli version doesn't mean I should be forced to.
• When you run pacstrap, despite having a 600+Mb ISO image of Arch at its disposal, it insists on pulling every package (223Mb for the base install) from the network.  This is awesome fun when you have to blow it away and do it again.  And again.  And again.  And again....
• VMWare Fusion doesn't use UEFI by default, but you can enable it by shutting down the VM, editing the .vmx file in a text editor, and adding "firmware=efi".  This only really mattered because I setup the partitions with GPT like all sensible people, then realised the VMWare guest defaulted to old BIOS.
• Because there is no setup tool, nothing is setup.  All the config files are defaults.  You will need to manually tell your bootloader to pull in all the required system modules to boot (e.g. lvm2).  You will need to tell it to tell the kernel your root fs lives off in lvm-land.  You will need to ensure your filesystems are compatible with whatever lives on them, e.g. if grub can't use f2fs then, well, you shouldn't either.  Nothing is going to stop you from making an unbootable system.  Yes, you can excuse that away with blame-the-monkey-at-the-wheel.  No, its not mentioned anywhere in the installation guide.
• Because it has to pull gobs of packages every install, it takes so long the console screensaver activates.  For some reason they chose to run with "garble the left third of the screen".  What, haven't heard of just blanking it?
• Arch has no versions.  This makes it utter nonsense for systems deployment, because by necessity pacman -Syu will just grab whatever is current, not whatever you built that worked last week.  If you are a hobbyist this is probably great (until it breaks, just like running Debian sid or similar).  If you need to do serious work, you should run a serious distro like Debian, Ubuntu, or $DEITY forbid, RHEL/Centos.  You know something is horrid when even RHEL gets it right over that thing.
• The undocumented pacstrap tool doens't like it if initrd / vmlinuz already exist in /boot (from the hundreds of other times you've tried to get Arch running).  You need to rm these files or it fails.  There is no "force overwrite" option (like mkfs has -f)
• pacstrap doesn't load packages in dependency order.  This means when it runs mkinitcpio to create the initrd, it will fail.  You'll need to be paying attention to the copious output and hope your scrollback buffer goes back far enough to capture it (ok on VMware but bad luck on raw metal) or you'll wonder why the system didn't boot.
• Well, seems that's not the only thing to fail.  Apparently it doesn't know where the heck it put init.  I'd love to scroll back and look but I've had to boot twice since that pacstrap run & of course there's no logs.  Its probably worth hooking up a serial console and having syslogd drop everything there.
• Oh, LOLLOCOPTERS.  Of course, it breaks standard Unix / POSIX compliance.  /bin and /sbin are symlinks to /usr/bin and /usr/sbin.  Guess what happens when you try to boot /sbin/init and /usr isn't mounted, as totally expected on a Unix system at boot time?  Fail, Arch Linux, epic fail.  The fix is to add yet more modules into mkinitcpio.conf (usr, fsck and shutdown).

Manjaro, take 2

Well, at least this one boots.  So I took the advanced partitioning option and setup lvm and reboot.  I took care to leave /boot with ext4 for maximum bootloader compatibility.  Everything else is either btrfs or (for its special case) /var is xfs

It whines about not being able to mount the root filesystem.

Darn, looks like you actually do have to edit the mkinitcpio.conf and enable lvm2 in the HOOKS (autodetect doesn't pick it up, despite the disk partition being clearly flagged lvm).  Also, for good measure, I forced it to load the dm_mod kernel module so there'd be no mistake what I'm after.

So I reboot (VM is still set to boot from the CD, whose boot loader offers a boot from hard drive option so its actually kinda great as a backup fixit tool).

mount /dev/mapper/VolGroup00-lv_root /mnt

[... and so on for all the other partitions, /usr, /var]

mount -t proc proc /mnt/proc

mount --rbind /sys /mnt/sys

mount --rbind /dev /mnt/dev

mount --rbind /run /mnt/run

chroot /mnt /bin/bash

Turns out not only is lvm2 not in the initrd, but none of the tools are installed.  Not for it, not for any of the filesystems, not even vim for crying out loud.  I have to "nano" (ugh the pain) the initrd config, but luckily pacman works so:

pacman -S xfsprogs btrfs-progs lvm2

Reboot, all's well.

I've taken a snapshot so next one I need to pull up is all ready to go, as pacman had 260Mb of packages to pull on update.

Splunk app for Stream for passive DNS monitoring

This is a killer app.

Forget everything I said in my last blog article.

Drop Splunk App for Stream into your system, put a stream forwarder on (or between) your DNS server, and look at the pretty pictures.

(snip of the full page, there's also tables on domains and so forth)

For added value, the app appears to be CIM compliant, so you can use e.g. PCAP Analyzer and Splunk DBIR App with the feed as well.

I'll dive into this some more later on with examples on using it for analytics, e.g. checking for zeus infection and so forth.

Passive DNS monitoring with dnsmasq, rsyslog and Splunk

Passive DNS monitoring with dnsmasq, rsyslog and Splunk

There's not too many articles about this out on the internet, so I thought I'd give it a go.  Configuration is based on a Raspberry Pi running Raspbian (Jessie) and Splunk 6.2.4.

What is passive DNS monitoring?

It's the practice of recording DNS query replies on your network.  It's a small bit of information which can help diagnose network issues, and of course identify potentially compromised hosts through either overt detection (think: x837fnv9ebfj38390nf78d7.evil-domain.ru) or baseline anomalies.

About the tools

• dnsmasq – a home network's best friend – caching DNS, DHCP & tftp server
  
• rsyslog – a system logging daemon with powerful features.  Default syslogd in Debian.
  
• splunk – commercial log analysis platform, free for up to 500Mb/day intake
  

dnsmasq setup

Just enable the log-queries option to your existing setup and you're good to go.
$ echo "log-queries" > /etc/dnsmasq.d/log-queries
$ service dnsmasq restart

rsyslog setup

Observe that dnsmasq logs DNS queries now into /var/log/daemon.log.  The two lines we're most interested in contain the texts "query" and "reply", so lets snarf them in rsyslog.  We'll add a little context to keep it real.

$ cat /etc/rsyslog.d/00-dnsmasq-splunk.conf
if $programname == 'dnsmasq' and ($msg contains ' query[' or $msg contains ' reply ') then @splunkserver:5141

Obviously substitute the IP of your real splunk index node for splunkserver above.  Why IP?  Because rsyslog doing a DNS lookup inserts a log entry in the DNS log which rsyslog parses, does a DNS lookup, which inserts an entry in the log which rsyslog parses, does a DNS lookup, which inserts an entry in the log which rsyslog parses, does a DNS lookup, which inserts ...

splunk setup

We'll be doing multiple things in Splunk, so lets take it one step at a time.

intake

Splunk now has to have a syslog listener on port 5141.  Navigate to Settings -> Data Inputs and add a new UDP listener.  (you could also go TCP, use @@splunkserver:5141 in rsyslog, and shake hands all around).  You'll accept messages from raspberrypi (restrict to host) and set the source type to linux_messages_syslog.  As per best practice log it into its own index (e.g. 'dns') with a source name override of 'dns' too.

You can pull up the search app now and test that you're seeing events matching e.g. "index=dns" &/or "source=dns"

Field Extractions

There's four main important data elements in the dnsmasq logs:

• dns_query_clientip - ^.+query\[.+\s(?P.+)$
• dns_query_hostname - ^(?:[^\]\n]*\]){2}\s+(?P[^ ]+)
• dns_reply_hostname - ^[^\]\n]*\]:\s+\w+\s+(?P[^ ]+)
• dns_reply_ip - ^.+reply.+\s(?P.+)$

You can easily make these in Splunk 6.2.4 by clicking on "Extract New Fields" in the field list on the left side.  You can cut/paste the above regexps in by choosing "I prefer to write the regular expression myself", or have fun like I did by poking around with the mouse then fixing it up later.

Data model

To make things easier, we can create a data model for the DNS information which automagically extracts the fields we like.  You'll probably like the date information and the four fields we created above.

dashboard

You can now create a custom dashboard to let you easily interrogate DNS traffic.  Start with a Pivot on the data model, and play around with the stats and charting as you wish.  Edit panels and add inputs for the time and the extracted data fields.  Create some visualisations.

Interesting info for analysis:

• TLD.  Create a search macro dns_tld defined as:  index=dns | rex field=dns_hostname "(?P[A-Za-z0-9]+\.[A-Za-z]+)$" | transaction dns_hostname startswith="query" 
• Top queried domains.  Try: `dns_tld` | top 25 domain
• Anomalies.  Try: `dns_tld`| anomalies field=dns_hostname | sort - unexpectedness

Don't use systemd

Upgraded to jessie.  On reboot;

* primary network interface is not configured.  This caused total network outage as this system is the DHCP server for the network.  Root cause = systemd.  It messed up the network config.

* Full GUI loads at boot.  RPi was set to be headless, but systemd pulled in dependencies requiring all of X11 (probably for the hideously awful systemd-ui app, the most user-unfriendly thing I've used.  And I've used Microsoft Windows.)

* systemd-logind is broken by default, there's no login consoles.

* load is constantly over 1.5 because of this:

top - 15:21:53 up 4 days,  2:12,  2 users,  load average: 1.58, 1.58, 1.50
Tasks: 127 total,   2 running, 125 sleeping,   0 stopped,   0 zombie
%Cpu(s): 10.4 us, 14.2 sy,  0.0 ni, 75.2 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem:    949408 total,   711404 used,   238004 free,    88224 buffers
KiB Swap:  1918972 total,        0 used,  1918972 free.   247176 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
    1 root      20   0    6416   4972   2748 R  77.6  0.5   4493:41 systemd

Hovers between 77 and 90% CPU utilisation.  There's absolutely NOTHING logged as to why.

* shutdown is broken because systemd refuses to shut the system down until one of its broken components responds - which it doesn't in 7 hours of waiting.  Only solution is hard poweroff.


Whoever decided this should be the default init in Debian needs to be taken out to pasture and shot thrice.

Idea of the day: Stop spam

Apparently 71% of email traffic globally is spam.  The usual MO is the spammer finds an open relay, does a cheap dump & run, and your YouTube video buffers while the networks deal with this crap.

So my idea is pretty simple; if that amount of email is spam, drop 3/4 emails.  A legitimate user getting a bounce message will try again.  A spammer has already long gone.  This reduces their business model to a 25% chance of getting caught in your spam filter regardless.

No doubt the Universe will build a bigger idiot, but if this is sufficient to starve them and force them to move on, job done!

No, I'm sorry, you've got the wrong number.

This is nine one... two.