blogg nr. 176 671 167

I’m neither a mathematician, or statistician, but these things fascinate me. For the remaining people still awake after that statement, I want to share the (insanely great) RadioLab podcast titled ‘Numbers’, where I first heard of a phenomenon called Benford’s Law ..and maybe a wallpaper, too.

Benford’s Law says:

in lists of numbers from many (but not all) real-life sources of data, the leading digit is distributed in a specific, non-uniform way. According to this law, the first digit is 1 almost one third of the time, and larger digits occur as the leading digit with lower and lower frequency, to the point where 9 as a first digit occurs less than one time in twenty.

What this means is numbers starting with 1 are more likely to crop up in large datasets, than numbers starting with the other eight digits.

As I am a sceptical geek I had to test this out for myself.

Large sets of numbers, eh?

I could go out and find data from data.gov, or statistics from web-servers, but why leave the comfort of my own chair, when a simple bash-command would gather large numbers for me?

sudo ls -A -o -R / > benford.txt

That command runs through my entire harddrive, searching through each and every folder it finds and spitting out lines and lines of filenames and sizes into the aptly named file ‘benford.txt’. Actually it generated one million, thirty one thousand, three hundred and twenty two lines, equal to a whopping 68.2 MB of text and almost as large as an empty canvas in Illustrator :p

What now? The lines look like this: -rw-r--r-- 1 root 150 Jul 3 2009 InfoPlist.strings

I’m guessing, not being a unix professional, that there are ways to make the ls command only print the filesize and the name, and saving me the time of editing one million, thirty one thousand, three hundred and twenty two lines to only have lines like these 150 InfoPlist.strings

Here’s where regular expressions make their appearance. And the excellent RegExr Desktop is a great tool for it! Seeing as I’m no RegEx pro, either, I want to test my expressions before they run amok on my dataset :)

After testing my regular expressions in RegExr, I went into the great TextWrangler and started to find and replace. It went very well, as you can see :)

My filelist is now complete, and it’s also a tab-delimited textfile, which is great for filling into a [MySQL database][]. The great thing with having this data inside a database, is the ease of which you can pull data out, and I’m looking for the amount of lines starting with 1,2,3 etc. So I made a database-table with three columns: Id, filesize, name, and started to import my filelist into it. After all the find/replace in TextWrangler, the filesize had shrunk to 25.1 MB.

Now I can start datamining, to see if this all was for nothing, or if my computers files also adhered to Benford’s Law.

Squeal or Sequel?

I like to use Sequel Pro for all MySQL needs, and it’s open-source and free. It lets you manage the database easily in a GUI, and also makes it easy to type your own SQL-statements and run them against the server. Sequel Pro is a fork of the discontinued CocoaMySQL.

Running select filesize from benford where filesize rlike '^[1-9]' on the database yields ninehundred sixty thousand twohunded seventy three rows. This shows all lines and is equal to 100%. Now the calulations begin.

So what does the number s look like? It was fun to see that the numbers were distributed mostly like Benford’s Law dictated. Check it out. Benford percentage on the left, my numbers in the middle, and the total amount to the right :)

1. 30,10 / 35,14 / 337 445
2. 17,60 / 17,04 / 163 654
3. 12,50 / 12,03 / 115 551
4. 9,70 / 10,47 / 100 493
5. 7,90 / 7,29 / 70 044
6. 6,70 / 5,92 / 56 815
7. 5.80 / 4,70 / 45 114
8. 5,10 / 4,10 / 39 363
9. 4,60 / 3,31 / 31 795

Mathematics and statistics can be fun, especially when they are this close to /home :p

I’ve been using WordPress for a while now, and I’m doing development on a local apacheserver. Since I’m used to not having NAT loopback on my network, I usually setup wordpress on the local dev-server and pass the blog URL as localhost:portnumber. When clients want to see progress, I upload to the www, and give them the link.

But I wanted the clients to be able to check out what I’m doing without uploading, so I had to find a way to enter the blog URL so it would work from outside too. If I put in localhost:portnumber, WordPress reroutes the visitor to that internal address which is not available from outside the LAN.

Enter DynDNS!

Dynamic DNS was the solution: So I entered myurl.dyndns.org:portnumber. It works like a charm, but since I’m inside a network without NAT-loopback, when I try to go to myurl.dyndns.org:portnumber, it won’t resolve for me. Another thing to solve.

Enter /etc/hosts!

If I modify my hosts file on my developer-machine, I can reroute all traffic to the dynamic dns URL, to the internal one, which is localhost:portnumber. So I added this to my /etc/hosts file:

//webdev locally on network without NAT-loopback
//10.0.0.2 is my dev-server
10.0.0.2 myurl.dyndns.org

Enter Terminal

To use these hosts-settings without a reboot, enter terminal on macosx and type sudo dscacheutil -flushcache

So this is how I’ll do it going forward. Any tips and ideas can be put in the comments below.