Benford’s Law says:

in lists of numbers from many (but not all) real-life sources of data, the leading digit is distributed in a specific, non-uniform way. According to this law, the first digit is 1 almost one third of the time, and larger digits occur as the leading digit with lower and lower frequency, to the point where 9 as a first digit occurs less than one time in twenty.

What this means is numbers starting with 1 are more likely to crop up in large datasets, than numbers starting with the other eight digits.

As I am a sceptical geek I had to test this out for myself.

Large sets of numbers, eh?

I could go out and find data from data.gov, or statistics from web-servers, but why leave the comfort of my own chair, when a simple bash-command would gather large numbers for me?

sudo ls -A -o -R / > benford.txt

That command runs through my entire harddrive, searching through each and every folder it finds and spitting out lines and lines of filenames and sizes into the aptly named file ‘benford.txt’. Actually it generated one million, thirty one thousand, three hundred and twenty two lines, equal to a whopping 68.2 MB of text and almost as large as an empty canvas in Illustrator :p

What now? The lines look like this: -rw-r--r-- 1 root 150 Jul 3 2009 InfoPlist.strings

I’m guessing, not being a unix professional, that there are ways to make the ls command only print the filesize and the name, and saving me the time of editing one million, thirty one thousand, three hundred and twenty two lines to only have lines like these 150 InfoPlist.strings

Here’s where regular expressions make their appearance. And the excellent RegExr Desktop is a great tool for it! Seeing as I’m no RegEx pro, either, I want to test my expressions before they run amok on my dataset :)

After testing my regular expressions in RegExr, I went into the great TextWrangler and started to find and replace. It went very well, as you can see :)

My filelist is now complete, and it’s also a tab-delimited textfile, which is great for filling into a [MySQL database][]. The great thing with having this data inside a database, is the ease of which you can pull data out, and I’m looking for the amount of lines starting with 1,2,3 etc. So I made a database-table with three columns: Id, filesize, name, and started to import my filelist into it. After all the find/replace in TextWrangler, the filesize had shrunk to 25.1 MB.

Now I can start datamining, to see if this all was for nothing, or if my computers files also adhered to Benford’s Law.

Squeal or Sequel?

I like to use Sequel Pro for all MySQL needs, and it’s open-source and free. It lets you manage the database easily in a GUI, and also makes it easy to type your own SQL-statements and run them against the server. Sequel Pro is a fork of the discontinued CocoaMySQL.

Running select filesize from benford where filesize rlike '^[1-9]' on the database yields ninehundred sixty thousand twohunded seventy three rows. This shows all lines and is equal to 100%. Now the calulations begin.

So what does the number s look like? It was fun to see that the numbers were distributed mostly like Benford’s Law dictated. Check it out. Benford percentage on the left, my numbers in the middle, and the total amount to the right :)

1. 30,10 / 35,14 / 337 445
2. 17,60 / 17,04 / 163 654
3. 12,50 / 12,03 / 115 551
4. 9,70 / 10,47 / 100 493
5. 7,90 / 7,29 / 70 044
6. 6,70 / 5,92 / 56 815
7. 5.80 / 4,70 / 45 114
8. 5,10 / 4,10 / 39 363
9. 4,60 / 3,31 / 31 795

Mathematics and statistics can be fun, especially when they are this close to /home :p

But I wanted the clients to be able to check out what I’m doing without uploading, so I had to find a way to enter the blog URL so it would work from outside too. If I put in localhost:portnumber, WordPress reroutes the visitor to that internal address which is not available from outside the LAN.

Enter DynDNS!

Dynamic DNS was the solution: So I entered myurl.dyndns.org:portnumber. It works like a charm, but since I’m inside a network without NAT-loopback, when I try to go to myurl.dyndns.org:portnumber, it won’t resolve for me. Another thing to solve.

Enter /etc/hosts!

If I modify my hosts file on my developer-machine, I can reroute all traffic to the dynamic dns URL, to the internal one, which is localhost:portnumber. So I added this to my /etc/hosts file:

//webdev locally on network without NAT-loopback
//10.0.0.2 is my dev-server
10.0.0.2 myurl.dyndns.org

Enter Terminal

To use these hosts-settings without a reboot, enter terminal on macosx and type sudo dscacheutil -flushcache

So this is how I’ll do it going forward. Any tips and ideas can be put in the comments below.

Mange filer, liten tid Når jeg holder på med et prosjekt er det endel mapper og filer som går igjen på nesten alle sites. Dette er en struktur på filene som jeg trives med.

Eksempler på dette kan være: .htaccess – for å kontrollere apache-serveren

‘ilegg/’ – mappe for php-includes

‘ilegg/start.php’ – som er en php-fil med diverse variabler som skal inn i alle dokumenter på siten

‘grafikk/bilder’ – mappe for bilder til siten

‘grafikk/logo’ – mappe for logoer til siten (forskjellige størrelser til ulike formål)

‘grafikk/css’ – mappe for bilder som blir brukt som bakgrunner ol. i css-filen

‘stilsett/’ – mappe for css

‘stilsett/skjerm.css’ – stilsett for skjermvisning

‘stilsett/print.css’ – stilsett for utskrift

‘js/’ – mappe for javascript

‘js/start.js’ – kildefil for eksterne javascript

‘_arbeidsfiler/’ – for arbeidsfiler (.psd, .ai og denslags).

‘index.php’ – indexfilen som også er en malfil for resten av siten. Denne filen inneholder også include-kommando og javascript-kommandoer som henter inn de to startfilene ovenfor, samt style-tagg med attributt som henviser til de ulike css filene

Dette er jo en hel haug med filer som skal opprettes og kan fort gå litt tid dersom du skal opprette alt dette og skrive all koden for å hente inn de ulike stilsett, php-filer og javascript-kodefiler som skal henge med. Heldigvis kan QS gjøre dette nesten automatisk for meg.

QuickSilver fikser. Opprett alle disse filene i en mappe du kaller f.eks. ‘ny web’. I denne mappen setter du opp alle filene slik du ønsker å ha de, dvs. at index-filen kan ha doctype som validerer den mot xhtml-strict og inkludere start.php/start.js og begge stilsettene.

Stilsettene kan du sette opp slik at alle marger og luft rundt alle objekter er fjernet * { margin:0;padding:0; } og denslags. Du skjønner tegningen..?

Deretter kopierer du hele denne mappen inn i ~/Library/Application Support/Quicksilver/Templates. Om ikke templates mappen er der, kan du opprette den.

Do or Do Not. There is no Try Deretter er det enkelt i Quicksilver å opprette et nytt webprosjekt i mappen Kunder/Kundenavn ved å gjøre følgende:

Åpne Quicksilver (ctrl-space som standard), skriv ‘Kunder. - QS gir deg da mappen kunder og du kan bla deg nedover til den kunden du skal opprette web for. - Deretter tab’er du til neste og skriver ‘Make New’ (eller bare ‘mn’). - QS vil da gi deg en liste over alle filer som ligger i mappen Templates vi lagde tidligere. Om du har flere maler i denne mappen, blar du deg nedover til du finner ‘Ny web’, og trykker ‘enter’.

Nå har QS opprettet mappen ‘untitled.’ i den mappen du valgte, og ‘untitled.’ inneholder alle filene som er i ‘ny web’. Merk: .htaccess er en skjult fil, ettersom den starter med ‘.’, men Quicksilver kopierer den likevel. I bildet over heter filen kun ‘htaccess’ uten ‘.’ fordi den skal vises på bildet.

Hva heter barnet..? Det som er flott nå er at QS åpner seg selv igjen, med ‘untitled.’ valgt som ikon og du kan enkelt tab’e bort til neste vindu i QS og skrive ‘rename’ og døpe om ‘untitled.’ til navnet på domenet du jobber med f.eks. 3djegrad.net.

Programmér Mappen 3djegrad.net er opprettet med alle filene du ønsket i mappen Kunder. Ferdig!

–Noen siste tanker– Utvidelser av denne funksjonen kan være: ‘ny web mysql’ – mappesett som inneholder php-filer med informasjon du trenger for å koble til databaser, og for innloggingssystem. ‘ny web html’ – mappesett som kun er et standard statisk nettsted med kun rene html filer ‘ny web xhtml strict’ – mappesett som kan være et utgangspunktet for et nettsted som validerer med korrekt Doctype mot XHTML 1.0 Strict ‘ny web xhtml trans’ – mappesett som kan være et utgangspunktet for et nettsted som validerer med korrekt Doctype mot XHTML 1.0 Transitional

Om filen du renamer til allerede finnes vil QuickSilver ikke gjøre noe, bortsett fra å gi deg en feilmelding neste gang du åpner QuickSilver. Men du skal jo ha såpass oversikt over hvilke sider du har laget tidligere, at du kaller evt nye versjoner av nettsder for v2 på slutten? :)

Jeg har også en egen QS Template som heter ‘ny kunde’, og den fungerer på akkurat samme måten som ovenfor, men lager en mappe i ‘Kunder’, som inneholder bl.a. mappene ‘_logoprofil/’ og ‘prosjekter/’, som brukes til å lagre logomateriell for ny kunde og en egen undermappe for prosjekter. Da kan jeg, om det kommer en ny kunde som skal ha nettside, vha. QS lage en mappe for denne nye kunden med egen mappe for logoer og prosjekter. Ettersom QS viser det siste elementet du brukte, om du ikke venter for lenge, kan jeg dermed åpe QS engang til øyeblikkelig og navigere til ‘prosjekter/’ og lage en ‘ny web’ mappe.

Kjapt og gale. Dette tar ikke engang et minutt, og du er sikret lik struktur på alle jobber, og ikke minst. det er superlett å arkivere slike strukturer når du skal ta backup.

Merk: .htaccess er en skjult fil, ettersom den starter med ‘.’, men Quicksilver kopierer den likevel.

Håper dette var til hjelp for noen. Det hjelper ihvertfall meg.

Lykke til!

• christian

Jakob Nielsen made an alertbox entry about password fields, recently.

The gist of what he said, is that it’s user-unfriendly to obscure the passwords, on websites, which presents a password field to their users. He proposes to just use cleartext and not bullets, in these fields, and also suggests a solution to the secrecy problem on public computers: Have an option to show and hide the password field, in case someone looks over your shoulder.

Jakob Nielsen propose that the password is cleartexted as default, but I, and others disagree. Chris Heilmann shows a way to make a show / hide link on all password fields, on a page.

It looks like an elegant script. Kudos to him, for sharing it.

What about password reminders? They only trigger on password-fields.
Password reminders, I believe, will remember passwords if input type=’password’, when you hit submit.
There should be a script that checks to see if the field is text or password, and turns it into password on submit, so that password reminders can remember the passwords even though you use the cleartext method.

Thanks to Chris Heilmann for making the script in the first place.


UPDATE: And btw: I tried my hand at an icon for show/hide password link. It’s of-course under Creative Commons.

Hide/show password icon. With a Creative Commons license.

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.

You, and I, read every privacy section on every beta we apply for, to feel secure that our email address won’t get in the Infernal Spam Database(TM). It pays to be diligent, and safeguarding your email address is a smart thing to do, and you should, but you can’t do it forever, you know.

Some day, on some website, you need to use that precious email address, so People actually can contact you, and People generally like to do so, since that is what the internet is all about. These same People also don’t like to fill out forms, they like to click links, especially mailto: links . This gives them a warm and fuzzy feeling, and also let’s them keep a copy in their Sent folder.

You and I have a strained relationship with mailto:links. So we have devised schemes (like mail-obfuscation) and voodoo-trickery to let People use mailto:links so long as they themselves use voodoo, in the form of JavaScript-enabled browsers. This leaves The Spammers in the dark. At least for now.

The Spammer Never Sleeps

Spammers don’t like to be in the dark, and so They strive for perfection in their spam-robots. Using all Their knowledge about searching text, and using a heap of regex The Spammers usually manage to suck out all the email addresses They want. According to the report from Centre for Democracy & Technology, published in 2003, e-mail obfuscation seems to work, but for how long? Probably only so long as it takes Them to make a regex converting obfuscated email addresses into correct email addresses. Considering 2009 – 2003 = 6 years They are probably on to Us.

The goal must be to hide that mailto-link and the email address with it, but still make it possible for the People to click links.

For those about to be spammed, we salute you

We have two requirements for our email-address:
1. It has to be easy to type for an author / contributor. No large codes!
2. We have to be able to use any email on whatever domain.

Seeing as every email-address under a domain is unique, we can simplify emails that belongs to the current domain to only the username.

I’ll be using the address ‘iluvspam@3djegrad.net’ for this article. On the demo-site, the link for this email-address will let People click it, open Their favourite email-app, send the email and continue on their Quest to find the End of the Internet.

To do this we’ll use Apache’s mod_rewrite capabilities, to rewrite URLs on the fly, and we will be able to write the email above as:

<a href="contact/iluvspam" rel="no-follow">iluvspam</a>

Open up your favourite text-editor (notepad, or textedit will do fine though I prefer the excellent and gratis TextWrangler from BareBones Software), and start a new file called ‘.htaccess’. Save this file in your root directory on the webserver.

If you have used mod_rewrite before, the already existing .htaccess will probably look something like this.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
</IfModule>

This enables the RewriteEngine in Apache and sets the RewriteBase to /, sort of like BASEHREF in HTML, but not exactly the same.

What we need to add is a RewriteRule which fires every time someone clicks the iluvspam link.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

#Email-harvest prevention for any domain
RewriteRule contact/([a-zA-Z0-9-_\.]+)/([a-zA-Z0-9-_\.]+) email.php?email=$1&domain=$2  

# Email-harvest for this domain
RewriteRule contact/([a-zA-Z0-9-_\.]+) email.php?email=$1

</IfModule>

Rewrite-email-to-the-what-now?

Before we start, grab a box of concentration and open your head.

All those characters jumbled together like an imploded anthill, are regex. As I said earlier, regex are regular expressions which lets Spammers work with text. We can also use them for Good.

• ^ (the caret sign) means ‘the start of the text’
• () parenthesis groups text together in a variable which can be used again.
• $ means the end of the string.
• [] groups together a string of letters to be handled later by another parameter, like ‘+’ or ‘*’.

As an example: - ^abc matches abcdef but not aabcdef, since the match doesn’t start with abc - abc$ matches aabc, but not abcd, since the string ‘abcd’ doesn’t end in abc - ^abc$ only matches abc, seeing as we tell the regex-engine to look for a string that starts with abc and ends with abc.

Back to our .htaccess-file:

# Email-harvest for this domain
RewriteRule contact/([a-zA-Z0-9-_\.]+) email.php?email=$1

The RewriteRule above tells Apache to forward all links which refer to ‘contact/(something)’ to the email.php script which parses the URL and returns the correct email to the (hopefully shiny and happy) People.

Phew! Take a deep breath before we dive back in … Ready? Let’s go!

The RewriteRule utilize regex for pattern recognition. The pattern it looks for is ‘contact/’ followed by any letter between a-z in either lower- or UPPERCASE and/or any digits from 0-9 and/or the characters ‘-’, ‘_’ and ‘.’.

This should cover all email-adresses. The plus sign at the end tells the Apache-server that we are looking for multiple characters. If you’re observant you might have noticed that the period-sign (.) has a backslash infront of it (.), and a bracket encloses the text inside the parentheses. The backslash is an escape-character in regex, and tells the regex-engine that it should allow the literal character ‘.’ (period), and not treat it like a wildcard, which is the default behavior for ‘.’ in Regular Expressions. The bracket groups text together to be worked on by parameters later, like the ‘+’

# Email-harvest for this domain
RewriteRule contact/([a-zA-Z0-9-_\.]+) email.php?email=$1

The $1 in /email.php?email=$1 tells the regex-engine to use the found-text variable we grouped with parenthesis ([a-zA-Z0-9-_.]+) and set the variable email to the found text, which in our case is iluvspam. The rewritten URL will then be: /email.php?email=iluvspam

So when People click ‘iluvspam’ they send the variable ‘iluvspam’ into email.php. email.php in turn is configured to append 3djegrad.net after all email variables, unless something else is defined.

As you’ve seen, we have two RewriteRules in our .htaccess-file and the other one let’s us type and send email to addresses on other domains, besides our own. That line looks like this.

#Email-harvest prevention for any domain
RewriteRule contact/([a-zA-Z0-9-_\.]+)/([a-zA-Z0-9-_\.]+) email.php?epost=$1&domene=$2

Using the knowledge you’ve gained above, you should be able to recognize that we now have two variables in the RewriteRule. The first variable is the email and the second is the domain. This RewriteRule let’s us write email-adresses to other domains like so:

Contact him via<a href="contact/chrleon/gmail.com" rel="no-follow">his email-adress</a>

Just one thing. The more advanced RewriteRule has to be defined before the simpler ones, or the regex-engine will recognize the simpler form before the more advanced one and fire, never activating the more advanced one. So in this case the .htaccess looks like this:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

#Email-harvest prevention for any domain
RewriteRule contact/([a-zA-Z0-9-_\.]+)/([a-zA-Z0-9-_\.]+) email.php?epost=$1&domene=$2 

# Email-harvest for this domain
RewriteRule contact/([a-zA-Z0-9-_\.]+) email.php?epost=$1
</IfModule>

You can learn more about Regular Expressions at regular-expressions.info and at weblogtoolscollection. Or do you need to read it again?

.htaccess files are extra configuration-files for the Apache web-server. To learn more about them, got to the Apache documentation website.

So, what does email.php look like?

The file is relatively simple. All it conceptually does is to get the user-name from the email/iluvspam and append the domain address. Try to also check the email-variable for non-allowed characters, or better yet do a white-list check where you only allow the characters a-zA-Z0-9.-_. That way you guard against injection-attacks. Any comments on the security issues are most welcome.

Here is the code for email.php, with comments.

//  set two variables to hold the generated email-adress
//  the $domain variable is used for email-addresses on THIS domain
//  if no domain is entered in the link like contact/abc123
//  then the email will be sent to 'abc123@3djegrad.net'
$recipient = " ";
$domain = "3djegrad.net";


//  The email-address is gathered from the GET-request
//  This GET-request is from the link you clicked
//  An if-statement checks to see that the email is set
if ($_GET['email']) {
$recipient = $_GET['email'];
}

//  The domain is gathered from the GET-request.
//  if it is set, we replace the $domain variable with the correct domain
if($_GET['domain']) {
$domain = $_GET['domain'];
}

This is the gist of it, but we still haven’t made any headway on opening uur Readers’ email-application. That’s the next step.

We open the Readers’ email.application with a header-change in php, like so:

header("Location: mailto:$recipient@$domain");

This will open the mail-app with the correct email-address, already filled in. But there’s just a blank page in the browser.

All the People who want to contact you, don’t like to see blank pages. They’re a tough crowd.

For this to work we must rely on voodoo-trickery, since PHP won’t let us send two header(location)-commands to the browser. If we try to send two header(location), only the last one fires. Even with the PHP function ob_start(). This is for security-reasons.

echo '<script type="text/javascript">history.back()</script>';

$explanation = file_get_contents("spamexplain.html");
echo '<noscript>';
echo $explanation;
echo '</noscript>';

The code above does just that. The first line tells the browser to go back one page. The following lines prepare a page for those Readers who for some reason have Javascript turned off or there is no Javascript available to them. This is our graceful fail.

But we’re not in Kansas yet, Dorothy.

Oi! What about my statistics!?

It seems that Firefox and Opera generates a hit when you use the back-button, and so does using the history.back() function. Safari is the only browser not doing this, as I can tell from my tests.

The solution for this is to set a $_SESSION-variable in the email.script and test for that variable on the article page. If the variable is set, then don’t load the counters. After that test is run, destroy the session-variable. Now the count won’t register.

For this example I just use a date and time-stamp in a textfile

if (!isset($_SESSION['url'])) { 
$filename = fopen("counter.txt", a);
//  This could be Google Analytics or others
//  like so: include('google-analytics-script.php');

//  write the date and time in the logfile, followed by a newline 
$dateandtime = date("d-m-Y - H:i:s",time());

fwrite($filename, $dateandtime . "\n");

fclose($filename); // close the file

unset($_SESSION['url']); 
//  unset the session-variable so that the counting still works on 
//  the rest of the page. Remember, we only want to drop the count
//  on this one page, after People have clicked their happy 
//  mailto:links.

    } 

Try it out

Demo-site: 3djegrad.net/moat

So there you have it. Hopefully spamfree, and humanreadable email-addresses on the internet.
The People will be glad they can click on mailto-links again. Wayhey!

Thanks for reading. Any comments? Submit them below.

Creative Commons photos by: http://www.flickr.com/photos/wheatfields/

Jeg har bestemt meg for å installere Subversion for å drive versjonskontroll på kode jeg skriver. Subversion er OpenSource Version Control System, som lar deg ta vare på alle versjoner av kode, og gjør at du kan dele prosjektet som et team og alle kan sjekke inn og ut kode. For min del er versjons-styringen det viktigste, ettersom jeg jobber alene på programmeringsprosjekter, og har foreløpig ikke så mye bruk for team-delen. Les mer på subversion.tigris.org.

Installasjonen av Subversion skjer gjennom MacPorts, et prosjekt som er laget for å installere og administrere fri kildekode under OS X.

Følg med fremover for mer informasjon om hvordan jeg opplever Subversion.

Installasjon av Subversion

subversion.tigris.org – her finner du Subversion
MacPorts – er programvare for å installere FLOSS (free software / opensource) under OS X

GUI for Subversion i OS X – Både fri og lukket

svnX – open source
Versions – ser unektelig sexy ut. €39 – demoversjon
Syncro SVN Client – $59 – årlig avgift

Hva er dine erfaringer med Subversion? Noen gode tips eller fallgruver jeg bør være obs på?

Problemer du kan støte på underveis:

1. ./apachectl -S gir en bus error
2. Kan ikke koble til localhost
3. Error i syntax

1. Bus error

Dette kan være mye. Det jeg har erfart er at det noen ganger mangler en innstilling i httpd.conf filen som sier hvilken fil serveren skal lete etter som startfil (index.htm/index.php/index.html)

2. kan ikke koble til localhost:

Dette betyr at serveren ikke klarer å koble seg til den adressen du har oppgitt. Bruk porter over 1024 i OS X. Jeg bruker fra 3000 og oppover. Du kan også på Mac gå til Verktøy > Nettverksverktøy og klikke på Port Scan for å se om noen programmer på maskinen bruker portene du velger. Da velger su 127.0.0.1 i feltet ‘Oppgi en Internett- eller IP-adresse for å søke etter åpne utganger’ og haker av ‘Test kun utganger mellom’ og skriver inn 3000, 3100 i de to feltene. Du vil da få opp alle programmer som bruker port 3000 – 3100. Mest sannsynlig ingen.

3. Error i syntax

Her er det viktig at filer faktisk finnes At du fører DocumentRoot-banen inn i anførselstegn. DocumentRoot “/Bane/til/fil” Du trenger ikke / etter siste mappenavn.

Husk at httpd.conf filen er case-sensitiv, dvs at den skiller mellom små og store bokstaver.

4. Siste tips

Legg alle Virtual Hosts i egen fil, eksempelvis conf/vhosts.conf, og legg inn en ekstra linje i httpd.conf som henter inn denne filen Include conf/vhosts.conf

Ikke bruk Unicode UTF-8 NO-BOM, då blir det noen feil med æøå. Bildet viser innstillingene i TextWrangler fra BareBones Software.

Her er det satt i TextWrangler