Since you’re a band member in our hypothetical mega-band, what this means to you in practical terms is that bloggers cannot put your videos in their blogs, among other things. Naturally, the music label wants to disable sharing, because they want fans to be dependent on the label to get band updates. In Internet terms, this is a de facto walled garden of content, which in the music labels’ ideal world would be something completely separate from the Internet.

Ideally, as a band member, you’ll get a cut of everything the Label sells, so there’s a lot to say for the walled garden concept. The big problem is this: by definition, sharing is disabled between the Internet and the walled garden. For as long as the Label was the best way to promote your music, there has never been any benefit to sharing content. Ever since broadcast radio music was used to promote albums, and even through the whole music video era, sharing has never played into the promotion scheme.

Maybe sharing should be a part of music promotion.

I’m not talking about ripping whole albums or bittorrent filesharing, per se, although there are some people would riff on this. They might go so far as to argue that if you give an artist some money as a result of downloading an entire album worth of mp3s, then that artist got some free promotion via filesharing. It’s happened before.

But I’m not even going to touch that. The situation I am talking about is when a music label on youtube clicks that one checkbox to “disable embedding.” In the picture below, these are some of the options youtube gives you.

It’s worth considering why someone would ever decide to disable sharing, but the inescapable observation is that many music labels have made this choice. A practical consequence is that they are missing out on potentially free promotion through the Internet. More on this point later.

This worked acceptably well, but I frequently had to ask myself: “which password did I use when I signed up for this service?” In response to having to guess my own passwords, I made two decisions: I would start writing my passwords down, and I would make them all unique and randomly generated. Four years later, I am using a totally different system, and I’ll explain all of my reasoning.

To facilitate my random password approach, I started using 3×5 index cards and a card filer. I added A-Z tabs, and I generally filed cards according to the domain name of the service (e.g. paypal.com is filed under P). I wrote a quick perl script to make 10 random passwords at a time, and I would pick one from the list and write it down on the index card. I really liked the concept of a purely non-digital password storage system, because it would be essentially unhackable without physical access. Essentially unhackable – more on this later.

There were several drawbacks to the index card system. For brevity, I’ll just list them:

• writing some characters by hand is ambiguous. I confused capital I, lowercase L, and numeral 1 all the time. Capital O and numeral 0 are also a trick.
• it’s possible to copy the password incorrectly
• it is extremely difficult to create a backup copy, so catastrophic loss is a possibility
• if someone has physical access to the index cards, they have access to your accounts
• it’s tedious to type in a random password every time you log in
• it doesn’t scale well after about 400 accounts

The scaling problems were the real killer. For example, did I file sandbox.paypal.com under P for paypal or S for sandbox? I don’t remember, so I need to perform a linear search through both letters. Or, since a disproportionate number of words start with S, then it became a more tedious task to flip through all the S cards in order to find an S site, whereas a site that started with Y would be pretty quick to look up since there were fewer. Eventually, it got to the point that I knew it was too much of a chore to look up cards, and on that basis, I became too lazy to log in to my accounts! Total failure.

The solution for me is to use Apple Keychain. If you’re a GTD adherent, then you’ll understand what I mean when I say this is my trusted system for account information. How did I reconcile a digital password storage with my original goal of keeping my passwords offline in order to make it unhackable? It was when I realized that both offline passwords and the keychain can be successfully attacked with a keystroke logger. If someone went to those lengths to get a password, then it wouldn’t matter how it was originally stored; the password could be intercepted regardless.

Why use Apple Keychain? Based on my list of drawbacks for the index cards, here’s a list of pro-Keychain points:

• built-in random password generator
• keyword search
• simple cut-and-paste workflow makes it very easy to enter passwords without typing
• keychain itself is password protected
• passwords are Triple DES encrypted (which should be acceptable until the year 2030)
• simple to back up keychain file
• slick integration with many applications, including Mail.app, subversion, and Safari/Chrome.

I’m currently at about 900 accounts (yes – this is deserving of a separate post unto itself) and the system is working great. I think this scales to meet my requirements, and probably beyond. In practical terms, a password that used to take 30 second to retrieve is now instant. I probably save 5 minutes per day by switching away from index cards, and I am avoiding untold frustrations. In all, I recommend Apple Keychain highly.   

In the Internet video world, there are two kinds of creatures: streaming video (which is appropriate for live events) and buffered video (which is for recorded things, like youtube). “Buffering” means it’s actually downloading a file in the background, and if it can download a little faster than you can watch it, then everything plays smoothly. If the video pauses suddenly and restarts after a few seconds, that’s because it’s rebuffering.

Have you ever noticed how the youtube progress bar slowly fills in with a pinkish color? It’s at about 20% in the picture, above. That indicates how much of the file has been buffered, and when it reaches the end, it means the file is fully downloaded. In other words, you don’t need some special plugin or service to “download a video from youtube.” Your browser does it automatically! Even better, this happens for any website that uses flash and .flv files to deliver buffered video.

The key is to use lsof (which is a mnemonic for “list open files”). I’m demonstrating this on OS X, but the process is basically the same with *nixes and Cygwin. If you don’t have lsof installed by default, just use your package manager to install it. (e.g. apt-get install lsof).

So, the magical incantation is:

lsof |grep lash

I grep for “lash” instead of “Flash” since you never know if the F will be capitalized or not, and this is the laziest way to get the desired results. Here is an example of the output:

Notice the files FlashTmp0 and FlashTmp1? That’s where the video files are saved, so long as you keep your browser window and video tabs open. There’s no need to “download” a video that you just watched. Instead, simply copy the file straight to your Desktop:

cp /private/var/folders/.../TemporaryItems/FlashTmp1 ~/Desktop/rickroll.flv

Now, you can open the local file with VLC:

You might need to try multiple FlashTmp files before you find the one containing the video you want (i.e. is it FlashTmp0 or FlashTmp1) but there usually aren’t many. On many non-youtube sites, this is the only way you’re going to get access to a buffered flash video (since there aren’t handy pwnyoutube.com clones for everything).

Once you have copied the file to your desktop, why not convert it to mp4 and edit it in iMovie?

ffmpeg -i rickroll.flv rickroll.mp4

And now you know how to access videos you just watched, as well as convert them into a format you can edit.

Configuration

For starters, I’m testing this out with OS X 10.5:

$ uname -a
Darwin osx.example.com 9.7.0 Darwin Kernel Version 9.7.0: Tue Mar 31 22:52:17
PDT 2009; root:xnu-1228.12.14~1/RELEASE_I386 i386 i386

For all of the output in this post, I used this version of curl:

$ curl --version
curl 7.19.6 (i386-apple-darwin9.7.0) libcurl/7.19.6 zlib/1.2.3
Protocols: tftp ftp telnet dict http file
Features: Largefile libz

Next up, we have the web server I was testing with:

$ curl -I osx.example.com:8000
HTTP/1.0 302 FOUND
Date: Fri, 18 Sep 2009 13:57:22 GMT
Server: WSGIServer/0.1 Python/2.6.2

I’ve been using wireshark and tcpdump to watch the traffic.  Here’s an example invocation of tcpdump that you can work with to replicate the issue:

sudo tcpdump -X -s 1500 -i lo0 tcp port 8000

Obviously, you might run a testing webserver on port 80, and you might send your traffic over lo, eth0, or en0.  If you’re reading this post, you probably know what’s what, and how to modify the command accordingly.

Issuing a simple multipart/form-data POST

It starts when I try to issue a multipart form POST:

curl -F name=somevalue http://osx.example.com:8000

This means “set the field called ‘name’ to ’somevalue’ and instead of url encoding it, post it as a multipart MIME message.”  Your browser does this any time you upload a file to a website.  In my case, my REST API lets me upload PDF files, so curl needs to use multipart instead of url encoding for this purpose.

Curl only generates one TCP packet based on this command (even though it should generate multiple) and this is what that one packet looks like:

10:07:00.971856 IP osx.57777 > osx.8000: P 1:260(259) ack 1 win 65535
<nop,nop,timestamp 1076257225 1076257225>
 0x0000:  4500 0137 03b9 4000 4006 0000 7f00 0001  E..7..@.@.......
 0x0010:  7f00 0001 e1b1 1f40 3f35 c1d8 0bb4 4ba7  .......@?5....K.
 0x0020:  8018 ffff ff2b 0000 0101 080a 4026 61c9  .....+......@&a.
 0x0030:  4026 61c9 504f 5354 202f 2048 5454 502f  @&a.POST./.HTTP/
 0x0040:  312e 310d 0a55 7365 722d 4167 656e 743a  1.1..User-Agent:
 0x0050:  2063 7572 6c2f 372e 3139 2e36 2028 6933  .curl/7.19.6.(i3
 0x0060:  3836 2d61 7070 6c65 2d64 6172 7769 6e39  86-apple-darwin9
 0x0070:  2e37 2e30 2920 6c69 6263 7572 6c2f 372e  .7.0).libcurl/7.
 0x0080:  3139 2e36 207a 6c69 622f 312e 322e 330d  19.6.zlib/1.2.3.
 0x0090:  0a48 6f73 743a 2031 3237 2e30 2e30 2e31  .Host:.127.0.0.1
 0x00a0:  3a38 3030 300d 0a41 6363 6570 743a 202a  :8000..Accept:.*
 0x00b0:  2f2a 0d0a 436f 6e74 656e 742d 4c65 6e67  /*..Content-Leng
 0x00c0:  7468 3a20 3134 380d 0a45 7870 6563 743a  th:.148..Expect:
 0x00d0:  2031 3030 2d63 6f6e 7469 6e75 650d 0a43  .100-continue..C
 0x00e0:  6f6e 7465 6e74 2d54 7970 653a 206d 756c  ontent-Type:.mul
 0x00f0:  7469 7061 7274 2f66 6f72 6d2d 6461 7461  tipart/form-data
 0x0100:  3b20 626f 756e 6461 7279 3d2d 2d2d 2d2d  ;.boundary=-----
 0x0110:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
 0x0120:  2d2d 2d2d 2d2d 2d37 3937 3037 6465 6438  -------79707ded8
 0x0130:  6339 640d 0a0d 0a                        c9d....

We can tell a few things from this.  First, the packet is 260 bytes long, the HTTP Content-Length header indicates a forthcoming payload of 148 bytes, and curl has set the HTTP Expect header to:

Expect: 100-continue

Notice that it gets all the way up to the MIME boundary, which is fine.  The server responds with an ACK of the 260 bytes received, then sends back an HTTP reply:

10:07:00.971880 IP osx.8000 > osx.57777: . ack 260 win 65535 <nop,nop,
timestamp 1076257225 1076257225>
 0x0000:  4500 0034 cda2 4000 4006 0000 7f00 0001  E..4..@.@.......
 0x0010:  7f00 0001 1f40 e1b1 0bb4 4ba7 3f35 c2db  .....@....K.?5..
 0x0020:  8010 ffff fe28 0000 0101 080a 4026 61c9  .....(......@&a.
 0x0030:  4026 61c9                                @&a.
10:07:00.973365 IP osx.8000 > osx.57777: P 1:21(20) ack 260 win 65535
<nop,nop,timestamp 1076257225 1076257225>
 0x0000:  4500 0048 1bbf 4000 4006 0000 7f00 0001  E..H..@.@.......
 0x0010:  7f00 0001 1f40 e1b1 0bb4 4ba7 3f35 c2db  .....@....K.?5..
 0x0020:  8018 ffff fe3c 0000 0101 080a 4026 61c9  .....<......@&a.
 0x0030:  4026 61c9 4854 5450 2f31 2e30 2033 3032  @&a.HTTP/1.0.302
 0x0040:  2046 4f55 4e44 0d0a                      .FOUND..

Ah!  The server didn’t respond with HTTP/1.1 100 CONTINUE.  Curl will wait until it receives a 100 CONTINUE before it sends its 148 byte payload.  If your server never sends that response, curl will never send the payload.  This can be a problem if your server or your application doesn’t know about this HTTP 1.1 behavior.

Here is the same communication, as seen from curl’s perspective with the -v flag (for verbose output):

$ curl -v -F name=somevalue http://127.0.0.1:8000
* About to connect() to 127.0.0.1 port 8000 (#0)
*   Trying 127.0.0.1... connected
* Connected to 127.0.0.1 (127.0.0.1) port 8000 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.19.6 (i386-apple-darwin9.7.0) libcurl/7.19.6 zlib/1.2.3
> Host: 127.0.0.1:8000
> Accept: */*
> Content-Length: 148
> Expect: 100-continue
> Content-Type: multipart/form-data; boundary=----------------------------d70cdce71857
>
* HTTP 1.0, assume close after body
< HTTP/1.0 302 FOUND
< Date: Fri, 18 Sep 2009 15:03:20 GMT
< Server: WSGIServer/0.1 Python/2.6.2
< Vary: Cookie
< Content-Type: text/html; charset=utf-8
< Location: http://127.0.0.1:8000/crm/form/login?next=/
<
* Closing connection #0

A brief note about HTTP 1.1

So, the fact that the server responds with a 302 FOUND instead of a 100 CONTINUE is really a feature of HTTP 1.1 because it is at this point that the conversation stops.  The 302 response will redirect the client to another resource on the server, so why not wait until you actually hit the resource that will receive your POST before you POST your payload?  If you’re POSTing a file to a URL redirect, then you will end up uploading your file at least twice, and that’s a waste of precious upstream bandwidth.

The HTTP/1.1 100 CONTINUE can potentially save you bandwidth, but read the coda at the end of this post for a cautionary tale.

Issuing a multipart/form-data POST without Expect

Let’s try this again without the Expect header:

curl -H "Expect:" -F name=somevalue http://osx.example.com:8000

The command above is identical to the previous one with the exception of the -H flag.  By setting “Expect:” to have no value after the colon, curl will interpret this as deleting the Expect header.  Sure enough, when we look at the TCP packet, the Expect header is gone:

10:11:59.308674 IP osx.57803 > osx.8000: P 1:238(237) ack 1 win 65535
<nop,nop,timestamp 1076260192 1076260192>
 0x0000:  4500 0121 c4b5 4000 4006 0000 7f00 0001  E..!..@.@.......
 0x0010:  7f00 0001 e1cb 1f40 5bb2 5099 2f1c 8bad  .......@[.P./...
 0x0020:  8018 ffff ff15 0000 0101 080a 4026 6d60  ............@&m`
 0x0030:  4026 6d60 504f 5354 202f 2048 5454 502f  @&m`POST./.HTTP/
 0x0040:  312e 310d 0a55 7365 722d 4167 656e 743a  1.1..User-Agent:
 0x0050:  2063 7572 6c2f 372e 3139 2e36 2028 6933  .curl/7.19.6.(i3
 0x0060:  3836 2d61 7070 6c65 2d64 6172 7769 6e39  86-apple-darwin9
 0x0070:  2e37 2e30 2920 6c69 6263 7572 6c2f 372e  .7.0).libcurl/7.
 0x0080:  3139 2e36 207a 6c69 622f 312e 322e 330d  19.6.zlib/1.2.3.
 0x0090:  0a48 6f73 743a 2031 3237 2e30 2e30 2e31  .Host:.127.0.0.1
 0x00a0:  3a38 3030 300d 0a41 6363 6570 743a 202a  :8000..Accept:.*
 0x00b0:  2f2a 0d0a 436f 6e74 656e 742d 4c65 6e67  /*..Content-Leng
 0x00c0:  7468 3a20 3134 380d 0a43 6f6e 7465 6e74  th:.148..Content
 0x00d0:  2d54 7970 653a 206d 756c 7469 7061 7274  -Type:.multipart
 0x00e0:  2f66 6f72 6d2d 6461 7461 3b20 626f 756e  /form-data;.boun
 0x00f0:  6461 7279 3d2d 2d2d 2d2d 2d2d 2d2d 2d2d  dary=-----------
 0x0100:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
 0x0110:  2d65 3932 3861 6430 3332 3262 340d 0a0d  -e928ad0322b4...
 0x0120:  0a

The packet is now 238 bytes long, which is exactly right.  “Expect: 100-continue” is 20 bytes long, plus 2 bytes for \r\n (0d0a in hex), which accounts for the packet being 22 bytes shorter than the previous 260 byte packet.   As before, the content length is 148 bytes.  As before, the packet goes all the way up to the MIME boundary.

As before, the server sends back a TCP ACK of the 238 bytes received, but here’s the difference.  Critically, without the Expect header, curl sends the entire payload before the server responds:

10:11:59.308693 IP osx.8000 > osx.57803: . ack 238 win 65535 <nop,nop,
timestamp 1076260192 1076260192>
 0x0000:  4500 0034 05cf 4000 4006 0000 7f00 0001  E..4..@.@.......
 0x0010:  7f00 0001 1f40 e1cb 2f1c 8bad 5bb2 5186  .....@../...[.Q.
 0x0020:  8010 ffff fe28 0000 0101 080a 4026 6d60  .....(......@&m`
 0x0030:  4026 6d60                                @&m`
10:11:59.308751 IP osx.57803 > osx.8000 P 238:386(148) ack 1 win 65535
<nop,nop,timestamp 1076260192 1076260192>
 0x0000:  4500 00c8 45bc 4000 4006 0000 7f00 0001  E...E.@.@.......
 0x0010:  7f00 0001 e1cb 1f40 5bb2 5186 2f1c 8bad  .......@[.Q./...
 0x0020:  8018 ffff febc 0000 0101 080a 4026 6d60  ............@&m`
 0x0030:  4026 6d60 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  @&m`------------
 0x0040:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
 0x0050:  2d2d 6539 3238 6164 3033 3232 6234 0d0a  --e928ad0322b4..
 0x0060:  436f 6e74 656e 742d 4469 7370 6f73 6974  Content-Disposit
 0x0070:  696f 6e3a 2066 6f72 6d2d 6461 7461 3b20  ion:.form-data;.
 0x0080:  6e61 6d65 3d22 6e61 6d65 220d 0a0d 0a73  name="name"....s
 0x0090:  6f6d 6576 616c 7565 0d0a 2d2d 2d2d 2d2d  omevalue..------
 0x00a0:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
 0x00b0:  2d2d 2d2d 2d2d 2d2d 6539 3238 6164 3033  --------e928ad03
 0x00c0:  3232 6234 2d2d 0d0a                      22b4--..

Whoah!  Did you notice the “name” … somevalue above?  That’s our payload, and it’s finally being transmitted.  This all happens before the server respond with an HTTP status code.  When sending the response, the TCP header is now an ACK to the total size of the POST-related packets (238 and 148, which is a total size of 386 bytes).

10:11:59.308768 IP osx.8000 > osx.57803: . ack 386 win 65535 <nop,nop,
timestamp 1076260192 1076260192>
 0x0000:  4500 0034 d082 4000 4006 0000 7f00 0001  E..4..@.@.......
 0x0010:  7f00 0001 1f40 e1cb 2f1c 8bad 5bb2 521a  .....@../...[.R.
 0x0020:  8010 ffff fe28 0000 0101 080a 4026 6d60  .....(......@&m`
 0x0030:  4026 6d60                                @&m`
10:11:59.316155 IP osx.8000 > osx.57803: P 1:21(20) ack 386 win 65535
<nop,nop,timestamp 1076260192 1076260192>
 0x0000:  4500 0048 e1c5 4000 4006 0000 7f00 0001  E..H..@.@.......
 0x0010:  7f00 0001 1f40 e1cb 2f1c 8bad 5bb2 521a  .....@../...[.R.
 0x0020:  8018 ffff fe3c 0000 0101 080a 4026 6d60  .....<......@&m`
 0x0030:  4026 6d60 4854 5450 2f31 2e30 2033 3032  @&m`HTTP/1.0.302
 0x0040:  2046 4f55 4e44 0d0a                      .FOUND..

So removing the Expect header allowed curl to send an HTTP 1.1 POST, with its payload, before the server generated HTTP 1.0 302 FOUND.  The fact that the server responded with 302 FOUND means the entire POST data was ignored on the server side, but the client DID send it!  In other words, we just wasted some bandwidth, and we are going to need to POST the data at least one more time.  When curl was expecting an HTTP 1.1 100 CONTINUE instead, it never sends the rest of the payload, and curl never complains, not even with -v.

Issuing a multipart/form-data POST using HTTP 1.0

I’ll spare you the complete packet dump, but suffice to say that when curl is invoked with the HTTP 1.0 flag (-0, as in “dash zero”), it works just like when the Expect header is absent.  In other words, the following example also sends the payload before waiting for the server to respond.

curl -0 -F name=somevalue http://osx.example.com:8000

This results in:

10:33:25.942611 IP osx.57900 > osx.8000 P 1:238(237) ack 1 win 65535
<nop,nop,timestamp 1076272988 1076272988>
 0x0000:  4500 0121 f225 4000 4006 0000 7f00 0001  E..!.%@.@.......
 0x0010:  7f00 0001 e22c 1f40 0fcb 74fc 46cf 148b  .....,.@..t.F...
 0x0020:  8018 ffff ff15 0000 0101 080a 4026 9f5c  ............@&.\
 0x0030:  4026 9f5c 504f 5354 202f 2048 5454 502f  @&.\POST./.HTTP/
 0x0040:  312e 300d 0a55 7365 722d 4167 656e 743a  1.0..

After a little conversation with the server, the payload is transmitted before the HTTP response, just like in the previous example.

Conclusion

What is the takeaway message from all of this?  If you’re using curl to test your REST interface, then make sure you are aware of the behavior HTTP 1.1 100 CONTINUE.  You might notice it because your server receives a blank POST payload.  Your HTML forms will appear to have not been filled in, even though you specified one or more -F arguments on the curl command line.

The solution for the versions of curl I’ve tested is to either remove the Expect header, or to tell curl to use HTTP 1.0 (since curl will default to 1.1 otherwise).  Once again, here are those examples:

curl -H "Expect:" -F name=somevalue http://osx.example.com:8000
curl -0 -F name=somevalue http://osx.example.com:8000

This forces curl to POST the payload without waiting for the 100 CONTINUE response, and it is suitable for servers that don’t know how to provide a 100 CONTINUE.  I hope this helps someone out there to avoid the trouble I had debugging my REST interface.

Coda: fix your server!

The “right” way to handle this situation is to make sure your server will send a 100 CONTINUE.  curl is smart enough, but your application might not be.

In a specific instance, my Django/Django-Piston/mod_wsgi application having trouble with the 100 CONTINUE when the client sets an Expect: 100-continue header. It turns out this was a problem with mod_wsgi 2.5 and the solution is to update to 3.0 (it’s RC4 as of this post). Until I realized this was the deeper issue, I was able to get around the problem by preemptively POSTing the entire payload.  This worked, but as I said before, it frequently resulted in duplicate uploads (like in 302 and 401 situations).

Before I started preemptively POSTing the whole payload, my application appeared to be receiving blank POST data, so it was responding with a 400 BAD REQUEST.   In truth, the POST was blank, so it technically was a bad request.  This is not curl’s fault, though – it was just being extremely polite, waiting for a proper handshake before sending the payload.

Just watch out, because curl might be too polite – it didn’t even tell me that my server was refusing the 100 CONTINUE handshake.  This masked a much deeper problem with mod_wsgi, which took several days to sort out.

Once upon a time, I took a class based on  a single question: “what is freedom?”  We meandered through US history, identifying several distinct stages in the evolution of the definition of “freedom.”  I was horrified to learn, during a discussion, that so many of my classmates wanted what I will call “freedom from information.” Ah yes – Professor Sandage had a way of bringing the ugliest truths to the surface, for all to witness.

On the one hand, I can understand this desire for freedom from information: telemarketing, advertising, spam, the scrolling headlines at the bottom of a newscast…  well, any unsolicited attempt at selling things you don’t care about.  On the other hand, I think we need more information instead of less, and we need effective tools to filter and manage that information so we only see what we care about.

The term “freedom” is muddied by historical contexts, but also through the process of etymological erosion.  With that said, I want to take a moment to discuss the expression, “free as in speech, not beer.”

Free as in speech, not beer

“Free as in speech, not beer” is an expression that comes up in open source discussions all the time.  It’s a little hard to unpack, unless you really dig into the dual meaning of the word “free.”  Thanks to Wikipedia, we’re part of the way there: the word “free” is used to mean two things: Gratis versus Libre.  We call both of these terms “free” nowadays, but once upon a time, there were different words because they are totally different concepts.  Gratis means “without charge” whereas Libre is more like “liberty” or “freedom.”

So what is free speech?  Of course, that’s the freedom to say what you want (so long as you accept the consequences for what you’ve said).  And free beer?  Well, that would mean beer that is provided at no cost.  I think the key is this: although you are free to say what you want, you could well end up in court for it (e.g. slander) and your expression won’t come free of charge.  On the flipside, you can provide beer free of charge, but not to someone who is 15 years old, so you may not freely provide beer to anyone you wish.

In other words, speech embodies Libre (but not necessarily Gratis) perfectly.  Likewise, beer embodies Gratis very well, at the same time that beer is so closely regulated by many governments that it is hardly “libre.”  Nevertheless, everybody likes a good party with some beer pro gratis.

The Free House, and the Public House

Speaking of free beer, the Free House is definitely not a place to find such a zero-cost beverage.  For starters, the term Free House is mostly British, and always beer-related.  It refers to a Public House (which you may know as a “pub”) that will sell any kind of beer they can get people to buy.  Contrast this with a Tied House, which sells beer manufactured by a single brewer, and you find that the Free House will have several brands on tap.  Here, the term “Free” is more like Libre, and is used in the context of the “free market.”  …and we all know that the free market isn’t composed of things that are zero-cost.

When I was living in Berkeley, California there were two particularly good “Tied House” pubs that brewed and sold only their own brands of beer: Jupiter and Triple Rock.  I should also mention Pyramid, which had a pretty cool restaurant with their own beverages on tap.  This kind of pub is fun because they’ll often have a sampler option to let you taste a small glass of everything they brew.  It’s a great way to experience the full spectrum of beers, but a word of advice: start with the lightest stuff and progress towards darker.  The one exception to this rule is for hoppy beverages (e.g. IPA or APA), which might be light but which may have a pronounced bitter taste.  You might want to close it off with an APA, even after drinking the stouts.

Open Source Software

There’s nothing that goes quite so well with open source software as a tasty hoppy beverage.  I like pairing Stone Brewing Company’s Arrogant Bastard with GnuPG, the open source implementation of Phil Zimmerman’s PGP (pretty good privacy) software.  Another favorite of mine is the Spaten Optimator paired with Wordpress.  More recently, I’ve taken a liking to Unibroue, the French Canadian brewer, who offers such brews as Tres Pistoles, which is an excellent complement to Python.  This last combination is probably the most dangerous of the group, because you might end up with excellent code, and you might end up with British comedy.

http://en.wikipedia.org/wiki/Monty_Python

In the end of the day, free speech and free beer have a lot to do with open source software.  You see, licenses such as the GNU General Public License actually permit developers to charge for their software, while simultaneously requiring all GPL software to be published with its source code.  In this sense, the “free beer” part means the software isn’t necessarily without cost, and the “free speech” part means you are required to publish the source code.  In other words, the Libre aspect of the GPL has an important restriction: you are not free to not publish the source code, which in turn provides the most fundamental tenet of open source software: you are free to read and distribute the source code.

I want to hedge my previous statement: the GPL is a famous topic of debate, so there’s plenty of room to criticize anyone who says anything – at all – about the GPL or about open source software, either according to the letter of the license, or according to the spirit of the movement.

Let me sum it up like this: “free” means many things to many people throughout many time-periods, but for some reason, it almost always comes down to a matter of speech and beer.

It worked…  mostly.  Partially by design, I chose to not migrate some command line tools, but now I find that every so often, I want to accomplish some task and I can’t … quite … do it, because I need to reinstall something, or perhaps reconfigure something.  I’d say 95% of the old functionality is still there, but the remaining 5% comes up often enough that it feels like something more than 5%.  The feeling is this lurking suspicion that I can’t trust my computer to do something that I know it used to be capable of, and it reminded me of a disease called Semantic Dementia. I don’t have semantic dementia in the sense of the neurological disease, but I’d like to start this off with a story about it.

Rumelhart and McClelland

Back in 2002, I was taking a class called something like “Cognitive Neuroscience” from Carl Olson and Jay McClelland.  I already knew  about McClelland’s work from a previous class on neural networks, which relied heavily on a textbook written by McClelland and his former colleague Rumelhart.  While the neuroscience class was fascinating, the most salient memory I have of the class involves a fairly personal reflection by McClelland on the current state of Rumelhart.

Although Rumelhart is relatively young (aged 67, as of 2009) he suffers from a form of semantic dementia called Pick’s Disease (or, if it’s not a form of semantic dementia, it’s related to it).  In any case, Pick’s Disease is a neuro-degenerative disorder.  The deep irony of this is that in the 1980s, Rumelhart and McClelland provided the first and possibly still-best description of neural networks, which could only have come from a profound insight into how the brain functions.

I can’t say much about what Rumelhart thinks of his disease, but I know McClelland thought about it, and if I’m not mistaken, this ultimately lead to a bridge between neural networks and an older model of semantic cognition called ISA networks.  As I recall it, it’s in terms of ISA networks that semantic dementia can be understood, which provides an intuitive way to describe how categories of items can become indistinct.

Consider: is a cat an animal?  Is a bird an animal?  Is a canary a bird?  Is a penguin a bird?  Is a penguin an animal?  It’s when we say “a penguin ISA bird,” and “a bird ISA animal” that we’re establishing categories, but lots of categories have weird and one-of-a-kind rules that lead us to say, “a penguin is a bird, even though it doesn’t fly” or “even though a cat is an animal that doesn’t fly, it is not a bird.”  Kids have to learn this through trial and error (e.g. a horse has four legs, but it isn’t a cat).

from http://www.uark.edu/misc/lampinen/sm.html

Semantic dementia is a condition where concepts lose their meaning, and a possible explanation for this comes from the inability to categorize concepts anymore.  This might have to do with making new distinctions between concepts, and it might relate to the retrieval of concepts that were previously distinct.  I think of semantic dementia as being like a car crash victim who is paralyzed and unable to communicate with the world, even though they are not cognitively impaired in any other way.  I imagine semantic dementia as being the frustrating condition of “knowing” what you want to say, but being unable to find the right words to say it.  Maybe it’s like having every word on the “tip of your tongue” and trying to construct sentences in spite of it.

Corrupted Greenthink

Earlier this year, I read a book by Vernor Vinge called Marooned in Realtime about life in a post-singularity enclave.  First off, I recommend Marooned in Realtime highly, but the book contains a cautionary tale about relying on an external system to store your personal database.  Whereas humans once kept their thoughts in their brains, and later turned to notebooks for assistance, in the MiR universe the pinnacle of personal database technology is called Greenthink.  While Greenthink has some similarities to 2009’s Wikipedia, I think it also includes personal notes, and presents a deeply personalized interface for interacting with these information objects.

Because the characters in MiR departed from Earth at different times over the span of about a century, and because technology continued to improve at an exponential rate, those who departed 10 years later benefited from vastly superior technology than those who were 10 years earlier.  One of the characters is so advanced as to be barely recognizable as a human, and this character has the deepest and most integrated relationship with Greenthink.

At one point, another character tries to use the futuristic Greenthink database, but since they were familiar with an older version of Greenthink, the interface is so foreign and personalized that they are unable to accomplish much of anything.  With some practice, they get up to speed, but the learning curve is steep.  During a fascinating future-combat scene, parts of Greenthink are corrupted, and the futuristic character struggles to retrieve information objects that were previously available.

Again, this reminds me of semantic dementia.  In the case of Greenthink it’s the retrieval cues that no longer lead to the right information objects; perhaps the interface doesn’t link correctly, perhaps the “menus” don’t have the right choices, or perhaps the choices are all there but the information objects aren’t there.  Whatever the case is, the character is frustrated and somewhat crippled by the inability to “think” the way they are used to, or to think at the speed they are accustomed to.

Although Vinge wrote Marooned in Realtime back in 1986, it is startling prescient.  As is the case with many series, I only found out too late that MiR is the third book in the Across Realtime trilogy, and I haven’t read the first two books yet.

Thinking in real-time

I can imagine some of the frustration of semantic dementia, or with aphasia (another language disorder), in terms of the amount of time it takes to accomplish something.  I definitely don’t think this provides any insight to the disorders themselves, but just consider the “tip of the tongue” phenomenon.  There’s a word that means something, and you know exactly what that thing is, but you just can’t think of the word.  If it weren’t that you are in the middle of a conversation, there would be no problem with sitting down to ponder the concept until you remember precisely the word you were looking for.  However, there’s another person waiting for you to finish your thought, and you are so compelled to express what it is that you have to say.  This results in a feeling of frustration, because it’s so irritatingly imprecise to not have access to the word you need, to the extent that it won’t do justice to the concept you are expressing.

In other words, we like to think and talk in real-time, but when something delays us, we get frustrated.  Likewise, I like to compute in realtime, and I strongly doubt I’m alone on this one.  Although I have many stories about my frustration with an early-1990s 286 computer (which I used until about 1996), I experienced the same problem in 2006 with my shiny new laptop.  It showed up with 512MB of RAM, which was not my intention.  I waited several months to upgrade to 1GB, but during this time-period, I experienced the profound frustration of swap disk hell.  Let me explain.

You might think of computers as a pyramid of buckets, where the fewest buckets at the top of the pyramid are the fastest to put things into and to get things out of.  As you go down the pyramid, you find that the buckets are both more numerous and also slower.  At the top of this pyramid is the storage that is physically located on the CPU; this is very small and extremely fast (called L1 Cache).  Depending on your CPU type, there might be two more levels of storage here (called L2 and L3), and just slightly slower is that type of storage called RAM.  Almost at the bottom of the pyramid is the hard drive, which is about 1000 times slower than RAM, but is correspondingly cheaper and much larger.

So there I was, using a totally modern machine that had way less RAM than it needed.  To run some software quickly might require lots of RAM, but if you don’t have that, modern operating systems elegantly “swap” data from RAM to the hard disk.  Recall that the hard drive is 1000 times slower than RAM, so swapping can be a very slow process.  However, this is much better than being restricted from running certain software because there isn’t enough RAM.

My typical usage pattern required vastly more RAM than I had available, such that I had to wait as much as 30 seconds to switch from my web browser to my word processor.  If I were leisurely wasting time, I might not mind this so much, but the fact of the matter is that I was working really hard at the time, and I didn’t have time for all of those 30-second delays.  As a result, I was frustrated, which is really an understatement.  I was able to think so much faster than my computer, and it felt like I was crippled somehow because I wasn’t free to think as fast as I wanted.  I call these 30-second delays “swap disk hell.”

For what it’s worth, I’ve maxed out my laptop at 2GB, and I’m quite happy, because I’ve caught back up with real-time.

The tip of the fingers phenomenon

On Thursday, I upgraded to OS X 10.5, and as I said, I migrated about 95% of my old functionality to the new system.  It’s that lingering 5% that comes up way more often than it should.  The 5% of the time that I try to do something (say, run a Python script I rely on) but I find that I can’t, it’s at once surprising (since I can’t predict when it will happen) and frustrating (since it will invariably delay me from accomplishing my task).

In some senses, this is like the tip of the tongue problem: I know what I want to do, and I could describe exactly what needs to happen, but I am looking for the command to actually accomplish my goal and the command just isn’t there.  It’s like a corrupted Greenthink database, and the frustration is just as bad as living in swap disk hell.  It’s like thinking and knowing everything you always used to know, and finding that it just doesn’t matter how it used to work, because it doesn’t work that way anymore.  This leads to suspicion and distrust, like I can’t totally rely on my ability to think anymore, because my thoughts don’t map onto actions the way they used to.

Semantic dementia can be a progressive disease, meaning it doesn’t necessarily happen all at once, and instead comes on in degrees.  Over time, I will rehabilitate my computing environment to be back at 100% – this is one sense in which my problem is totally unlike semantic dementia, because I can recover from reformatting my computer.  Really, there are many senses in which everything I’ve said is just a metaphor for semantic dementia; I don’t know what it’s like, and even what I do know about it is still quite different from what I am experiencing now.

Still, in everything that I do, I find that I hesitate slightly, since I am not certain I will accomplish the goal I have in mind as long as my computer is partially corrupt.  This is the “tip of the fingers phenomenon” … and the only cure is the slow and deliberate process of rehabilitation.

Network Topology: equal distrust for the Internet as for wireless

A series of events brought me to the point where this became realistic, the most important of which is that I got an extra router for the LAN.  Let me briefly explain the current network topology I use, which allows me to equally distrust public Internet traffic as much as I distrust my wireless router.

- We connect to the internet via DSL and a router, which uses NAT to provide a private address space behind the router (10.0.50.x)

- The “wired” router connects to the DSL+router, which uses NAT to create a separate private address space (10.0.51.x)

- The wireless router also connects to the DSL+router, and as you might have guessed, there is yet another private address space behind this router (10.0.52.x)

So, if you’re using an ethernet cable, your connection cannot be routed to a machine connected via wireless, and vise versa.  Barring an attack against the wired router, the address space is simply not routable.  I’ll eventually provide a VPN into the wired network, so I can print using a wireless connection (since the printer is only connected to the wired network).

At this point, I was pretty happy about running a wireless access point, because I was really no worse off if someone attacked the wired LAN via wireless or via the public Internet.  Basically, both vectors are equally untrusted.

Digging into WPA2

Still, I was uneasy about actually using my wireless network, and I hoped that wireless security had advanced beyond the famous WEP debacle, which made it downright trivial to attack older wireless access points.  The solution is to use WPA2, which is a better protocol that only runs on newer hardware.  This is not without its pitfalls, and some impressive work has been undertaken to attack WPA2.  Notably, the pyrit project has made great progress using 3d acceleration hardware to create downright feasible attacks against WPA2 with a pre-shared key (WPA2-PSK).

An alternative to using a pre-shared key with WPA2 is to use a key server technology called Radius, but because I didn’t wish to run another server, I needed to learn more about the pyrit approach so that I could still use WPA2-PSK.

The Pyrit Approach

Pyrit can make use of multiple 3d accelerator cards, and now can even cluster machines for parallel processing, in order to pre-calculate values that are useful in attacking a wireless network.  In other words, it is plausible for anyone with enough friends (or perhaps a government budget) to get the raw computing power required to crunch the numbers.  After saving these computed values to disk (a process that takes hours or days), they can be rapidly transmitted to the access point in a few minutes, and the attack will have been executed.

The key here comes down to disk storage, instead of processor power, because we might as well assume that processor power isn’t realistically limited anymore.  From the pyrit blog itself, it appears PSK values longer than 10 ASCII characters cannot affordably be stored on current hard drives, even though it is definitely possible to perform the necessary calculations.

The pyrit attack is further thwarted by the incorporation of the wireless access point’s SSID in the WPA2 calculations, so while it is possible to pre-calculate an attack for common SSIDs (like “linksys” or “default”) it is only possible to attack a novel SSID after some reconnaissance to determine that value of the target SSID.  Most impromptu pyrit attacks will probably involve common SSIDs that ship as the default setting for wireless access points.

Other considerations

There is also the issue of traffic over the air, where the question is to either use TKIP or AES.  This one is easy: there is a weakness in TKIP, so don’t use it.

If you know ahead of time which machines will exclusively use your access point, then MAC address filtering will be an extra security measure.  While MAC addresses can be spoofed, it takes extra time to do so and can be a hassle to brute force your way through the address space.  MAC address filtering is an option on my wireless router, so I have chosen to disallow all network access except for the few wireless devices that I know the MAC address of.

So you know, it can become a hassle to keep your MAC address whitelist up to date if you keep adding new wireless devices, like if you have friends who drop by with their laptops.  It’s probably worth the 60 seconds it takes to add a new device, but YMMV.

Recommendations

After all is said and done, it looks like it’s possible to create a relatively secure wireless access point.  Here are my recommendations:

- Use WPA2-PSK

- Use a PSK that is the maximum allowable length (probably around 63.)  Use a completely random method that includes all allowable ASCII characters (mixed case, numbers, and symbols).  Your wireless access point will probably call this a “password” or something, but just know that this is the “pre-shared key” (PSK).

- Encrypt all traffic with AES instead of TKIP

- Use a randomly generated SSID to name your access point

- tell your access point to NOT broadcast its SSID.  This will prevent it from showing up in the list of available access points when someone clicks on their wireless network card to scan.  This won’t deter the most determined attackers, but do this if it’s an option.

- Use MAC address filtering.  Disallow all by default, and whitelist the devices you want to explicitly allow.

This should be a pretty good starting point, and it works with my $32 wireless router.  There may be new attacks in the future, and hard drive space will obviously get cheaper, but I feel pretty comfortable at this precise moment.