• On monday the 5th of november I installed a new firewall. I tried changing the MAC address linked to my fixed IP on their web management application. Of course (I tried this with customers before) this didn’t work.
• I called the help desk to notify them of the change of my MAC address. I’ve done this before for customers, usually it is fixed within 5 minutes. This time there seemed to be a problem. The help desk guy (very friendly) said that for some reason he could not change things in my account. I had been assigned a fixed ip address, but not the one I had before. He would escalate the problem to the appropriate department and call me back the next day (it was after business hours and that department was not available). I told him I needed my fixed ip address because of an array of VPN connections and I was not keen on reconfiguring all of them.
• Since they did not contact me the next 2 days, I called them again on Thursday. The help desk employee checked the status of my problem with the appropriate department while I held, but could not report any progress. All he could tell me was “They’re working on it”. My problem would be fixed today and he would call me back as soon as possible. So I agreed, hung up the phone and waited. But guess what: he did not call back.
• So Friday, 5 days after reporting the issue, I called them, again! Same procedure, checking with department, no news, will call back later today! This time he did call back, only to report that there was no news and he would check back in on monday.
• On monday I was at the TEDxBrussels event, so my phone was off most of the day. But my voicemail had no message from my internet provider. I was starting to get pissed. By now a full week has gone by, and not a hint of a solution. Even worse, I started noticing connection drops. The internet connection would go down anywhere from 1 to 5 times a day.
• I was out all day on tuesday so I called back on wednesday. The normal ritual occurred (check with department, no news, working on it). I pointed out that I have that business account for a reason. I pay extra to have a fixed ip and a help desk that can solve a problem within a reasonable time. Both of which were not being delivered. I also notified them of the connection drops, they said it was a problem with my firewall, I disagreed. They told me they could send a technician, but if he found no problem with their installation, I would have to pay for the costs.
• I started monitoring those connection drops, trying to prove it was not a problem with my installation. When I had collected enough compelling proof that it was not a problem with my installation, I called them again. Suddenly they were able to see the errors on their systems so they added that problem to my outstanding (10 days by now) ticket.
• Several days went by, repeating the same ritual over and over again. They promise to call me, they don’t call me, I call them, they have no news, I get pissed. Ultimatly friday evening I sent an email to the CEO of Telenet explaining the situation and asking him if he could get his helpdesk to fix the problems.

By now it is Sunday, November 18. Almost a full two weeks after I first reported the problem. I realize that, if they would have told me on day 1 that they could not fix the problem, I would have been pissed, but I would have lost less time. I could have reconfigured all of my VPN’s in less time then I would have spent on the phone with the help desk.

Update:

Apparently my mail the the CEO had effect. Today (monday) by 6 in the evening my problem was fixed.

Presenting SshGuard

Since Mac OS X has no solution for this out of the box, I created one. SshGuard is a script that monitors the log file /var/log/secure.log for ssh errors containing “Invalid user” or “authentication error”. When a host has too many failed authentications, his ip address gets blocked by the firewall.

SshGuard is written in ruby and distributed as a gem. Installation is simple:

1
2

sudo gem install ssh_guard
sudo ssh_guard -i #to install and load the launchd plist file

After this launchd will keep SshGuard running at all times. It will keep monitoring the log files for authentication errors and block hosts that are trying to guess passwords. Host blocking rules will be added with rule number 100 to keep it among the first rules to be applied.

At the moment it only works on Mac OS X. It’s only been tested on Mac OS X Server 10.6, but it should work on any Mac OS X system. It could probably be ported to other Unix system without much trouble. The source code is available here. Any bugs or feature request can be reported here.

Nov 24 21:15:53 mini sshd[55670]: error: PAM: authentication error for root from 200.106.67.47 via 192.168.1.2
Nov 24 21:15:54 mini sshd[55682]: in pam_sm_authenticate(): Failed to determine Kerberos principal name.
Nov 24 21:15:54 mini sshd[55670]: error: PAM: authentication error for root from 200.106.67.47 via 192.168.1.2
Nov 24 21:15:54 mini sshd[55687]: in pam_sm_authenticate(): Failed to determine Kerberos principal name.
Nov 24 21:15:54 mini sshd[55670]: error: PAM: authentication error for root from 200.106.67.47 via 192.168.1.2
...
Nov 24 21:15:56 mini sshd[55670]: Failed keyboard-interactive/pam for root from 200.106.67.47 port 4910 ssh2
Nov 24 21:16:09 mini sshd[55715]: reverse mapping checking getaddrinfo for client-200.106.67.47.speedy.net.pe [200.106.67.47] failed - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 15:30:35 mini sshd[92616]: Address 212.156.122.94 maps to static.turktelekom.com.tr, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 18:57:19 mini sshd[1595]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
...
Nov 25 18:57:20 mini sshd[1605]: Invalid user camilla from 81.181.90.106
Nov 25 18:57:20 mini sshd[1611]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 18:57:21 mini sshd[1615]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 18:57:21 mini sshd[1619]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 18:57:21 mini sshd[1619]: Invalid user vava from 81.181.90.106
Nov 25 18:57:22 mini sshd[1625]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 18:57:22 mini sshd[1625]: Invalid user vava from 81.181.90.106
Nov 25 18:57:23 mini sshd[1631]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
...
Nov 25 19:31:46 mini sshd[8201]: User postgres not allowed because shell /dev/null is not executable
Nov 25 19:31:46 mini sshd[8207]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 19:31:46 mini sshd[8207]: User postgres not allowed because shell /dev/null is not executable
Nov 25 19:31:47 mini sshd[8213]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Nov 25 19:31:47 mini sshd[8213]: User postgres not allowed because shell /dev/null is not executable
Nov 25 19:31:47 mini sshd[8219]: Address 81.181.90.106 maps to host2.openmind.is.ew.ro, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
...

These are just some snippets from the log (/var/log/secure.log).
If one of these IP addresses is you then get those trojans of your computer.

Counter measures

Thanks to well chosen passwords, none of my accounts seemed to be compromised. Nevertheless I disabled password login for ssh. This is how it’s done:

1. Make sure public key authentication works. If it doesn’t you will lock yourself out when disabling password authentication.
2. Open /etc/sshd_config with your favorite editor.
3. Uncomment the line where it says “PasswordAuthentication no”
4. Disable ssh’s PAM support by uncommenting the line “UsePAM no”
5. Restart sshd

Now ssh now longer accepts password authentication and hackers can try to guess passwords as long as they want.

NB: Since disabling password authentication the attacks continue. At least I’m confident there are no passwords to be guessed. I’m still searching for a way to dynamicly block these ip addresses with the firewall.

pdfbook --short-edge file.pdf

This creates a file name file-book.pdf. This file is ready for duplex printing and folding into a booklet afterwards.

1. It is very similar to bash what makes it very easy to learn. There is nothing to learn when you’re used to bash because everything works exactly like bash but you get a lot of extra’s.
2. It has really cool autocomplete functionality.
   • It can present suggestions in a menu that you can navigate in with your arrow keys
   • It can complete command options (typing ls -<tab> gives you possible actions and it’s meaning).
   • It can complete hosts from your hosts file when using network commands or ssh’s known_hosts file when using ssh, …
3. Typo correction
   How many times have you typed small errors like ssj remote_server instead of ssh remote_server? When that happens with zsh I get a question like zsh: correct 'ssj' to 'ssh' [nyae]? Very cool indeed.
4. Intelligent history
   • Shared history across sessions: How many times have you lost that super complex command from your history because you had more than one terminal window open? No more with zsh!
   • prefix filtered history: No more browsing through your complete history to find that complex command you can’t remember completely. Type the start of a command and then use your up arrow key to navigate through your filtered history containing only commands starting with the part you already typed.
5. Oh-my-zsh: This is an awesome extension for zsh. It features extra functions and auto-complete helpers, themes and a configurable plugin system. You can find it here. You can find my fork with my custom theme and a plugin for iTerm here. You should try it, it really makes switching to zsh worth it.

Create a StartupItem with one line:

ifconfig en0 1.2.3.4

This will force your host online, ignoring the fact that there is another host online with the same ip address. After this the failover host “sees” the host coming online and will release the ip address.

Usualy AFP refuses to start automatically after booting so I start it manually with the same script:

/usr/sbin/serveradmin start afp

An example StartupItem can be found here.

It requires you to hack around some of the security features of kerberos, so please do not try this if you do not have extensive experience with kerberos.

Step 1: Setting up DNS

The first thing to configure when setting up kerberos is always DNS. Usually when using round robin you configure a different A record for each host. Then you would setup 1 CNAME record pointing to those A records. But: Kerberos is very dependent a A records and PTR records for each host. So to satisfy both I created 2 A records, both with the same name, but a different ip address. Then I created a PTR record for each ip address, both pointing to the same name.

• Two A records for homes.example.com with 2 ip adresses 192.168.0.2 and 192.168.0.3
• PTR records for both ip addresses pointing to homes.example.com

Step 2: Setting up Kerberos

I tested this setup on Mac OS X Server 10.5. This means kerberos setup is a little different than on other platforms. The principle remains the same on other platforms.

• First configure both servers to use kerberos. On Mac OS X this is as simple as clicking a button.
• If you’re working with Mac OS X Server your kerberos should be working automatically. After binding your second server to the directory, the kerberos configuration on the first host will break. This is due to the way the keytab is created. Normally it is created by doing ktadd in kadmin. Mac OS X Server does this automatically for you. This causes the version number of the hosts principle to be incremented. This means the keytab of host 1 is out of date.
• Next, copy the necessary keytab entries using ktutil or copy the complete keytab file from host 2 back to host 1. Since you already have extensive knowledge of kerberos I assume you know how it’s done. For those of you who don’t:
  • to start ktutil on the commandline just open a terminal window and type

    ktutil

  • to read in the keytab of both hosts type

    rkt /path/to/keypath

  • list all the loaded entries and their slot numbers with

    list

  • delete all the unnecessary entries with (replace [slot with the correct slot number from the previous step])

    delent [slot]

  • write out the remaining entries to a new keytab file with

    wkt /path/to/new/keytab

• install the new keytab file on both servers

This should get you up and running. You should now have a load-balanced, kerberized server setup. I use it for afp home directories, but it should work for any service.

I only tested this with Mac OS X 10.5 Server, but the principles of kerberos are the same on every platform, so it should work on other platforms but your milage may vary.

Updated: a subversion repository has been set up. You can now install the plugin in your rucola application with the following command:

./script/plugin install https://svn.hcode.be/trunk/alt_activerecord_proxies

At first I tried fixing ActiveRecordProxy myself, but that didn’t work. So I decided to start from scratch. The result of can be downloaded here

What’s different from the normal ActiveRecordProxy class:

• support for save and reload.
• minimal support for validations. If you use saveWithError, an NSError object is returned when saving fails. The userDict of this error should contain the validation errors.
• rails like setterMethods: you can now use

  CustomerProxy.name = "test"

  instead of

  CustomerProxy.setValue_forKey("test", :name)

  (Updated: this already seems to work on normal ActiveRecordProxy objects)

It appears there is a record in the local DirectoryService that overrules the old configuration file. If you try to find it using dscl or Workgroup Manager you can find it at /Local/Default/config/Kerberos:EXAMPLE.COM. This record contains an XMLPlist object with the kerberos configuration. This is what the XMLPlist attribute should look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>KADM_List</key>
    <array>
      <string>kadm.example.com</string>
    </array>
    <key>KDC_List</key>
    <array>
      <string>kdc1.example.com</string>
      <string>kcc2.example.com</string>
    </array>
  </dict>
</plist>

Mac OS X Server has used this format before to distribute kerberos configuration to clients. On Tiger this record in the Open Directory server would be used to create the file /Library/Preferences/edu.mit.Kerberos. On Leopard this record is duplicated to the local directory and this record overrules the configuration in the edu.mit.Kerberos file. If you are changing your kerberos configuration by changing the edu.mit.Kerberos file and your changes do not seem to work, check your local directory for kerberos records. The easiest way to do this is by using Workgroup Manager. You can also edit the directory service record directly. It is stored in a text file at /var/db/dslocal/nodes/Default/config/Kerberos\:HCODE.BE.plist. The record itself is also a plist file so all the xml entities should be encoded. This is what the file should look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>XMLPlist</key>
  <array>
    <string>
      &lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;
      &lt;!DOCTYPE plist PUBLIC &quot;-//Apple//DTD PLIST 1.0//EN&quot; &quot;http://www.apple.com/DTDs/PropertyList-1.0.dtd&quot;&gt;
      &lt;plist version=&quot;1.0&quot;&gt;
        &lt;dict&gt;
          &lt;key&gt;KADM_List&lt;/key&gt;
          &lt;array&gt;
            &lt;string&gt;kadm.example.com&lt;/string&gt;
          &lt;/array&gt;
          &lt;key&gt;KDC_List&lt;/key&gt;
          &lt;array&gt;
            &lt;string&gt;kdc1.example.com&lt;/string&gt;
            &lt;string&gt;kcc2.example.com&lt;/string&gt;
          &lt;/array&gt;
        &lt;/dict&gt;
      &lt;/plist&gt;
    </string>
  </array>
  <key>name</key>
  <array>
    <string>Kerberos:EXAMPLE.COM</string>
  </array>
</dict>
</plist>