2009/10/21

OCaml for the impatient - part 2, reading standard input

Now that "Hello, world" is out of the way, let's look at the next step in writing our log processing script. We want to be able to read lines from standard input.
OCaml has an "input_line" function, which takes a channel as a parameter. Standard input is available without doing any extra work as the channel 'stdin'. So, to read a line of text from standard input, we just need to call;

input_line stdin

In OCaml, parameters to functions are not enclosed with braces. There are plenty of places you do need to use braces, but surrounding parameters is not one of them.
To do something useful with our line of text, we'll need to assign it to a variable. OCaml uses the "let" keyword for that, but we'll also need to declare a scope for our variable, using "in". So, the code we want is something like this;

let line = input_line stdin in
... a block of code ...


To read all the lines of text from standard input, until we run out, we'll need a loop of some kind. OCaml does allow us to write code in an imperative style, so we can just use a while loop. While loops are pretty basic in OCaml (and in functional languages in general), because you're meant to do much cleverer things with recursion.
Our loop will need to terminate when we run out of lines to read. The simplest way to do that in OCaml is to catch the "End_of_file" exception. I'm not a big fan of using exceptions for normal control flow, but we can live with it for now.
So, a simple program to read lines from standard input and echo them to standard output might look like this;

try
while true do
let line = input_line stdin in
Printf.printf "%s\n" line
done;
None
with
End_of_file -> None
;;


There are a few points to note here. The semi-colon after "done" is necessary to tell OCaml that it should evaluate everything before the semi-colon first, and then evaluate the stuff after it. Without the semi-colon, you'll get a syntax error. It needs to be ";" and not ";;" because we're not terminating a block of code.
We're using "End_of_file -> None" to discard the exception we get when "input_line" tries to read a line that isn't there. "None" is a bit like "nil" in Ruby or "undef" in Perl.
The "None" at the end of the block is required to keep the return type consistent. OCaml, like Perl or Ruby, returns whatever is the last thing evaluated in the block. OCaml requires that the try block return the same type of value as we will return if we catch an exception and end up in the with block. If you try running the code without the "None" before with, you'll get an error saying "This expression has type 'a option but is here used with type unit" (OCaml error messages are translated from French, so they're a little idiosyncratic).
The type "unit" is the empty type, like void in Java. Our with block is returning "None", so it's return type is unit, and the try block must return the same type.
If we change the with to say;

End_of_file -> "whatever"

Then the error becomes "This expression has type string but is here used with type 'a option". So, we can make it go away by replacing the earlier None with any string constant (like "hello" - try it).
The last thing we're going to do is to take our inline "Printf.printf" statement and turn it into a function call, so that we can do something more interesting with line later.
In OCaml, functions are values we can assign to variables. So, to define a function, we use the same let statement as we used to define line. Here's a function to print out our line;

let out = Printf.printf "%s\n";;

Notice that we terminated the statement without specifying what is supposed to be printed. If you type the code above into the interactive ocaml interpreter, you get this;

# let out = Printf.printf "%s\n";;
val out : string -> unit =


That's saying "the value out is a function which takes a single string and doesn't return anything". OCaml decided we were defining a function because we didn't specify all the arguments. If we had, it would have simply evaluated it and assigned the result to 'out'.
Now, we can simplify our program a little;

let out = Printf.printf "%s\n";;

try
while true do
let line = input_line stdin in
out line
done;
None
with
End_of_file -> None
;;


Try running the program like this "ls | ocaml foo.ml", or by compiling it as shown in part 1.
So far, we haven't done anything very useful overall, but we've covered reading from standard input and writing to standard output, looping over all the available input, assigning each line to a variable and calling a function with that variable.
In part 3, we'll actually do something!

2009/10/17

OCaml for the impatient - part 1, "Hello, world!"

After an inspiring presentation by Tom Stuart at LRUG earlier this week, I decided to have a go at learning a functional programming language. I picked OCaml, partly on recommendation from Tom, although he's since changed his advice to recommend Clojure because of it's "smart and comprehensive treatment of mutable state".

I found quite a few introductions to OCaml, but many of them seem quite theoretical and a bit dry. Plus, I'm really impatient. I tend to learn best by just diving in and trying to perform a real-world task. So, I decided to start by writing a basic filter to take apache access log lines and spit out some fields in CSV form.


I know this is not necessarily the best task for OCaml, and I'm sure my coding style is missing the point and doing many things the 'wrong' way. But, at least it's a practical way to get my hands dirty with the language.

First of all, let's do the traditional "Hello, world!" exercise;

1. Installing OCaml

Lots of instructions for various platforms here;

http://wiki.cocan.org/getting_started_with_ocaml

I used the INRIA binary on Mac OS X.

2. Hello World

A basic hello world program in OCaml. Using a text editor (preferably vim, but I've heard that other programs sort of work), put the following into a file called "hello.ml"


Printf.printf "Hello, world!\n";;


Some points to note;

1. The 'printf' function comes from the 'Printf' module. This module is available to all programs, so there's no need to do anything special to gain access to it.

2. ";;" denotes the end of a chunk of code.

You can run this code interactively using the "ocaml toplevel"



$ ocaml
Objective Caml version 3.10.1

# Printf.printf "Hello, world!\n";; <--- type this and press enter Hello, world! - : unit = () # <-------- Ctrl-D to exit $


To run the program from the command-line there are several options.

Option 1


$ ocaml hello.ml


This is the easiest way, using the ocaml interpreter.

Option 2


$ ocamlc hello.ml -o hello
$ ./hello


This creates an executable "hello" file of OCaml bytecode, which should run on any machine with the OCaml bytecode interpreter, "ocamlrun" installed. It also creates a "hello.cmo" and "hello.cmi" file, which are OCaml object and interface files, respectively. This seems to be an intermediate step, since "hello" runs just fine if you delete them.

Option 3


$ ocamlopt hello.ml -o hello
$ ./hello


This time, "hello" is a compiled binary which can run standalone. There will also be the "hello.cmi" file, as well as "hello.cmx" (OCaml *native* object file), and "hello.o" (object file for your OS).

The interactive toplevel is great for noodling around, and option 1 is what I spend most of my time doing. Option 3 is what I will use if and when I write something that I want to use in production.

So, that's "Hello, world" out of the way. In the next part, we'll look at reading from standard input and writing to standard output.

OCaml for the impatient - part 2, reading standard input

2009/05/30

Quick DB check script without loading Rails

I use Zabbix to monitor my servers. It works by having a central zabbix server make calls to zabbix-agent processes on the monitored servers. The zabbix-agent can invoke a user-created script on the server and return whatever one-line output the script provides.

I needed a quick script to check the server to see if a set of database records exist, matching a given set of conditions. In this case, I'm checking to see if some log data representing yesterday's web traffic has been imported correctly, on the monitored server.

To start with, I used something like this;


#!/usr/bin/env ruby

ENV['RAILS_ENV'] = 'production'

require File.dirname(__FILE__) + '/../config/environment'

day = 1.day.ago.to_date
hostname = `hostname`.chomp

puts LogEntry.all(:conditions => {:day => day, :hostname => hostname}, :limit => 1).size


There could be a whole lot of log_entries records for yesterday, for this hostname. All I really want is to confirm that the importer script ran OK, so checking for the existence of one record that matches these criteria is sufficient and much faster than doing an aggregate query like a count.

This script outputs a '1' if everything is OK, and a '0' if not.

Hooking up a zabbix agent check to the script is easy enough, but whenever the server tries to call it, the check times out. The problem is that loading the entire Rails stack takes too long. Besides, it's overkill when all I want to do is run a single database query.

So, with help from here I changed it to this;


#!/usr/bin/env ruby

RAILS_ROOT = File.dirname(__FILE__) + '/..'
require 'rubygems'
require RAILS_ROOT + '/vendor/rails/activerecord/lib/active_record'
yaml_file = RAILS_ROOT + '/config/database.yml'
db = YAML::load(File.read(yaml_file))['production']
ActiveRecord::Base.establish_connection(db)

class LogEntry < ActiveRecord::Base; end

hostname = `hostname`.chomp
yesterday = 1.day.ago.to_date

puts LogEntry.all(:conditions => {:day => day, :hostname => hostname}, :limit => 1).size


If you've got Rails installed as a gem on your server, you don't need to jump through all the 'RAILS_ROOT' hoops. But, I vendor everything, so I need to tell ruby where to find active_record.

Now the script is fast enough to respond to zabbix checks. If I needed it to be even faster, I could use lower-level ruby database code. But, this is quick enough, and importing ActiveRecord gives us the "1.day.ago" stuff too, keeping the code clean.

2009/04/25

Setting up Ubuntu 9.04 for Ruby on Rails development

I've just installed Ubuntu 9.04 "Jaunty Jackalope", and so far I'm really impressed. It's slick and fast, with some really nice little touches (e.g. my laptop wakes up when I open the lid - I used to have to hit the power button to wake it up from suspend).

Here are the steps I went through to set it up for Rails development;

  • Launch a terminal and become root
  • aptitude install build-essential
This gives you a c compiler and a bunch of other stuff essential to allow you to build other packages (hence the name)
  • aptitude install git-core
I use git for version control. Of course, there are a bunch of subversion and other packages available too. In Ubuntu 9.04, the version of git is 1.6.0.4, which is good enough for me.
  • aptitude install phpunit php5-curl
I do some PHP development too. You can ignore this and the php mysql package later on, if you don't.
  • aptitude install ttf-mscorefonts-installer
This makes fonts look a lot nicer in firefox.
  • aptitude install rails vim-rails ruby1.8-dev
This will install rails 2.1.0. A later version would be nicer, but all my projects have their own version of rails in the vendor directory, so this is good enough.

I use gvim as my editor. You can skip the vim-rails if you don't.

The "ruby1.8-dev" package is important. Without this, you'll get failures loading "mkmf" whenever you try to install gems.
  • aptitude install mysql-server-5.0 php5-mysql mysql-client-5.0 libmysqlclient15-dev
Ubuntu 9.04 comes with mysql 5.1, but a lot of my projects are still using 5.0, so I want to stick to that for now.

"libmysqlclient15-dev" is required when we want to switch to a native mysql library for ruby. If you don't do this, you'll get deprecation warnings because the non-native, pure ruby library is going to be dropped soon.
  • gem install ruby-debug hpricot mongrel
So far, those have been the only gems I needed to install to get to the point where my specs would run.

Happy installing, and please add a comment if you've got any suggestions.

2009/04/06

Private web browsing via SOCKS proxy

So, Big Brother is now requiring ISPs to keep a log of all your emails, internet phone calls and web browsing activity. This is, of course, to keep us safe from terrorists. How we stay safe from Big Brother is less clear.

Anyway, with this in mind, here is a quick guide to using a SOCKS proxy to keep your web browsing private by routing all your activity via an encrypted ssh tunnel to a server elsewhere.

When you browse the internet normally, e.g. to www.google.com, connections go like this;

Your machine -> www.google.com

Your ISP provides the connection in between, so they can log everything for those lovely government people. When you use a SOCKS proxy, it works like this;

Your machine (1)-> Your proxy (2)-> www.google.com

Connection (1) still goes via your ISP, but it's only a connection from your machine to your proxy, so they can't log where you're browsing to (in this case, www.google.com). All they can see is that you made a connection from your machine to your proxy. The connection is via SSH, so they have no way of knowing what information is travelling up and down the pipe.

Connection (2) is from your proxy to www.google.com Whichever ISP connects your proxy to the Internet can log your browsing activity. This is why it's important to have your proxy in a country where they won't do that.

Let's get started. You will need;

  • A server, located somewhere with a less Stalinist government. I use a virtual server in the US, which is quite ironic, really. This machine needs to be running an SSH server.
  • You need to be able to ssh onto this server from the machine you want to browse from. I've tried this from Mac OSX and Linux machines, where it's easy enough. If you're unlucky enough to be using Windows, it's probably possible to do this via cygwin.

  • Optionally, the "SwitchProxy" add-on for Firefox makes it easy to switch from browsing via your SSH tunnel to browsing normally.

First of all, you need to have things set up so that you can access your remote server va SSH. i.e. you can start a terminal window and type "ssh yourserver" and you've got a terminal session on your remote server. If you can't do this, you're screwed, and Big Brother knows about all the nasty websites you visit.

Now, open up a terminal and type;

ssh -CND 9999 yourserver

"-N" tells SSH that we don't want to run any commands on the remote box (such as a login shell).

"-D 9999" says "listen on local port 9999 and forward whatever connections come in to that port, using the appropriate protocol, via the remote server. So, if you use local port 9999 to try to connect to www.google.com on port 80, you actually end up connecting to your remote server via SSH and then *from there* to port 80 on www.google.com

Currently, the -D option to SSH only supports running as a SOCKS proxy, but since that's all we need, that's fine.

"-C" compresses all data sent over the SSH tunnel. Browsing via a remote proxy server is a bit slower than browsing direct, so compression may help speed things up a bit.

You will need to have a terminal window open, running this command, whenever you browse the net. There are clever ways to make this automatic, but I prefer to keep things simple.

Now we just need to tell our web browser to connect via our new SOCKS proxy, instead of going straight to the target website.

In Firefox, use the Edit menu and select;

Edit -> Preferences -> Advanced -> Network -> Settings


Configure the settings shown. i.e. a SOCKS host on localhost port 9999. You can use more or less any port you like, but any port number below 1024 can only be forwarded by root, so it's easier to use a higher number. Whatever number you used in the "ssh -ND ..." command is the port you need to put here.

For any sites where you don't want to use the SOCKS proxy, add them to the "No proxy for" section. For example, if you use BBC iPlayer, it won't work via a US host, so you need to not use the proxy for bbc.co.uk

There are similar options for other browsers such as Safari and, if you really have to use it, Internet Explorer.

Once you've configured your proxy settings, you're good to go. Try browsing to a website and it should be going via your SSH tunnel. To confirm, try closing the terminal window and refreshing the page - you should see an error saying, "Proxy Server Refused Connection". Restart the tunnel and refresh again and it should work.

The SwitchProxy add-on for Firefox makes it really easy to switch proxy configurations - hence the name - via a new entry in your "Tools" menu.

Happy private web-browsing.

2009/03/28

Redundancy using Nginx for failover


I've got a website running on two servers, for redundancy, with an nginx load-balancer in front of them. If one server goes down, all requests will be sent to the other using this stanza in the nginx config;




upstream webservers {
server 1.2.3.4; # web1
server 2.3.4.5; # web2
}



This uses a round-robin system to send alternate requests to alternate servers. That's great for spreading the load around, but not exactly what I want.




The webservers are not heavily loaded, but they do run some quite complex reports against the database, the results of which are cached on the webserver's local filesystem. The cache is expired periodically.




Using a round-robin algorithm, we end up calculating each report twice per period - i.e. once on each webserver. What I really want is for all requests to go to web1, so that it's using its cache as efficiently as possible. web2 should only start receiving traffic if web1 goes down.




I don't know of a way to make nginx do exactly this, but you can use the 'weight' parameter to fake it, like this;




upstream webservers {
server 1.2.3.4 weight=100000; # web1
server 2.3.4.5; # web2
}



Now, only one in every 100,000 web requests will go to web2, so the cache on web1 is being utilised to a much greater extent. If web1 goes down, web2 gets all the traffic.




Redundancy plus efficient caching.

2009/03/08

Rewired State

I went to the Rewired State hack day yesterday.



A brilliant time was had by all. Photos here, video here.


Our hack was What's On Their Minds? (if that URL doesn't work, try this link).

2008/11/30

Converting iTunes playlists with ruby

Just posting this here in case anyone else has a similar setup.

I've got an xbox running XBMC hooked up to my home stereo system, but all my music is stored and organised in iTunes. The folder containing the music is shared so the xbox can see it, so all my music is available in XBMC with no problems. The one thing I was missing is all my iTunes playlists.

Along with virtually every other music player, XBMC uses a standard .m3u text file format to define playlists. I think the '#EXT...' stuff is specific to XBMC, but I'm not sure.

Needless to say, iTunes is not compatible with this at all. But, it is possible to export a playlist from iTunes, in its own bizarre XML format

One quick ruby script later, and I can convert the iTunes XML to M3U, including a remapping of file locations because iTunes has a different name for the root music folder (in fact, it has 2 different names for it because I moved the folder once, but the older playlists seem to still use the original location, somehow. Hence the alternatives in the ITUNES_ROOT regexp).

Finally, a noddy bash script to convert all the .xml playlist files in a directory to .m3u and transfer them to the xbox using ncftp

I still have to right-click on my playlists in iTunes and choose 'Export Song List' for each one, but that's not too arduous. I could probably write an AppleScript or something to do that for me, but in my case it's not worth the extra effort.

If you find any of this stuff useful, please drop a note in the comments.

Cheers

David

2008/11/12

Client collaboration via github



UPDATE: The post-receive URL on github has now been moved to a new "Services" tab on the project edit page.





It can be a hassle setting up and maintaining development environments
for clients; particularly if they're not very technical, or if they
use Windows.


I usually set up a separate VPS for each client's project as a
development/staging server, and it would be great if they could
just edit the source files on there. Then they could play around
and see the effect of their changes in realtime. The trouble is
that this requires some kind of remote editing or file-syncing setup
(e.g. SSH+vi), which can be troublesome to set up.


So, I'm trying out a combination of github's in-browser source
editing, plus a post-receive URL pointing to the client's development
VPS. Here's how it should work.


  • I set up the client's project as a github repository, and add the client as a collaborator.

  • The client makes changes to the view/css files using github's in-browser editor.

  • When they commit their changes, github posts to a listener on the client's development VPS (using a post-receive hook)

  • The listener on the VPS pulls the client's changes

  • The client views their development site to see the changes in (nearly) realtime




This is surprisingly easy to set up. I'll walk you through setting this up for a dummy "Hello, world" rails application. The ingredients we'll be using are;


  • github - a free account will do fine, for this demo

  • A development server - github will be posting to a URL on this server, so you need to be able to access it from the internet

  • A listener to pull changes from github, whenever there's a new commit. I'm going to use a very simple sinatra application, running via a command-line mongrel instance.

  • An application to act as our super-important client project




Setup a simple rails project as a github repository. You can just use a
bare "rails foobar" project - no need to set up databases.



Now, ssh to your development server (the one github will be posting to when your client makes changes). Setup a clone of the github repo.



For this demo, use the "public" clone url (the one that starts with "git://github.com").


You can use this technique for a private repo - otherwise it would be a bit pointless - but that takes a bit more SSH setup than I want to get into in this post.


I'm using "screen" so that I can start the demo application running
in a terminal and have it keep running after I log out. If you don't
know about the unix screen command, go learn about it right now.
After SSH it's one of the most useful tools to have in your toolbox.



screen
cd dstest
script/server
[Ctrl-A D to detach from screen]


So, now the application is running, and you can open a browser window to;



http://your.development.server:3000/


and see the "Welcome to Ruby on Rails" page.


Now for the listener which will pull our client's changes.


If you haven't already got them, you will need sinatra and mongrel on your development server;



sudo gem install sinatra
sudo gem install mongrel


Now, put the following into a file called "gitmon.rb"



require 'rubygems'
require 'sinatra'
post '/update_site' do
dir = '/home/david/dstest'
system "cd #{dir}; git pull"
"Site updated\n"
end


You will need to change the "dir" line to
be the full path to your rails project.


Now use screen again to start a listener that will continue to run after you log out;



screen
ruby gitmon.rb
[Ctrl-A D to detach from screen]


The listener will start up on Sinatra's
default port, 4567. You can test it by
visiting



http://your.development.server:4567


You should see "Sinatra doesn't know this diddy."


Now we just need to tell github to post to the listener whenever it gets new commits.


Login to your github account and go to
your repository's edit page.





Setup the "Post-receive URL" as;



http://your.development.server:4567/update_site





That should do it.


Now, edit a file (e.g. public/index.html) using the github in-browser editor (if you're not logged-in, you won't see the edit link). Commit your change.





Refresh the browser window you've got open to;



http://your.development.server:3000/


and you should see your changes (it can take a few seconds before your site updates).


This is just the tip of the iceberg, and there are a lot of ways this technique could be extended (e.g. hooks to run migrations after a "git pull" on the development server, or getting the sinatra listener to post to Campfire whenever a client commits something).


Also, there is a lot of information in the POST from github, which we're just ignoring here.


I hope you find this useful. I'll be setting it up for one of my clients very soon, so I'm keen to see how it works out.


2008/11/01

Setting up Ubuntu 8.10 for Ruby on Rails development

Here's how I've set up my laptop, running Ubuntu 8.10 (Intrepid Ibex) for Ruby on Rails development. You'll need to run the following commands from the command-line as root. These commands worked for me, but I take no responsibility if you trash your box by doing this ;)

There is some duplication here, because I sometimes setup servers with different sub-sets of the software I use on my development laptop. Don't worry about this. It does no harm if you tell apt-get or aptitude to install the same package twice - it will just ignore you the second time.

Not all of the gems here are required for RoR development, but I use cucumber for integration testing, and it has some pre-requisites.



# General/Pre-requisite packages
aptitude install \
build-essential \
screen \
subversion \
mysql-client \
telnet \
meld \
vim \
vim-gnome \
exuberant-ctags \
tk8.5 \
apache2-prefork-dev \
rcov

# mysql server
# use apt-get to avoid installing exim4
DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes \
mysql-server mysql-client \
libmysqlclient15-dev libmysql-ruby1.8


# git
mkdir gitcore
cd gitcore
wget http://kernel.org/pub/software/scm/git/git-1.6.0.3.tar.gz
apt-get build-dep git-core --assume-yes
tar xzvf git-1.6.0.3.tar.gz
cd git-1.6.0.3/
./configure
make
make install
cd

# Ruby
aptitude --assume-yes install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 \
irb1.8 libreadline-ruby1.8 libruby1.8 libopenssl-ruby
ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby;
ln -s /usr/bin/ri1.8 /usr/local/bin/ri;
ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc;
ln -s /usr/bin/irb1.8 /usr/local/bin/irb;
ln -s /usr/local/bin/ruby /usr/bin/ruby

# Rubygems. You REALLY don't want to let aptitude install rubygems.
wget http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz
tar xzf rubygems-1.3.1.tgz
cd rubygems-1.3.1
ruby setup.rb
ln -s /usr/bin/gem1.8 /usr/bin/gem
for g in rails rake capistrano capistrano-ext hpricot treetop ruby-debug term-ansicolor mongrel cheat passenger annotate-models rak; do gem install $g; done



Have fun.

David

2008/07/23

Finding files in vim

Some of my vim-challenged colleagues, when they try switching to vim, miss the Cmd-T function that lets them type parts of a filename and open the corresponding file.

After a quick search of the tips on vim.org, it turns out that someone has implemented exactly this function;

http://www.vim.org/tips/tip.php?tip_id=1432

This has the drawback that it will find blah.rb.svn-base when you look for blah.rb. So, I've tweaked it a little to skip any svn files, and also not to look in the 'vendor' directory, so it doesn't find rails source files.

http://gist.github.com/1625

Have fun

2008/07/22

Automatically "git rm" all deleted files

Love git. Hate having to "git rm" individual files whenever I delete a bunch of stuff (yes, I know I can "git rm foo/*", but what if I didn't remove *all* the files in foo/ ?).

So, a quick ruby script: gitrmall.rb

2008/07/15

Disabling tests in Shoulda

I'm using Shoulda on a client project, although I prefer rspec myself.

One of the things I miss from rspec is the ability to turn off a test, temporarily, by changing;


it "should do something"


...to;


xit "should do something"


Tests which are disabled in this way show up as "Example disabled: it should do something", when the specs are executed.

Using this with autotest and a couple of quick vim macros, my workflow goes something like this;

1. See that there is a failure in a particular test
2. Position the cursor in that test and hit ",ofs" this triggers my macro to disable all specs except the current one, and save the file so that autotest runs it again.
3. Hack, hack, hack to fix the bug, or develop the feature that the failing test relates to.
4. Hit ",ons" to turn all the specs back on and save the file again.

In Shoulda, renaming a test from;


should "do something"


...to;


xshould "do something"


Gives you a nice NoMethodError.

But, this is ruby, so we just add this to our test/test_helper.rb



class ActiveSupport::TestCase
def self.xshould(name, &block)
puts "disabled test: #{name}"
end
end



Now, we get pretty much the same behaviour as with 'xit' in rspec.

For any vim users in the audience, here are the macros (actually, abbreviations) I use to turn tests, specs and shouldas on and off. Just add to your .gvimrc (or .vimrc) file;


"rspec
map ,ofs :%s/ it / xit ''?xitx:w
map ,ons :%s/ xit / it /'':w

"shoulda
map ,ofh :%s/ should / xshould ''?xshouldx:w
map ,onh :%s/ xshould / should /'':w

"test/unit
map ,oft :%s/def test/def xtest''?xtestx:w
map ,ont :%s/def xtest/def test/'':w

2008/07/01

ActiveRecord commit timing

I'm building an application where we're using a message queue.

New objects add their IDs to a beanstalkd queue via an after_create callback. Some daemons monitor beanstalk, and grab objects to process, as soon as they hit the queue.

We were getting errors from the daemons, saying "record with ID 'N' not found", but when we looked for object N via script/console, or the database, there it was.

It turns out that the after_create callback (along with all the other object creation callbacks) occurs *inside* a transaction. So, this is what was happening;



There are a couple of ways around this;

Call "self.class.connection.commit_db_transaction" in the model, after the after_create method. This works, but it smells really bad.

A nicer way is to add an "after_commit" callback, like this

Pat Allan has some modifications to this, to make it play nicer with Rails 2.0 and 2.1

Or, there is a version here which handles nested transactions.

Many thanks to Pat for showing me the after_commit stuff.

2008/06/07

Using non-standard primary keys with ActiveRecord in Ruby on Rails

A Rails application I'm currently working on has a 'User' model. For the purposes of this app. a 'user' is uniquely identified by their mobile phone number - a user must have a mobile phone number, and may only have a single one. A second mobile phone number must be a second user.

I'm going to need the mobile phone number in other models - e.g. "User has_many :messages", and I'd like to have a "mobile_phone_number" attribute in my Message model, so that I can say "message.mobile_phone_number" without having to join the users table all the time, like this; "message.user.mobile_phone_number".

I can get what I want if the mobile_phone_number is the primary key of the User model. Let's try it.

Here is the migration (this is specific to mysql);



class CreateUsers < ActiveRecord::Migration
def self.up
create_table :users, :id => false do |t|
t.integer :mobile_phone_number
end
execute "alter table users modify column mobile_phone_number bigint unsigned primary key"
end

def self.down
drop_table :users
end
end



(The alter table statement is a bit of a hack. Another way to achieve the same result would be to use the mysql_bigint gem)

Here is the model;



class User < ActiveRecord::Base
set_primary_key :mobile_phone_number
end



Looks OK, but this is what happens when you try to create a User;



>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 0>



Where did that '0' come from, and what happened to the mobile_phone_number?

A quick look in the logs shows this;



WARNING: Can't mass-assign these protected attributes: mobile_phone_number
SQL (0.000146) BEGIN
User Create (0.000222) INSERT INTO `users` VALUES(DEFAULT)
SQL (0.000422) COMMIT



So, Rails is not sending the primary key column value in the SQL create statement. This would make sense if, as usual, our primary key was an autoincrement column. The database would supply that value for us, so we don't want to send it. But, in our case, we want to set the primary key ourselves, so this behaviour is wrong. (The 0 comes from the default value of the column)

You can try using 'attr_accessible' to allow you to assign to the mobile_phone_number, but it doesn't work (presumably because it's the primary key that we're trying to assign to).

We need some way to override the standard Rails primary key behaviour so it does what we want. That sounds a lot like the composite_primary_keys gem.

So, after installing the gem (in my case, after installing Rick Olson's 'gems' plugin so that I can put the composite_primary_keys gem in my project's vendor directory), here is the updated model;



require 'composite_primary_keys'
class User < ActiveRecord::Base
set_primary_keys :mobile_phone_number
end



Note that 'set_primary_key' has changed to 'set_primary_keys'.
Now, we get this;



>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 447123456789>



Much better. Even though we're not using a composite_primary_key, the gem is overriding the default Rails behaviour so that the key we want is being set by the create statement.

If you know a more elegant way to do this, please let me know.

2008/04/26

Gotcha: composite_primary_keys gem

Dr. Nic's composite_primary_keys gem is incredibly helpful if you're writing rails code against a legacy database, or in any other situation where you can't or won't follow the rails convention of having a single field as your model's primary key.
After doing the usual;

sudo gem install composite_primary_keys
...and requiring it in you environment.rb file, you can define a model like this;
class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
...
end
Brilliant.
But, there is one subtlety you need to be aware of. If your model mixes in any modules, you might end up writing something like this;
class Membership < ActiveRecord::Base
include MyAwesomeModule
set_primary_keys :user_id, :group_id
...
end
Looks fine, doesn't it? Unfortunately, your tests will now break with lots of errors like this;
ActiveRecord::StatementInvalid in 'Membership should foobar'
Mysql::Error: Column count doesn't match value count at row 1: INSERT INTO memberships (`user_id`, `group_id`, ... , id) VALUES (...whatever...)
/Users/david/myproj/vendor/composite_primary_keys-0.9.90/lib/composite_primary_keys/base.rb:106:in `create_without_callbacks'
See that last id, just before VALUES? It shouldn't be there. Something weird is going on, because composite_primary_keys doesn't seem to be doing its thing.
The solution is to make sure the call to "set_primary_keys" is the first thing executed in your model;
class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
include MyAwesomeModule
...
end
Remember this, and everything works fine. Forget it, and you'll have lots of fun with the debugger.
If I were braver, smarter and kinder, I would dive headfirst into the code and try to fix it. But, I've got work to do.

2008/04/13

Testing file uploads with RSpec on Rails

I was writing specs for a controller that processes uploaded images, and came up with this way of mocking a file upload control. So, I thought I'd share it.
Say you have a WibbleController which can replace the image belonging to a particular wibble. In your wibble/edit view you probably have something like this;


<% form_for(@wibble, :html => { :multipart => true }) do |f| %>
...
Replace with new image:
<%= f.file_field :new_file %>
...
<% end -%>

In your controller, there will be something that reads the streamed file data that the browser posts when you choose and submit a file.
In your spec, you can create an ActionController::UploadedStringIO object (which is what your controller will see), but it won't have access to the file data. So, we need to monkey patch it a bit.
The two methods we need to override are "read", which will return the file contents, and "size" (guess what that does). So, define a method in your spec file like so;

def mock_uploader(file, type = 'image/png')
filename = "%s/%s" % [ File.dirname(__FILE__), file ]
uploader = ActionController::UploadedStringIO.new
uploader.original_path = filename
uploader.content_type = type
def uploader.read
File.read(original_path)
end
def uploader.size
File.stat(original_path).size
end
uploader
end

Then, you could write a spec like this ('foo.png' should be a suitable image file in your spec/controllers directory);

it "should upload an image" do
Image.delete_all
uploader = mock_uploader 'foo.png'
post :update, {
:id => @object.id,
:wibble => { :image_file => uploader }
}
response.should be_success
Image.count.should == 1
i = Image.find(:first)
i.filename.should == uploader.original_path
i.contents.length.should == uploader.size
end

There may well be a better way to do this, in which case please let me know via the comments.

2008/04/08

Auto-rotate Rails log files

I've been searching for a way to make my rails apps rotate their log files automatically, the way apache does it. In my case, I get logs like this;

  • access_log.20080407
  • error_log.20080407
i.e. a log file per day, with the date as the suffix of the filename.
When a new day rolls around, a new file is started. Simple.
Hunting around on the web, most people seem to be writing a /etc/logrotate.conf file to achieve something like this. This has a couple of problems;
  • After switching to a new log file, all your mongrel/fastcgi processes need to be restarted. Otherwise, you get a segfault when they try to log to a file that's not there anymore.
  • The /etc/logrotate.conf file is yet another file whose deployment needs to be managed (potentially along with a cron entry to kick off logrotate at the appropriate time).
So, I kept digging and eventually found this post
So, Rails can be told to log to a pipe, just like apache. I'm not familiar with cronolog, so this is what I ended up putting in my config/environments/production.rb file;
config.active_record.colorize_logging = false

log_pipe = IO.popen("/usr/sbin/rotatelogs #{RAILS_ROOT}/log/production_log.%Y%m%d 86400", 'a')

config.logger = Logger.new(log_pipe)
[the second entry is supposed to be a single line from 'log_pipe = ' to ')' ]
  1. Turn off colorized logs, because they irritate me.
  2. Open a pipe to rotatelogs (I use 'a' for 'append', instead of 'w' for 'write', but it doesn't seem to make any difference on my system - the file is appended to when I restart rails, whichever I choose).
  3. Point the rails logger to the pipe.
This seems to do exactly what I want, with minimal mucking about.

2008/03/18

Cloning Ubuntu Gutsy virtual machines on OSX with VMWare Fusion

I use VMWare Fusion on my Mac to create virtual machines so that I can see exactly how my code will behave when it's deployed on Ubuntu Gutsy server.
I want to setup a few servers, so that I can experiment with different load-balancing solutions. So, I took a virtual machine that I had already configured, and copied it using Finder. When I launched the VM, it asked me if I had moved it or copied it. I clicked "I copied it", and the machine carried on booting.

Unfortunately, the networking was broken - the machine kept insisting that eth0 was not present. After a bit of research, I found this thread on the VMWare forums.
The problem is that VMWare assigns a new MAC address to the machine when you copy it - which is sensible, since you can't have two instances of the same MAC address on the same network. But, Linux stores the MAC address of the card when the network interface is configured, and VMWare has no way of telling Linux about the new MAC address. So, Linux thinks it's original network card has disappeared, and that a new card with a different MAC address has been installed, and gets very confused.
The solution, for Ubuntu 7.10 (Gutsy Gibbon), is to edit this file as root;

/etc/udev/rules.d/70-persistent-net.rules
Here is the file before editing;
# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:ba:e7:7a", NAME="eth0"

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth1"
(The SUBSYSTEM ... NAME stuff is all on a single line, but the blog theme causes it to wrap)
The red MAC address is the original one, the same address as in the original virtual machine we created the copy from. As far as the Linux server is concerned, this network card has just vanished, because it can't see any card installed with that MAC address.
The green address is the new MAC address that VMWare created. The Linux server thinks it has a network card plugged in, with this MAC address (because VMWare is telling it so), but it's not configured.
All we need to do is to edit the file so that the card which is installed is known as "eth0", and get rid of the old MAC address that our original source VM is still using. i.e. we delete the eth0 line and rename eth1 to eth0.
So, after editing, the file looks like this;
# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth0"

Now, reboot the VM (/etc/init.d/networking restart is not enough - you need to reboot), and all should be fine. Rinse and repeat for as many instances of your server as you need.

2008/03/12

Hiding vi/vim leftovers in OSX

I love my tricked-out MacVim editor. A friend keeps trying to get me to convert to using NetBeans and, while it's an impressive product, I keep coming back to my trusty and blisteringly fast combo of MacVim (with a shedload of plugins), a narrow tree-view Finder window so I can drag files into the editor, and a tabbed iTerm window for running tests, servers and executing commands.


One thing that really irritates me though is the xxxx~ files that vi/vim leaves lying around. When you edit and save file 'foo.txt', you end up an extra file 'foo.txt~' cluttering up your Finder. If the Finder window is too narrow to display the end of the filename, you have to be careful not to start editing the ~ file by mistake.

But, I've finally found a way to make those files disappear (temporarily) from Finder windows. If you've installed the developer tools on OS X, then you have the SetFile utility, which lets you set the Mac-specific (well, HFS-specific, if you want to be picky) file attributes. To make a file invisible;

SetFile -a V some/file/name
(Replacing V with v will make it visible again).

So, a little alias line in your .bashrc file;
alias hidejunk="find . -name '*~' | xargs SetFile -a V"
Now, typing hidejunk on the command-line will tidy up your Finder windows.

Unfortunately, the file becomes visible again when vim recreates it, but it would be simple enough to use cron or stakeout to re-run the SetFile command.