Converting iTunes playlists with ruby

Just posting this here in case anyone else has a similar setup.

I've got an xbox running XBMC hooked up to my home stereo system, but all my music is stored and organised in iTunes. The folder containing the music is shared so the xbox can see it, so all my music is available in XBMC with no problems. The one thing I was missing is all my iTunes playlists.

Along with virtually every other music player, XBMC uses a standard .m3u text file format to define playlists. I think the '#EXT...' stuff is specific to XBMC, but I'm not sure.

Needless to say, iTunes is not compatible with this at all. But, it is possible to export a playlist from iTunes, in its own bizarre XML format

One quick ruby script later, and I can convert the iTunes XML to M3U, including a remapping of file locations because iTunes has a different name for the root music folder (in fact, it has 2 different names for it because I moved the folder once, but the older playlists seem to still use the original location, somehow. Hence the alternatives in the ITUNES_ROOT regexp).

Finally, a noddy bash script to convert all the .xml playlist files in a directory to .m3u and transfer them to the xbox using ncftp

I still have to right-click on my playlists in iTunes and choose 'Export Song List' for each one, but that's not too arduous. I could probably write an AppleScript or something to do that for me, but in my case it's not worth the extra effort.

If you find any of this stuff useful, please drop a note in the comments.

Cheers

David

Client collaboration via github

UPDATE: The post-receive URL on github has now been moved to a new "Services" tab on the project edit page.

---

It can be a hassle setting up and maintaining development environments
for clients; particularly if they're not very technical, or if they
use Windows.


I usually set up a separate VPS for each client's project as a
development/staging server, and it would be great if they could
just edit the source files on there. Then they could play around
and see the effect of their changes in realtime. The trouble is
that this requires some kind of remote editing or file-syncing setup
(e.g. SSH+vi), which can be troublesome to set up.


So, I'm trying out a combination of github's in-browser source
editing, plus a post-receive URL pointing to the client's development
VPS. Here's how it should work.

• I set up the client's project as a github repository, and add the client as a collaborator.


• The client makes changes to the view/css files using github's in-browser editor.


• When they commit their changes, github posts to a listener on the client's development VPS (using a post-receive hook)


• The listener on the VPS pulls the client's changes


• The client views their development site to see the changes in (nearly) realtime

This is surprisingly easy to set up. I'll walk you through setting this up for a dummy "Hello, world" rails application. The ingredients we'll be using are;

• github - a free account will do fine, for this demo


• A development server - github will be posting to a URL on this server, so you need to be able to access it from the internet


• A listener to pull changes from github, whenever there's a new commit. I'm going to use a very simple sinatra application, running via a command-line mongrel instance.


• An application to act as our super-important client project

Setup a simple rails project as a github repository. You can just use a
bare "rails foobar" project - no need to set up databases.

Now, ssh to your development server (the one github will be posting to when your client makes changes). Setup a clone of the github repo.

For this demo, use the "public" clone url (the one that starts with "git://github.com").


You can use this technique for a private repo - otherwise it would be a bit pointless - but that takes a bit more SSH setup than I want to get into in this post.


I'm using "screen" so that I can start the demo application running
in a terminal and have it keep running after I log out. If you don't
know about the unix screen command, go learn about it right now.
After SSH it's one of the most useful tools to have in your toolbox.

screen
cd dstest
script/server
[Ctrl-A D to detach from screen]

So, now the application is running, and you can open a browser window to;

http://your.development.server:3000/

and see the "Welcome to Ruby on Rails" page.


Now for the listener which will pull our client's changes.


If you haven't already got them, you will need sinatra and mongrel on your development server;

sudo gem install sinatra
sudo gem install mongrel

Now, put the following into a file called "gitmon.rb"

require 'rubygems'
require 'sinatra'
post '/update_site' do
dir = '/home/david/dstest'
system "cd #{dir}; git pull"
"Site updated\n"
end

You will need to change the "dir" line to
be the full path to your rails project.


Now use screen again to start a listener that will continue to run after you log out;

screen
ruby gitmon.rb
[Ctrl-A D to detach from screen]

The listener will start up on Sinatra's
default port, 4567. You can test it by
visiting

http://your.development.server:4567

You should see "Sinatra doesn't know this diddy."


Now we just need to tell github to post to the listener whenever it gets new commits.


Login to your github account and go to
your repository's edit page.





Setup the "Post-receive URL" as;

http://your.development.server:4567/update_site

That should do it.


Now, edit a file (e.g. public/index.html) using the github in-browser editor (if you're not logged-in, you won't see the edit link). Commit your change.





Refresh the browser window you've got open to;

http://your.development.server:3000/

and you should see your changes (it can take a few seconds before your site updates).


This is just the tip of the iceberg, and there are a lot of ways this technique could be extended (e.g. hooks to run migrations after a "git pull" on the development server, or getting the sinatra listener to post to Campfire whenever a client commits something).


Also, there is a lot of information in the POST from github, which we're just ignoring here.


I hope you find this useful. I'll be setting it up for one of my clients very soon, so I'm keen to see how it works out.

Setting up Ubuntu 8.10 for Ruby on Rails development

Here's how I've set up my laptop, running Ubuntu 8.10 (Intrepid Ibex) for Ruby on Rails development. You'll need to run the following commands from the command-line as root. These commands worked for me, but I take no responsibility if you trash your box by doing this ;)

There is some duplication here, because I sometimes setup servers with different sub-sets of the software I use on my development laptop. Don't worry about this. It does no harm if you tell apt-get or aptitude to install the same package twice - it will just ignore you the second time.

Not all of the gems here are required for RoR development, but I use cucumber for integration testing, and it has some pre-requisites.

# General/Pre-requisite packages
aptitude install \
  build-essential \
  screen \
  subversion \
  mysql-client \
  telnet \
  meld \
  vim \
  vim-gnome \
  exuberant-ctags \
  tk8.5 \
  apache2-prefork-dev \
  rcov

# mysql server
# use apt-get to avoid installing exim4
DEBIAN_FRONTEND=noninteractive apt-get install --assume-yes \
  mysql-server mysql-client \
  libmysqlclient15-dev libmysql-ruby1.8


# git
mkdir gitcore
cd gitcore
wget http://kernel.org/pub/software/scm/git/git-1.6.0.3.tar.gz
apt-get build-dep git-core --assume-yes
tar xzvf git-1.6.0.3.tar.gz
cd git-1.6.0.3/
./configure
make
make install
cd

# Ruby
aptitude --assume-yes install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 \
  irb1.8 libreadline-ruby1.8 libruby1.8 libopenssl-ruby
ln -s /usr/bin/ruby1.8 /usr/local/bin/ruby;
ln -s /usr/bin/ri1.8 /usr/local/bin/ri;
ln -s /usr/bin/rdoc1.8 /usr/local/bin/rdoc;
ln -s /usr/bin/irb1.8 /usr/local/bin/irb;
ln -s /usr/local/bin/ruby /usr/bin/ruby

# Rubygems. You REALLY don't want to let aptitude install rubygems.
wget http://rubyforge.org/frs/download.php/45905/rubygems-1.3.1.tgz
tar xzf rubygems-1.3.1.tgz
cd rubygems-1.3.1
ruby setup.rb
ln -s /usr/bin/gem1.8 /usr/bin/gem
for g in rails rake capistrano capistrano-ext hpricot treetop ruby-debug term-ansicolor mongrel cheat passenger annotate-models rak; do gem install $g; done

Have fun.

David

Finding files in vim

Some of my vim-challenged colleagues, when they try switching to vim, miss the Cmd-T function that lets them type parts of a filename and open the corresponding file.

After a quick search of the tips on vim.org, it turns out that someone has implemented exactly this function;

http://www.vim.org/tips/tip.php?tip_id=1432

This has the drawback that it will find blah.rb.svn-base when you look for blah.rb. So, I've tweaked it a little to skip any svn files, and also not to look in the 'vendor' directory, so it doesn't find rails source files.

http://gist.github.com/1625

Have fun

Automatically "git rm" all deleted files

Love git. Hate having to "git rm" individual files whenever I delete a bunch of stuff (yes, I know I can "git rm foo/*", but what if I didn't remove *all* the files in foo/ ?).

So, a quick ruby script: gitrmall.rb

Disabling tests in Shoulda

I'm using Shoulda on a client project, although I prefer rspec myself.

One of the things I miss from rspec is the ability to turn off a test, temporarily, by changing;


it "should do something"


...to;


xit "should do something"


Tests which are disabled in this way show up as "Example disabled: it should do something", when the specs are executed.

Using this with autotest and a couple of quick vim macros, my workflow goes something like this;

1. See that there is a failure in a particular test
2. Position the cursor in that test and hit ",ofs" this triggers my macro to disable all specs except the current one, and save the file so that autotest runs it again.
3. Hack, hack, hack to fix the bug, or develop the feature that the failing test relates to.
4. Hit ",ons" to turn all the specs back on and save the file again.

In Shoulda, renaming a test from;


should "do something"


...to;


xshould "do something"


Gives you a nice NoMethodError.

But, this is ruby, so we just add this to our test/test_helper.rb




class ActiveSupport::TestCase
  def self.xshould(name, &block)
    puts "disabled test: #{name}"
  end
end




Now, we get pretty much the same behaviour as with 'xit' in rspec.

For any vim users in the audience, here are the macros (actually, abbreviations) I use to turn tests, specs and shouldas on and off. Just add to your .gvimrc (or .vimrc) file;


"rspec
map ,ofs :%s/ it / xit ''?xitx:w
map ,ons :%s/ xit / it /'':w

"shoulda
map ,ofh :%s/ should / xshould ''?xshouldx:w
map ,onh :%s/ xshould / should /'':w

"test/unit
map ,oft :%s/def test/def xtest''?xtestx:w
map ,ont :%s/def xtest/def test/'':w


ActiveRecord commit timing

I'm building an application where we're using a message queue.

New objects add their IDs to a beanstalkd queue via an after_create callback. Some daemons monitor beanstalk, and grab objects to process, as soon as they hit the queue.

We were getting errors from the daemons, saying "record with ID 'N' not found", but when we looked for object N via script/console, or the database, there it was.

It turns out that the after_create callback (along with all the other object creation callbacks) occurs *inside* a transaction. So, this is what was happening;



There are a couple of ways around this;

Call "self.class.connection.commit_db_transaction" in the model, after the after_create method. This works, but it smells really bad.

A nicer way is to add an "after_commit" callback, like this

Pat Allan has some modifications to this, to make it play nicer with Rails 2.0 and 2.1

Or, there is a version here which handles nested transactions.

Many thanks to Pat for showing me the after_commit stuff.

Using non-standard primary keys with ActiveRecord in Ruby on Rails

A Rails application I'm currently working on has a 'User' model. For the purposes of this app. a 'user' is uniquely identified by their mobile phone number - a user must have a mobile phone number, and may only have a single one. A second mobile phone number must be a second user.

I'm going to need the mobile phone number in other models - e.g. "User has_many :messages", and I'd like to have a "mobile_phone_number" attribute in my Message model, so that I can say "message.mobile_phone_number" without having to join the users table all the time, like this; "message.user.mobile_phone_number".

I can get what I want if the mobile_phone_number is the primary key of the User model. Let's try it.

Here is the migration (this is specific to mysql);

class CreateUsers < ActiveRecord::Migration
  def self.up
    create_table :users, :id => false do |t|
      t.integer :mobile_phone_number
    end
    execute "alter table users modify column mobile_phone_number bigint unsigned primary key"
  end

  def self.down
    drop_table :users
  end
end

(The alter table statement is a bit of a hack. Another way to achieve the same result would be to use the mysql_bigint gem)

Here is the model;

class User < ActiveRecord::Base
  set_primary_key :mobile_phone_number
end

Looks OK, but this is what happens when you try to create a User;

>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 0>

Where did that '0' come from, and what happened to the mobile_phone_number?

A quick look in the logs shows this;

  WARNING: Can't mass-assign these protected attributes: mobile_phone_number
  SQL (0.000146)   BEGIN
  User Create (0.000222)   INSERT INTO `users` VALUES(DEFAULT)
  SQL (0.000422)   COMMIT

So, Rails is not sending the primary key column value in the SQL create statement. This would make sense if, as usual, our primary key was an autoincrement column. The database would supply that value for us, so we don't want to send it. But, in our case, we want to set the primary key ourselves, so this behaviour is wrong. (The 0 comes from the default value of the column)

You can try using 'attr_accessible' to allow you to assign to the mobile_phone_number, but it doesn't work (presumably because it's the primary key that we're trying to assign to).

We need some way to override the standard Rails primary key behaviour so it does what we want. That sounds a lot like the composite_primary_keys gem.

So, after installing the gem (in my case, after installing Rick Olson's 'gems' plugin so that I can put the composite_primary_keys gem in my project's vendor directory), here is the updated model;

require 'composite_primary_keys'
class User < ActiveRecord::Base
  set_primary_keys :mobile_phone_number
end

Note that 'set_primary_key' has changed to 'set_primary_keys'.
Now, we get this;

>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 447123456789>

Much better. Even though we're not using a composite_primary_key, the gem is overriding the default Rails behaviour so that the key we want is being set by the create statement.

If you know a more elegant way to do this, please let me know.

Gotcha: composite_primary_keys gem

Dr. Nic's composite_primary_keys gem is incredibly helpful if you're writing rails code against a legacy database, or in any other situation where you can't or won't follow the rails convention of having a single field as your model's primary key.
After doing the usual;

sudo gem install composite_primary_keys

...and requiring it in you environment.rb file, you can define a model like this;

class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
...
end

Brilliant.
But, there is one subtlety you need to be aware of. If your model mixes in any modules, you might end up writing something like this;

class Membership < ActiveRecord::Base
include MyAwesomeModule
set_primary_keys :user_id, :group_id
...
end

Looks fine, doesn't it? Unfortunately, your tests will now break with lots of errors like this;

ActiveRecord::StatementInvalid in 'Membership should foobar'
Mysql::Error: Column count doesn't match value count at row 1: INSERT INTO memberships (`user_id`, `group_id`, ... , id) VALUES (...whatever...)
/Users/david/myproj/vendor/composite_primary_keys-0.9.90/lib/composite_primary_keys/base.rb:106:in `create_without_callbacks'

See that last id, just before VALUES? It shouldn't be there. Something weird is going on, because composite_primary_keys doesn't seem to be doing its thing.
The solution is to make sure the call to "set_primary_keys" is the first thing executed in your model;

class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
include MyAwesomeModule
...
end

Remember this, and everything works fine. Forget it, and you'll have lots of fun with the debugger.
If I were braver, smarter and kinder, I would dive headfirst into the code and try to fix it. But, I've got work to do.

Testing file uploads with RSpec on Rails

I was writing specs for a controller that processes uploaded images, and came up with this way of mocking a file upload control. So, I thought I'd share it.
Say you have a WibbleController which can replace the image belonging to a particular wibble. In your wibble/edit view you probably have something like this;

<% form_for(@wibble, :html => { :multipart => true }) do |f| %>
...
Replace with new image:
<%= f.file_field :new_file %>
...
<% end -%>

In your controller, there will be something that reads the streamed file data that the browser posts when you choose and submit a file.
In your spec, you can create an ActionController::UploadedStringIO object (which is what your controller will see), but it won't have access to the file data. So, we need to monkey patch it a bit.
The two methods we need to override are "read", which will return the file contents, and "size" (guess what that does). So, define a method in your spec file like so;

def mock_uploader(file, type = 'image/png')
filename = "%s/%s" % [ File.dirname(__FILE__), file ]
uploader = ActionController::UploadedStringIO.new
uploader.original_path = filename
uploader.content_type = type
def uploader.read
  File.read(original_path)
end
def uploader.size
  File.stat(original_path).size
end
uploader
end

Then, you could write a spec like this ('foo.png' should be a suitable image file in your spec/controllers directory);

it "should upload an image" do
 Image.delete_all
 uploader = mock_uploader 'foo.png'
 post :update, {
   :id => @object.id,
   :wibble => { :image_file => uploader }
 }
 response.should be_success
 Image.count.should == 1
 i = Image.find(:first)
 i.filename.should == uploader.original_path
 i.contents.length.should == uploader.size
end

There may well be a better way to do this, in which case please let me know via the comments.

Auto-rotate Rails log files

I've been searching for a way to make my rails apps rotate their log files automatically, the way apache does it. In my case, I get logs like this;

• access_log.20080407
• error_log.20080407

i.e. a log file per day, with the date as the suffix of the filename.
When a new day rolls around, a new file is started. Simple.
Hunting around on the web, most people seem to be writing a /etc/logrotate.conf file to achieve something like this. This has a couple of problems;

• After switching to a new log file, all your mongrel/fastcgi processes need to be restarted. Otherwise, you get a segfault when they try to log to a file that's not there anymore.
• The /etc/logrotate.conf file is yet another file whose deployment needs to be managed (potentially along with a cron entry to kick off logrotate at the appropriate time).

So, I kept digging and eventually found this post
So, Rails can be told to log to a pipe, just like apache. I'm not familiar with cronolog, so this is what I ended up putting in my config/environments/production.rb file;

config.active_record.colorize_logging = false

log_pipe = IO.popen("/usr/sbin/rotatelogs #{RAILS_ROOT}/log/production_log.%Y%m%d 86400", 'a')

config.logger = Logger.new(log_pipe)

[the second entry is supposed to be a single line from 'log_pipe = ' to ')' ]

1. Turn off colorized logs, because they irritate me.
2. Open a pipe to rotatelogs (I use 'a' for 'append', instead of 'w' for 'write', but it doesn't seem to make any difference on my system - the file is appended to when I restart rails, whichever I choose).
3. Point the rails logger to the pipe.

This seems to do exactly what I want, with minimal mucking about.

Cloning Ubuntu Gutsy virtual machines on OSX with VMWare Fusion

I use VMWare Fusion on my Mac to create virtual machines so that I can see exactly how my code will behave when it's deployed on Ubuntu Gutsy server.
I want to setup a few servers, so that I can experiment with different load-balancing solutions. So, I took a virtual machine that I had already configured, and copied it using Finder. When I launched the VM, it asked me if I had moved it or copied it. I clicked "I copied it", and the machine carried on booting.

Unfortunately, the networking was broken - the machine kept insisting that eth0 was not present. After a bit of research, I found this thread on the VMWare forums.
The problem is that VMWare assigns a new MAC address to the machine when you copy it - which is sensible, since you can't have two instances of the same MAC address on the same network. But, Linux stores the MAC address of the card when the network interface is configured, and VMWare has no way of telling Linux about the new MAC address. So, Linux thinks it's original network card has disappeared, and that a new card with a different MAC address has been installed, and gets very confused.
The solution, for Ubuntu 7.10 (Gutsy Gibbon), is to edit this file as root;

/etc/udev/rules.d/70-persistent-net.rules

Here is the file before editing;

# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:ba:e7:7a", NAME="eth0"

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth1"

(The SUBSYSTEM ... NAME stuff is all on a single line, but the blog theme causes it to wrap)
The red MAC address is the original one, the same address as in the original virtual machine we created the copy from. As far as the Linux server is concerned, this network card has just vanished, because it can't see any card installed with that MAC address.
The green address is the new MAC address that VMWare created. The Linux server thinks it has a network card plugged in, with this MAC address (because VMWare is telling it so), but it's not configured.
All we need to do is to edit the file so that the card which is installed is known as "eth0", and get rid of the old MAC address that our original source VM is still using. i.e. we delete the eth0 line and rename eth1 to eth0.
So, after editing, the file looks like this;

# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth0"

Now, reboot the VM (/etc/init.d/networking restart is not enough - you need to reboot), and all should be fine. Rinse and repeat for as many instances of your server as you need.

Hiding vi/vim leftovers in OSX

I love my tricked-out MacVim editor. A friend keeps trying to get me to convert to using NetBeans and, while it's an impressive product, I keep coming back to my trusty and blisteringly fast combo of MacVim (with a shedload of plugins), a narrow tree-view Finder window so I can drag files into the editor, and a tabbed iTerm window for running tests, servers and executing commands.


One thing that really irritates me though is the xxxx~ files that vi/vim leaves lying around. When you edit and save file 'foo.txt', you end up an extra file 'foo.txt~' cluttering up your Finder. If the Finder window is too narrow to display the end of the filename, you have to be careful not to start editing the ~ file by mistake.

But, I've finally found a way to make those files disappear (temporarily) from Finder windows. If you've installed the developer tools on OS X, then you have the SetFile utility, which lets you set the Mac-specific (well, HFS-specific, if you want to be picky) file attributes. To make a file invisible;

SetFile -a V some/file/name

(Replacing V with v will make it visible again).

So, a little alias line in your .bashrc file;

alias hidejunk="find . -name '*~' | xargs SetFile -a V"

Now, typing hidejunk on the command-line will tidy up your Finder windows.

Unfortunately, the file becomes visible again when vim recreates it, but it would be simple enough to use cron or stakeout to re-run the SetFile command.

YAML FAQs plugin

I've just released my first rails plugin - yaml_faqs

It's a modest little number that adds yaml-based FAQs to your rails apps.

Hope people find it useful.

Rails2 and boolean radio buttons

Just came across a difference in the way rails 2 handles radio buttons, compared with rails 1, and I don't think I've seen it documented anywhere.

If you've got something like this;

<% form_for(@hello) do |f| %>
<p> <%= f.radio_button :foo, true %>True </p>
<p> <%= f.radio_button :foo, false %>False </p>
<p> <%= f.submit "Create" %> </p>
<% end %>

Then, if you click on the "True" radio button, and submit the form, you get;

"hello"=>{"foo"=>"true"}

...submitted to your controller. All fine.

Now, click on the "False" button and submit, and you get;

"hello"=>{"foo"=>"on"}

Not quite so good. This can really mess you up if your controller method has something like;

if params[:foo] == "false"
... do something cool

Do the same thing in Rails 1.2.6 and you get;

"hello"=>{"foo"=>"true"}

and

"hello"=>{"foo"=>"false"}

The moral of the story? Do this instead;

<% form_for(@hello) do |f| %>
<p> <%= f.radio_button :foo, "true" %>True </p>
<p> <%= f.radio_button :foo, "false" %>False </p>
<p> <%= f.submit "Create" %> </p>
<% end %>

i.e. Use string values, not logical values, in the view file. This works fine in either version.

London Ruby User Group

I was among several people to give a 20x20 presentation at the London Ruby User Group on Monday.

Great fun, albeit slightly nerve-wracking. Particularly when I ended up going first, and Muz's screensaver came on halfway through my presentation!

Beware of Rails date arithmetic

Having time and date methods on integers in Rails is nifty. Being able to say things like;

>> Date.today.to_time + 7.days
=> Thu Feb 07 00:00:00 +0000 2008

...is nice.

But, be careful about relying on this when doing arithmetic with months.

>> Date.new( 2008, 1, 1 ).to_time + 1.month
=> Thu Jan 31 00:00:00 +0000 2008

Wrong.

The problem is this;

>> 1.month / 86400.0
=> 30.0

i.e. a "month" is just 30 days' worth of seconds.

Similarly;

>> ( Date.new( 2008, 2, 1 ).to_time + 1.month )
=> Sun Mar 02 00:00:00 +0000 2008

There is a way to do month calculation correctly - use the built-in ActiveSupport::CoreExtensions::DateTime::Calculations like this;

>> Date.new( 2008, 1, 1 ).to_time.advance( :months => 1 )
=> Fri Feb 01 00:00:00 +0000 2008

>> Date.new( 2008, 2, 1 ).to_time.advance( :months => 1 )
=> Sat Mar 01 00:00:00 +0000 2008

Much better.