Automatically "git rm" all deleted files

Love git. Hate having to "git rm" individual files whenever I delete a bunch of stuff (yes, I know I can "git rm foo/*", but what if I didn't remove *all* the files in foo/ ?).

So, a quick ruby script: gitrmall.rb

Disabling tests in Shoulda

I'm using Shoulda on a client project, although I prefer rspec myself.

One of the things I miss from rspec is the ability to turn off a test, temporarily, by changing;


it "should do something"


...to;


xit "should do something"


Tests which are disabled in this way show up as "Example disabled: it should do something", when the specs are executed.

Using this with autotest and a couple of quick vim macros, my workflow goes something like this;

1. See that there is a failure in a particular test
2. Position the cursor in that test and hit ",ofs" this triggers my macro to disable all specs except the current one, and save the file so that autotest runs it again.
3. Hack, hack, hack to fix the bug, or develop the feature that the failing test relates to.
4. Hit ",ons" to turn all the specs back on and save the file again.

In Shoulda, renaming a test from;


should "do something"


...to;


xshould "do something"


Gives you a nice NoMethodError.

But, this is ruby, so we just add this to our test/test_helper.rb




class ActiveSupport::TestCase
  def self.xshould(name, &block)
    puts "disabled test: #{name}"
  end
end




Now, we get pretty much the same behaviour as with 'xit' in rspec.

For any vim users in the audience, here are the macros (actually, abbreviations) I use to turn tests, specs and shouldas on and off. Just add to your .gvimrc (or .vimrc) file;


"rspec
map ,ofs :%s/ it / xit ''?xitx:w
map ,ons :%s/ xit / it /'':w

"shoulda
map ,ofh :%s/ should / xshould ''?xshouldx:w
map ,onh :%s/ xshould / should /'':w

"test/unit
map ,oft :%s/def test/def xtest''?xtestx:w
map ,ont :%s/def xtest/def test/'':w


ActiveRecord commit timing

I'm building an application where we're using a message queue.

New objects add their IDs to a beanstalkd queue via an after_create callback. Some daemons monitor beanstalk, and grab objects to process, as soon as they hit the queue.

We were getting errors from the daemons, saying "record with ID 'N' not found", but when we looked for object N via script/console, or the database, there it was.

It turns out that the after_create callback (along with all the other object creation callbacks) occurs *inside* a transaction. So, this is what was happening;



There are a couple of ways around this;

Call "self.class.connection.commit_db_transaction" in the model, after the after_create method. This works, but it smells really bad.

A nicer way is to add an "after_commit" callback, like this

Pat Allan has some modifications to this, to make it play nicer with Rails 2.0 and 2.1

Or, there is a version here which handles nested transactions.

Many thanks to Pat for showing me the after_commit stuff.

Using non-standard primary keys with ActiveRecord in Ruby on Rails

A Rails application I'm currently working on has a 'User' model. For the purposes of this app. a 'user' is uniquely identified by their mobile phone number - a user must have a mobile phone number, and may only have a single one. A second mobile phone number must be a second user.

I'm going to need the mobile phone number in other models - e.g. "User has_many :messages", and I'd like to have a "mobile_phone_number" attribute in my Message model, so that I can say "message.mobile_phone_number" without having to join the users table all the time, like this; "message.user.mobile_phone_number".

I can get what I want if the mobile_phone_number is the primary key of the User model. Let's try it.

Here is the migration (this is specific to mysql);

class CreateUsers < ActiveRecord::Migration
  def self.up
    create_table :users, :id => false do |t|
      t.integer :mobile_phone_number
    end
    execute "alter table users modify column mobile_phone_number bigint unsigned primary key"
  end

  def self.down
    drop_table :users
  end
end

(The alter table statement is a bit of a hack. Another way to achieve the same result would be to use the mysql_bigint gem)

Here is the model;

class User < ActiveRecord::Base
  set_primary_key :mobile_phone_number
end

Looks OK, but this is what happens when you try to create a User;

>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 0>

Where did that '0' come from, and what happened to the mobile_phone_number?

A quick look in the logs shows this;

  WARNING: Can't mass-assign these protected attributes: mobile_phone_number
  SQL (0.000146)   BEGIN
  User Create (0.000222)   INSERT INTO `users` VALUES(DEFAULT)
  SQL (0.000422)   COMMIT

So, Rails is not sending the primary key column value in the SQL create statement. This would make sense if, as usual, our primary key was an autoincrement column. The database would supply that value for us, so we don't want to send it. But, in our case, we want to set the primary key ourselves, so this behaviour is wrong. (The 0 comes from the default value of the column)

You can try using 'attr_accessible' to allow you to assign to the mobile_phone_number, but it doesn't work (presumably because it's the primary key that we're trying to assign to).

We need some way to override the standard Rails primary key behaviour so it does what we want. That sounds a lot like the composite_primary_keys gem.

So, after installing the gem (in my case, after installing Rick Olson's 'gems' plugin so that I can put the composite_primary_keys gem in my project's vendor directory), here is the updated model;

require 'composite_primary_keys'
class User < ActiveRecord::Base
  set_primary_keys :mobile_phone_number
end

Note that 'set_primary_key' has changed to 'set_primary_keys'.
Now, we get this;

>> u = User.create :mobile_phone_number => 447123456789
=> #<User mobile_phone_number: 447123456789>

Much better. Even though we're not using a composite_primary_key, the gem is overriding the default Rails behaviour so that the key we want is being set by the create statement.

If you know a more elegant way to do this, please let me know.

Gotcha: composite_primary_keys gem

Dr. Nic's composite_primary_keys gem is incredibly helpful if you're writing rails code against a legacy database, or in any other situation where you can't or won't follow the rails convention of having a single field as your model's primary key.
After doing the usual;

sudo gem install composite_primary_keys

...and requiring it in you environment.rb file, you can define a model like this;

class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
...
end

Brilliant.
But, there is one subtlety you need to be aware of. If your model mixes in any modules, you might end up writing something like this;

class Membership < ActiveRecord::Base
include MyAwesomeModule
set_primary_keys :user_id, :group_id
...
end

Looks fine, doesn't it? Unfortunately, your tests will now break with lots of errors like this;

ActiveRecord::StatementInvalid in 'Membership should foobar'
Mysql::Error: Column count doesn't match value count at row 1: INSERT INTO memberships (`user_id`, `group_id`, ... , id) VALUES (...whatever...)
/Users/david/myproj/vendor/composite_primary_keys-0.9.90/lib/composite_primary_keys/base.rb:106:in `create_without_callbacks'

See that last id, just before VALUES? It shouldn't be there. Something weird is going on, because composite_primary_keys doesn't seem to be doing its thing.
The solution is to make sure the call to "set_primary_keys" is the first thing executed in your model;

class Membership < ActiveRecord::Base
set_primary_keys :user_id, :group_id
include MyAwesomeModule
...
end

Remember this, and everything works fine. Forget it, and you'll have lots of fun with the debugger.
If I were braver, smarter and kinder, I would dive headfirst into the code and try to fix it. But, I've got work to do.

Testing file uploads with RSpec on Rails

I was writing specs for a controller that processes uploaded images, and came up with this way of mocking a file upload control. So, I thought I'd share it.
Say you have a WibbleController which can replace the image belonging to a particular wibble. In your wibble/edit view you probably have something like this;

<% form_for(@wibble, :html => { :multipart => true }) do |f| %>
...
Replace with new image:
<%= f.file_field :new_file %>
...
<% end -%>

In your controller, there will be something that reads the streamed file data that the browser posts when you choose and submit a file.
In your spec, you can create an ActionController::UploadedStringIO object (which is what your controller will see), but it won't have access to the file data. So, we need to monkey patch it a bit.
The two methods we need to override are "read", which will return the file contents, and "size" (guess what that does). So, define a method in your spec file like so;

def mock_uploader(file, type = 'image/png')
filename = "%s/%s" % [ File.dirname(__FILE__), file ]
uploader = ActionController::UploadedStringIO.new
uploader.original_path = filename
uploader.content_type = type
def uploader.read
  File.read(original_path)
end
def uploader.size
  File.stat(original_path).size
end
uploader
end

Then, you could write a spec like this ('foo.png' should be a suitable image file in your spec/controllers directory);

it "should upload an image" do
 Image.delete_all
 uploader = mock_uploader 'foo.png'
 post :update, {
   :id => @object.id,
   :wibble => { :image_file => uploader }
 }
 response.should be_success
 Image.count.should == 1
 i = Image.find(:first)
 i.filename.should == uploader.original_path
 i.contents.length.should == uploader.size
end

There may well be a better way to do this, in which case please let me know via the comments.

Auto-rotate Rails log files

I've been searching for a way to make my rails apps rotate their log files automatically, the way apache does it. In my case, I get logs like this;

• access_log.20080407
• error_log.20080407

i.e. a log file per day, with the date as the suffix of the filename.
When a new day rolls around, a new file is started. Simple.
Hunting around on the web, most people seem to be writing a /etc/logrotate.conf file to achieve something like this. This has a couple of problems;

• After switching to a new log file, all your mongrel/fastcgi processes need to be restarted. Otherwise, you get a segfault when they try to log to a file that's not there anymore.
• The /etc/logrotate.conf file is yet another file whose deployment needs to be managed (potentially along with a cron entry to kick off logrotate at the appropriate time).

So, I kept digging and eventually found this post
So, Rails can be told to log to a pipe, just like apache. I'm not familiar with cronolog, so this is what I ended up putting in my config/environments/production.rb file;

config.active_record.colorize_logging = false

log_pipe = IO.popen("/usr/sbin/rotatelogs #{RAILS_ROOT}/log/production_log.%Y%m%d 86400", 'a')

config.logger = Logger.new(log_pipe)

[the second entry is supposed to be a single line from 'log_pipe = ' to ')' ]

1. Turn off colorized logs, because they irritate me.
2. Open a pipe to rotatelogs (I use 'a' for 'append', instead of 'w' for 'write', but it doesn't seem to make any difference on my system - the file is appended to when I restart rails, whichever I choose).
3. Point the rails logger to the pipe.

This seems to do exactly what I want, with minimal mucking about.

Cloning Ubuntu Gutsy virtual machines on OSX with VMWare Fusion

I use VMWare Fusion on my Mac to create virtual machines so that I can see exactly how my code will behave when it's deployed on Ubuntu Gutsy server.
I want to setup a few servers, so that I can experiment with different load-balancing solutions. So, I took a virtual machine that I had already configured, and copied it using Finder. When I launched the VM, it asked me if I had moved it or copied it. I clicked "I copied it", and the machine carried on booting.

Unfortunately, the networking was broken - the machine kept insisting that eth0 was not present. After a bit of research, I found this thread on the VMWare forums.
The problem is that VMWare assigns a new MAC address to the machine when you copy it - which is sensible, since you can't have two instances of the same MAC address on the same network. But, Linux stores the MAC address of the card when the network interface is configured, and VMWare has no way of telling Linux about the new MAC address. So, Linux thinks it's original network card has disappeared, and that a new card with a different MAC address has been installed, and gets very confused.
The solution, for Ubuntu 7.10 (Gutsy Gibbon), is to edit this file as root;

/etc/udev/rules.d/70-persistent-net.rules

Here is the file before editing;

# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:ba:e7:7a", NAME="eth0"

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth1"

(The SUBSYSTEM ... NAME stuff is all on a single line, but the blog theme causes it to wrap)
The red MAC address is the original one, the same address as in the original virtual machine we created the copy from. As far as the Linux server is concerned, this network card has just vanished, because it can't see any card installed with that MAC address.
The green address is the new MAC address that VMWare created. The Linux server thinks it has a network card plugged in, with this MAC address (because VMWare is telling it so), but it's not configured.
All we need to do is to edit the file so that the card which is installed is known as "eth0", and get rid of the old MAC address that our original source VM is still using. i.e. we delete the eth0 line and rename eth1 to eth0.
So, after editing, the file looks like this;

# This file was automatically generated by the /lib/udev/write_net_rules
# program, probably run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single line.

# PCI device 0x1022:0x2000 (pcnet32)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:0c:29:e4:ee:d4", NAME="eth0"

Now, reboot the VM (/etc/init.d/networking restart is not enough - you need to reboot), and all should be fine. Rinse and repeat for as many instances of your server as you need.

Hiding vi/vim leftovers in OSX

I love my tricked-out MacVim editor. A friend keeps trying to get me to convert to using NetBeans and, while it's an impressive product, I keep coming back to my trusty and blisteringly fast combo of MacVim (with a shedload of plugins), a narrow tree-view Finder window so I can drag files into the editor, and a tabbed iTerm window for running tests, servers and executing commands.


One thing that really irritates me though is the xxxx~ files that vi/vim leaves lying around. When you edit and save file 'foo.txt', you end up an extra file 'foo.txt~' cluttering up your Finder. If the Finder window is too narrow to display the end of the filename, you have to be careful not to start editing the ~ file by mistake.

But, I've finally found a way to make those files disappear (temporarily) from Finder windows. If you've installed the developer tools on OS X, then you have the SetFile utility, which lets you set the Mac-specific (well, HFS-specific, if you want to be picky) file attributes. To make a file invisible;

SetFile -a V some/file/name

(Replacing V with v will make it visible again).

So, a little alias line in your .bashrc file;

alias hidejunk="find . -name '*~' | xargs SetFile -a V"

Now, typing hidejunk on the command-line will tidy up your Finder windows.

Unfortunately, the file becomes visible again when vim recreates it, but it would be simple enough to use cron or stakeout to re-run the SetFile command.

YAML FAQs plugin

I've just released my first rails plugin - yaml_faqs

It's a modest little number that adds yaml-based FAQs to your rails apps.

Hope people find it useful.

Rails2 and boolean radio buttons

Just came across a difference in the way rails 2 handles radio buttons, compared with rails 1, and I don't think I've seen it documented anywhere.

If you've got something like this;

<% form_for(@hello) do |f| %>
<p> <%= f.radio_button :foo, true %>True </p>
<p> <%= f.radio_button :foo, false %>False </p>
<p> <%= f.submit "Create" %> </p>
<% end %>

Then, if you click on the "True" radio button, and submit the form, you get;

"hello"=>{"foo"=>"true"}

...submitted to your controller. All fine.

Now, click on the "False" button and submit, and you get;

"hello"=>{"foo"=>"on"}

Not quite so good. This can really mess you up if your controller method has something like;

if params[:foo] == "false"
... do something cool

Do the same thing in Rails 1.2.6 and you get;

"hello"=>{"foo"=>"true"}

and

"hello"=>{"foo"=>"false"}

The moral of the story? Do this instead;

<% form_for(@hello) do |f| %>
<p> <%= f.radio_button :foo, "true" %>True </p>
<p> <%= f.radio_button :foo, "false" %>False </p>
<p> <%= f.submit "Create" %> </p>
<% end %>

i.e. Use string values, not logical values, in the view file. This works fine in either version.

London Ruby User Group

I was among several people to give a 20x20 presentation at the London Ruby User Group on Monday.

Great fun, albeit slightly nerve-wracking. Particularly when I ended up going first, and Muz's screensaver came on halfway through my presentation!

Beware of Rails date arithmetic

Having time and date methods on integers in Rails is nifty. Being able to say things like;

>> Date.today.to_time + 7.days
=> Thu Feb 07 00:00:00 +0000 2008

...is nice.

But, be careful about relying on this when doing arithmetic with months.

>> Date.new( 2008, 1, 1 ).to_time + 1.month
=> Thu Jan 31 00:00:00 +0000 2008

Wrong.

The problem is this;

>> 1.month / 86400.0
=> 30.0

i.e. a "month" is just 30 days' worth of seconds.

Similarly;

>> ( Date.new( 2008, 2, 1 ).to_time + 1.month )
=> Sun Mar 02 00:00:00 +0000 2008

There is a way to do month calculation correctly - use the built-in ActiveSupport::CoreExtensions::DateTime::Calculations like this;

>> Date.new( 2008, 1, 1 ).to_time.advance( :months => 1 )
=> Fri Feb 01 00:00:00 +0000 2008

>> Date.new( 2008, 2, 1 ).to_time.advance( :months => 1 )
=> Sat Mar 01 00:00:00 +0000 2008

Much better.

Rails Importing invoices from Blinksale

I use Blinksale for all my invoicing, it's great. I use a Ruby on Rails application, that I wrote myself, to handle my accounts. It's great too. What isn't so great is re-typing my invoices into my accounts application after creating them in Blinksale. That's not even a little bit great, in fact it sucks.

Earlier this year Blinksale released a REST API so I decided to see if I can use that to allow my accounts application to import invoices from Blinksale.


Getting Started

BTW, Blinksale recommend setting up and using a separate login for any API integration.

I'm going to use the ruby library that Blinksale provide, so I need to download the following files to my Rails Application's lib directory;

• http://www.blinksale.com/api/blinksale.rb


• http://www.blinksale.com/api/rest_client.rb


• http://www.blinksale.com/api/xml_node.rb

Talking to Blinksale

Let's start with a quick test script in the application's script directory;

    #!/usr/bin/env ruby

    require File.dirname(__FILE__) + '/../config/environment'

    blinksale = Blinksale.new |company|, |user|, |password|, |use_ssl|

    invoice = blinksale.invoices[ |a blinksale invoice id| ]

    puts invoice.data

Replace the elements between |pipes| with values for your own Blinksale account. The "use_ssl" value should be set to true if you're a paying customer using the SSL option. If not, you can just leave it off altogether.

The "blinksale invoice id" is NOT the value you see in the "ID" column when looking at a list of invoices. Instead, it's a numeric value assigned when you create an invoice. To find this ID just click on any Blinksale invoice to view it. The URL you are looking at should be something like;

    https://[your account].blinksale.com/invoices/123456

If you're using a free account, or if you don't use https, it will be an http url.

The ID we want is the number at the end - in this case, 123456


Blinksale Invoice XML

Run the test script and you should see a stream of XML printed out, something like this example from the Blinksale API document;

    <?xml version="1.0" encoding="UTF-8"?>
    <invoice xmlns="http://www.blinksale.com/api" uri="http://example.blinksale.com/invoices/1" 
             status="pastdue" subtotal="19.00" total="19.00" paid="2.00" total_due="17.00"
             surplus="0.00" updated_at="2006-09-20T17:27:48Z" created_at="2006-06-27T22:43:13Z">
      <client name="Acme">http://example.blinksale.com/clients/2</client>
      <number>100001</number>
      <po_number>123456</po_number>
      <date>2006-06-27</date>
      <terms due_date="2006-07-12">15</terms>
      <currency>USD</currency>
      <tax amount="0.00">8.75%</tax>
      <freight>0.00</freight>
      <late_fee amount="0.00">0%</late_fee>
      <tags>bob, scott</tags>
      <lines total="882.00">
        <line>
          <name>French Hens</name>
          <quantity>3.0</quantity>
          <units>Product</units>
          <unit_price>19.00</unit_price>
          <taxed>false</taxed>
        </line>
        <line>
          <name>Piper-Piping</name>
          <quantity>11.0</quantity>
          <units>Service</units>
          <unit_price>75.00</unit_price>
          <taxed>false</taxed>
        </line>
      </lines>
      <deliveries uri="http://example.blinksale.com/invoices/1/deliveries">
        <delivery uri="http://example.blinksale.com/invoices/1/deliveries/3" created_at="2006-09-22T23:51:42Z">
          <body>Here's the invoice for the latest work - thanks!</body>
          <recipient name="John Doe" email="john@acme.com">http://example.blinksale.com/clients/2/people/2</recipient>
          <recipient name="Bob Smith" email="bob@example.com">http://example.blinksale.com/users/1</recipient>
        </delivery>
      </deliveries>
      <payments uri="http://example.blinksale.com/invoices/1/payments" total="10.00">
        <payment uri="http://example.blinksale.com/invoices/1/payments/5" created_at="2006-09-25T18:01:33Z">
          <amount>10.00</amount>
          <date>2006-09-25</date>
          <payment_method>Check</method>
          <number>10001</number>
        </payment>
      </payments>
      <notes>Please reference this invoice number in your check memo.</notes>
      <include_payment_link>true</include_payment_link>
    </invoice>

The information I need for the object in my accounts system (in my case, a "Revenue" object) is all buried in there, but there's a lot more detail than I need, and I have to get at the stuff I do want, somehow.


Hpricot

The Hpricot parser is a great way of extracting information from XML (including HTML). If you haven't already, you will need to install it using;

    sudo gem install hpricot -y

(or whatever you do on Windows, if that's your OS)

After a few minutes playing with a parsed version of the downloaded invoice, I can figure out how to get the fields I want. Now, let's expand our test script so that it can pull out the key details;

    #!/usr/bin/env ruby

    require File.dirname(__FILE__) + '/../config/environment'
    require 'hpricot'

    blinksale = Blinksale.new |company|, |user|, |password|, |use_ssl|

    invoice = blinksale.invoices[ |a blinksale invoice id| ]

    data = Hpricot.parse invoice.data

    puts "Invoice Number: %s" % (data/'number')[0].innerHTML
    puts "Date: %s" % (data/'date')[0].innerHTML
    puts "Client: %s" % (data/'client')[0]['name']
    puts "Net amount: %s" % (data/'lines')[0]['total']
    puts "VAT: %s" % (data/'tax')[0]['amount']
    puts "Total: %s" % (data/'invoice')[0]['total']

Integrating with the Rails App

Now that I know how to get an invoice from Blinksale, and what to do with it when I get it, I want to build this capability into my Rails accounts system.

My Revenue model gets a new class method (don't forget to add require 'hpricot' at the top);

    def self.new_from_blinksale( url )

      return nil unless url =~ /http.*\/(\d+)$/
      id = $1

      blinksale = Blinksale.new BLINKSALE[:company], BLINKSALE[:user], BLINKSALE[:password], BLINKSALE[:use_ssl]
      invoice = blinksale.invoices[ id ]
      data = Hpricot.parse invoice.data

      revenue = self.new
      revenue.invoice_number = (data/'number')[0].innerHTML
      revenue.date = (data/'date')[0].innerHTML
      revenue.amount = (data/'lines')[0]['total']
      revenue.vat = (data/'tax')[0]['amount']
      revenue.supplier = (data/'client')[0]['name']
      revenue.description = ((data/'lines')[0]/'name')[0].innerHTML

      # expect that any payment represents payment in full
      if (data/'payment').size > 0
        revenue.paid = true
        revenue.bank_date = ((data/'payment')[0]/'date').innerHTML
      end

      revenue
    rescue Net::HTTPServerException
      logger.error "Failed to fetch blinksale invoice: #{ url }"
      nil
    end

I've defined my Blinksale credentials in a BLINKSALE hash in my config/environment.rb file.

I'm going to import an invoice by pasting the Blinksale invoice url into a form text field, so my class method takes a url as a parameter, extracts the ID with a regular expression and adds the data to a new Revenue object.

Blinksale records the payment of an invoice, so I check to see if there are any payments. In my case, clients only ever pay invoices in a single payment. So, my method assumes that if there are any payments recorded, then the invoice has been paid on the date of the first payment.

My Revenue model has a description field. There is nothing directly equivalent in a Blinksale invoice, so I take the text of the first invoice line and use that as the description.

At this point I can fire up script/console and use my new method to pull in invoice data from Blinksale.


The RevenuesController

Now I want to add this function to the web interface of my accounts system.

Initially, my RevenuesController's "new" method looks like this;

    def new
      @revenue = Revenue.new
      respond_to do |format|
        format.html
        format.xml  { render :xml => @revenue }
      end
    end

I'm going to extend this so that, if a blinksale_url parameter was passed in, the new Revenue object is created via my new class method;

    def new
      @revenue = Revenue.new

      if !params[:blinksale_url].blank?
        if rev = Revenue.new_from_blinksale( params[:blinksale_url] )
          @revenue = rev
        else
          @revenue.errors.add_to_base "Failed to import Blinksale invoice from URL: #{ params[:blinksale_url] }"
        end
      end

      respond_to do |format|
        format.html
        format.xml  { render :xml => @revenue }
      end
    end

If the Blinksale import fails, for whatever reason, I fall back to simply having a new Revenue object, but with an error message that will be displayed by the usual "error_messages_for :revenue" call in my view.


The Revenues View

My app/views/revenues/index.html.erb has a "New Revenue" link like this;

    link_to 'New revenue', new_revenue_path

I want an additional form, also targeting the 'new' method, which passes the 'blinksale_url' parameter;

  <%= form_tag new_revenue_path, :method => :get %>
    URL: <%= text_field_tag 'blinksale_url' %>
    <%= submit_tag "Import from Blinksale" %>
  </form>

And that's it.

Please let me know if you found this post useful, or if you have any other ideas for integration with Blinksale, or any other cool web apps.

Configuring Zabbix

Configuring Zabbix

Zabbix is configured via a PHP web interface. This is either good or bad news, depending on your point of view. Personally, I prefer to configure things with text files. This is mainly because I can look at them, when I come back to the system after some time, and read them to get an idea of what I was trying to do.

Zabbix Concepts

A quick run-through of the basic conceptual model of a zabbix installation is a big help, when you're getting started.

The Zabbix Server

The zabbix server is the heart of a monitoring setup. The zabbix server holds the configuration database and serves the web interface that you use both to configure your monitoring and to see status information and graphs of historical data. The zabbix server fires 'questions' to a zabbix agent running on each monitored host. A question might be "what is your current CPU load?", or "what percentage of /home is free disk space?"

The Zabbix Agent

This runs on all monitored hosts, listening (on port 10050, by default, although you can change that) for questions from the zabbix server, and responding with the answers.

Whatever port the agent is listening on (10050, by default) must be accessible from the zabbix server. So, you may need to open up a hole in your firewall to allow the server to talk to the agent.

Configuration Entities

On the configuration side of things, these are the main entities you need to know about;

• Hosts - these are the servers (or routers, switches or whatever) that you are monitoring. Hosts can be grouped into, surprise surprise, host groups.

• Items - these are the data items or properties that you are monitoring. e.g. free disk space, server load, network traffic or a whole host of other items. Items are grouped into "Applications".

• Triggers - conditions that you care about. e.g. the server load on a particular host is higher than 5, or there is less than 10% disk space free on the home partition. Triggers apply to items, and may be simple or complex thresholds. Triggers have a severity, so less than 10% free disk space could be a "warning", but the web server going down might be a "disaster".
  
  

• Actions - things that you want to happen in response to a trigger switching on or off. e.g. send a warning message to all the users in the "sysadmin" group if any triggers switch on whose severity is "warning" or higher.

• Templates - templates simplify the configuration process by allowing you to define sets of items, triggers and graphs (which we haven't talked about yet), which you can then apply to one or several hosts at the same time. Zabbix comes with some pre-defined templates, such as one to set up typical monitoring of a Linux host, and you can easily define your own.

Setting up monitoring of a host

This is a quick walk-through of setting up monitoring of a single host by applying a pre-defined zabbix template. I'm using Zabbix version 1.4.1, which is the version you currently get if you install zabbix on Ubuntu 7.10 Gutsy Gibbon via apt.


Client configuration

The zabbix agent must be running on the host you want to monitor. On a Fedora, or similar, Linux server, you should be able to install the agent using the "yum" package manager;

   $ sudo yum install zabbix-agent

On an ubuntu or other Debian-type server, use apt;

   $ apt-get install zabbix-agent

The zabbix agent needs very little configuration, but you do need to tell it the zabbix server's IP number. The zabbix agent will only answer questions it receives from this IP.

   $ sudo vi /etc/zabbix/zabbix_agentd.conf

Look for the line "Server=127.0.0.1" and change the IP to that of your zabbix server. If you want the zabbix agent to listen on a non-default port, change the value on the "ListenPort" line and uncomment it.

You should also add the zabbix agent to your system startup, so that it runs on boot;

   $ sudo vi /etc/rc.local

Add this line;

   /etc/init.d/zabbix-agent start

That's it for the client-side configuration. You should check that the agent is running by doing a "ps aux | grep zabbix-agent".

Server Configuration

Now we need to set up the server so that it will periodically ask the host some questions, and show us the answers.

1. Login as an administrator

Login to the web interface of your zabbix server.




The default zabbix login is "admin", with no password. If your zabbix server is exposed to the Internet, or even if it's not, you really should change that.

2. Click on "Configuration" then "Hosts", then "Create Host"




Make sure the selection box next to the "Create Host" button has "Hosts" selected.

Enter the details for your new host. The "Connect to" drop-down is used to tell zabbix how it should try to reach your host when it gathers item data. In my case, I'm using the IP number.

Port 10050 is the default port on which the zabbix agent listens. If you are planning to use a different port, enter it here.

3. Add a template

In the "Link with Template" section, click on the "Add" button.



Check the box next to the default "Template_Linux" template and click the "Select" button.

Now click the "Save" button on the host details page.



You should see a screen like this, showing that a host of monitoring items have been added to your server.

If you click on "Monitoring", "Latest data", and select your server from the drop-downs, you should see the data being gathered from your server.



This is a very basic introduction to setting up monitoring using zabbix. There is a lot more that can be done, and I'm planning to cover some of them in future posts.

Zabbix on Ubuntu 7.10 (Gutsy) Server

A simple walk-through of installing a Zabbix monitoring server on Ubuntu 7.10 server.

Choosing a monitoring system

I've been meaning to upgrade the level of monitoring I do for Admoda. So, I've been looking at monitoring software over the last couple of days. I narrowed down the options to Nagios plus some extras (possibly Groundwork), or Zabbix.

In the end, I decided to go with Zabbix, mainly because a client uses it too, so I'll probably end up writing some custom tests for them.

Zabbix is a fully-featured monitoring system, and it's quite easy, once you get a grasp of the concepts, to do some powerful monitoring. It will also draw graphs showing you how values vary over time,

In this post, I'll go through the (very simple) steps required to install Zabbix on Ubuntu 7.10 server (Gutsy Gibbon).

I did try to install Zabbix 1.4.2 via macports on my Macbook Pro and, although the installation seemed to be successful, I found that none of the popups on the web interface came up. Rather than spend the time to track down the problem and fix it, I decided to go for a clean install on Ubuntu, because I won't be using my Mac as the zabbix server anyway.

In my case, I'm installing on a new virtual machine under VMWare Fusion. Later on, I will move the whole virtual server to my main server, and leave it running there. But, for now, it's easier to do everything on my mac.

Installing on Ubuntu 7.10 server

So, step one is to do a clean install of Ubuntu 7.10 server. When given the choice of the type of services you want to install, select "LAMP" (so that you get apache, php and mysql), and "OpenSSH Server" (assuming you will be connecting to it remotely via SSH - if not, you don't necessarily need this). If you want your zabbix server to send you alerts by email, you should also select "Mail Server", and choose the option to send and receive by SMTP.

When the Ubuntu installation is finished, you should be able to point your web browser to the server's IP number, and see a directory listing of the sites you have configured. Initially, this should show "apache2-default", which will simply show a page saying "It works!" if you click on it.

Now login to the server and execute the following command;

• sudo aptitude install zabbix-server-mysql zabbix-frontend-php

This will install the zabbix server, configured to use mysql as its database, and the PHP frontend gui for zabbix. At the time of writing, the zabbix package available for Ubuntu 7.10 is version 1.4.1. I stuck with this, rather than building the current 1.4.2 version from source, but there is a good walk-through here if you'd prefer to do that.

After this, there are a couple of changes you need to make in the PHP configuration.

• sudo vi /etc/php5/apache2/php.ini

• Set the "date.timezone" value ( in my case, to Europe/London, but you can find a list of values here )
• Set the "max_execution_time" to 300

Restart apache so that it picks up these changes

• sudo /etc/init.d/apache2 restart

Now, visit http://[your server ip]/zabbix with a browser, and go through the configuration steps.

At the end of this process, the GUI lets you download your zabbix.conf.php file using your browser. Save the generated file, transfer it to the server, and then

• sudo mv zabbix.conf.php /usr/share/zabbix/conf/zabbix.conf.php

That's it. You should now be able to login to the zabbix web interface as 'admin' (with no password), and get started.

SvnRepository.com - Part II

So, I just got this response from SvnRepository.com

Hi David,

I apologize for the delayed response. We generally try to have all support
issues resolved within 24 hours. We are working on a way to automate the
importing of dump files into existing repositories, we chose not to initial for
security and stability reasons.

--
Joe Clarke

And, checking my server logs, I can see that someone downloaded the tarball of my SVN dump.

A few minutes later;

Hi David,

Your repository has been imported.

--
Joe Clarke

So, we're up and running after the promised 5 minutes and an additional 24 hours.

In the meantime, I found DevjaVu. Not quite as cheap as I was hoping for, but hey - it's Ninja-Powered!

From the look of their blog, things are moving pretty fast. Also, there seems to be a lot of recent activity on their forums too (are you listening, SvnRepository.com? They have forums (fora?)).

So, if I do end up switching to another SVN host, I think I might drop those ninjas an email.

But, hopefully, SvnRepository.com will be fine, from now on. I hope so - they really are awfully cheap!

Hosted Subversion (SVN) Services

I've been using a dedicated, hosted server to host my subversion repository, so that I always have an off-site backup of my source code. I used to need the server anyway, so this seemed the simplest solution.

But, I really don't need to be paying for a dedicated box anymore when there are so many online services that offer low-cost subversion hosting. So, I did a bit of research and found this thread, among others.

I'm a bit limited in my choices, since the first project I want to migrate is currently using 262MB of disk space by itself, and I want to use the hosted service for multiple projects.

My priorities;

1. Regularly backed up
   
2. Enough disk space
3. Cheap
4. Ability to create additional repositories
5. Reasonable limits on the number of projects and users
6. Add-ons (e.g. Trac, Wikis) are a bonus

So, I decided to try SvnRepository.com Their Level 2 pricing plan is excellent value for money - 2GB of storage, Trac and unlimited repositories and developers for $7/month.

But, Matt Raible was so not kidding about their slow response time.

The first thing I want to do is to migrate my existing repository into my shiny new hosted service. As per their website blurb, the new service was set up in 5 minutes, and they offer "Free Migration services for your Subversion repositories". Unfortunately, that doesn't mean they have a nice system set up for you to migrate your repository into their service - they have an easy way for you to migrate out of it.

I figured a quick posting to their support site would have this sorted out in no time. After all, this has got to be the single most common thing a new customer wants to do, no?

So, I opened a support ticket;

Hi there

I would like to migrate my existing SVN repository into my new, hosted
service.

I've created a tarball via "svnadmin dump". How can I load this into
my new svnrepository.com repository?

Regards

David

Four hours later, I get this response;

David,

Please place the .dump file somewhere that I can download it to our
server, as well as the repository name you wish it to be imported to
and I will take care of it for you.

Please let me know if you have any further questions.

Danny Vigil
SVNrepository.com

OK - four hours is a bit slow, but helpful enough. An hour or so later, I post this response;

Hi Danny

You can download the tarball from this URL;

http://qqqqqqqqq

I'd like it loaded into the repository at;

http://xxxxxxxxxx

Please let me know when you've done this, so that I can stop the web server
that's serving that tarball.

Thanks

David

That was about 12 hours ago. Since then, no response at all, despite chasing them twice.

Now, this is a cheap service - $7/month - so I'm not expecting them to be super-efficient, or to keep a dedicated support person on call to cater to my every whim, despite having "Free 24/7 personal technical support" as one of their standard features, apparently.

But, this is an internet business, providing a service to developers, so I can't understand why;

• They haven't built a web interface to allow developers to upload their old repositories. This has got to be the first thing that most of their customers want. Failing that, at least a tutorial or a faq entry would be something.
  
• They built a system to automate leaving their service instead. Sure, I want to be able to take my repository with me when I leave (which could be really soon), but how does it make business sense to invest your effort in making it easier for customers to leave than to join?
• Their customer support is so incredibly slow. This is the Internet, after all. Taking this long to handle a simple, common request for a new customer just seems unacceptable. This is especially true since, now that I've provided a dump of my SVN repository, there's no point checking anything into the old one, because I'll just have to check it into the new one all over again. So, right now, I'm effectively without version control.
  

So, I'm just venting here, while I'm waiting for them to get me up and running. In the meantime, I think I'll go and re-visit a few alternatives, and remind myself why I chose SvnRepository.com in the first place;

• Unfuddle - a good service, and I use their free plan for one small project. But, you don't get enough storage space for a large project, even if you get one of their more expensive plans.
• CVS Dude - A bit pricy, and you don't get Trac unless you pay $30/month
• SourceHosting.net - very expensive
  
• Wush.net - not too pricy, but only 500MB of storage, and a single repository, unless you stump up more cash.
• DevGuard - Looks good, if only my project were smaller
• AVLUX - pricy
• ProjectLocker.com - really expensive, and their opaque pricing system puts me off
• Code Spaces - Looks pretty good. Not quite as cheap as SvnRepository.com, but not bad.
  

So, the moral of this rant is that, surprise surprise, the cheapest is not always the best option.

I'm going to give SvnRepository.com another day or so to sort me out and, if I'm still not happy, give Code Spaces a try.

Using capistrano with SSH-agent

I develop on my laptop, and deploy to an application server which I connect to via SSH. The laptop and the application server both talk to my subversion repository using urls like "svn+ssh://my.svn.server/repository/project/...", which is on a different server. For security and simplicity, my private key is only stored on my laptop. Only my public SSH key is on the application server.

When I fire up the laptop, I also setup an ssh-agent to which I authenticate using my private key, then I use agent forwarding so that I can ssh to all my servers without re-typing my passphrase, and so that I can checkout code from subversion while I am logged into the application server.

Capistrano uses Net::SSH to provide SSH connectivity. Unfortunately, Net::SSH completely ignores your .ssh/config file, so you have to specify all your servers via resolvable names or IP numbers. In other words, if you have an application server that you refer to as 'wibble' in your .ssh/config, and which points to IP number 123.124.125.126, that won't help capistrano deploy your application.

So, in your config/deploy.rb you will either need to have something like this;

    set :app, "123.124.125.126" 

...which is really ugly. Or, you need to put something in your /etc/hosts file that gives you a nicer name that resolves to that IP number. e.g.;

    123.124.125.126 wibble.mydev.pri

...and then in your config/deploy.rb

    set :app, "wibble.mydev.pri"

Most importantly, you need to have this line in your config/deploy.rb

    set :ssh_options, { :forward_agent => true }

Happy Capping!

Sortable headings

In my application, I want to be able to list records in tables, and have headings that the user can click on to sort the table in ascending or descending order.

• Rather than clicking on the heading to sort and toggle the sort direction, I want 'up' and 'down' links for each column heading.
• I haven't decided what the links are going to look like yet, so I want to use something simple for now, and be able to change them easily later on.
• In true DRY fashion, I want to be able to reuse as much of this code as possible, between my various controllers.

To start with, I need a way to pass a field and a sort direction into my controller. So, my revenues controller's index method looks like this;

  def index
  find_params = { :include => :client }.merge( order_by )
  @revenues = Revenue.find :all, find_params
end

The ' :include => :client ' gives me cheaper access to revenue.client.name, so that I can put the name of the owning client into each table row, without going back to the database every time. It also joins the clients table in the underlying SQL, which allows me to order by 'clients.name'. i.e. I can sort my revenues table by the parent clients' names, rather than the 'client_id' property of the revenue object, which wouldn't be a very useful thing to sort on.

The ' order_by ' is a method that I'm going to re-use in all my controllers, so it goes into controllers/application.rb;

  def order_by
  return {} if params[:order].blank?
  direction = params[:direction] || "ASC"
  { :order => "#{params[:order]} #{direction}" }
end

My column heading links need to send in an 'order' parameter, which is the name of the field we want to order by, and a 'direction' parameter containing either 'asc' or 'desc'.

The column headings, and their associated fields to sort by are;

Date => :invoiced_on
Reference => :reference
Client => 'clients.name'
Net => :amount
VAT => :vat_amount

Before re-factoring, we might want something like this for our Date table header;

<th>

<%= link_to( '^', revenues_path( :order => :invoiced_on, :direction => 'asc' ) ) %>

Date

<%= link_to( 'v', revenues_path( :order => :invoiced_on, :direction => 'desc' ) ) %>

</th>

I'm using RESTful routes, so I can say 'revenues_path', instead of having a ' link_to ( :controller => 'revenues', :action => 'index' )'. But the principle is the same, even if your application isn't RESTful.

But, if I want to reuse my sortable headings code in different controllers, It won't be a link to 'revenues_path'. So, my code needs to be able to take the link destination method as a parameter. Fortunately, ruby makes that very easy with 'proc'. So, here is my view code, using my sortable_headings helper method;

<% link_proc = proc { |params| revenues_path params } -%>
<table id="revenues">
<tr>
  <th> <%= sortable_heading( 'Date', :invoiced_on, link_proc ) %> </th>
  <th> <%= sortable_heading( 'Reference', :reference, link_proc ) %> </th>
  <th> <%= sortable_heading( 'Client', 'clients.name', link_proc ) %> </th>
  <th> <%= sortable_heading( 'Net', :amount, link_proc ) %> </th>
  <th> <%= sortable_heading( 'VAT', :vat_amount, link_proc ) %> </th>
  <th> Gross </th>
  <th>   </th>
</tr>

<%= render :partial => 'revenue', :collection => @revenues %>

</table>

And the 'sortable_heading' method goes in my app/helpers/application_helpers.rb file;

  def sortable_heading( label, field, make_link )
  up_params = { :order => field, :direction => 'asc' }
  down_params = { :order => field, :direction => 'desc' }
  "%s #{label} %s" % [
    link_to( '^', make_link.call( up_params ) ),
    link_to( 'v', make_link.call( down_params ) )
  ]
end

Voila! Bi-directionally sortable column headings, very DRY, and I can easily change the way they look by replacing the '^' and 'v' in my helper with suitable image links, or whatever, later on.