Rails Importing invoices from Blinksale

I use Blinksale for all my invoicing, it's great. I use a Ruby on Rails application, that I wrote myself, to handle my accounts. It's great too. What isn't so great is re-typing my invoices into my accounts application after creating them in Blinksale. That's not even a little bit great, in fact it sucks.

Earlier this year Blinksale released a REST API so I decided to see if I can use that to allow my accounts application to import invoices from Blinksale.

Getting Started

BTW, Blinksale recommend setting up and using a separate login for any API integration.

I'm going to use the ruby library that Blinksale provide, so I need to download the following files to my Rails Application's lib directory;

  • http://www.blinksale.com/api/blinksale.rb

  • http://www.blinksale.com/api/rest_client.rb

  • http://www.blinksale.com/api/xml_node.rb

Talking to Blinksale

Let's start with a quick test script in the application's script directory;

#!/usr/bin/env ruby

require File.dirname(__FILE__) + '/../config/environment'

blinksale = Blinksale.new |company|, |user|, |password|, |use_ssl|

invoice = blinksale.invoices[ |a blinksale invoice id| ]

puts invoice.data

Replace the elements between |pipes| with values for your own Blinksale account. The "use_ssl" value should be set to true if you're a paying customer using the SSL option. If not, you can just leave it off altogether.

The "blinksale invoice id" is NOT the value you see in the "ID" column when looking at a list of invoices. Instead, it's a numeric value assigned when you create an invoice. To find this ID just click on any Blinksale invoice to view it. The URL you are looking at should be something like;

https://[your account].blinksale.com/invoices/123456

If you're using a free account, or if you don't use https, it will be an http url.

The ID we want is the number at the end - in this case, 123456

Blinksale Invoice XML

Run the test script and you should see a stream of XML printed out, something like this example from the Blinksale API document;

<?xml version="1.0" encoding="UTF-8"?>
<invoice xmlns="http://www.blinksale.com/api" uri="http://example.blinksale.com/invoices/1"
status="pastdue" subtotal="19.00" total="19.00" paid="2.00" total_due="17.00"
surplus="0.00" updated_at="2006-09-20T17:27:48Z" created_at="2006-06-27T22:43:13Z">
<client name="Acme">http://example.blinksale.com/clients/2</client>
<terms due_date="2006-07-12">15</terms>
<tax amount="0.00">8.75%</tax>
<late_fee amount="0.00">0%</late_fee>
<tags>bob, scott</tags>
<lines total="882.00">
<name>French Hens</name>
<deliveries uri="http://example.blinksale.com/invoices/1/deliveries">
<delivery uri="http://example.blinksale.com/invoices/1/deliveries/3" created_at="2006-09-22T23:51:42Z">
<body>Here's the invoice for the latest work - thanks!</body>
<recipient name="John Doe" email="john@acme.com">http://example.blinksale.com/clients/2/people/2</recipient>
<recipient name="Bob Smith" email="bob@example.com">http://example.blinksale.com/users/1</recipient>
<payments uri="http://example.blinksale.com/invoices/1/payments" total="10.00">
<payment uri="http://example.blinksale.com/invoices/1/payments/5" created_at="2006-09-25T18:01:33Z">
<notes>Please reference this invoice number in your check memo.</notes>

The information I need for the object in my accounts system (in my case, a "Revenue" object) is all buried in there, but there's a lot more detail than I need, and I have to get at the stuff I do want, somehow.


The Hpricot parser is a great way of extracting information from XML (including HTML). If you haven't already, you will need to install it using;

sudo gem install hpricot -y

(or whatever you do on Windows, if that's your OS)

After a few minutes playing with a parsed version of the downloaded invoice, I can figure out how to get the fields I want. Now, let's expand our test script so that it can pull out the key details;

#!/usr/bin/env ruby

require File.dirname(__FILE__) + '/../config/environment'
require 'hpricot'

blinksale = Blinksale.new |company|, |user|, |password|, |use_ssl|

invoice = blinksale.invoices[ |a blinksale invoice id| ]

data = Hpricot.parse invoice.data

puts "Invoice Number: %s" % (data/'number')[0].innerHTML
puts "Date: %s" % (data/'date')[0].innerHTML
puts "Client: %s" % (data/'client')[0]['name']
puts "Net amount: %s" % (data/'lines')[0]['total']
puts "VAT: %s" % (data/'tax')[0]['amount']
puts "Total: %s" % (data/'invoice')[0]['total']

Integrating with the Rails App

Now that I know how to get an invoice from Blinksale, and what to do with it when I get it, I want to build this capability into my Rails accounts system.

My Revenue model gets a new class method (don't forget to add require 'hpricot' at the top);

def self.new_from_blinksale( url )

return nil unless url =~ /http.*\/(\d+)$/
id = $1

blinksale = Blinksale.new BLINKSALE[:company], BLINKSALE[:user], BLINKSALE[:password], BLINKSALE[:use_ssl]
invoice = blinksale.invoices[ id ]
data = Hpricot.parse invoice.data

revenue = self.new
revenue.invoice_number = (data/'number')[0].innerHTML
revenue.date = (data/'date')[0].innerHTML
revenue.amount = (data/'lines')[0]['total']
revenue.vat = (data/'tax')[0]['amount']
revenue.supplier = (data/'client')[0]['name']
revenue.description = ((data/'lines')[0]/'name')[0].innerHTML

# expect that any payment represents payment in full
if (data/'payment').size > 0
revenue.paid = true
revenue.bank_date = ((data/'payment')[0]/'date').innerHTML

rescue Net::HTTPServerException
logger.error "Failed to fetch blinksale invoice: #{ url }"

I've defined my Blinksale credentials in a BLINKSALE hash in my config/environment.rb file.

I'm going to import an invoice by pasting the Blinksale invoice url into a form text field, so my class method takes a url as a parameter, extracts the ID with a regular expression and adds the data to a new Revenue object.

Blinksale records the payment of an invoice, so I check to see if there are any payments. In my case, clients only ever pay invoices in a single payment. So, my method assumes that if there are any payments recorded, then the invoice has been paid on the date of the first payment.

My Revenue model has a description field. There is nothing directly equivalent in a Blinksale invoice, so I take the text of the first invoice line and use that as the description.

At this point I can fire up script/console and use my new method to pull in invoice data from Blinksale.

The RevenuesController

Now I want to add this function to the web interface of my accounts system.

Initially, my RevenuesController's "new" method looks like this;

def new
@revenue = Revenue.new
respond_to do |format|
format.xml { render :xml => @revenue }

I'm going to extend this so that, if a blinksale_url parameter was passed in, the new Revenue object is created via my new class method;

def new
@revenue = Revenue.new

if !params[:blinksale_url].blank?
if rev = Revenue.new_from_blinksale( params[:blinksale_url] )
@revenue = rev
@revenue.errors.add_to_base "Failed to import Blinksale invoice from URL: #{ params[:blinksale_url] }"

respond_to do |format|
format.xml { render :xml => @revenue }

If the Blinksale import fails, for whatever reason, I fall back to simply having a new Revenue object, but with an error message that will be displayed by the usual "error_messages_for :revenue" call in my view.

The Revenues View

My app/views/revenues/index.html.erb has a "New Revenue" link like this;

link_to 'New revenue', new_revenue_path

I want an additional form, also targeting the 'new' method, which passes the 'blinksale_url' parameter;

<%= form_tag new_revenue_path, :method => :get %>
URL: <%= text_field_tag 'blinksale_url' %>
<%= submit_tag "Import from Blinksale" %>

And that's it.

Please let me know if you found this post useful, or if you have any other ideas for integration with Blinksale, or any other cool web apps.


Configuring Zabbix

Configuring Zabbix

Zabbix is configured via a PHP web interface. This is either good or bad news, depending on your point of view. Personally, I prefer to configure things with text files. This is mainly because I can look at them, when I come back to the system after some time, and read them to get an idea of what I was trying to do.

Zabbix Concepts

A quick run-through of the basic conceptual model of a zabbix installation is a big help, when you're getting started.

The Zabbix Server

The zabbix server is the heart of a monitoring setup. The zabbix server holds the configuration database and serves the web interface that you use both to configure your monitoring and to see status information and graphs of historical data. The zabbix server fires 'questions' to a zabbix agent running on each monitored host. A question might be "what is your current CPU load?", or "what percentage of /home is free disk space?"

The Zabbix Agent

This runs on all monitored hosts, listening (on port 10050, by default, although you can change that) for questions from the zabbix server, and responding with the answers.

Whatever port the agent is listening on (10050, by default) must be accessible from the zabbix server. So, you may need to open up a hole in your firewall to allow the server to talk to the agent.

Configuration Entities

On the configuration side of things, these are the main entities you need to know about;

  • Hosts - these are the servers (or routers, switches or whatever) that you are monitoring. Hosts can be grouped into, surprise surprise, host groups.
  • Items - these are the data items or properties that you are monitoring. e.g. free disk space, server load, network traffic or a whole host of other items. Items are grouped into "Applications".
  • Triggers - conditions that you care about. e.g. the server load on a particular host is higher than 5, or there is less than 10% disk space free on the home partition. Triggers apply to items, and may be simple or complex thresholds. Triggers have a severity, so less than 10% free disk space could be a "warning", but the web server going down might be a "disaster".

  • Actions - things that you want to happen in response to a trigger switching on or off. e.g. send a warning message to all the users in the "sysadmin" group if any triggers switch on whose severity is "warning" or higher.
  • Templates - templates simplify the configuration process by allowing you to define sets of items, triggers and graphs (which we haven't talked about yet), which you can then apply to one or several hosts at the same time. Zabbix comes with some pre-defined templates, such as one to set up typical monitoring of a Linux host, and you can easily define your own.

Setting up monitoring of a host

This is a quick walk-through of setting up monitoring of a single host by applying a pre-defined zabbix template. I'm using Zabbix version 1.4.1, which is the version you currently get if you install zabbix on Ubuntu 7.10 Gutsy Gibbon via apt.

Client configuration

The zabbix agent must be running on the host you want to monitor. On a Fedora, or similar, Linux server, you should be able to install the agent using the "yum" package manager;

$ sudo yum install zabbix-agent

On an ubuntu or other Debian-type server, use apt;

$ apt-get install zabbix-agent

The zabbix agent needs very little configuration, but you do need to tell it the zabbix server's IP number. The zabbix agent will only answer questions it receives from this IP.

$ sudo vi /etc/zabbix/zabbix_agentd.conf

Look for the line "Server=" and change the IP to that of your zabbix server. If you want the zabbix agent to listen on a non-default port, change the value on the "ListenPort" line and uncomment it.

You should also add the zabbix agent to your system startup, so that it runs on boot;

$ sudo vi /etc/rc.local

Add this line;

/etc/init.d/zabbix-agent start

That's it for the client-side configuration. You should check that the agent is running by doing a "ps aux | grep zabbix-agent".

Server Configuration

Now we need to set up the server so that it will periodically ask the host some questions, and show us the answers.

1. Login as an administrator

Login to the web interface of your zabbix server.

The default zabbix login is "admin", with no password. If your zabbix server is exposed to the Internet, or even if it's not, you really should change that.

2. Click on "Configuration" then "Hosts", then "Create Host"

Make sure the selection box next to the "Create Host" button has "Hosts" selected.

Enter the details for your new host. The "Connect to" drop-down is used to tell zabbix how it should try to reach your host when it gathers item data. In my case, I'm using the IP number.

Port 10050 is the default port on which the zabbix agent listens. If you are planning to use a different port, enter it here.

3. Add a template

In the "Link with Template" section, click on the "Add" button.

Check the box next to the default "Template_Linux" template and click the "Select" button.

Now click the "Save" button on the host details page.

You should see a screen like this, showing that a host of monitoring items have been added to your server.

If you click on "Monitoring", "Latest data", and select your server from the drop-downs, you should see the data being gathered from your server.

This is a very basic introduction to setting up monitoring using zabbix. There is a lot more that can be done, and I'm planning to cover some of them in future posts.


Zabbix on Ubuntu 7.10 (Gutsy) Server

A simple walk-through of installing a Zabbix monitoring server on Ubuntu 7.10 server.

Choosing a monitoring system

I've been meaning to upgrade the level of monitoring I do for Admoda. So, I've been looking at monitoring software over the last couple of days. I narrowed down the options to Nagios plus some extras (possibly Groundwork), or Zabbix.

In the end, I decided to go with Zabbix, mainly because a client uses it too, so I'll probably end up writing some custom tests for them.

Zabbix is a fully-featured monitoring system, and it's quite easy, once you get a grasp of the concepts, to do some powerful monitoring. It will also draw graphs showing you how values vary over time,

In this post, I'll go through the (very simple) steps required to install Zabbix on Ubuntu 7.10 server (Gutsy Gibbon).

I did try to install Zabbix 1.4.2 via macports on my Macbook Pro and, although the installation seemed to be successful, I found that none of the popups on the web interface came up. Rather than spend the time to track down the problem and fix it, I decided to go for a clean install on Ubuntu, because I won't be using my Mac as the zabbix server anyway.

In my case, I'm installing on a new virtual machine under VMWare Fusion. Later on, I will move the whole virtual server to my main server, and leave it running there. But, for now, it's easier to do everything on my mac.

Installing on Ubuntu 7.10 server

So, step one is to do a clean install of Ubuntu 7.10 server. When given the choice of the type of services you want to install, select "LAMP" (so that you get apache, php and mysql), and "OpenSSH Server" (assuming you will be connecting to it remotely via SSH - if not, you don't necessarily need this). If you want your zabbix server to send you alerts by email, you should also select "Mail Server", and choose the option to send and receive by SMTP.

When the Ubuntu installation is finished, you should be able to point your web browser to the server's IP number, and see a directory listing of the sites you have configured. Initially, this should show "apache2-default", which will simply show a page saying "It works!" if you click on it.

Now login to the server and execute the following command;

  • sudo aptitude install zabbix-server-mysql zabbix-frontend-php

This will install the zabbix server, configured to use mysql as its database, and the PHP frontend gui for zabbix. At the time of writing, the zabbix package available for Ubuntu 7.10 is version 1.4.1. I stuck with this, rather than building the current 1.4.2 version from source, but there is a good walk-through here if you'd prefer to do that.

After this, there are a couple of changes you need to make in the PHP configuration.

  • sudo vi /etc/php5/apache2/php.ini
  • Set the "date.timezone" value ( in my case, to Europe/London, but you can find a list of values here )
  • Set the "max_execution_time" to 300

Restart apache so that it picks up these changes

  • sudo /etc/init.d/apache2 restart

Now, visit http://[your server ip]/zabbix with a browser, and go through the configuration steps.

At the end of this process, the GUI lets you download your zabbix.conf.php file using your browser. Save the generated file, transfer it to the server, and then

  • sudo mv zabbix.conf.php /usr/share/zabbix/conf/zabbix.conf.php

That's it. You should now be able to login to the zabbix web interface as 'admin' (with no password), and get started.


SvnRepository.com - Part II

So, I just got this response from SvnRepository.com

Hi David,

I apologize for the delayed response. We generally try to have all support
issues resolved within 24 hours. We are working on a way to automate the
importing of dump files into existing repositories, we chose not to initial for
security and stability reasons.

Joe Clarke

And, checking my server logs, I can see that someone downloaded the tarball of my SVN dump.

A few minutes later;

Hi David,

Your repository has been imported.

Joe Clarke

So, we're up and running after the promised 5 minutes and an additional 24 hours.

In the meantime, I found DevjaVu. Not quite as cheap as I was hoping for, but hey - it's Ninja-Powered!

From the look of their blog, things are moving pretty fast. Also, there seems to be a lot of recent activity on their forums too (are you listening, SvnRepository.com? They have forums (fora?)).

So, if I do end up switching to another SVN host, I think I might drop those ninjas an email.

But, hopefully, SvnRepository.com will be fine, from now on. I hope so - they really are awfully cheap!

Hosted Subversion (SVN) Services

I've been using a dedicated, hosted server to host my subversion repository, so that I always have an off-site backup of my source code. I used to need the server anyway, so this seemed the simplest solution.

But, I really don't need to be paying for a dedicated box anymore when there are so many online services that offer low-cost subversion hosting. So, I did a bit of research and found this thread, among others.

I'm a bit limited in my choices, since the first project I want to migrate is currently using 262MB of disk space by itself, and I want to use the hosted service for multiple projects.

My priorities;

  1. Regularly backed up
  2. Enough disk space
  3. Cheap
  4. Ability to create additional repositories
  5. Reasonable limits on the number of projects and users
  6. Add-ons (e.g. Trac, Wikis) are a bonus

So, I decided to try SvnRepository.com Their Level 2 pricing plan is excellent value for money - 2GB of storage, Trac and unlimited repositories and developers for $7/month.

But, Matt Raible was so not kidding about their slow response time.

The first thing I want to do is to migrate my existing repository into my shiny new hosted service. As per their website blurb, the new service was set up in 5 minutes, and they offer "Free Migration services for your Subversion repositories". Unfortunately, that doesn't mean they have a nice system set up for you to migrate your repository into their service - they have an easy way for you to migrate out of it.

I figured a quick posting to their support site would have this sorted out in no time. After all, this has got to be the single most common thing a new customer wants to do, no?

So, I opened a support ticket;

Hi there

I would like to migrate my existing SVN repository into my new, hosted

I've created a tarball via "svnadmin dump". How can I load this into
my new svnrepository.com repository?



Four hours later, I get this response;


Please place the .dump file somewhere that I can download it to our
server, as well as the repository name you wish it to be imported to
and I will take care of it for you.

Please let me know if you have any further questions.

Danny Vigil

OK - four hours is a bit slow, but helpful enough. An hour or so later, I post this response;

Hi Danny

You can download the tarball from this URL;


I'd like it loaded into the repository at;


Please let me know when you've done this, so that I can stop the web server
that's serving that tarball.



That was about 12 hours ago. Since then, no response at all, despite chasing them twice.

Now, this is a cheap service - $7/month - so I'm not expecting them to be super-efficient, or to keep a dedicated support person on call to cater to my every whim, despite having "Free 24/7 personal technical support" as one of their standard features, apparently.

But, this is an internet business, providing a service to developers, so I can't understand why;

  • They haven't built a web interface to allow developers to upload their old repositories. This has got to be the first thing that most of their customers want. Failing that, at least a tutorial or a faq entry would be something.
  • They built a system to automate leaving their service instead. Sure, I want to be able to take my repository with me when I leave (which could be really soon), but how does it make business sense to invest your effort in making it easier for customers to leave than to join?
  • Their customer support is so incredibly slow. This is the Internet, after all. Taking this long to handle a simple, common request for a new customer just seems unacceptable. This is especially true since, now that I've provided a dump of my SVN repository, there's no point checking anything into the old one, because I'll just have to check it into the new one all over again. So, right now, I'm effectively without version control.

So, I'm just venting here, while I'm waiting for them to get me up and running. In the meantime, I think I'll go and re-visit a few alternatives, and remind myself why I chose SvnRepository.com in the first place;

  • Unfuddle - a good service, and I use their free plan for one small project. But, you don't get enough storage space for a large project, even if you get one of their more expensive plans.
  • CVS Dude - A bit pricy, and you don't get Trac unless you pay $30/month
  • SourceHosting.net - very expensive
  • Wush.net - not too pricy, but only 500MB of storage, and a single repository, unless you stump up more cash.
  • DevGuard - Looks good, if only my project were smaller
  • AVLUX - pricy
  • ProjectLocker.com - really expensive, and their opaque pricing system puts me off
  • Code Spaces - Looks pretty good. Not quite as cheap as SvnRepository.com, but not bad.
So, the moral of this rant is that, surprise surprise, the cheapest is not always the best option.

I'm going to give SvnRepository.com another day or so to sort me out and, if I'm still not happy, give Code Spaces a try.


Using capistrano with SSH-agent

I develop on my laptop, and deploy to an application server which I connect to via SSH. The laptop and the application server both talk to my subversion repository using urls like "svn+ssh://my.svn.server/repository/project/...", which is on a different server. For security and simplicity, my private key is only stored on my laptop. Only my public SSH key is on the application server.

When I fire up the laptop, I also setup an ssh-agent to which I authenticate using my private key, then I use agent forwarding so that I can ssh to all my servers without re-typing my passphrase, and so that I can checkout code from subversion while I am logged into the application server.

Capistrano uses Net::SSH to provide SSH connectivity. Unfortunately, Net::SSH completely ignores your .ssh/config file, so you have to specify all your servers via resolvable names or IP numbers. In other words, if you have an application server that you refer to as 'wibble' in your .ssh/config, and which points to IP number, that won't help capistrano deploy your application.

So, in your config/deploy.rb you will either need to have something like this;

    set :app, "" 
...which is really ugly. Or, you need to put something in your /etc/hosts file that gives you a nicer name that resolves to that IP number. e.g.; wibble.mydev.pri
...and then in your config/deploy.rb
    set :app, "wibble.mydev.pri"
Most importantly, you need to have this line in your config/deploy.rb
    set :ssh_options, { :forward_agent => true }
Happy Capping!


Sortable headings

In my application, I want to be able to list records in tables, and have headings that the user can click on to sort the table in ascending or descending order.

  • Rather than clicking on the heading to sort and toggle the sort direction, I want 'up' and 'down' links for each column heading.
  • I haven't decided what the links are going to look like yet, so I want to use something simple for now, and be able to change them easily later on.
  • In true DRY fashion, I want to be able to reuse as much of this code as possible, between my various controllers.
To start with, I need a way to pass a field and a sort direction into my controller. So, my revenues controller's index method looks like this;

  def index
find_params = { :include => :client }.merge( order_by )
@revenues = Revenue.find :all, find_params

The ' :include => :client ' gives me cheaper access to revenue.client.name, so that I can put the name of the owning client into each table row, without going back to the database every time. It also joins the clients table in the underlying SQL, which allows me to order by 'clients.name'. i.e. I can sort my revenues table by the parent clients' names, rather than the 'client_id' property of the revenue object, which wouldn't be a very useful thing to sort on.

The ' order_by ' is a method that I'm going to re-use in all my controllers, so it goes into controllers/application.rb;

  def order_by
return {} if params[:order].blank?
direction = params[:direction] || "ASC"
{ :order => "#{params[:order]} #{direction}" }

My column heading links need to send in an 'order' parameter, which is the name of the field we want to order by, and a 'direction' parameter containing either 'asc' or 'desc'.

The column headings, and their associated fields to sort by are;

Date => :invoiced_on
Reference => :reference
Client => 'clients.name'
Net => :amount
VAT => :vat_amount

Before re-factoring, we might want something like this for our Date table header;


<%= link_to( '^', revenues_path( :order => :invoiced_on, :direction => 'asc' ) ) %>


<%= link_to( 'v', revenues_path( :order => :invoiced_on, :direction => 'desc' ) ) %>


I'm using RESTful routes, so I can say 'revenues_path', instead of having a ' link_to ( :controller => 'revenues', :action => 'index' )'. But the principle is the same, even if your application isn't RESTful.

But, if I want to reuse my sortable headings code in different controllers, It won't be a link to 'revenues_path'. So, my code needs to be able to take the link destination method as a parameter. Fortunately, ruby makes that very easy with 'proc'. So, here is my view code, using my sortable_headings helper method;

<% link_proc = proc { |params| revenues_path params } -%>
<table id="revenues">
<th> <%= sortable_heading( 'Date', :invoiced_on, link_proc ) %> </th>
<th> <%= sortable_heading( 'Reference', :reference, link_proc ) %> </th>
<th> <%= sortable_heading( 'Client', 'clients.name', link_proc ) %> </th>
<th> <%= sortable_heading( 'Net', :amount, link_proc ) %> </th>
<th> <%= sortable_heading( 'VAT', :vat_amount, link_proc ) %> </th>
<th> Gross </th>
<th> </th>

<%= render :partial => 'revenue', :collection => @revenues %>


And the 'sortable_heading' method goes in my app/helpers/application_helpers.rb file;

  def sortable_heading( label, field, make_link )
up_params = { :order => field, :direction => 'asc' }
down_params = { :order => field, :direction => 'desc' }
"%s #{label} %s" % [
link_to( '^', make_link.call( up_params ) ),
link_to( 'v', make_link.call( down_params ) )

Voila! Bi-directionally sortable column headings, very DRY, and I can easily change the way they look by replacing the '^' and 'v' in my helper with suitable image links, or whatever, later on.