2006/12/02

Test coverage for multiple projects


I've started using rcov to get a view of just how many tests I forgot to write the first time. It's fantastic, and gives a beautiful HTML stats page with a visual representation of how well (or not) your tests exercise your code.




gem install rcov

rcov -t test/*/*_test.rb



Creates a "coverage/index.html" which looks like this;


But, I've got a lot of Ruby on Rails projects. So, I wanted the same sort of summary "one level up". What I really want is exactly the same sort of page, but showing the "TOTAL" line from each project, with a link to each project so that I can drill down. There may be a way to get rcov to do this automatically, but I'm too dumb to figure it out. So, I decided to write my own.

I want to use the same look and feel for my summary page, so the "coverage/index.html" seems like a good place to start. A quick look inside shows that the meat of the page is a bunch of table rows inside tbody tags. So, if we cut those out and replace them with an Erb tag, we have a perfect template to use for our summary page;





template = <<-TEMPLATE
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

...snip...

<thead>
<tr>
<td class='heading'>Name</td>
<td class='heading'>Total lines</td>
<td class='heading'>Lines of code</td>
<td class='heading'>Total coverage</td>
<td class='heading'>Code coverage</td>
</tr>
</thead>
<tbody>

<%= approws %>

</tbody>

...snip...

TEMPLATE




We'll fill in the 'approws' tag with a collection of rows, where each row has summary information for a project.



Since the rows will look very much like the table rows in index.html, we can use one of those as a template too;






<tr class='light'>
<td>TOTAL</td>
<td class='value'>
<tt>839</tt>
</td>
<td class='value'>
<tt>553</tt>
</td>
<td>
<table cellspacing='0' cellpadding='0' align='right'>
<tr>
<td>
<tt>53.2%</tt> </td>
<td>
<table cellspacing='0' class='percent_graph' cellpadding='0' width='100'>
<tr>
<td class='covered' width='53' />
<td class='uncovered' width='47' />
</tr>
</table>
</td>
</tr>
</table>
</td>
<td>
<table cellspacing='0' cellpadding='0' align='right'>
<tr>
<td>
<tt>30.2%</tt> </td>
<td>
<table cellspacing='0' class='percent_graph' cellpadding='0' width='100'>
<tr>
<td class='covered' width='30' />
<td class='uncovered' width='70' />
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>





Turning that into a simple template, we get this;



apptemplate = <<-APPSECTION

<tr class='<%= rowcolour %>'>
<td>
<a href='<%= appname %>/coverage/index.html'> <%= appname %> </a>
</td>
<td class='value'>
<tt><%= summary[:lines] %></tt>
</td>
<td class='value'>
<tt><%= summary[:codelines] %></tt>
</td>
<td>
<table cellspacing='0' cellpadding='0' align='right'>
<tr>
<td>
<tt><%= summary[:total_coverage] %></tt> </td>
<td>
<table cellspacing='0' class='percent_graph' cellpadding='0' width='100'>
<tr>
<td class='covered' width='<%= summary[:total_coverage].to_i %>' />
<td class='uncovered' width='<%= 100 - summary[:total_coverage].to_i %>' />
</tr>
</table>
</td>
</tr>
</table>
</td>
<td>
<table cellspacing='0' cellpadding='0' align='right'>
<tr>
<td>
<tt><%= summary[:code_coverage] %></tt> </td>
<td>
<table cellspacing='0' class='percent_graph' cellpadding='0' width='100'>
<tr>
<td class='covered' width='<%= summary[:code_coverage].to_i %>' />
<td class='uncovered' width='<%= 100 - summary[:code_coverage].to_i %>' />
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>

APPSECTION





So, we need to pass in 'rowcolour' to get the alternating light/dark rows, 'appname' - the name of the project - which is used to find the location of the coverage/index.html page for this particular project, and a summary hash with the linecount, code-linecount and test coverage stats.


Here's what we've got so far;






#!/usr/bin/env ruby

require 'erb'

apptemplate = <<-APPSECTION
... snip ...
APPSECTION

template = <<-TEMPLATE
... snip ...
TEMPLATE

#######################################################

appsection = ERB.new apptemplate

rowcolour = 'dark'

approws = ''

project_directories.map { |appname|

index = "%s/coverage/index.html" % appname

if File.exist?(index)
summary = parse( index )
rowcolour = (rowcolour == 'light') ? 'dark' : 'light'
approws += appsection.result( binding )
end

}

render = ERB.new template

puts render.result( binding )




Iterate through all the project directories, and if you find a coverage/index.html file, parse it to get the summary data for that project. Then, when a suitable appname, rowcolour and summary are all in scope, create an HTML table row for this application using the appsection

erb template.

When all the rows are in a variable called approws, render the whole template.

So, we need a simple list of our project directories which, in my case, is just every directory below the current one;




def project_directories
directories = Dir.entries(".").collect { |entry|
entry if File.directory?(entry)
}
directories.compact!
end




Finally, we need to parse the coverage/index.html file to get the summary information.




def parse( index_html )

stats = []

f = File.open( index_html )
while line = f.gets do
if line =~ /TOTAL/ .. line =~ /href/
if line =~ %r[<tt>(\d+)</tt>]
stats << $1
end

if line =~ %r[<tt>(.*)%</tt>]
stats << $1
end
end
end
f.close

{
:lines => stats[0],
:codelines => stats[1],
:total_coverage => stats[2],
:code_coverage => stats[3],
}
end




In other words, look at all lines from the one containing TOTAL to the next href tag. Grab the first two integers and the first two percentages you find inside <tt> tags. Those are, in order, the total lines, total lines of code, total test coverage percentage and code coverage percentage. So, return those as a suitable hash.
That's it. Run...


./coverage.rb > index.html

...in the directory above all your Rails projects, and you get a summary page that looks like this;



Except, of course, it won't because you'll have far more tests than I do.

Happy testing.

David





Digg!



Here is the full listing



coverage.rb







#!/usr/bin/env ruby

require 'erb'

apptemplate = <<-APPSECTION

<tr class='<%= rowcolour %>'>
<td>
<a href='<%= appname %>/coverage/index.html'> <%= appname %> </a>
</td>
<td class='value'>
<tt><%= summary[:lines] %></tt>
</td>
<td class='value'>
<tt><%= summary[:codelines] %></tt>
</td>
<td>
<table cellspacing='0' cellpadding='0' align='right'>
<tr>
<td>
<tt><%= summary[:total_coverage] %></tt> </td>
<td>
<table cellspacing='0' class='percent_graph' cellpadding='0' width='100'>
<tr>
<td class='covered' width='<%= summary[:total_coverage].to_i %>' />
<td class='uncovered' width='<%= 100 - summary[:total_coverage].to_i %>' />
</tr>
</table>
</td>
</tr>
</table>
</td>
<td>
<table cellspacing='0' cellpadding='0' align='right'>
<tr>
<td>
<tt><%= summary[:code_coverage] %></tt> </td>
<td>
<table cellspacing='0' class='percent_graph' cellpadding='0' width='100'>
<tr>
<td class='covered' width='<%= summary[:code_coverage].to_i %>' />
<td class='uncovered' width='<%= 100 - summary[:code_coverage].to_i %>' />
</tr>
</table>
</td>
</tr>
</table>
</td>
</tr>

APPSECTION

template = <<-TEMPLATE
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html lang='en' xml:lang='en' xmlns='http://www.w3.org/1999/xhtml'>
<head>
<title>Code coverage information</title>
<style type='text/css'>body { background-color: rgb(240, 240, 245); }</style>
<style type='text/css'>span.marked0 {
background-color: rgb(185, 210, 200);
display: block;
}
span.marked1 {
background-color: rgb(190, 215, 205);
display: block;
}
span.inferred0 {
background-color: rgb(175, 200, 200);
display: block;
}
span.inferred1 {
background-color: rgb(180, 205, 205);
display: block;
}
span.uncovered0 {
background-color: rgb(225, 110, 110);
display: block;
}
span.uncovered1 {
background-color: rgb(235, 120, 120);
display: block;
}
span.overview {
border-bottom: 8px solid black;
}
div.overview {
border-bottom: 8px solid black;
}
body {
font-family: verdana, arial, helvetica;
}
div.footer {
font-size: 68%;
margin-top: 1.5em;
}
h1, h2, h3, h4, h5, h6 {
margin-bottom: 0.5em;
}
h5 {
margin-top: 0.5em;
}
.hidden {
display: none;
}
div.separator {
height: 10px;
}
/* Commented out for better readability, esp. on IE */
/*
table tr td, table tr th {
font-size: 68%;
}
td.value table tr td {
font-size: 11px;
}
*/
table.percent_graph {
height: 12px;
border: #808080 1px solid;
empty-cells: show;
}
table.percent_graph td.covered {
height: 10px;
background: #00f000;
}
table.percent_graph td.uncovered {
height: 10px;
background: #e00000;
}
table.percent_graph td.NA {
height: 10px;
background: #eaeaea;
}
table.report {
border-collapse: collapse;
width: 100%;
}
table.report td.heading {
background: #dcecff;
border: #d0d0d0 1px solid;
font-weight: bold;
text-align: center;
}
table.report td.heading:hover {
background: #c0ffc0;
}
table.report td.text {
border: #d0d0d0 1px solid;
}
table.report td.value {
text-align: right;
border: #d0d0d0 1px solid;
}
table.report tr.light {
background-color: rgb(240, 240, 245);
}
table.report tr.dark {
background-color: rgb(230, 230, 235);
}
</style>
</head>
<body>
<h3>Code coverage information</h3>
<p>Generated on Fri Dec 01 13:53:57 GMT 2006 with <a href='http://eigenclass.org/hiki.rb?rcov'>rcov 0.5.0</a>
</p>
<hr /> <table class='report'>
<thead>
<tr>
<td class='heading'>Name</td>
<td class='heading'>Total lines</td>
<td class='heading'>Lines of code</td>
<td class='heading'>Total coverage</td>
<td class='heading'>Code coverage</td>
</tr>
</thead>
<tbody>

<%= approws %>

</tbody>
</table><hr /> <p>Generated using the <a href='http://eigenclass.org/hiki.rb?rcov'>rcov code coverage analysis tool for Ruby</a> version 0.5.0.</p><p>
<a href='http://validator.w3.org/check/referer'>
<img src='http://www.w3.org/Icons/valid-xhtml11' height='31' alt='Valid XHTML 1.1!' width='88' />
</a>
<a href='http://jigsaw.w3.org/css-validator/check/referer'>
<img src='http://jigsaw.w3.org/css-validator/images/vcss' alt='Valid CSS!' style='border:0;width:88px;height:31px' />
</a>
</p>
</body>
</html>
TEMPLATE


def project_directories
directories = Dir.entries(".").collect { |entry|
entry if File.directory?(entry)
}
directories.compact!
end

def parse( index_html )

stats = []

f = File.open( index_html )
while line = f.gets do
if line =~ /TOTAL/ .. line =~ /href/
if line =~ %r[<tt>(\d+)</tt>]
stats << $1
end

if line =~ %r[<tt>(.*)%</tt>]
stats << $1
end
end
end
f.close

{
:lines => stats[0],
:codelines => stats[1],
:total_coverage => stats[2],
:code_coverage => stats[3],
}
end

#######################################################

appsection = ERB.new apptemplate

rowcolour = 'dark'

approws = ''

project_directories.map { |appname|
index = "%s/coverage/index.html" % appname
if File.exist?(index)
summary = parse( index )
rowcolour = (rowcolour == 'light') ? 'dark' : 'light'
approws += appsection.result( binding )
end

# Flag up projects whose tests aren't running yet
if File.exist?( "%s/skip.nightly.tests" % appname )
summary = {
:lines => 0,
:codelines => 0,
:total_coverage => 0,
:code_coverage => 0,
}
rowcolour = (rowcolour == 'light') ? 'dark' : 'light'
approws += appsection.result( binding )
end

}

render = ERB.new template

puts render.result( binding )