Saturday, June 22, 2013

Rakefiles and Generated Sources

A quick note on something that confused me.  I was working on a Rakefile for a project that has generated sources.  Generated sources aren't that unusual.  For example, people will often write a gemspec file from their Rakefile.

In my case, it started with a generated source file that needed to be added to my gemspec.  I had done the usual "file = Dir["**/*/*.rb"] and assumed that the file would be picked up when the :gem task was run. In other words, when the gem task is finally called it should figure out what should be included.  Yet it didn't do that.  Why?

My code to fill in the gemspec is an anonymous function (aka do-block) that is getting passed into the constructor for the gem task.  That function is being evaluated when the object is being constructed.  As a result, my generated sources didn't exist and thus weren't making it into the gem.

Rakefile Snippets

First, I explicitly constructed the list of generated sources:
GSRC_VERSION = File.join('lib', 'amatchpp', 'version.rb') GSRC_PROTODOC = "protodoc.rb" GSRCS = [GSRC_VERSION, GSRC_PROTODOC]

Then I construct a file-rule to construct a file
file GSRC_VERSION => "Rakefile" do desc m = "Writing version.rb information for #{PKG_VERSION}" puts m File.open(GSRC_VERSION, 'w') do |v| v.puts version_text end end
I add a task which depends on the file.
task :version => GSRC_VERSION

I have a :generated task that depends on all constructed sources.
task :generated => GSRCS

I add the :generated task as a dependency to other tasks as needed. For example, I add it to the :doc task as show below.
############################################################ # Documentation ############################################################ RDoc::Task.new(:doc) do |rd| rd.main = "README" rd.title = "#{PKG_NAME} - Enhanced Approximate Matching" rd.rdoc_files.include( PKG_DOC_FILES) rd.rdoc_dir = "doc" end task :doc => :generated

Wednesday, June 12, 2013

Ruby Project Directory Organization

NOTE: This post will be updated from time-to-time as I learn more.  Right now, it is collection of tidbits gathered together.  There is some conflicting information between a Rails project and a Ruby project.

TODO:  References

Ruby Project

A good Ruby project is organized into one or more modules.  A module's component files are placed into the project hierarchy by what they do.

project
  lib
    module1
      class_a.rb
      class_b.rb
      ...
    module1.rb

  test
    module1
      class_a_test.rb
      class_b_test.rb

We see that a module's files aren't kept together.  Copying a module from project-to-project will involve copy multiple directories and a file or two.

The motivation behind this organization appears to be search path and file name globbing.  One usually runs Ruby with the "-Ilib" command line option set.  A require statement for a module is:

    require 'module1'

Ruby will then locate and identify the code file as:

    lib/module1.rb

The file module1.rb will either be a complete implementation or simply a set of require statements:

    require 'module1/class_a'
    require 'module1/class_b'

Globbing is important as it allows other utilities to act on the appropriate subset of files.  For example, RDoc should be run on all Ruby source files under the lib/ directory but we don't want it to publicly document our test code.  Test runners are similar.

Files are organized into the directory structure for easy and consistent globbing by scripts and other utilities.

Sunday, June 9, 2013

Distributed your Git repository to RubyForge (Linux)


There are some notes on the RubyForge site but it wasn't obvious to someone has doesn't live and breath SCM.  I found the following useful:
NOTE:  If you are already using GitHub, you probably already have a ~/.ssh/id_rsa.pub key.  That didn't work.  My suspicion is that RubyForge wants your account name in the key, not your e-mail address.

Part 1: Make a RubyForge key
ssh-keygen -t rsa -C your-account-name -f id_rubyforge
Part 2: Add the key to your account
  • You add the key to your account using the high level "My Account"
    • edit keys is at the bottom of the screen
  • I use xclip to copy the contents of a file to the clipboard:
xclip ~/.ssh/id_rubyforge.pub
Part 3:  Add a rubyforge.org section to your ~/.ssh/config file

Host rubyforge.org
     user your-account-name
     IdentityFile ~/.ssh/id_rubyforge.pub
Part 4: Verify that sftp works
sftp -v rubyforge.org
The -v option produces useful trace information about the connection process.  An ssh guru/wizard may be able to help you if they have that output.

Once that is done, you can add RubyForge as a remote repository:

git remote add rubyforge gitosis@rubyforge.org:YOURPROJECT.git

and then:
git push rubyforge master

Gem Hosting

Gem Hosting is confusing.  There is a lot of conflicting information out there and it is very hard to tell what is the "right thing" to do.  I use GitHub by default for my projects as I can keep them private and GitHub focuses on version control.  When I'm working in Ruby, there are things I'd like to contribute to the community as a gem.  What is the "right way" to do this?

I looked at various sites as they had mentions when Google'd:
  • GitHub
  • RubyForge
  • RubyGems
GitHub is no longer building gems (whatever that means exactly) so they are out of the running.  There is no obvious way to mark your repository as being a Ruby Gem so other can find it.

RubyGems.org sounded like they had an easier to use interface.  However, parts of the site haven't been updated in a while and the single-signon with a RubyForge didn't work.

That left RubyForge which is the default hosting site for "gem install".  Account creation worked and the blogs/forums look current.  I do like having the projects moderated and approved.  RubyForge it is!

I filled a project request for my C++/Rice documentation tool.  That got approved on a Sunday morning in about 10 minutes.

There are a couple of next steps for me.  I will be keeping my project on GitHub and publishing to RubyForge.  This is akin to using Heroku for application hosting.

To do:
  • Making the gem/package structure standards-compliant
  • Figure out how to push to RubyForge
  • Public documentation

Thursday, June 6, 2013

Pulling on a thread - amatch, rice, rdoc

Ever have a sweater with a loose thread?  You pull on it and the next thing you know you've got a ball of yarn and no sweater.

My loose thread starts with trying to write a Ruby-on-Rails application.  It seemed so straightforward.  Along the way, I needed do approximate text-matching so I Google'd and found the amatch gem.  Installed it, hooray!

But not so fast.  I was doing search using approximate match and Amatch::Sellers#search returns a score but doesn't tell you where it found the string.

So I downloaded the source code, figuring "how hard could it be to add the position"?  Let me tell you a couple of ways:

  • The code is a few years old and the Ruby infrastructure has changed since it was written.  The lesson here is to get a clean run of all rake tasks before you hack the code.
    • Unit tesing is now done via Minitest, for example.
  • The C-code is in one large file.  No the original author wasn't being silly, RDoc couldn't handle cross file references at the time.
  • Macros and lots of them.  Not my cup of tea.
The macro thing was the deal-breaker on this piece of source.  The Levenshtein and Sellers algorithm's are calculated using dynamic programming.  In order to figure out the search position, you have to navigate a cost matrix looking for binding constraints.  The macro-based version didn't retain the calculated cost matrix so I couldn't backtrace it to find the position.

So I ported the code to Rice/C++ and split it into multiple small files.  Rice was pretty easy to use and handled all of the type juggling between Ruby and C++.

I got the code working (the original author had a decent test suite) so refactoring of this sort went pretty well, maybe 8-10 hours.

Then it was time to produce the documentation.  Queue exploding sounds.  RDoc doesn't understand Rice/C++!
And so it begins...

One can say that I've been learning Ruby but that wouldn't really be true. I've discovered that programming in Ruby is actually pretty easy if you already know another Object-Oriented language  What is remarkably painful is actually setting up a project according to "standards".  So I'm recording my experiences in using Ruby (and related technology) to do something useful.