Capybara HTML View Checking

I had wanted to do a decent job writing BDD tests for my views.  I was a bit surprised that the matchers were using the string form of the HTML.  I converted the string to a Capybara node in order to use the Capybara matchers.

HTML is tree-structured and my specs mirror  the HTML structure.  I check the parent node for correctness and then verify the children.  I had thought Capybara supported hierarchical specs using "within" but that doesn't seem to work for views.  It could be cockpit error on my part but Google searches didn't show any better way.

So I spent a little time adding to the RSpec DSL.  I wanted to have describe "scope into a subnode" of the current subject (the current subject being the equivalent of a DOM node).  I named the DSL "with_subnode".  With_subnode acts like describe with the ability to change the subject to a subnode.

The current subject is assumed to be a Capybara node.  with_subnode takes in the name of a Capybara::Node method (op:) to apply to the current subject plus any arguments (args:)

One starts by constructing a root_node using let( :root_node) do ... end.  This should be the string form as present in the rendered attribute.

The with_root_node method converts the root_node string into a Capybara::Node::Simple and makes it the subject of the do...end.

The example:

The code:

The code looks a bit odd. That has to do with the way RSpec builds tests. Each example/example_group is a meta-programmed class and one loses the lexical scoping. Each with_subnode clause is assigned a unique symbol. That symbol is used to retrieve the Capybara subject node from an associative array.

Unique symbols with an external associative array are used instead of let(). let() order evaluation along with inadvertent infinite recursion made that solution problematic.

The with_subnode clause is converted into two describe clauses. The inner describe clause is what the user placed in the do...end. The outer describe clause establishes the new subject.

An extra "it should_not be_nil" clause is added after the subject clause. This clause ensures the subject clause will be evaluated before the inner describe clause.

Controller Parameter Checking

Reference: rails/strong_parameters
Reference: active_record_validations.html#format
Reference: activemodel
Reference:

There are two main use cases.  The first to make sure that no extra parameters are accidentally passed into create/update operations.  This is to address any "mass assignment" vulnerability.

The second case is that RESTful controllers often have parameters not contained in the URL and hence not checked by the routing logic.  To validate those, we first construct a white-list:

nparams = params.permit( :my_param_a, :my_param_b, ...)

Note these are somewhat validated as being a basic type (see the reference above), but they aren't fully validated.

raw_parameters = { :my_param_a => "john@example.com", :my_param_b => "John", :admin => true } parameters = ActionController::Parameters.new(raw_parameters) puts "params: #{parameters}" cleaned = parameters.permit( :my_param_a,:my_param_b) puts "cleaned: #{cleaned}"

When executed:

params: {"my_param_a"=>"john@example.com", "my_param_b"=>"John", "admin"=>true} cleaned: {"my_param_a"=>"john@example.com", "my_param_b"=>"John"}

Oddly, input validation is not considered part of a controller; it seems to be model logic (see how-can-a-controller-manually-set-validation-errors-for-a-certain-field).  So we have to develop a Model that includes ActiveModel::Base in order to check the parameters.

Here is an example model which includes ActiveModel::Model

require 'active_record' require 'action_controller' class ParamCheck include ActiveModel::Model attr_accessor :my_param_a attr_accessor :my_param_b validates :my_param_a, presence: true, length: { minimum: 4 } validates :my_param_b, presence: true end

Some test code:

pc = ParamCheck.new puts pc.valid? pc.errors.full_messages.each do |err| puts err.inspect end

And the output:

false "My param a can't be blank" "My param a is too short (minimum is 4 characters)" "My param b can't be blank"

Using the cleaned parameters in the constructor:

pc2 = ParamCheck.new( cleaned) puts pc2.valid? pc2.errors.full_messages.each do |err| puts err.inspect end

Results in:

true

Adding checking to a controller

Each controller action is responsible for handling parameter white-list and required presence.

The idea is to define a nested class within the controller that is responsible for parameter checking. To avoid having a class per action, one ensures that named parameters are used consistently across all actions. For example, params[:email] should always be an e-mail address. Don't use names like :arg1 across multiple actions where it is an integer in one and an e-mail address in another.

The nested class then supplies validators for each parameter name. The validates should not check for presence as that was already handled by the action.

After the white-list and requires, one instantiates the CheckParams class using the cleaned-up parameters. One then checks to see if the instantiation is valid. If not, one produces a response with status 400 (bad request).

render :text => "400 Bad Request", :status => 400

Toggle Panel for Twitter Bootstrap

I had searched for a toggle panel widget and obtained one that sort-of worked from: where-are-collapsible-panels-in-this-framework

It was designed for JQuery and my toggle icon kept appearing on a line by itself.  I'm a relative JQuery newbie but old enough to know that writing highly chained JQuery code is someone's way of ensuring job security.

I refactored the code to use variables, added comments and substituted bootstrap icons for JQuery icons.  Here is the revised widget:

$.fn.togglepanels = function() { return this.each(function() { $(this).addClass("ui-accordion ui-accordion-icons ui-widget ui-helper-reset"); $h3 = $(this).find("h3"); $h3.addClass("ui-accordion-header ui-helper-reset ui-state-default ui-corner-top ui-corner-bottom"); // Add hover so this is apparent that it is a control $h3.hover(function() { $(this).toggleClass("ui-state-hover"); }); // Add the decorative icon $h3.prepend('<span class="toggle-panel-icon icon-plus-sign"></span>'); $h3.click(function() { $h3 = $(this); $h3.toggleClass("ui-accordion-header-active ui-state-active ui-state-default ui-corner-bottom"); $icons = $h3.children(".toggle-panel-icon"); $icons.toggleClass("icon-plus-sign icon-minus-sign"); // Stop any animations on content $content = $h3.next(); $content.stop(true, true); // Show/hide using a sliding animation $content.slideToggle(); return false; }); // Add styling classes to the content and initial hide it $content = $h3.next(); $content.addClass("ui-accordion-content ui-helper-reset ui-widget-content ui-corner-bottom"); $content.hide(); // Content is returned by default }); };

Note to self: SCSS and background images

I use background images in my .css.  When I deployed my app to Heroku, those images weren't showing up.

I used Google's developer tools and looked at the page's resources.  Those images are flagged as "not found".  Looking at the links, they were different-looking from my other precompuled assets.

Turned out that rather than url("image.png"), I needed url( image_path( "image.png")) in my .scss files.

Forms with Twiiter Bootstrap and Rails

I was using Rails's "form_for".  I had styled my form using 'form-horizontal'.  Everything looked good until I submitted a form with errors.  The form styling looked awful.

The solution can be found at the RailsApps project on GitHub.  I changed to SimpleForm using the directions provided and the presentation looks good.

Note to self: run rspec in project directory only

I noticed this when developing routing specs. Usually I run the command from the directory I'm working in. In this case, running "bundle exec rspec foo_routing_spec.rb" would fail. Running it in the parent directory as "bundle exec rspec spec/routing/foo_routing_spec.rb" works.

Templatized / Metaprogramming Testing

We often think of rules/specifications as applying to more than one object.  When I first began learning Ruby, I had followed Hartl's examples in writing tests and discovered that I was doing a lot of copy-and-paste.  That didn't feel right but I didn't know to do.  I wanted to do something like "foreach sign-in state { foreach page { verify page }}".

During the conversion of project to Cabybara 2.0, I noticed that descriptions could be tagged with additional hash key-value pairs.  They had something like:

describe "this test", :type => "feature" do

Interestingly, there is a note in rspec / rspec-core on metadata.  After some playing, I discovered a working design pattern.

  user_states = ["visiting","signed in", "admin"]

  user_states.each do |user_state|
    describe "for #{user_state} users", :user_state => user_state do
      let( :user_state) { example.metadata[:user_state] }

      let(:user) do
        user = nil
        case user_state
        when "signed in"
          user = FactoryGirl.create(:user)
        when "admin"
          user = FactoryGirl.create(:admin)
        end
        user
      end

      before do
        sign_in user unless user.nil?
      end

      all_static_page_names.each do |pg|
        describe pg, :page_name => pg do

          let(:page_name) { example.metadata[:page_name] }
          let(:pa) { PageAttributes.new( page_name) }

          before do
            # puts "          user state #{user_state}"
            pa.path = path_for_static_page( page_name)
            # puts "        pa: #{pa.inspect}"
            visit pa.path
          end

          it { should have_title( pa.title) }
          it { should_not have_title(full_title('Edit user')) }

          it { should have_link('Contact', href: contact_path) }
          it { should have_link('About', href: about_path) }
          
          if( user_state=="visiting") then
            it { should have_link('Sign in', href: signin_path) }
          else
            it { should have_link('Sign out', href: signout_path) }
          end

        end
      end

    end

  end

I have a few helper routines in my utilities. One provides the array of page names, another returns the page attributes for a page. The last returns the path to a page given a page name. I had to keep the path one separate as when I tried to include in the PageAttribute constructor, something was wrong and I couldn't access the *_path variables.

Getting my project to work with Capybara 2.0

My project was originally designed following Michael Hartl's excellent tutorial.  I was using a 1.x verison of Capybara and only had "cockpit error" types of problems - my own fault.

When I moved to Rails 4.0, my tests stopped working.  So I upgraded to Capybara 2.1 and dug in.  I must say I was surprised at the nature of the changes between 1.x and 2.1.

Some symptoms:

• Accessors like "root_path" stopped working
• visit stopped working

In general, you will need to modify your tests.

• the requests directory should be named features
• require 'spec_helper' needs to be require_relative '../spec_helper'
• I haven't figured out how to adjust the search path to allow the original construct to work.
• The outermost "describe" blocks should be "features"

I discovered that features don't nest, nor do scenarios.  See the references below.  In order to make 'visit' work, I turn the outermost 'describe' block into a feature block.

I tried using scenarios for inner blocks.  This was very painful and I went back to describe for inner blocks.

Reference links

• How to fix “Undefined method ‘visit’” with Capybara 2.0
• rspec-rails and capybara 2.0: what you need to know
• Capybara 2.0 Upgrade Guide
• Introducing Capybara 2.1
• End-to-end testing....
• Capybara project on GitHub
• Stack overflow on nested featues
• secret_key_base not defined

An incomplete list of what I changed

Added to spec_helper:

require 'capybara/rspec' require 'capybara/rails' ... config.include Rails.application.routes.url_helpers

I had several checks for title that needed to be updated. They were

it { should have_selector('title', text: 'Sign up') }

Now:

it { should have_title('Sign up') }

An update to this post will follow as i organize my notes.

Note to Self: Rails Generate Scaffold

I learned about Rails by working through Michael Hartl's tutorial.  Generating scaffolding is not something that I do every day or even every week, so I rely on my notes.

I recently shifted to Rails 4.0 and constructed a scaffold per Hartl's tutorial.  I added the pages to my so I could use the generated MVC.  I discovered that while I could create objects, the contents/attributes were blank.

I traced it back and discovered that my attributes were not added to the model as "attr_accessible".  Added those and the MVC now works.

Rubygems, Heroku, Upgrading to Rails 4.0

School of hard knocks

In a previous post, I had mentioned that I hosting my gem(s) on RubyForge. Well, that didn't work out for me as couldn't get "git" to push to the repository. I've managed to get Heroku working, starting with a project on GitHub. But no luck with RubyForge. I had opened a support ticket some time ago & no response. So I will be moving my public projects to RubyGems.
Rubygems has some tooling and documentation for Gem building. I installed the tooling and my development environment broke. Yes, likely my fault as I was lazy and ignored all of the "wax-on/wax-off" about setting up a Ruby environment.
Lesson #1: Use rvm (Ruby Version Manager). This tool installs ruby to your own private area so you don't mess up your system installation. Particularly good advice if you are on a Macintosh. For me, I'm running Fedora and I had to do the standalone install of rvm. It worked, although I had to properly modify the search path. rvm is truly private to you.
Lesson #2: Install gems locally, not into your system. When things are looking very confused and your development environment is messed up, I find it useful to delete my '~/.gem' directory and use bundle install to rebuild it.
Lesson #3: Use gem version numbers in your Gemfile. Things aren't as stable as one would like, so pinning things down is helpful.

Upgrading to Rails 4.0

I had followed this post: http://blog.barbershoplabs.com/blog/2013/02/27/upgrading-from-rails-32-to-rails-40

The biggest culprit was in my config/application.rb file.   require "active_resource/railtie" doesn't work and I just commented it out. Rake now works and I'm keeping my fingers crossed.

I'm now just cleaning up after the new set of error messages.  Those seem obvious.

Update:
I generated a new Rails 4.0 project. I wrote a small ruby script to compare and modify my existing project based upon the new reference projects. It added missing directories and copied over any missing ruby files. For ruby files that existed in both and were different, I had the script invoke the merge tool "meld". The deltas weren't all that bad to merge.

#! /usr/bin/env ruby

require 'fileutils'

ref_dir = "reference/"
src_dir = "target_project/"

# okay, let's start with directories

l4 = Dir[ "#{ref_dir}**/**/**/**" ]

# puts l4

l4.each do |n|
  result =  File.stat( n)
  if File.stat(n).directory? then

    # Take off the front part
    p = n[(ref_dir.length..-1)]
    # puts "subpath #{p}"

    d = File.join( src_dir, p)
    if !File.directory?( d) then
      puts "Directory #{d} does not exist, creating it"
      `mkdir -p #{d}`
    end

    next
  end

end


l4.each do |n|
  result =  File.stat( n)
  if File.file?(n) then

    # Take off the front part
    p = n[(ref_dir.length..-1)]
    # puts "subpath #{p}"

    d = File.join( src_dir, p)
    if !File.file?( d) then
      puts "File #{d} does not exist, copying it over"
      `cp #{n} #{d}`
    end

    next
  end

end

l4.each do |n|
  result =  File.stat( n)
  if File.file?(n) then

    # Take off the front part
    p = n[(ref_dir.length..-1)]
    # puts "subpath #{p}"

    d = File.join( src_dir, p)
    if File.file?( d) then
      if d.match( /\.rb$/) then
        result = FileUtils.cmp( n, d)
        if( !result) then
          puts "Ruby file #{d} does not match"
          r = `diff #{n} #{d}`
          `meld #{n} #{d}`
        end
      end
    end

    next
  end

end

Rakefiles and Generated Sources

A quick note on something that confused me.  I was working on a Rakefile for a project that has generated sources.  Generated sources aren't that unusual.  For example, people will often write a gemspec file from their Rakefile.

In my case, it started with a generated source file that needed to be added to my gemspec.  I had done the usual "file = Dir["**/*/*.rb"] and assumed that the file would be picked up when the :gem task was run. In other words, when the gem task is finally called it should figure out what should be included.  Yet it didn't do that.  Why?

My code to fill in the gemspec is an anonymous function (aka do-block) that is getting passed into the constructor for the gem task.  That function is being evaluated when the object is being constructed.  As a result, my generated sources didn't exist and thus weren't making it into the gem.

Rakefile Snippets

First, I explicitly constructed the list of generated sources:

GSRC_VERSION = File.join('lib', 'amatchpp', 'version.rb') GSRC_PROTODOC = "protodoc.rb" GSRCS = [GSRC_VERSION, GSRC_PROTODOC]

Then I construct a file-rule to construct a file

file GSRC_VERSION => "Rakefile" do desc m = "Writing version.rb information for #{PKG_VERSION}" puts m File.open(GSRC_VERSION, 'w') do |v| v.puts version_text end end

I add a task which depends on the file.

task :version => GSRC_VERSION

I have a :generated task that depends on all constructed sources.

task :generated => GSRCS

I add the :generated task as a dependency to other tasks as needed. For example, I add it to the :doc task as show below.

############################################################ # Documentation ############################################################ RDoc::Task.new(:doc) do |rd| rd.main = "README" rd.title = "#{PKG_NAME} - Enhanced Approximate Matching" rd.rdoc_files.include( PKG_DOC_FILES) rd.rdoc_dir = "doc" end task :doc => :generated

Ruby Project Directory Organization

NOTE: This post will be updated from time-to-time as I learn more.  Right now, it is collection of tidbits gathered together.  There is some conflicting information between a Rails project and a Ruby project.

TODO:  References

Ruby Project

A good Ruby project is organized into one or more modules.  A module's component files are placed into the project hierarchy by what they do.

project
  lib
    module1
      class_a.rb
      class_b.rb
      ...
    module1.rb

  test
    module1
      class_a_test.rb
      class_b_test.rb

We see that a module's files aren't kept together.  Copying a module from project-to-project will involve copy multiple directories and a file or two.

The motivation behind this organization appears to be search path and file name globbing.  One usually runs Ruby with the "-Ilib" command line option set.  A require statement for a module is:

    require 'module1'

Ruby will then locate and identify the code file as:

    lib/module1.rb

The file module1.rb will either be a complete implementation or simply a set of require statements:

    require 'module1/class_a'
    require 'module1/class_b'

Globbing is important as it allows other utilities to act on the appropriate subset of files.  For example, RDoc should be run on all Ruby source files under the lib/ directory but we don't want it to publicly document our test code.  Test runners are similar.

Files are organized into the directory structure for easy and consistent globbing by scripts and other utilities.

Distributed your Git repository to RubyForge (Linux)

There are some notes on the RubyForge site but it wasn't obvious to someone has doesn't live and breath SCM.  I found the following useful:

•  Dr Nic's post on this subject somewhat helpful
• simplify your life with an ssh config file

NOTE:  If you are already using GitHub, you probably already have a ~/.ssh/id_rsa.pub key.  That didn't work.  My suspicion is that RubyForge wants your account name in the key, not your e-mail address.

Part 1: Make a RubyForge key

ssh-keygen -t rsa -C your-account-name -f id_rubyforge

Part 2: Add the key to your account

• You add the key to your account using the high level "My Account"
• edit keys is at the bottom of the screen
• I use xclip to copy the contents of a file to the clipboard:

xclip ~/.ssh/id_rubyforge.pub

Part 3:  Add a rubyforge.org section to your ~/.ssh/config file

Host rubyforge.org
     user your-account-name
     IdentityFile ~/.ssh/id_rubyforge.pub

Part 4: Verify that sftp works

sftp -v rubyforge.org

The -v option produces useful trace information about the connection process.  An ssh guru/wizard may be able to help you if they have that output.

Once that is done, you can add RubyForge as a remote repository:

git remote add rubyforge gitosis@rubyforge.org:YOURPROJECT.git

and then:

git push rubyforge master

Gem Hosting

Gem Hosting is confusing.  There is a lot of conflicting information out there and it is very hard to tell what is the "right thing" to do.  I use GitHub by default for my projects as I can keep them private and GitHub focuses on version control.  When I'm working in Ruby, there are things I'd like to contribute to the community as a gem.  What is the "right way" to do this?

I looked at various sites as they had mentions when Google'd:

• GitHub
• RubyForge
• RubyGems

GitHub is no longer building gems (whatever that means exactly) so they are out of the running.  There is no obvious way to mark your repository as being a Ruby Gem so other can find it.

RubyGems.org sounded like they had an easier to use interface.  However, parts of the site haven't been updated in a while and the single-signon with a RubyForge didn't work.

That left RubyForge which is the default hosting site for "gem install".  Account creation worked and the blogs/forums look current.  I do like having the projects moderated and approved.  RubyForge it is!

I filled a project request for my C++/Rice documentation tool.  That got approved on a Sunday morning in about 10 minutes.

There are a couple of next steps for me.  I will be keeping my project on GitHub and publishing to RubyForge.  This is akin to using Heroku for application hosting.

To do:

• Making the gem/package structure standards-compliant
• Figure out how to push to RubyForge
• Public documentation

Pulling on a thread - amatch, rice, rdoc

Ever have a sweater with a loose thread?  You pull on it and the next thing you know you've got a ball of yarn and no sweater.

My loose thread starts with trying to write a Ruby-on-Rails application.  It seemed so straightforward.  Along the way, I needed do approximate text-matching so I Google'd and found the amatch gem.  Installed it, hooray!

But not so fast.  I was doing search using approximate match and Amatch::Sellers#search returns a score but doesn't tell you where it found the string.

So I downloaded the source code, figuring "how hard could it be to add the position"?  Let me tell you a couple of ways:

• The code is a few years old and the Ruby infrastructure has changed since it was written.  The lesson here is to get a clean run of all rake tasks before you hack the code.
• Unit tesing is now done via Minitest, for example.
• The C-code is in one large file.  No the original author wasn't being silly, RDoc couldn't handle cross file references at the time.
• Macros and lots of them.  Not my cup of tea.

The macro thing was the deal-breaker on this piece of source.  The Levenshtein and Sellers algorithm's are calculated using dynamic programming.  In order to figure out the search position, you have to navigate a cost matrix looking for binding constraints.  The macro-based version didn't retain the calculated cost matrix so I couldn't backtrace it to find the position.

So I ported the code to Rice/C++ and split it into multiple small files.  Rice was pretty easy to use and handled all of the type juggling between Ruby and C++.

I got the code working (the original author had a decent test suite) so refactoring of this sort went pretty well, maybe 8-10 hours.

Then it was time to produce the documentation.  Queue exploding sounds.  RDoc doesn't understand Rice/C++!

And so it begins...

One can say that I've been learning Ruby but that wouldn't really be true. I've discovered that programming in Ruby is actually pretty easy if you already know another Object-Oriented language  What is remarkably painful is actually setting up a project according to "standards".  So I'm recording my experiences in using Ruby (and related technology) to do something useful.