curb to github

I've made a few more improvements to curb and decided keeping track of all these patches is too much. I'm now tracking my changes on github, and hopefully the curb author will reappear to review the changes soon, so we can get them released in gem form.

New curb patch, on_succes, and on_failure

I added a new callback hook to curb on_success and on_failure. They work as advertised and are esspecially useful when using the Curl::Multi interface from my last patch.

gc = Curl::Easy.new("http://www.google.com/")
gc.on_success{|curl| puts curl.body_str }
yc = Curl::Easy.new("http://www.yahoo.com/")
yc.on_success{|curl| puts curl.body_str }
   
mc = Curl::Multi.new
mc.add(gc)
mc.add(yc)
   
mc.perform

What's nice about this, is imagine you have some process running and want to add requests to it. When you add the request you also want to register a callback to handle the success or failure cases. Before this patch you'd have to check the content length and listen to the on_body handler. The problem with that, is the content length header is not always correct. These callbacks are guaranteed, because they happen during the cleanup process for each handle.

As an aside they'll work using the Easy handle diretly.

Here's the patch, enjoy!

Freezing compiled gems in merb

Vendoring gems definitely makes deployment easier for both rails and merb. For merb, the solution is to create a local gem repository within your project.

  gem install -i gems/ --no-ri --no-rdoc #{gem_to_install}

This works great, once you update your merb/config/init.rb to include the new gem repository.

Gem.clear_paths
Gem.path.unshift(Merb.root / "gems")

I also recommend saving space by deleting the gem cache.

rm -rf gems/cache/*

This creates one problem, if the gems you have installed include binaries (e.g. C extensions), then you'll have an issue if you develop and deploy to different architectures. I think the most common would be developing on Mac OSX and deploying to Linux. There are many different ways to solve this problem. One might use gem pristine, or as I do below maintain a list of the gems that need compiling and rebuild them for the deployment target.

GEM_BUILDS=['rbtagger','hpricot','mongrel','fastthread']
GEM_ROOT=File.join(File.dirname(__FILE__),'gems')

desc 'Refresh gem builds for the current system'
task :build_refresh do
  gem_dir_list = File.join(GEM_ROOT,'gems')
  gems_built = GEM_BUILDS
  Dir["#{gem_dir_list}/*"].each do|gem_path|
    gems_built.each do|gem|
      if gem_path.match(gem)
        if File.exist?("#{gem_path}/Rakefile")
          puts "building #{gem_path}"
          system("cd #{gem_path} && rake compile")
        else
          base = File.basename(gem_path)
          version = base.split('-').last
          cmd = "gem install -i gems --no-ri --no-rdoc #{gem} --version #{version}"
          puts cmd.inspect
          system(cmd)
        end
        gems_built.reject!{|g| g == gem}
      end
    end
  end
end

Now to run the task:

rake build_refresh

To change the list of compiled gems just modify the GEM_BUILDS constant.

Updated curb multi interface patch

I've updated my patch for curb multi interface and tested against valgrind for memory leaks. So far the patch looks stable. I'm still waiting to hear from the curb author. In the meantime here's how to apply the patch if you're interested.

Check out curb trunk

  svn co svn://rubyforge.org/var/svn/curb/TRUNK/curb

Grab my latest patch

  wget http://taf2-patches.s3.amazonaws.com/curb-multi.patch

Apply the patch

  patch -p0 < curb-multi-3.patch

Rebuild and test curb

  rake test

Package and install

  rake package
  sudo gem install pkg/curb-0.2.0.gem

In the patch I updated the curb version number. This may of course change, based on whether the patch is accepted or not. In the meantime, would be great to get a few people to try the patch and provide feedback and or bugs. Thanks!

Update (example usage):

  responses = {}
  requests = ["http://www.google.co.uk/", "http://www.ruby-lang.org/"]
  m = Curl::Multi.new
  # add a few easy handles
  requests.each do |url|
    responses[url] = ""
    c = Curl::Easy.new(url) do|curl|
      curl.follow_location = true
      curl.on_body{|data| responses[url] << data; data.size }
    end
    m.add(c)
  end

  m.perform do
    puts "idling... can do some work here, including add new requests"
  end

  requests.each do|url|
    puts responses[url]
  end

Here's a performance comparison using ruby's native threads vs curb multi interface to make up to 400 concurrent requests. You can see a huge pay off in performance at the 50 concurrent requests.