January 2008 Archives

Recently there have been a slew of posts complaining that rails doesn't scale and is hard to deploy.  I think the root of this is that people come to rails after experience with Java, PHP or similar frameworks.   Rails is neither of these.  It is heavier than PHP, and it seems that this makes its fit with the typical PHP deployment methods slow at best.  At the same time it has an entirely different deployment model than Java web applications.  The Java application containers are heavier, and designed to support a much more highly available application within  that JVM instance.

But rails still has its sweet spot.  So why would I use rails instead of Java? It's not as light as PHP, but from a development perspective it's usually close enough.  Bouncing webrick or mongrel takes a second or two, and for most changes I don't even need to do this.  To accomplish the same code/test cycle in Java takes  at least 10 times as long, often much worse.  At the same time rails offers.  Plus I can do a lot of things I can't do with Java, thanks to dynamic typing, better metaprogramming facilities, and things like the rails console.  Plus, from a hosting perspective you can get much better deals on hosting a rails application than a full Java servlet environment, which tells you something about the relative difficulty of hosting rails and J2EE.

Why would I use it instead of PHP?  The framework does enough magic that getting an application up and running is faster than raw PHP (disclaimer:  I haven't used any of the newer PHP frameworks like CakePHP . Maybe they're more on par with rails here.).  MVC is nicely broken out.  Ruby itself is a wonderful language to develop in.  Yes, you're going to have to do more work around deployment.  I can't imagine going live using FastCGI, SCGI, or that sort of thing.  But learn how to set up a mongrel cluster.  That seems to work.

I think Java and PHP still have their places as well.  Java provides much better concurrency in a single process, better built-in capabilities for high availability, security, and a larger library/tool ecosystem.  Enterprises know how to deploy and maintain Java applications, and in general feel more comfortable putting those in production than a rails system.  PHP is lighter weight, will likely scale better out of the box, and is better understood by most hosting companies.

At the end of the day the decision comes down to what you're building and where you're deploying.  I think rails works in wide range of situations, but there is an equally wide range where PHP or Java may be a better choice. 

I'm in the process of building a REST service using rails and ActiveResource. On the server side everything was fairly straightforward, but I hit a snag when I moved to the web tier. For various reasons I wanted to have two separate servers, one providing the REST service and the other serving HTML content. The rub is that ActiveResource assumes that credentials are a class level attribute. I'd like the calls to the ActiveResource client to use the credentials of the currently logged on user, but there doesn't appear to be a straightforward way to do this out of the box. Metaprogramming to the rescue!

Using the REST example from my last post, I extended the client in the following way.

require 'activeresource'

module Sample
  module Client
    class API
      #
      # Creates a module that serves as an ActiveResource
      # client for the specified user
      #
      def self.create_api(login = nil, password = nil)
        # specify the site.  Default to no credentials
        @url_base = "http://localhost:3000"
        @url_base = "http://#{login}:#{password}@localhost:3000" if login
        # build a module name.  This assumes that logins are unique.
        # it also assumes they're valid ruby module names when capitalized
        @module = login ? login.capitalize : "Default"
        class_eval <<-"end_eval",__FILE__, __LINE__
        module #{@module}
          class Post < ActiveResource::Base
            self.site = "#{@url_base}"
          end
          class Comment < ActiveResource::Base
            self.site = "#{@url_base}/posts/:post_id"
          end
          # return the module, not the last site String
          self
        end
        end_eval
      end
    end
  end
end

This new version allows you to create an api for a specific user. You can squirrel this away and use it for specific users, allowing you to use multiple connections to your rest service. A quick irb session follows

>> require 'ares_sample_client'
=> ["Sample"]
>> api = Sample::Client::API.create_api
=> Sample::Client::API::Default
>> p = api::Post.find(1)
=> #<Sample::Client::API::Default::Post:0xb715af74 @attributes={"updated_at"=>Wed Jan 09 02:36:34 UTC 2008, "id"=>1, "content"=>"The first post", "user_id"=>1, "created_at"=>Wed Jan 09 02:36:34 UTC 2008}, @prefix_options={}>
>> p = api::Post.create(:content => "should fail")
ActiveResource::UnauthorizedAccess: Failed with 401 Unauthorized
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:125:in `handle_response'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:112:in `request'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:101:in `post'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:803:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:636:in `save_without_validation'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/validations.rb:262:in `save'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/core_ext/object/misc.rb:28:in `returning'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from (irb):4
>> auth_api = Sample::Client::API.create_api('test1','test1')
=> Sample::Client::API::Test1
>> p = auth_api::Post.find(1)
=> #<Sample::Client::API::Default::Post:0xb713dde8 @attributes={"updated_at"=>Wed Jan 09 02:36:34 UTC 2008, "id"=>1, "content"=>"The first post", "user_id"=>1, "created_at"=>Wed Jan 09 02:36:34 UTC 2008}, @prefix_options={}>
>> p = auth_api::Post.create(:content => "should succeed!")
=> #<Sample::Client::API::Test1::Post:0xb713312c @attributes={"updated_at"=>Thu Jan 10 04:01:53 UTC 2008, "id"=>7, "content"=>"should succeed!", "user_id"=>nil, "created_at"=>Thu Jan 10 04:01:53 UTC 2008}, @prefix_options={}>
>>

I recently came across an article questioning whether Maven is too complex and bloated. The short answer is it can be. If you aren't taking some steps up front to deal with some of Maven's quirks, the benefit it provides can quickly be outweighed by the administrative overhead and performance hit its project management scheme imposes.

At my job we have a fairly sophisticated Maven project. It involves custom code generation plugins, cross-platform C++ compiles and deployments, assemblies, you name it. The number of engineers working on this project is probably around 80, and the number of Maven projects for this system is probably around 50.

It's not the most beautiful system in the world, but at the end of the day a lot of Maven's benefits are realized. Your average developer can check out an individual component from src, run "mvn test", and be reasonably assured
that things will compile, run, and execute unit tests, with minimal up front configuration of the build environment.

This is no small feat. It didn't start off easy. Maven is complicated. It's large. Documentation could no doubt be better. But at the end of the day, we're in a better place than we would have been with ant or make. Some of the steps we've taken to make things easier follow.

   1. Create your own repository. If you want to be able to do repeatable builds, don't let your projects access any repositories outside your control. Poms get broken, sites go offline... this causes all sorts of chaos. We ended up with a single repository server, but multiple repositories. The salient ones are maven-releases, maven-snapshots, our-releases, and our-snapshots. We put the artifacts we needed into maven-releases and maven-snapshots manually. We added profiles that can be enabled to get to the public repositories, but these are never enabled on our automated build machines.
   2. Use inheritance correctly. It's really tempting to use pom inheritance to capture project structure. This is what module tags are for. Pom inheritance is to allow you to apply similar configurations easily (like Java inheritance). It took us a long time to unwind this mess. We have a base-java pom that sets up all the reports we want to run for java, a base-model one to handle domain models,
   3. Fight the urge to tightly couple your large project. Maven leads you to fine grained componentization, which leads to a looser coupling of components in the build/release sense. There is a natural tendency to be uncomfortable with this (what's really going into my final build), but fight the urge to make it one giant system that gets built from the ground up. Executing releases is a nightmare with one giant system.
   4. Use version ranges where possible. This makes dealing with #3 easier. The odds that a component needs that specific release of a component (especially if you're doing agile and releasing every 30 days) is pretty slim. Most just need the latest. Also get familiar with the dependency convergence report.
   5. Make sure people understand what Maven goals are necessary for doing work. If lots of people are complaining about site generation taking too long, you have a clue that people don't get this. Your average developer should be running test and install, very rarely site.
   6. Decide on a versioning scheme up front, and make sure you can execute on it. Nothing is more frustrating than realizing it takes two weeks to get all your poms revved to the next revision. The maven release plugin has been fairly unreliable, but it leads you into a set of best practices that work even if you're taking the steps manually.

After taking these steps, the nearly universal sentiment is that while Maven is indeed complicated, it's a step forward when compared to ant/make. At the end of the day, designing a build process for large systems is difficult and rarely gets the attention from the development org it deserves. Maven doesn't make it dirt simple, but it makes the overall management of the build system simpler.

I don't think there are that many developers out there these days that don't think that automated unit testing is a good idea. Most seem to agree that automated functional and integration testing are worthwhile as well, even if they're generally harder to achieve. Being able to be reasonably confident that the change you just made didn't disrupt far flung parts of the project just by executing a test suite is great. However it's not a tangible, quantifiable thing you can sell to management.

There are generally two cases where you need to sell the effort involved in building and maintaining these suites. The first (and easier) is when you already have the suites in place, but need to invest in maintaining them. This is an easier sell because usually management will have seen the benefit and/or the ongoing expense is fairly small, and justifying the ongoing effort isn't all that bad. I'll throw new development in here as well... writing the suites as you go usually isn't as bad as retrofitting suites onto an existing code base.

Which brings us to the second case. When you have a legacy product, adding automated test suites is usually time consuming at best, and technically challenging in most cases. Your legacy product probably wasn't built in a way that encourages automated testing. The interfaces may not be there, it's tough to run in a test sandbox, etc. If you want to get automated tests working, you're probably facing a significant investment in people terms, and could be facing some infrastructure expenditures as well.

The wrong way to pitch this is to go to management, roll out your proposal and say that you need to spend $500k over the next three months building something that provides purely qualitative benefit. Unfortunately most attempts at quantification are difficult. You can talk about improved quality, improved productivity, increased agility... but at the end of the day that $500k is staring them down and making everybody in the room uncomfortable.

A better way is to talk about it in terms that can be strictly quantified, and the easiest one I've found is around regression testing. Figuring out how much you spend each year on regression testing is usually pretty straightforward.

   1. Write down how many times a year you execute a full regression test
   2. Write down many people a single test cycle takes
   3. Write down how long it takes
   4. Figure out how much a tester's time is worth.

Doing the math hear should give you a dollar amount for your regression expenditure. The next step is simpler, all though less precise. Estimate how much you think you could cut the time you got for #3. From what I've seen this should be on the order of 50%-80%. This gives you a total savings realized from automated testing.

A quick example may help

   1. We release twice a year, and execute on average two regression test cycles per release (one beta, one GA)
   2. Each test requires four test engineers
   3. The test cycle takes four weeks
   4. Each tester costs the org $100k a year

this means we have a total cost of 4*4*(4/52)*100k which is about $123k. If I can cut my regression test time to 1 week, I save at least $92k per year. Keep the last two words in mind... that's per year. Forever.

Keep in mind this is a minimum. You still have all the intangible benefits. But $92k/year is something the business folks can plug into their project valuation tools and evaluate. When we ran these numbers for our project it became clear that dedicating people full time to build out automated test suites for existing projects made perfect financial as well as technical sense.

   
Posted by McWong 53 days ago
This walkthrough shows the creation of a simple REST API using ActiveResource. It includes a nested model, authorization scheme, and a client library. The intent is not to exhaustively explore ActiveResource, but to show how to quickly get up and running with a reasonable REST API.

Step 1: create an application

rails ares-demo

I'm using rails 2.0.2 here, but I don't think anything is version specific.

Step 2: install two great rest plugins
Resource This is a plugin that adds default rest-based actions to a controller. It has so-so support for nested resources... you won't be able to use it out of the box for HABTM relationships for instance, but for strict hierarchies it works well, and really beats hand coding the controllers. You can install it with

script/plugin install http://jnewland.com/svn/public/ruby/rails/plugins/resource_this/

Restful Authentication is derived from acts_as_authenticated, and adds a rest-ish session controller, as well as the usual controller helpers (login_required, etc.) You can install it with

script/plugin install http://svn.techno-weenie.net/projects/plugins/restful_authentication/

Step 3: setup restful authentication
This bit is easy. Just run the generator to get your migrations for users created, along with a couple other things. I honestly didn't dig into this too much, it just works.

./script/generate authenticated user sessions

Step 4: create application model
Next up is generating your model. We'll do a simple one. Posts contain many Comments. Both Posts and Comments belong to a user.

./script/generate scaffold post
./script/generate scaffold comment

Next set up your migrations. db/migrate/002_create_posts.rb should contain

class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.column :content, :text
      t.column :user_id, :integer
      t.timestamps
    end
  end

  def self.down
    drop_table :posts
  end
end

and db/migrate/003_create_comments.rb should contain

class CreateComments < ActiveRecord::Migration
  def self.up
    create_table :comments do |t|
      t.column :content, :text
      t.column :user_id, :integer
      t.column :post_id, :integer
      t.timestamps
    end
  end

  def self.down
    drop_table :comments
  end
end

Next you'll need to add the relationships to your model classes. Add the following bit to the front of app/models/user.rb

class User < ActiveRecord::Base
  # custom relationships
  has_many :posts
  has_many :comments

app/models/post.rb should look like

class Post < ActiveRecord::Base
  belongs_to :user
  has_many :comments
end

and app/models/comment.rb should look like

class Comment < ActiveRecord::Base
  belongs_to :user
  belongs_to :post
end

Now run migrate to get your database all configured

rake db:migrate

Step 5: populate dummy data
Next we'll want to get some default data in there. Execute script/console and run through the following

>> u1 = User.new(:login => "test1", :email => "test1@foo.com", :password => "test1", :password_confirmation => "test1")
=> #<User id: nil, login: "test1", email: "test1@foo.com", crypted_password: nil, salt: nil, created_at: nil, updated_at: nil, remember_token: nil, remember_token_expires_at: nil>
>> u1.save
=> true
>> u2 = User.new(:login => "test2", :email => "test2@foo.com", :password => "test2", :password_confirmation => "test2")
=> #<User id: nil, login: "test2", email: "test2@foo.com", crypted_password: nil, salt: nil, created_at: nil, updated_at: nil, remember_token: nil, remember_token_expires_at: nil>
>> u2.save
=> true
>> p = Post.new(:user => u1, :content => "The first post")
=> #<Post id: nil, content: "The first post", user_id: 1, created_at: nil, updated_at: nil>
>> p.save
=> true
>> c = Comment.new(:user => u2, :post => p, :content => "first!")
=> #<Comment id: nil, content: "first!", user_id: 2, post_id: 1, created_at: nil, updated_at: nil>
>> c.save
=> true
>>

Step 6: resource_this!
Now the boring part us over. So far we have a basic rails app. Now we get to the REST part! You're three edits away... Change app/controllers/posts_controller.rb to the following

class PostsController < ApplicationController
  resource_this
end

Change app/controllers/comments_controller.rb to the following

class CommentsController < ApplicationController
  resource_this :nested => [:post]
end

And finally update config/routes.rb

ActionController::Routing::Routes.draw do |map|
  map.resources :posts do |post|
    post.resources :comments
  end

  map.resources :users

  map.resource :session
end

That's it! You're ready to look through your API

Step 7: browse
Start up your server with script/server. You'll need to clean up the erb views if you want to use the HTML interface, but your xml interface should be working just fine. Try the following urls for verification

    * http://localhost:3000/posts/1.xml
    * http://localhost:3000/posts/1/comments.xml

Step 8: lock it down
So far our service is open to anybody. We'll add very simple authentication at this stage... we'll require a valid user for anything other than show and index requests. All we have to do for this is include AuthenticatedSystem in our application controller, and add a before_filter. Make app/controllers/application.rb look like the following

class ApplicationController < ActionController::Base
  include AuthenticatedSystem
  before_filter :login_required, :only => [:create,:edit,:new,:update,:destroy]
  helper :all # include all helpers, all the time

  # See ActionController::RequestForgeryProtection for details
  # Uncomment the :secret if you're not using the cookie session store
  protect_from_forgery # :secret => 'b845cf981afbfef06600e5d7f23e0fed'
end

Step 9: create client lib
We could test this with curl or a browser, but the real fun of ActiveResource is getting to use ruby objects. So... we'll create a simple client api. Create a new file in lib called 'ares_sample_client.rb' and add the following code

require 'activeresource'

module Sample
  module Client
    class API
      class Post < ActiveResource::Base
        self.site = "http://localhost:3000"
      end
      
      class Comment < ActiveResource::Base
        self.site = "http://localhost:3000"
      end
    end
  end
end

Now we can play with our creation using the console. Make sure your server is running on port 3000, and startup script/console.

         
>> require 'ares_sample_client.rb'
=> ["Sample"]
>> posts = Sample::Client::API::Post.find(:all)
=> [#<Sample::Client::API::Post:0xb710bec4 @attributes={"updated_at"=>Wed Jan 09 02:36:34 UTC 2008, "id"=>1, "content"=>"The first post", "user_id"=>1, "created_at"=>Wed Jan 09 02:36:34 UTC 2008}, @prefix_options={}>]
>>

>> p = Sample::Client::API::Post.create
ActiveResource::UnauthorizedAccess: Failed with 401 Unauthorized
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:125:in `handle_response'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:112:in `request'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:101:in `post'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:803:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:636:in `save_without_validation'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/validations.rb:262:in `save'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/core_ext/object/misc.rb:28:in `returning'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from (irb):3
>>

Our API works, and the authentication is as it should be. We didn't specify credentials, so we can't actually create a post. Let's fix up our client lib to add credentials

module Sample
  module Client
    class API
      class Post < ActiveResource::Base
        self.site = "http://test1:test1@localhost:3000"
      end
     
      class Comment < ActiveResource::Base
        self.site = "http://test1:test1@localhost:3000"
      end
    end
  end
end

Note that credentials apply to the whole class... this makes the API fairly easy to use, but makes using multiple users difficult. Anyway, with our authentication in place, let's try again

>> p = Sample::Client::API::Post.create(:content => "Should succeed", :user_id => 1)
=> #<Sample::Client::API::Post:0xb71b81d8 @attributes={"updated_at"=>Wed Jan 09 02:55:22 UTC 2008, "id"=>3, "content"=>"Should succeed", "user_id"=>1, "created_at"=>Wed Jan 09 02:55:22 UTC 2008}, @prefix_options={}>
>> p.save
=> true
>>

Sweet! Now we can not only browse posts and comments, but create them as well!