McWong tag:www.themcwongs.com,2008-03-02:/mcblog//1 2008-05-15T02:14:10Z Movable Type Open Source 4.1 How J2EE set architecture back a ways tag:www.themcwongs.com,2008:/mcblog//1.43 2008-05-15T01:43:46Z 2008-05-15T02:14:10Z McWong
I'm currently involved in architecting a next generation systems management framework.  It has fairly hefty requirements... monitor and manage millions of objects distributed across the globe in real time.  Take out the "millions" bit, and this isn't so bad.  There are a bunch of standards out there (JMX, SNMP), there are some off the shelf tools, and you can knock something together pretty quickly that covers a reasonable subset of equipment for small installs.  But we need to do better.   We need to manage more than just "most" of the equipment, and we need to do it in some of the world's biggest data centers.  And as I talk about this with my peers from other groups,  I'm getting more convinced that J2EE has poisoned a good chunk of the current generation of architects.

I know this isn't how J2EE  has to work, but in most projects, you start them like this...
1) Build your object model, use magic to map it to the DB
2) Build your business logic
3) Build your front end
4) Ship it!

And there you are.  You have a nice central database so you don't have to worry about distributed data so much.  It's all nice and safe on the disk.  You have some adapters to get stuff in and out, and a cozy UI that serves it all up to the user.  And it scales... for as much load as you can throw at your desktop.

Then you take it to some place big.  And it just flat out doesn't work.  It's not a matter of getting a bigger database, because you just can't scale it up past a certain point. You can cluster, but now you need a professional services group to install your product, and the performance gains to be had aren't crystal clear.

It becomes a matter of deciding what goes in there.  And when, and how.  But looking forward to these issues is met with fierce resistance in design meetings... Building distributed systems is much harder than building something around a central store.  Objections are typically based on reading marketing propaganda from database vendors.  Responses of the form "well, we'll partition the data when we hit that load.  Or use a cluster... yeah, a cluster will fix everything" are extremely painful to work through.  After all, some "expert" from VendorX says it will work. Who are you, my peer, to assume you know more than me?  

But guess what?  Each of these objectors has a product in the field, based a central database, handling far less load than our targets, and none of them scale well.

Scaling out is beating scaling up.  Processors are going multi-core, not taking huge leaps forward in the GHz war.  Scala and Erlang are gaining in popularity, and with them a different model of parallelism (not new, just different) than that offered by J2EE.  The actor model embraced by both of those languages is primarily targeted at small processes within a service.  But as you look at larger chunks of the architecture, I think the sanest way forward is to embrace a similar model.  

ESB has been a buzzword for years now.  I don't buy all the marketing hype around it, but at its core there is a model that the current generation of architects needs to get a handle on.  It's very similar in concept to the Erlang/Scala actor model.  Yes, it's harder to design around asynchronous messages and a large number of distributed components.  But that's how you handle the big problems.  And that's why you hire good developers to work on them, not just some random CS grad who happens to be able to regurgitate the latest Spring book.  

I'm not saying there isn't a place for the traditional J2EE buildout (rails has a similar model, and works wonderfully for small-medium sized projects).  And in fact, even with these asynchronous systems there are almost certainly services that need a DB backing them.  But there are other tools that should be in your toolbelt.

My suggestion to Java architects today would be to pick up another language that fosters a totally different architecture.  I'm currently biased towards Ruby, Erlang or Scala.  There are no guarantees that any of these scale better than Java (and it's fairly easily arguable that Ruby doesn't).  But any of these will make you see the world in a different way, and allow you to more effectively consider and evolve your current architecture.
]]> Scala Lift Off - Static Companion to Ruby? tag:www.themcwongs.com,2008:/mcblog//1.42 2008-05-13T05:42:04Z 2008-05-13T06:50:44Z McWong Scala Lift Off on Saturday (only half, because the first half was taken up by my final MBA class.  Ever.).  I went primarily out of curiosity, not knowing much about Scala or Lift.  The main draw was the built in comet support for Lift, which seems to not be a focus in other frameworks... at least not for Rails.  We currently use Juggernaut for comet support, but depending on flash is something of a liability (see: iPhone), and Juggernaut itself isn't as smoothly integrated with Rails as i'd like.

I came away extremely impressed.  Scala is relatively unheralded in the world of alternative JVM languages (see Groovy, Jython and JRuby publicity), but shows a lot of promise.  It's a functional language with an expressive syntax that allows you to easily create code that looks DSL-ish.  These are the primary features that drew me to Ruby (ok, Ruby isn't a functional language, but you can sorta fake it).  But Scala has a better integration story with existing Java libraries, is strongly typed, and has a stronger functional bent.

I'm a big believer in the right tool for the job, and as such don't fall into a pure-dynamic or pure-static language camp.  I also don't fall into a single language camp. I really enjoy Ruby for quick prototyping, and love Rails for quick prototyping of webapps, and maintaining a nimble production face on web applications.  But Rails falls down when I need to run background processing.  The times I think hardest about moving back to a Java webapp environment are when I need to go write something that doesn't just receive a web request and terminate.  This is where concurrency issues get painful to deal with, Ruby daemons/DRb are painful, and starting up a whole Rails env for simple processing is rough.

So I'm hoping Scala/Lift fills that void.  I'm mentally sketching out a replacement of our background processing jobs (Twitter integration, email processing, etc.) with Scala, and in particular the Actors library.  These are relatively simple processing tasks, and should give me a decent feel for the language.  It should also improve the stability and scalability of our background processing.  It may also yield some reasonable libraries to contribute back (Scala Twitter library, Scala ActiveRecord bridge). 

Once I have that nailed down, an evaluation of how Lift can/should fit into the framework is in order... or maybe I'll have to start my Rettiwt side project based on Lift.


]]> System and Organizational Scaling - the Enterprise View tag:www.themcwongs.com,2008:/mcblog//1.41 2008-04-30T15:40:18Z 2008-04-30T17:21:56Z McWong post talking about the challenges faced by their startup portfolio, and how a vertical approach to subsystem division helps scaling the organization.  In fact the Web 2.0 landscape is very reflective of this approach.  10 years ago if Yahoo needed an authorization framework they would have built it themselves.  Today people use oAuth.  Flickr, Twitter, Delicious, Campfire, Meebo, S3... all very focused services that delegate non-core functionality to another service where possible.  For those things that are duplicated across services (web frameworks, database backends), nobody cares if they're different... it's hidden behind the service.

This is very attractive for the startup ecosystem.  The tangible results have been products that are cheaper to launch, quicker to market, and easy to adapt to customer feedback.  But how does it apply to the enterprise?

Issue 1:

For better or worse, in large engineering organizations people tend to care that the common horizontal components are indeed common.  If you work in a 100+ person engineering organization, and the rest of the team is using Spring + Struts for web development, it's usually a tough sell to start using Rails.  If the organizations insists on this homogeneity, your vertically sliced org is cross cut by the commonality police, hampering the agility of the vertical team.  There are good reasons for this.  Somebody needs to support and test the thing.  If nobody else can deal with what you just built, it doesn't have a path to market, and is therefore useless. 

Issue 2:

It's easier to add 1 person or 1 feature to an existing service than it is to spin up a whole new service team.  Most organizations usually don't have the luxury of five new reqs to apply to a service.  And they're also hesitant to carve out five people from existing teams to spin up a new service.  So teams and services are usually built by accretion.  The result is usually a gradual march to collapse, as services become bloated and so difficult to maintain they need to be replaced.

Issue 3:

The organization only has capacity for a limited number of products.  A startup of 5-10 people can fully support the development, launch, and marketing of a vertical service.  In a large organization, the amount of infrastructure required to take a product to market means that unless you have a multi-million dollar revenue stream guaranteed in year 1 your chances of getting marketing, sales, training, doc, etc. spun up to support you are very small.  That means actual releases are larger than individual services.

Issue 4:

Coordination challenges across vertical services.  This ties in to issue 3.  Because a product release is a composition of multiple services, there is a desire for tight coordination across these services.  Contrast this to the Web 2.0 world.  Basecamp uses Amazon's S3 storage service.  They are under no illusion that they can call the shots on S3's feature roadmap.  And even more important, they would not plan for a release based on features that S3 has not committed to.  In the enterprise, these rules don't apply.  Feature roadmaps for a collection of services are determined at the same time, and management hopes to coalesce them all into product at a predefined future date.

Possible Solutions

To address issue 1, the organization should focus on service capability, not underlying implementation.  To support this, service teams will need to be self sufficient, and should make implementation decisions as a team.  It needs to be tested.  Involve quality engineering in the technology selection.  It needs to be doced.  Involve doc.  At the end of the day the customer rarely cares if you use VB or Python to get them their value.

To address issue 2, the entire organization needs to be focused on keeping services focused on their core function.  When somebody needs feature X, run through the list of services.  Teams should be willing to say no to features not because they don't have capacity, but because the feature corrupts the purpose of the service. If the feature doesn't fit with any project, spin up a new service rather than accrete it onto an existing project.  This requires the organization to be flexible, to be willing to work on new things, and to give up old responsibilities.  It also demands a rejection of empire building.  You should take more pride in your project being small, focused, and absolutely fantastic at what it does, rather than measuring your worth by the number of people on your project. 

Issues 3 and 4 are the stickiest.  In the enterprise the opportunity for experimentation is much lower than it is for a consumer web offering.  Perhaps one option is to create a "labs" organization, that releases products for free for use by bleeding edge customers.  Services that are vetted through the "labs" channel could be productized at a later date. 

Coordination problems could be solved by eliminating planning that spans services. Take a snapshot of service capabilities, and plan from there.  This likely slows down development to a certain extent, but also makes individual plans more predictable, and significantly reduces coordination challenges.

Wrapup 

While challenges exist in the enterprise, I think it's worthwhile to look at how we can make vertical slicing of the org work.  The advantages realized by Web 2.0 companies are compelling, and we definitely have challenges with our current horizontally structured groups.
]]> Transparent PNGs in IE w/ Rails tag:www.themcwongs.com,2008:/mcblog//1.40 2008-04-27T20:58:10Z 2008-04-27T21:10:29Z McWong Kebima for several months now, using Firefox and Linux/OSX.  Chalk it up to not doing enough research, but I just figured transparent PNGs worked in IE.  Oh well.  They don't.  At least not in 6 and earlier.  So began my mission to get them to work.

I was already using a lightbox package that uses the technique mentioned in the MS support article, but I didn't want to have to apply a div-specific solution for every png on my page... and since i'm using the silk icon set, this would mean a lot a lot of specificity.

Googling gets you a lot of results.  The first hit is actually pretty good, in that it states that the script isn't maintained, but points you to 24 Ways, which has a pretty good solution.  This likely works out of the box for normal web development (I did run into one bug... for some reason the section of the script that sets root on line 17 failed... I took out the bit that allowed you to limit the div the script was applied to).  However rails likes to throw timestamps on the end of images, so instead of '/images/icon.png' you get '/images/icon.png?1209327623'

Because the 24 Ways script is looking for an img tag whose src attribute ends with '.png', this makes things not work.  My solution was to add a regex to match rails style image srcs

    var png_pattern        = /\.png(\?\d*)?$/i;

And then match against that in fnLoadPngs

            // background pngs
            if (obj.currentStyle.backgroundImage.match(png_pattern) !== null) {
                bg_fnFixPng(obj);
            }
            // image elements
            if (obj.tagName=='IMG' && obj.src.match(png_pattern) !== null){
                el_fnFixPng(obj);
            }

And it worked.  Plus I got a pretty decent workout running up and down the stairs between Mac and PC.
]]> Twitter, Jabber, Stability - Should Twitter be more ejabberd and less rails? tag:www.themcwongs.com,2008:/mcblog//1.39 2008-04-26T01:21:49Z 2008-04-26T01:34:30Z McWong Kebima from using the HTTP interface to using the XMPP interface.  We really wanted real time updates, and polling just seems so barbaric.  I found some code on how to create a twitter bot and got the conversion made surprisingly fast.  It's still ugly because auto-following has to be done through HTTP, but in a few hours I had a pretty simple bot going.

Then we went to the Web 2.0 Expo and tried it out.  And it didn't work.  Turns out Twitter's Jabber replies were delayed or somesuch... probably fallout of Twitter's other greyout problems this week.  But it got me thinking...

If you had to build twitter from scratch, how thin a veneer over XMPP could you do it with?  At its heart, Twitter is a message router.  There are interfaces with SMS systems, HTTP, and Jabber.  But messages come in, messages go out.  I've heard they run XMPP under the hood to handle this, but I've also heard they run rails for a good chunk of functionality.  I'm a rails fan as well, but from what I can see, perhaps there's too much rails and not enough ejabberd in the mix.

Conceptually it seems like you could set this up as a set of processes that each act as an internal Jabber client for an individual twitter user.  The process is responsible for receiving messages
  1. pushing them to SMS or the user's Jabber client
  2. Building the web page that people visit
  3. Handling API calls as they come in
  4. Dispatching messages to followers
In fact conceptually you could set up three clients, each to handle one of these jobs.  Each of these processes could in essence be a fully functional twitter service for an individual user.

In addition to this you need gateways for SMS and the HTTP API, but it seems like those could be scaled out fairly easily as they're not user specific.  The SMS gateway is just going to build an XMPP message and dispatch it to the matching twitter client.  The HTTP gateway is doing the same thing.

The beauty of this is that it's naturally sharded.  No shared data between users.  You have to parse messages for @responses and route them to appropriate destinations, but it seems like that could easily be written as an ejabberd plugin. 

Google has proven this scales to a fairly large user base.  Is Twitter already beyond that scale, or am I missing something?

Regardless, we'll be pushing ahead with our plans to add direct GTalk integration for our app.  It may go down at times as well, but it seems more robust than Twitter's infrastructure at this point.
]]> Standalone ActiveRecord and SQLite3 tag:www.themcwongs.com,2008:/mcblog//1.10 2008-03-05T23:50:33Z 2008-03-06T00:00:38Z I recently wanted to set up a quick daemon process that used ActiveRecord outside the rails framework. I also wanted it to use sqlite, just to keep install/dependencies simple. I found not so good documentation on how to do this...... McWong 
require 'rubygems'
require 'sqlite3'
require 'activerecord'

# connect to database.  This will create one if it doesn't exist
MY_DB_NAME = ".my.db"
MY_DB = SQLite3::Database.new(MY_DB_NAME)

# get active record set up
ActiveRecord::Base.establish_connection(:adapter => 'sqlite3', :database => MY_DB_NAME)

# create your AR class
class Update < ActiveRecord::Base

end

# do a quick pseudo migration.  This should only get executed on the first run
if !Update.table_exists?
  ActiveRecord::Base.connection.create_table(:updates) do |t|
    t.column :account_name, :string
    t.column :last_update_time, :timestamp
    t.column :last_update_id, :integer
  end
end
 ]]> Scaling and Deploying Rails: It's not Java. Or PHP. tag:www.themcwongs.com,2008:/mcblog//1.6 2008-01-09T05:14:12Z 2008-03-02T05:14:38Z McWong
But rails still has its sweet spot.  So why would I use rails instead of Java? It's not as light as PHP, but from a development perspective it's usually close enough.  Bouncing webrick or mongrel takes a second or two, and for most changes I don't even need to do this.  To accomplish the same code/test cycle in Java takes  at least 10 times as long, often much worse.  At the same time rails offers.  Plus I can do a lot of things I can't do with Java, thanks to dynamic typing, better metaprogramming facilities, and things like the rails console.  Plus, from a hosting perspective you can get much better deals on hosting a rails application than a full Java servlet environment, which tells you something about the relative difficulty of hosting rails and J2EE.

Why would I use it instead of PHP?  The framework does enough magic that getting an application up and running is faster than raw PHP (disclaimer:  I haven't used any of the newer PHP frameworks like CakePHP . Maybe they're more on par with rails here.).  MVC is nicely broken out.  Ruby itself is a wonderful language to develop in.  Yes, you're going to have to do more work around deployment.  I can't imagine going live using FastCGI, SCGI, or that sort of thing.  But learn how to set up a mongrel cluster.  That seems to work.

I think Java and PHP still have their places as well.  Java provides much better concurrency in a single process, better built-in capabilities for high availability, security, and a larger library/tool ecosystem.  Enterprises know how to deploy and maintain Java applications, and in general feel more comfortable putting those in production than a rails system.  PHP is lighter weight, will likely scale better out of the box, and is better understood by most hosting companies.

At the end of the day the decision comes down to what you're building and where you're deploying.  I think rails works in wide range of situations, but there is an equally wide range where PHP or Java may be a better choice.  ]]> Handling Multiple User Authentication With ActiveResource Clients tag:www.themcwongs.com,2008:/mcblog//1.5 2008-01-06T05:13:31Z 2008-03-02T05:14:03Z I'm in the process of building a REST service using rails and ActiveResource. On the server side everything was fairly straightforward, but I hit a snag when I moved to the web tier. For various reasons I wanted to have... McWong
Using the REST example from my last post, I extended the client in the following way.

require 'activeresource'

module Sample
  module Client
    class API
      #
      # Creates a module that serves as an ActiveResource
      # client for the specified user
      #
      def self.create_api(login = nil, password = nil)
        # specify the site.  Default to no credentials
        @url_base = "http://localhost:3000"
        @url_base = "http://#{login}:#{password}@localhost:3000" if login
        # build a module name.  This assumes that logins are unique.
        # it also assumes they're valid ruby module names when capitalized
        @module = login ? login.capitalize : "Default"
        class_eval <<-"end_eval",__FILE__, __LINE__
        module #{@module}
          class Post < ActiveResource::Base
            self.site = "#{@url_base}"
          end
          class Comment < ActiveResource::Base
            self.site = "#{@url_base}/posts/:post_id"
          end
          # return the module, not the last site String
          self
        end
        end_eval
      end
    end
  end
end

This new version allows you to create an api for a specific user. You can squirrel this away and use it for specific users, allowing you to use multiple connections to your rest service. A quick irb session follows

>> require 'ares_sample_client'
=> ["Sample"]
>> api = Sample::Client::API.create_api
=> Sample::Client::API::Default
>> p = api::Post.find(1)
=> #<Sample::Client::API::Default::Post:0xb715af74 @attributes={"updated_at"=>Wed Jan 09 02:36:34 UTC 2008, "id"=>1, "content"=>"The first post", "user_id"=>1, "created_at"=>Wed Jan 09 02:36:34 UTC 2008}, @prefix_options={}>
>> p = api::Post.create(:content => "should fail")
ActiveResource::UnauthorizedAccess: Failed with 401 Unauthorized
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:125:in `handle_response'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:112:in `request'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:101:in `post'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:803:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:636:in `save_without_validation'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/validations.rb:262:in `save'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/core_ext/object/misc.rb:28:in `returning'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from (irb):4
>> auth_api = Sample::Client::API.create_api('test1','test1')
=> Sample::Client::API::Test1
>> p = auth_api::Post.find(1)
=> #<Sample::Client::API::Default::Post:0xb713dde8 @attributes={"updated_at"=>Wed Jan 09 02:36:34 UTC 2008, "id"=>1, "content"=>"The first post", "user_id"=>1, "created_at"=>Wed Jan 09 02:36:34 UTC 2008}, @prefix_options={}>
>> p = auth_api::Post.create(:content => "should succeed!")
=> #<Sample::Client::API::Test1::Post:0xb713312c @attributes={"updated_at"=>Thu Jan 10 04:01:53 UTC 2008, "id"=>7, "content"=>"should succeed!", "user_id"=>nil, "created_at"=>Thu Jan 10 04:01:53 UTC 2008}, @prefix_options={}>
>>

]]> Dealing with Maven bloat and complexity tag:www.themcwongs.com,2008:/mcblog//1.3 2008-01-05T05:04:37Z 2008-03-02T05:05:15Z I recently came across an article questioning whether Maven is too complex and bloated. The short answer is it can be. If you aren't taking some steps up front to deal with some of Maven's quirks, the benefit it provides... McWong
At my job we have a fairly sophisticated Maven project. It involves custom code generation plugins, cross-platform C++ compiles and deployments, assemblies, you name it. The number of engineers working on this project is probably around 80, and the number of Maven projects for this system is probably around 50.

It's not the most beautiful system in the world, but at the end of the day a lot of Maven's benefits are realized. Your average developer can check out an individual component from src, run "mvn test", and be reasonably assured
that things will compile, run, and execute unit tests, with minimal up front configuration of the build environment.

This is no small feat. It didn't start off easy. Maven is complicated. It's large. Documentation could no doubt be better. But at the end of the day, we're in a better place than we would have been with ant or make. Some of the steps we've taken to make things easier follow.

   1. Create your own repository. If you want to be able to do repeatable builds, don't let your projects access any repositories outside your control. Poms get broken, sites go offline... this causes all sorts of chaos. We ended up with a single repository server, but multiple repositories. The salient ones are maven-releases, maven-snapshots, our-releases, and our-snapshots. We put the artifacts we needed into maven-releases and maven-snapshots manually. We added profiles that can be enabled to get to the public repositories, but these are never enabled on our automated build machines.
   2. Use inheritance correctly. It's really tempting to use pom inheritance to capture project structure. This is what module tags are for. Pom inheritance is to allow you to apply similar configurations easily (like Java inheritance). It took us a long time to unwind this mess. We have a base-java pom that sets up all the reports we want to run for java, a base-model one to handle domain models,
   3. Fight the urge to tightly couple your large project. Maven leads you to fine grained componentization, which leads to a looser coupling of components in the build/release sense. There is a natural tendency to be uncomfortable with this (what's really going into my final build), but fight the urge to make it one giant system that gets built from the ground up. Executing releases is a nightmare with one giant system.
   4. Use version ranges where possible. This makes dealing with #3 easier. The odds that a component needs that specific release of a component (especially if you're doing agile and releasing every 30 days) is pretty slim. Most just need the latest. Also get familiar with the dependency convergence report.
   5. Make sure people understand what Maven goals are necessary for doing work. If lots of people are complaining about site generation taking too long, you have a clue that people don't get this. Your average developer should be running test and install, very rarely site.
   6. Decide on a versioning scheme up front, and make sure you can execute on it. Nothing is more frustrating than realizing it takes two weeks to get all your poms revved to the next revision. The maven release plugin has been fairly unreliable, but it leads you into a set of best practices that work even if you're taking the steps manually.

After taking these steps, the nearly universal sentiment is that while Maven is indeed complicated, it's a step forward when compared to ant/make. At the end of the day, designing a build process for large systems is difficult and rarely gets the attention from the development org it deserves. Maven doesn't make it dirt simple, but it makes the overall management of the build system simpler. ]]> Selling Automated Testing to Management tag:www.themcwongs.com,2008:/mcblog//1.2 2008-01-05T05:03:04Z 2008-03-02T05:04:27Z I don't think there are that many developers out there these days that don't think that automated unit testing is a good idea. Most seem to agree that automated functional and integration testing are worthwhile as well, even if they're... McWong
There are generally two cases where you need to sell the effort involved in building and maintaining these suites. The first (and easier) is when you already have the suites in place, but need to invest in maintaining them. This is an easier sell because usually management will have seen the benefit and/or the ongoing expense is fairly small, and justifying the ongoing effort isn't all that bad. I'll throw new development in here as well... writing the suites as you go usually isn't as bad as retrofitting suites onto an existing code base.

Which brings us to the second case. When you have a legacy product, adding automated test suites is usually time consuming at best, and technically challenging in most cases. Your legacy product probably wasn't built in a way that encourages automated testing. The interfaces may not be there, it's tough to run in a test sandbox, etc. If you want to get automated tests working, you're probably facing a significant investment in people terms, and could be facing some infrastructure expenditures as well.

The wrong way to pitch this is to go to management, roll out your proposal and say that you need to spend $500k over the next three months building something that provides purely qualitative benefit. Unfortunately most attempts at quantification are difficult. You can talk about improved quality, improved productivity, increased agility... but at the end of the day that $500k is staring them down and making everybody in the room uncomfortable.

A better way is to talk about it in terms that can be strictly quantified, and the easiest one I've found is around regression testing. Figuring out how much you spend each year on regression testing is usually pretty straightforward.

   1. Write down how many times a year you execute a full regression test
   2. Write down many people a single test cycle takes
   3. Write down how long it takes
   4. Figure out how much a tester's time is worth.

Doing the math hear should give you a dollar amount for your regression expenditure. The next step is simpler, all though less precise. Estimate how much you think you could cut the time you got for #3. From what I've seen this should be on the order of 50%-80%. This gives you a total savings realized from automated testing.

A quick example may help

   1. We release twice a year, and execute on average two regression test cycles per release (one beta, one GA)
   2. Each test requires four test engineers
   3. The test cycle takes four weeks
   4. Each tester costs the org $100k a year

this means we have a total cost of 4*4*(4/52)*100k which is about $123k. If I can cut my regression test time to 1 week, I save at least $92k per year. Keep the last two words in mind... that's per year. Forever.

Keep in mind this is a minimum. You still have all the intangible benefits. But $92k/year is something the business folks can plug into their project valuation tools and evaluate. When we ran these numbers for our project it became clear that dedicating people full time to build out automated test suites for existing projects made perfect financial as well as technical sense. ]]> Quick REST with ActiveResource and rails tag:www.themcwongs.com,2008:/mcblog//1.4 2008-01-03T05:10:42Z 2008-03-02T05:13:00Z McWong Posted by McWong 53 days ago
This walkthrough shows the creation of a simple REST API using ActiveResource. It includes a nested model, authorization scheme, and a client library. The intent is not to exhaustively explore ActiveResource, but to show how to quickly get up and running with a reasonable REST API.

Step 1: create an application

rails ares-demo

I'm using rails 2.0.2 here, but I don't think anything is version specific.

Step 2: install two great rest plugins
Resource This is a plugin that adds default rest-based actions to a controller. It has so-so support for nested resources... you won't be able to use it out of the box for HABTM relationships for instance, but for strict hierarchies it works well, and really beats hand coding the controllers. You can install it with

script/plugin install http://jnewland.com/svn/public/ruby/rails/plugins/resource_this/

Restful Authentication is derived from acts_as_authenticated, and adds a rest-ish session controller, as well as the usual controller helpers (login_required, etc.) You can install it with

script/plugin install http://svn.techno-weenie.net/projects/plugins/restful_authentication/

Step 3: setup restful authentication
This bit is easy. Just run the generator to get your migrations for users created, along with a couple other things. I honestly didn't dig into this too much, it just works.

./script/generate authenticated user sessions

Step 4: create application model
Next up is generating your model. We'll do a simple one. Posts contain many Comments. Both Posts and Comments belong to a user.

./script/generate scaffold post
./script/generate scaffold comment

Next set up your migrations. db/migrate/002_create_posts.rb should contain

class CreatePosts < ActiveRecord::Migration
  def self.up
    create_table :posts do |t|
      t.column :content, :text
      t.column :user_id, :integer
      t.timestamps
    end
  end

  def self.down
    drop_table :posts
  end
end

and db/migrate/003_create_comments.rb should contain

class CreateComments < ActiveRecord::Migration
  def self.up
    create_table :comments do |t|
      t.column :content, :text
      t.column :user_id, :integer
      t.column :post_id, :integer
      t.timestamps
    end
  end

  def self.down
    drop_table :comments
  end
end

Next you'll need to add the relationships to your model classes. Add the following bit to the front of app/models/user.rb

class User < ActiveRecord::Base
  # custom relationships
  has_many :posts
  has_many :comments

app/models/post.rb should look like

class Post < ActiveRecord::Base
  belongs_to :user
  has_many :comments
end

and app/models/comment.rb should look like

class Comment < ActiveRecord::Base
  belongs_to :user
  belongs_to :post
end

Now run migrate to get your database all configured

rake db:migrate

Step 5: populate dummy data
Next we'll want to get some default data in there. Execute script/console and run through the following

>> u1 = User.new(:login => "test1", :email => "test1@foo.com", :password => "test1", :password_confirmation => "test1")
=> #<User id: nil, login: "test1", email: "test1@foo.com", crypted_password: nil, salt: nil, created_at: nil, updated_at: nil, remember_token: nil, remember_token_expires_at: nil>
>> u1.save
=> true
>> u2 = User.new(:login => "test2", :email => "test2@foo.com", :password => "test2", :password_confirmation => "test2")
=> #<User id: nil, login: "test2", email: "test2@foo.com", crypted_password: nil, salt: nil, created_at: nil, updated_at: nil, remember_token: nil, remember_token_expires_at: nil>
>> u2.save
=> true
>> p = Post.new(:user => u1, :content => "The first post")
=> #<Post id: nil, content: "The first post", user_id: 1, created_at: nil, updated_at: nil>
>> p.save
=> true
>> c = Comment.new(:user => u2, :post => p, :content => "first!")
=> #<Comment id: nil, content: "first!", user_id: 2, post_id: 1, created_at: nil, updated_at: nil>
>> c.save
=> true
>>

Step 6: resource_this!
Now the boring part us over. So far we have a basic rails app. Now we get to the REST part! You're three edits away... Change app/controllers/posts_controller.rb to the following

class PostsController < ApplicationController
  resource_this
end

Change app/controllers/comments_controller.rb to the following

class CommentsController < ApplicationController
  resource_this :nested => [:post]
end

And finally update config/routes.rb

ActionController::Routing::Routes.draw do |map|
  map.resources :posts do |post|
    post.resources :comments
  end

  map.resources :users

  map.resource :session
end

That's it! You're ready to look through your API

Step 7: browse
Start up your server with script/server. You'll need to clean up the erb views if you want to use the HTML interface, but your xml interface should be working just fine. Try the following urls for verification

    * http://localhost:3000/posts/1.xml
    * http://localhost:3000/posts/1/comments.xml

Step 8: lock it down
So far our service is open to anybody. We'll add very simple authentication at this stage... we'll require a valid user for anything other than show and index requests. All we have to do for this is include AuthenticatedSystem in our application controller, and add a before_filter. Make app/controllers/application.rb look like the following

class ApplicationController < ActionController::Base
  include AuthenticatedSystem
  before_filter :login_required, :only => [:create,:edit,:new,:update,:destroy]
  helper :all # include all helpers, all the time

  # See ActionController::RequestForgeryProtection for details
  # Uncomment the :secret if you're not using the cookie session store
  protect_from_forgery # :secret => 'b845cf981afbfef06600e5d7f23e0fed'
end

Step 9: create client lib
We could test this with curl or a browser, but the real fun of ActiveResource is getting to use ruby objects. So... we'll create a simple client api. Create a new file in lib called 'ares_sample_client.rb' and add the following code

require 'activeresource'

module Sample
  module Client
    class API
      class Post < ActiveResource::Base
        self.site = "http://localhost:3000"
      end
      
      class Comment < ActiveResource::Base
        self.site = "http://localhost:3000"
      end
    end
  end
end

Now we can play with our creation using the console. Make sure your server is running on port 3000, and startup script/console.

         
>> require 'ares_sample_client.rb'
=> ["Sample"]
>> posts = Sample::Client::API::Post.find(:all)
=> [#<Sample::Client::API::Post:0xb710bec4 @attributes={"updated_at"=>Wed Jan 09 02:36:34 UTC 2008, "id"=>1, "content"=>"The first post", "user_id"=>1, "created_at"=>Wed Jan 09 02:36:34 UTC 2008}, @prefix_options={}>]
>>

>> p = Sample::Client::API::Post.create
ActiveResource::UnauthorizedAccess: Failed with 401 Unauthorized
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:125:in `handle_response'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:112:in `request'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/connection.rb:101:in `post'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:803:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:636:in `save_without_validation'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/validations.rb:262:in `save'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from /usr/lib/ruby/gems/1.8/gems/activesupport-2.0.2/lib/active_support/core_ext/object/misc.rb:28:in `returning'
        from /usr/lib/ruby/gems/1.8/gems/activeresource-2.0.2/lib/active_resource/base.rb:339:in `create'
        from (irb):3
>>

Our API works, and the authentication is as it should be. We didn't specify credentials, so we can't actually create a post. Let's fix up our client lib to add credentials

module Sample
  module Client
    class API
      class Post < ActiveResource::Base
        self.site = "http://test1:test1@localhost:3000"
      end
     
      class Comment < ActiveResource::Base
        self.site = "http://test1:test1@localhost:3000"
      end
    end
  end
end

Note that credentials apply to the whole class... this makes the API fairly easy to use, but makes using multiple users difficult. Anyway, with our authentication in place, let's try again

>> p = Sample::Client::API::Post.create(:content => "Should succeed", :user_id => 1)
=> #<Sample::Client::API::Post:0xb71b81d8 @attributes={"updated_at"=>Wed Jan 09 02:55:22 UTC 2008, "id"=>3, "content"=>"Should succeed", "user_id"=>1, "created_at"=>Wed Jan 09 02:55:22 UTC 2008}, @prefix_options={}>
>> p.save
=> true
>>

Sweet! Now we can not only browse posts and comments, but create them as well!
]]>