Juggernaut, Push Server, The future of the web

My dear blog, it's been 3 long month since my last post. Forgive me.
I had so many stuff to do:

• Coding - have you seen the new forums at SA? (http://seekingalpha.com/symbol/aapl/mb/topics)
  Kudos to everyone in SA tech team
  
• Running
• Learning Italian
• A little more ...

But than - suddenly, unexpectedly, I ran across... The next hottest thing in WEB - JUGGERNAUT!

Yes, treat it as the new Gospel in our business... web.

1. What is it?
2. What's new? we had it since ever!
3. How will it change the web?
4. How can I start?

1. What is it?

It's more than Ajax.
Ajax is an HTTP connection that's opened especially per request and closes as it ends.
Juggernaut, with Flash OBJECT on the Client Side and Ruby Server (Push Server) on the Server Side, allows you to keep an open connection and thus allow real-time connection.



YU HU!!! what's so amazing about it???

Very high frequency (on my machine ~ 2000 req/second)
Scalability (based on EventMachine)
Low Hosting Costs (less servers and memory is needed than if Ajax was used)

2. What's new? we had it since ever!

Juggernaut does it well, easy, cheap and smart.

Client Side: JavaScript + Flash Object
Server Side: Ruby

3. How will it change the web?

Chat is nice but think of all the collaboration and real-time data services you know, think of it functioning 10x faster.

4. How can I start?

Luckily, Juggernaut code is updated very frequently - improvements are made and bugs are being treated.

Unfortunately, it changed it's already somewhat known API and so the already written tutorials are obsolete.

Luckily, I am going to explain how to get this sweety to work in my next post:

http://www.dorkalev.com/2008/04/juggernaut-up-to-date-tutorial-1042008.html

Kudos to Alex MacCaw from http://www.eribium.org/ for creating and maintaining Juggernaut

Standards VS. Performance

Today I had to create an XML with ~ 7000 records.

The first role looked like this:


Controller
def all_articles
@articles = Article.find_all
render :layout => false
end

View - articles.rxml
xml.instruct! :xml, :version => '1.0'
xml.root do
xml.articles do
@articles.each do |article|
xml.article do
xml.id article.id
xml.title article.title
end
end
end
end


The size of the XML was ~ 800kb and it took 7 seconds to render.
Horror, ah?

The truth is, after thinking, I don't really need an XML in this specific case. All the data is going to be read by a JavaScript that I can modify to get any format I like.

The solution:


Controller
def all_articles
ActiveRecord::Base.connection.execute("SET SESSION group_concat_max_len = 999999999;")
@articles_big_csv = ActiveRecord::Base.connection.execute("select group_concat(articles.id,'>',articles.name SEPARATOR '<') from articles").fetch_row.first render :layout => false
end

View - aricles.rxml
xml.instruct! :xml, :version => '1.0'
xml.root do
xml.articles @articles_big_csv
end


The output of "select group_concat(articles.id,'>',articles.name SEPARATOR '<') from articles" is in the format of "ARTICLE ID>ARTICLE TITLENEXT ARTICLE TITLE..."
(I can use the "<>" as separators since I dont allow them in an article name; else I could use any sepeartors I like)

Later, on the parsing JavaScript side I will split the string by '<' to get the rows and each row I will split by '>' to get it's columns.

The size of this output is now 200kb and rails can render it 11(req/s)

(...if you are lucky and your server gzips everything it goes down to 80kb)

What did we do?
1. Smallest size: we didn't use XML standards but a simple primitive string with separators.
2. Fastest tool: we did all the work in MySql which is much faster than Ruby.



Kudos to Roman Krom for the 'let sql do the work for you' philosophy
Kudos to Boris Peterbard for teaching me the group_concat magic

Why I love Ruby on Rails and what should be said on it's performance , part 1

In less than a week, you can establish a working model for your final product.

It is, however, very unwise to consider this is the end of the process.

Ruby on Rails allows ultra-fast development but all this comes in exchange to performance and efficiency.

At the end of the first development period you find yourself with a great working copy but with very low efficiency and as soon as your site starts growing you have to add more servers, more memory, more everything.

What can and should be done in this stage is the infamous 're-factoring' procedure with high attention on performance.

For example for Article and User where each user has many articles and each article
belongs to a user ( 1 -to- many relationship)

If we want to get an article's user email we railslishly write:

Article.find(123).user.email

Which feeds our logs with:

Article Load (0.002447) SELECT * FROM articles WHERE (articles.`id` = 123)
User Load (0.002856) SELECT * FROM users WHERE (users.`id` = 8)

This isn't a problem if you have a small application, this isn't a lot ah? 2 are 100% more SQL queries than only 1 query...

Actually, it's okay to live with it as long as you don't need to grow. When you do, you have to learn SQL.

The truth is even basic SQL can save you here (something PHP or ASP developers would never brag about...):

select * from articles inner join users on users.id = articles.user_id where articles.id = 123

Word is around that Ruby on Rails is slow because of Ruby. The truth for large scale websites is that Rails has much more to do with performance than Ruby.

Let's try a Tiny Profiling Utility on the the scenario described above, where in both cases I want to get an author email according according to it's ID:

>> prof(1000) { Article.find_by_sql('select * from articles inner join users on users.id = articles.user_id where articles.id = 123').first['email'] }
time: 0.004523629(s) . req/s: 221.061453094407
=> nil
>> prof(1000) { Article.find(123).user.email }
time: 0.005518112(s) . req/s: 181.221403262565
=> nil
>>

For one simple action we see a difference of around 20% in speed. Take into consideration that more time consumed by the SQL server is more time other queries are waiting in line, etc. etc.
Let's do the previous test for two articles (123 and 567):

>> prof(1000) {
a = Article.find_by_sql('select * from articles inner join users on users.id = articles.user_id where articles.id = 567 or articles.id = 123')
a[0]['email'] + a[1]['email']
}
time: 0.004959934(s) . req/s: 201.615586013846
=> nil
>> prof(1000) {
a = Article.find(567,123)
a[0].user.email + a[1].user.email
}
time: 0.008703669(s) . req/s: 114.894075130844=> nil

Now that's around 40% difference... imagine what would happen in the case of Many-to-Many connection...

The truth is that eager loading in Rails, should solver this issue - but for some reason it's slower... :
>> prof(1000) {
a = Article.find(567,123, :include => :user)
a[0].user.email+a[1].user.email
}
time: 0.006641206(s) . req/s: 150.575061216291
=> nil

Rails is a great framework and Ruby is a wonderful language, use them and learn some SQL too, they work much faster together...Start using it now: http://rubyonrails.com

Don't forget your sql... http://w3schools.com/sql/default.asp

enjoy!

Ruby 1.9.0

Have you heard it was released?

http://www.ruby-lang.org/en/news/2007/12/25/ruby-1-9-0-released/

howto install:

autoconf
./configure --prefix=/usr/local/ruby1.9
make
sudo make install



For a long while we were expecting this product that should increase Ruby performance tremendously.
Mind you, I had yet seen a web application running slow because of Ruby. Most of the cases it was misusage of database and memory structures, the major down-pit Rails developers fall into (newbies as well as experts).

Ruby 1.9 is here anyway (dev. version), let's give it a try!

Tiny profiling utility


def prof(t = 1)
  starting_time = Time.now
  t.times { yield }
  dlta = (Time.now - starting_time).to_f
  puts "time: #{dlta.to_s}(s) . req/s: #{(1/dlta)}"
end


• Use prof to measure how long a function or any block of code takes.
• Use parameter t>1 to run the block code t times and return the average time per test
  

Examples:

>> prof { Flight.find_first }
time: 0.002032(s) . req/s: 492.125984251969
=> nil
>> prof { Flight.find_first }
time: 0.00201(s) . req/s: 497.512437810945
=> nil
>> prof(10) { Flight.find_first }
time: 0.001351(s) . req/s: 740.19245003701
=> nil
>> prof(100) { Flight.find_first }
time: 0.0012727(s) . req/s: 785.731122809775
=> nil
>> prof(1000) { Flight.find_first }
time: 0.001415021(s) . req/s: 706.703292742652
=> nil
>> prof(10000) { Flight.find_first }
time: 0.0013922995(s) . req/s: 718.236270285237
=> nil
>>


ActiveRecord::Base#exists?

class ActiveRecord::Base
def self.exists?(conditions, unique_id = nil)
count1(conditions, unique_id)>0
end
end

Returns true/false if at least one row that follows ceratin conditions is found

• Can only be used along with count1
• unique_id - used when you still want to allow a certain raw (defined by id...) to be left out of the count. Useful for validations, ie:
  
User.exists? ['nick != ?',user.nick], user.id