On the web server scalability and speed are almost the same thing

Friday, 01 April 11
Today I complained on twitter that the obvious way to start writing an application with Ruby and Sinatra is too slow by default. Substituting a template calling the
erb :index
takes a few milliseconds (just substituting "hello" in toy template). Benchmarking it with apache benchmark shows how a similar hello world app written in PHP serves 1500 requests per second, while using erb this trivial substitution can handle 250 requests per second (both benchmarks ran on my MBA 11").

Probably there is some lame reason why this happens, like opening the template file again and again or alike (excluding the template the method dispatch of Sinatra + Mongrel is able to serve 600 requests per second, so it is the template substitution).

My point is, it is not ok that by default it is so damn slow. It's not Ruby that is slow, this is not computationally intensive code. Is is simply that nobody cares to provide the basic solution as a fast one apparently. I'm sure that with a few tricks I can handle that, and be happy using Ruby, that is a language that I love semantically, instead of PHP that is a language that I don't like at all.

Many twitter replies were in the tone like "but speed is not equal to scalability".

I disagree about that. If there is something cool about web programming is that often the web side is trivial to scale conceptually. Just add more web servers, there is no shared data.

Your only bottleneck in a web app should be: the databases, the workers. There are no excuses for the page generation to be slow. In the web speed is scalability because every web server is conceptually a parallel computation unit. So if the web page generation takes 10 ms instead of 100 ms I can server everything with just 10% of the hardware.

I love Ruby but this is just another instance of "slow by default" that I don't understand very well, and it is not a matter of optimizing for programmer's performance, you can substitute a simple template 1500 times per second in a MBA, how the lame PHP is teaching us.

Edit: the code I used as requested. The main point of this two code fragments is, it should be similar to what most people would write as a first example to accomplish that specific work. The code tries to also be conceptually equivalent, loading the template at every request, and performing a substitution involving parsing the template and evaluating the code.

PHP code:
$user = $_GET['username'];

The above runs at 1500 requests per second.

tempalte.php is:

<html> <body> <? echo("Hello ".$user); ?> </body> </html>

Ruby code:
require 'rubygems'
require 'sinatra'

if 1 require 'erubis' Tilt.register :erb, Tilt[:erubis] end

before do # Bla end

get '/slow/:username' do @var = "Hello #{params[:username]}" erb :index end

get '/fast/:username' do return "Hello #{params[:username]}" end

get '/subst/:username' do f = File.open("template.tpl") t = f.read res = t.sub("%content%","Hello #{params[:username]}"); f.close return res end

The template is:

Username = <%= @var %>

Yes, just one line with one var.

  • /fast is basically as fast as Sinatra/Mongrel dispatch, so reaches 550 requests/second.
  • /slow does what the PHP code actually does, and runs at 300 requests/second. This is the sad part. should be more or less like 'fast'.
  • /subst is (pretty incredible) much faster than /slow. 450 requests/second, even if the same file is opened again and again in ruby land.

Another data point is this very blog you are reading. It's the lamest PHP (written by me in a day just to have a blog engine with a few special features I liked), using many many MySQL queries per page. It serves 250 pages per second in a blog post with 15 comments, including parsing the post that is done with my own function that processes a markdown-alike stuff to convert it into HTML. The same 250 requests second we have with the default Hello World using the erb template.
Posted at 14:21:31 | permalink | 49 comments | print
Do you like this article?
Subscribe to the RSS feed of this blog or use the newsletter service in order to receive a notification every time there is something of new to read here.

Note: you'll not see this box again if you are a usual reader.


Dan writes:
01 Apr 11, 15:10:16
If you can throw money at a problem to make it go away, it scales.
Mikushi writes:
01 Apr 11, 15:11:54
Interesting, always had that feeling about Ruby.
Even your 1500req/sec seems low to me for a simple Hello World in PHP. Do you have the sample code somewhere?
With a simple framework, data fetching from MySQL (fairly basic though), no caching, i still can get to 1600req/sec on Apache2.2+PHP5.3 .
from a php developer who loves ruby writes:
01 Apr 11, 15:12:29
i really enjoy ruby, but unfortunately i have never been able to justify the time investment to use it for web applications. this is because it has always lacked maturity in critical areas such as unicode and performance. you are right to point out this problem, and it is a problem. good luck to the ruby community in resolving this...
Colin writes:
01 Apr 11, 16:49:26
First, anyone using any framework in any language in production will configure it to optimize performance and reduce page generation time. If your site expects any interesting request rate, then optimization will always be required, regardless of the framework. An Hello World example using a "default" configuration is never representative of how any real application will behave.

For your specific tests, could you please share the PHP and Ruby code you used so we can assess you are comparing oranges to oranges?
FunnyGuy writes:
01 Apr 11, 16:50:32
Testing on localhost doesn't give any indication of speed from the user perspective. Network congestion is far more damning, i.e., it doesn't matter how fast your app runs locally if the network is clogged...
HN Reader writes:
01 Apr 11, 16:52:01
Could you please post the version of Ruby and the Ruby server you used?
Chris writes:
01 Apr 11, 17:04:49
I think people asking about what is being used for Ruby are missing the point. I'm a PHP guy who also does Python stuff and has used Rails in the past. Antirez is right in asking why are things so slow out of the box for his Ruby solution compared to the PHP one.

As for those in the "oh anyone using a framework in production will optimize it" are also missing that point. What does it say about a particular framework that you have to optimize it to go from 250 requests per second to 1500 requests? If you are having to do things outside of the framework itself to make it happen (ie adding caching layers) doesn't that strike you as, I dunno, wrong?
Joran Greef writes:
01 Apr 11, 17:39:50
I have used Ruby and PHP. Have you tried Hello World in Node?
Kent writes:
01 Apr 11, 17:47:30
Apple to Oranges. Try using any PHP framework like Cake and you will get some comparative results. Using thin with plain ERB I get over 3000 req/s:
antirez writes:
01 Apr 11, 18:02:44
@Kent: I think by default Sinatra is processing the same template again and again, reading it, processing it, every time. This is what PHP is doing as well. So I think your example is apple to oranges ;)
Jimbob writes:
01 Apr 11, 18:13:46
"I think people asking about what is being used for Ruby are missing the point."

I imagine they just want to be sure you are using the latest and greatest - ie. Ruby 1.9.2 on Passenger 3.0.5. Even then, you are not talking about an apples-to-apples comparison, because you are still comparing a ruby "framework" (Sinatra), to (I assume) a single file php script. To be more apples-to-apples, you would have to use something like cakephp for the php part. Unfortunately, mod_ruby is pretty much dead, so there is no simple way to put up a single file ruby script on the web.

Also, as the web application size and complexity grows, I would imagine the speed difference between php and the ruby framework would diminish.
antirez writes:
01 Apr 11, 18:15:16
Oh! Somebody mentioned mod_ruby? it's dead but it is very fast! I used it in the past for large production sites with good results. Of course without erb or other things like this, but with my own micro framework.
Me writes:
01 Apr 11, 18:32:15
I've had similar issues in production environments where php just scaled better. The last time I touched RoR was 2 years ago when I tried to get our inhouse app to scale as well as the php stuff - the ruby side took a lot more creativity to get it to even come close. Looks like things haven't changed in those 2 years.
Me writes:
01 Apr 11, 18:36:21
@antirez: mod_ruby? maybe your definition of "fast" differs from mine.

One of our first mistakes was to use mod_ruby, after we switched to mongrel we had "better" performance, but not nearly great.
antirez writes:
01 Apr 11, 18:38:18
@Me: I used mod_ruby directly without any RoR in the middle, and I think that when RoR runs used to run on Mod Ruby did something strange that killed the performances or alike.

But per-se mod_ruby is very fast, it's the same interpreter again and again serving requests, so it is very dangerous (if you leak a file, it is leaked forever, not just for that request), but also very fast.
AX writes:
01 Apr 11, 18:45:50
You're right about performance, but "slow by default" is just wrong here. You're testing a production-type architecture (apache+PHP) against a development-type Ruby architecture (nobody uses Mongrel, for instance). More to the point, on either language you'd have caching, etc. enabled so even 1500 reqs/s would be slower than either site in reality.

Basically, you're doing a benchmark that doesn't correlate well with real world high-performance situations. You're right that nobody's going to optimize performance for a development environment.
watsonian writes:
01 Apr 11, 18:55:52
It seems to me that the primary issue here is that you're using a templating library in the slow Ruby test and are simply echoing the text out a la the /fast Ruby test in PHP. A fairer comparison would be to use some PHP templating system to render a single line with an evaluated variable

Also, what AB settings were you using (or did I miss them?)? In a similar /fast test using Mongrel I'm seeing ~875 reqs/sec.
etaty writes:
01 Apr 11, 18:57:00
For Mongrel2+PHP you can use the Photon Framework

"The baseline Photon returns about 2500 transactions per second. More than 90% of the PHP baseline!"
Ramon Leon writes:
01 Apr 11, 19:01:40
> In the web speed is scalability because every web server is conceptually a parallel computation unit. So if the web page generation takes 10 ms instead of 100 ms I can server everything with just 10% of the hardware.

That makes no sense at all. Scalability has shit to do with either speed or how much hardware you use.

Scalable means doubling the hardware nearly doubles the throughput. If your app can be sped up by a predictable amount by adding more machine, it's scalable; that's it.
BraveNewCurrency writes:
01 Apr 11, 19:41:51
I'm ecstatic that people like Salvatore obsess over every CPU instruction. (Remember when they were 1ms each?) But we shouldn't force *everyone* to obsess over performance, or nothing useful will ever get built.

Let's look at what happens when the ERB framework gets 50% faster: Going from 600 request per second to 1200 requests per second sounds impressive. But that's really only saving us 833 nanoseconds per request. For most non-trivial apps, that extra overhead is just a rounding error. Overall, the app doesn't get twice as fast -- it gets 1% faster.

I'm all for speeding up ERB (in fact, there are dozens of alternatives, can we get rid of ERB?). But let's not pretend that ERB is always going to be the bottleneck of your website.

A big thanks to the Redis team -- they worry about performance so we don't have to!
Trent Strong writes:
01 Apr 11, 19:54:46
> That makes no sense at all. Scalability has shit to do with either speed or how much hardware you use.

>Scalable means doubling the hardware nearly doubles the throughput. If your app can be sped up by a predictable amount by adding more machine, it's scalable; that's it.

I think you are vastly over simplifying the scalability concept. As engineers in the real world, not in some universe of semantics, the notion of scalability of client/server architecture is affected by many factors, definitely including the throughput of a single machine/process, but also including very important factors like cost. The cost to scale a web application serving R requests/s on a single server to R_TOT total requests/s is pretty simple:


If you are trying to scale a web application and can only serve some dismal throughput with each web server (say ~10 req/s), the cost to scale your app is going to be much higher if you actually have the need to scale, and could be the difference between a successfully bootstrapped company and burning through VC. Not to mention, even with load balancers like HAProxy and the like, adding more web servers to your architecture adds more complexity to your system, which will introduce more (vague, undefined, definitely real) costs down the road.

In the real world, we have to be scalable over many different factors.
Myxomatosis writes:
01 Apr 11, 22:01:32
Stop arguing about "scalability" and what it means, etc. The point here is that writing Ruby for the web is SLOW. It NEEDS to be faster.
derp writes:
01 Apr 11, 22:16:02
april fools? no?... well, ruby is plenty fast enough for plenty of companies to make money with. no amount of blog articles or incomprehensible comments is going to change that, so let's all just get back to work shall we?

wait, i forgot, my app is done because i wrote it in ruby, so i'm going to nap a bit instead while you force-pass php like a kidney stone
JavaRocks writes:
01 Apr 11, 22:51:34
thats it 1,500 requests per sec!! In JAVA i can get 80,000 requests per sec.

Yes, you sound just as stupid.
Glenn Rempe writes:
01 Apr 11, 23:35:52
I think you made a serious mistake when running this simple Sinatra app locally. We can't verify, since you broke the first rule of fake benchmarks and say nothing about how you ran this app on your local machine, but I'll assume for the moment that you were using the webrick Ruby server which is intended only for development and testing and is *never* used in a production like environment. This is why people who host real apps always choose Passenger, Thin, Mongrel, etc...

I just benchmarked your code on my local machine and achieved results that were definitely on par with the PHP script results you saw when I ran the code using the 'Thin' web server (which will get used by default if the gem is installed).

Here is the code for those who want to try it for themselves (a very slightly modified copy of the same code you posted above). Follow the instructions in the README.txt to try it for yourself. My machine is a new one, YMMV.



My Results? Between 800 and 1500+ requests per second as reported by Apache Bench which is roughly 3x (!) what you reported. Apache Bench was run with 1000 requests and a concurrency of 5 each time with a pre-measurement warmup run:

ab -n 1000 -c 10
=> Requests per second: 826.95 [#/sec] (mean)
=> Time per request: 12.093 [ms] (mean)

ab -n 1000 -c 10
=> Requests per second: 1362.72 [#/sec] (mean)
=> Time per request: 7.338 [ms] (mean)

ab -n 1000 -c 10
=> Requests per second: 1519.45 [#/sec] (mean)
=> Time per request: 6.581 [ms] (mean)

I'm kind of disappointed that you posted this rather FUD'y post in the first place. This is a partisan argument that has been hashed over a million times. Using PHP, Python, Ruby? Good! Use what makes you happy. These simple benchmarks are largely meaningless in a real production environment which is a much more complicated beast. Bottom line, Ruby is not 'slow' when compared to other dynamic languages. Your choice of language in the real world is much more affected by your skills and plans rather than any artificial and meaningless benchmark.

Matt McKnight writes:
01 Apr 11, 23:51:19
For apples to apples here you should run in Phusion Passenger, not mongrel. Indications are it would be faster. http://blog.phusion.nl/2010/06/10/the-road-to-pass...
Jimbob writes:
02 Apr 11, 01:20:22
@Glenn Rempe: any chance you could run the php app on that same hardware? I'd love to see how it compares, even if php still is faster.

Also, the article states he used mongrel (also not the best choice), not webrick.
Jimbob writes:
02 Apr 11, 01:25:39
@AX: At one time, mongrel was the cream of the crop for production ruby/rails/rack webservers. I ran one for a year or two before replacing it with Thin. Granted, today I don't know why anyone would pick mongrel for either production OR development, but I bet there are at least some leftovers from mongrel's heyday out there.
antirez writes:
02 Apr 11, 03:15:12
I used Mongrel, Sinatra run in production mode. The numbers are low just because it is a MBA 11" that is very very slow compared to an MBP.

I think many missed the point that if a template substitution takes all this time, substituting a real web page that involved a few templates N times in a single page completely trashed the performances to 10-30 requests/second in a *fast* server.

So the problem is not 300 or 600 requests per second, but a trivial site that is unable to serve more than 10 requests per second, with a latency penalty of 100 ms per user.
bjpirt writes:
02 Apr 11, 03:30:29
Whilst I agree with what you are saying...

speed != scalability

speed == efficiency

You could still have a fast web service that was architecturally unable to scale - it just might take you longer to get to the pain point. Equally you could have a well architected but slow app that continues to scale as you throw more hardware at it but ends up costing you more money.

I actually think that speed could be seen as more of a business metric because it boils down to what it will cost you to handle more traffic.

But I do share your frustration with the speed of Ruby web apps!
Glenn Rempe writes:
02 Apr 11, 03:44:54
Update to my previous comment. I actually squeezed an additional ~200 req/s by calling Thin directly with a config.ru rackup file and running thin daemonized.

~1796 req/s for the fast action in Sinatra.

See : http://bit.ly/g1XZw2

@jimbob : thx for pointing out he was using mongrel, somehow I missed that on first read. Your request for me to run the PHP on the same hardware is of course fair. As expected it is *very* fast. Likely as others have pointed out due to the fact that running this does not invoke any sort of PHP framework (e.g. Cake). Here are the results from my stock OS X 10.6.7 Apache with the same 10000 req and concurrency of 10:

Requests per second: 12608.59 [#/sec] (mean)

Fast, no doubt. Not as fast as if we had written it in C, but Fast. :-)

That being said, I would guess that a more Apples to Apples comparison of these two bits of code is likely the PHP script vs. a Ruby Rack application which doesn't include any web application framework (Sinatra or Rails) and seems more equivalent to straight up PHP code.

I whipped up a tiny Rack app, which I think is more similar to the /fast/:username action in the previous code for comparison (its also in the repository I linked to above). Many folks are using Rack apps (or Rails 'Metal') to avoid framework code for actions that need to be very very fast in the real world or in artificial benchmark like this one.

The rack app runs at about:

Requests per second: 4100.23 [#/sec] (mean)
Time per request: 2.439 [ms] (mean)

Not as fast as the PHP, but still *very* fast. Bottom line again is that this kind of test, while fun and provocative, is meaningless in the real world as these simple actions do not reflect the real world usage of templating frameworks, databases, network latency, etc. that will get in your way long before you get anywhere near these kinds of numbers. Not to mention the caching layer which is likely far faster than either of these tools and is a standard layer in any real world application at scale. The caching layer is really the equalizer between any two real world apps written using different tools.

My advice still stands, use the one that you feel makes you more productive, has the tools and extensions you need available, and is more fun.

Konstantin Haase writes:
02 Apr 11, 04:08:26
You are likely to run in development mode, which will indeed reload the template on every request, or you are running on Ruby 1.8.6, which will, due to a bug in Ruby, also cause templates to be reread (and reparsed, which is way slower than your simple subst). Running exactly your app on an two year old MBP on Ruby 1.9.2:

development: 654 req/sec
production: 1086.68 req/sec

development: 1168.39 req/sec
development: 1390.48 req/sec

development: 1006.90 req/sec
development: 1020.50 req/sec

You run in production mode with `-e production`.

Your PHP app gives me 1608.77 req/sec (PHP 5.3.3).
antirez writes:
02 Apr 11, 04:32:10
@Konstantin: you are contradicting yourself posting this numbers ;) If /slow and /fast are different then the template substitution is taking an huge time in your test as well. Did you noticed the multiplication factor is the same in your benchmark and in mine?

Substituting "Hello" in a one line template should not be measurable with apache benchmark unless there is something really odd going on.
John D. Rowell writes:
02 Apr 11, 05:57:27
Template rendering is not what Ruby frameworks are optimized for. Someone could dish out a pure C version of erb or haml and use it as a native extension. So, that would be faster, but in production it wouldn't make any difference. If you're rendering the same stuff over and over again in production you're Doing it Wrong (TM). That being a very popular (perhaps the most popular) use case for Redis--page, fragment, etc. caching--it's weird to see this discussed here.

There's a minimum performance level that you must achieve to have a scalable system. One can't take EC2's micro instances and scale anything compute-intensive, for instance. But 300 req/s (and my 2002 desktop dishes out more than that--shame on you Apple!) would still be pretty scalable if you could keep things in that ballpark. 3ms/req/core can handle more hits than probably 99.99% of today's websites get using only a handfull of nodes (if that). So, if you can cache most of your rendering and use async calls to other systems (like databases and queues), it's pretty easy to keep your average throughput at those levels, while keeping memory usage low (which PHP also does, but other ugly stuff like Java can't manage). Then again, if you're doing number crunching or complex document parsing and that sort of tasks, you shouldn't be using pure Ruby because your base performance will be too low to scale.

And so we get to what Ruby web frameworks are actually good at, which is orchestrating complex systems in a beautiful and maintainable way. As long as your supporting architecture is in place, including caching, fast and efficient database lookups, proper separation between static and dynamic content, appropriate planning for handling long lasting requests, and all of the other best practices that all scalable sites implement (or at least should), you'll be just fine.

As a last thought, the best way to prove any of the points is these comments would probably be setting up a few benchmarks on Github and running them on a cheap EC2 small instance. I've been doing a lot of Ruby benchmarking myself lately and got excellent results with Nginx + Passenger 3 + Ruby 1.9.2 + Sinatra which I'd be happy to make a recipe for if the Github benchmark idea gets any traction.
Niko writes:
02 Apr 11, 07:08:25
I've done some testing myself (no PHP, just Sinatra & Thin, ruby 1.9.2) on my MBP: https://gist.github.com/899394


* Simple substitution is faster than full fledged template engines
* Reading files or not doesn't matter (filesystem cache works)
* erb and erubis is fast compared to haml or slim

Other thoughts:

PHP perhaps uses a bytecode cache to cache the compiled template. Seem pretty efficient.

I recently benchmarked PHP vs. Sinatra with actions that do one single Redis lookup per request. PHP clocked in ~900req/second. Then we patched predis to use a unix domain socket connection to Redis. That improved things to about 1200req/sec. Today Sinatra is running driven by Unicorn with 4 workers delivering about 4200req/second. So depending on what you do one or the other language/library/template/driver combination is faster. (surprise, surpise). And Ruby isn't slow.
Niko writes:
02 Apr 11, 07:21:07
And I noticed another thing in my recent tests: Apache/PHP did well up to about 300 parallel connection. The throughput droppped significantly above. Unicorn and Thin don't degrade up to 1000 parallel requests. In my tests when using 1000 parallel connection Unicorn with 4 workers serving a JSON string defined inline was 50% faster than Apache served the same JSON as static file.

I didn't do any specific Apache configuration for these tests. So I can state with the same right as Antirez is stating "In Ruby the default is slow":

PHP/Apache can't handle high concurrency well. ;)
Ryan Tomayko writes:
02 Apr 11, 08:42:44
You're running under development mode which reloads a number of things (templates included) as a developer convenience. Set RACK_ENV=production (environment variable) and run the benchmarks again.
Niko writes:
02 Apr 11, 09:30:09
Ryan is right (at least in my case *blush*). Updated the ab numbers in the GIST with production numbers. https://gist.github.com/899394
antirez writes:
02 Apr 11, 09:32:24
@Ryan: this is how I run it:

$ export RACK_ENV=production
$ run app.rb

I also tested another version that loads the template just one time and put it into a global var, then I just call 'erubis' against that var. Again the trivial template substitution is absolutely measurable.

I'm lame as a Ruby programmer, but there is definitely something odd.
Nikkau writes:
02 Apr 11, 10:17:22
@antirez And inline template? It's not better than a global var?
sloser writes:
02 Apr 11, 11:19:54
IMHO It makes no sense to do such comparisons. HTML is two times faster than your PHP example. The point is that for any bit serious web site you must use the cache, whether it is written in PHP, Ruby, Python or LOLCODE.
antirez writes:
02 Apr 11, 11:21:31
I substituted the template stuff in my code with that:


And now the /fast nad /nolate take the same time with apache benchmark.
So I guess, there is some truth that template substitution should not be so slow I guess.

This is the code I added in the example posted:

get '/nolate/:username' do
@var = "Hello #{params[:username]}"
nlt "index.erb"
raggi writes:
02 Apr 11, 13:30:35
raggi writes:
02 Apr 11, 13:32:42
n.b. the config.ru is executable on any system which supports shbang args through execve.
Jimbob writes:
02 Apr 11, 22:50:34
@Glenn Rempe: You're running on Mac? I thought I heard Mac's have performance issues with ruby? That's awesome that you wrote a rack app. The speed increase really shows how much the framework slows things down. Even then you still have rack in the way, but I don't know how else you could get any closer without a descent modruby.

As has been said here and elsewhere, this is really all pointless though because ruby, rack, sinatra, etc are not designed for microbenchmarks - they are designed and optimized for full-featured web applications.
Alex Mikhalev writes:
08 Apr 11, 16:41:05
I think you will more enjoy pluggable CTPP template engine from http://sourceforge.net/projects/ctpp/ and C++ Application http://cas.havoc.ru/ CAS. It is hard to beat C++ in terms of speed. And people who created it are really care about scalability ( I have no association with them, but a great respect).
These fellows actively object an idea from the ruby world that hardware is cheap and developer's time is more expensive, their point is that for 25% reduction in hardware (consumption) devs can get a bonus of the size of their annual salary :)
myfreeweb writes:
19 Apr 11, 11:42:41
Shouldn't you use Unicorn for Ruby serving?

Also, Node.js on my '06 Mac mini with a lot of apps open: 3273 requests per second with this code: http://mfwb.us/lhj8
After adding some Mustache (with Milk), no caching, reading from a slow HDD on every request: 1404, code: http://mfwb.us/YjpI
And inlined: 3047 http://mfwb.us/EicL

Your PHP is just 1378, running 5.3.4 on Snow Leo's built-in Apache.

P.S. my post on Apache's I/O fail: http://floatboth.mfwb.us/on-io/
facciocose writes:
27 Apr 11, 03:44:56
I think that great part of you misses the point here. It's not about the webserver stack. It's the way erb substitutes the template in sinatra. So why do you keep talking about Apache, Webrick, Unicorn, etc.?
30 Apr 11, 05:41:31
At present, Major brand basketball shoes: Nike, Adidas, Nike Jordan, Converse, etc. Our site can provide you various kinds new and hot selling basketball shoes. Such as powerful and arrogant NBA star caliga: Kobe Bryant basketball shoes, James basketball shoes and unfailing Jordan basketball shoes. There are high quality, competitive prices of all basketball shoes on our site. Create super-class platform of the basketball shoes for you!http://www.basketballshoesmart.com/
comments closed