Another advantage is requiring cookie storage, which lots crawler seem be lacking for some reason. Since most common http clients support them. When the crawlers are doing lots of requests at once, needing JS engine would add a lot of overhead aswell. So its multiple factors which makes this viable method so far.