Komodo IDE 7 has just been released! It offers groundbreaking new technologies and innovative enhancements. Here’s an introduction to the most prominent features it includes.

• Code Collaboration is one of the coolest things to use in a team development environment. You can easily share your files and collaborate in real time with your colleagues and friends. Peer reviews and pair programming become a much more intuitive and interactive experience with Komodo’s collaboration tools.

  
  
  

• 

  For those that use multiple machines, Komodo’s Synchronization feature gives you the ability to copy your Komodo settings (remote server accounts, toolbox, file templates…) between all the Komodo instances. So you can customize Komodo in one place and have the same setup appear in another!

• The cloud is a new and exciting playground for developers. Komodo’s Stackato interface provides the ability to quickly deploy web based applications onto cloud infrastructure – you can have your application running on the cloud within minutes! A slew of languages and frameworks are supported – such as Python, Django, Ruby, Rails, PHP, Drupal, Perl, Mojolicious, Node.js and even Java!

  
  
  

• Node.js support – Komodo IDE brings in debugging, code intelligence, editor and syntax checking support for Node applications. As well as this, Komodo IDE 7 provides additional support for web and template languages like CoffeeScript, LESS, SCSS, EJS and Mojolicious.

  
  
  
  

• With the ability to Code Profile your application – you can now find out why that PHP or Python code is taking so long to run! Komodo shows a graphical representation of the methods and calls made by your program, as well as outlining the CPU time spent executing your code. Automatic time sorting makes it easy to see the code hot-spots at a single glance.

  
  
  

• Now, with every major Komodo IDE release we’ve aimed to make things work smarter and faster than they were working before. Komodo IDE 7 is no exception to this rule and you’ll notice a lot snappier Komodo startup time, lower CPU utilization – particularly when idle, and lower memory usage for large projects. Additionally, tools like the fast-find bar make it even quicker to do the common things you use the most.

  

There are way more features in Komodo IDE 7 than I can outline in a single blog post, so check out the full release notes to see other great enhancements like multi-language syntax checking, HTML end-tag completions, moveable side panes and much more.

If you’d like to test out these features yourself, you can download a 21-day free trial of Komodo IDE 7.

It’s also currently on sale, starting at $245 (reg. $295) until March 15th. Visit our store to purchase today.

A special thanks goes out to all the testing feedback provided by the Komodo 7 beta users during the Komodo 7 development process – you all rock!

We hope you enjoy Komodo IDE 7!

<!–

–>

We have a number of Stackato deployments here at ActiveState, some larger than others, but one stands out as enticingly quick to get setup and scaling with little effort.

Our Stackato vSphere cluster serves as one of our primary testing platforms and at any given time. It’s running at least 25 Stackato nodes, one ‘cloud controller’ and the rest mostly ‘DEA‘ nodes.

In a real elastic cloud you wouldn’t want to waste the resources or expense of running these DEA’s perpetually, you would want them only when there is a current or trending resource demand for them. We mainly idle these VMs for QA testing and benchmarking, so we are ready to instantly scale up or down.

Elasticity should be no stranger to those of us working on PaaS or IaaS products, and if you’re not already familiar with our Stackato Sandbox, it runs on Amazon EC2 with EBS backed storage. To meet user demand at any point in time, we can have the cloud controller automatically spin up new DEA instances taking into account projected load on resources and use the EC2 API to spin up a new DEA.

vSphere can also serve your own privately managed cloud in this exact manner, and I’ll be giving a short summary on how we did this on our own vSphere setup.

Taming the vSphere API

It should come as no surprise to more experienced vSphere users that to achieve scaling we made use of the vSphere API. As Cloud Foundry itself was conceived in Ruby, we choose to use RbVmomi, an open source Ruby interface to the vSphere API. If you’ve had previous dealings with the vSphere API, fear not – this Ruby interface actually makes the coalescence of the two a breeze.

For the connecting vCenter user, I recommend creating a new user with the permissions to create, start, stop and deploy new VMs, and access to any of resource pools it might need.

Diving into the Cloud Controller

I reused existing modifications to the health monitor component from our EC2 (Sandbox) scaling code, to incorporate the ‘events’ necessary to trigger a scaling activity.

These previous modifications to the health monitor are designed to probe all the available DEAs at runtime intervals and check their current resource capabilities and limits, including the DEA’s current RAM metrics. This, of course, can also include the condition when there are simply no DEAs available for deployment due to failure or full capacity.

Using a small analysis of past, current and projected app deployments, a ‘scale up’ event will be triggered and passed along the NATS message bus, which is the default PubSub messaging mechanism used by Cloud Foundry.

At the point of receiving the event message, the process flow follows something like this:

• The Health Monitor component keeps track of the health of any deployed apps and the availability of DEAs
• If under load, the ‘scale up’ event is triggered from the Health Monitor to the cloud controller via NATS
• The cloud controller will load up the vCenter YAML configuration (more on this later) and connect to vCenter via the API
• It will round-robin all hypervisor hosts currently available for deployment, so no single one becomes overloaded.
• Upon finding an available host, the DEA is deployed from the template to the host and it’s corresponding resource pool and datastore.
• The MRU index of the last deployed to host is kept in the cloud controllers database.
• When the DEA starts up, it is already configured to register with your cloud controller and is instantly ready to start receiving new app deployments.

Scaling from a pre-built template

On the ‘subscribing’ end of these event messages is the cloud controller itself, which upon receiving a ‘scaleup’ operation, will proceed to connect to vCenter, and deploy a new DEA from a pre-defined template. To make the template, we take a new Stackato VM and run:

  stackato-admin become dea -m <cloud_controller_ip> -e <api_url> 

This is the same step you would take when setting up a cluster manually. Once you’ve run this, you can shut that VM down, then right click ‘Save as template’. Once converted into a template, it as available for instant deployment across all your ESXi’s / vHosts.

Balancing the VM load on your ESXi’s

Checking the hosts status

A useful feature of the vSphere API is the ability to probe your hosts for their runtime metrics, so you can make an informed decision about where the next VM should be deployed to. We make use of the vSphere API’s HostSystem() object to get some information about which hosts are available, and which hosts are in good health.

Using the API makes it easy to locate a new host to deploy to. You don’t need to supply any host configurations, the cloud controller will take care of keeping tabs of hosts in the CC Database. The scaling code automatically chooses the next available vSphere host at runtime, so you don’t need to manually specify your hosts in a config file.

Wrapping it up in a configuration file

To ease maintenance and provide flexibility, the scale-up control settings are in a simple YAML configuration file that you upload to the stackato user’s $HOME directory, which is then parsed by the cloud controller. This configuration contains parameters like vCenter server, authentication details and template information. It is quite straight forward:

---
  server: vcenter.domain.com
  user: username
  password: password
  https: true
  port: 443
  insecure: true
  path: /sdk
  datacenter: DataCenter
  template: "Your-Template-Name"

We use SSL to connect to vCenter (recommended), but I’m also connecting with ‘insecure’ set to true. That simply means we are using self-signed SSL certificates, and forces RbvMomi to accept them.

Expanding the featureset

With the abundance of documentation on the vSphere API we could take the scaling process even further when working with different setups of datastores architecture.

Some items for consideration could be:

• Limiting the maximum number of VMs on any single host
• Allowing deployments from multiple/different templates
• Allowing multiple vCenters in the configuration

If you have any thoughts or questions about how this could apply to your own vSphere setup please let us know, we’d love to hear them!

<!–

–>

The ever-expanding OpenStack ecosystem now has an enterprise-ready PaaS. Stackato, built on Cloud Foundry and hardened for the enterprise, now supports vSphere, Amazon/AMI, HP Cloud Services, OpenStack, Citrix XenServer, and KVM infrastructure models.

ActiveState joined the OpenStack community in 2011, and committed publicly to making PaaS a reality for OpenStack deployments. As showcased with  Stackato’s successful deployment to HP’s OpenStack-based cloud, we’ve made good on this commitment, and are now pleased to offer OpenStack-ready Stackato for download with our latest release of Stackato.

Stackato—ActiveState’s commercial-ready, secure, multi-tenant platform for creating a private PaaS—is now the linchpin connecting Cloud Foundry and OpenStack.  Stackato’s enterprise-ready OpenStack support ensures that these two important ecosystems can be securely deployed together.

With Stackato, customers can deploy applications to either a private internal cloud (like ones powered by vSphere, XenServer, KVM, or OpenStack) or one hosted with a third-party cloud-hosting provider (like those powered by Amazon, RackSpace, or HP Cloud Services).

Stackato also now has SSH support, so you can have a secure interactive shell in any of your application instances. (Personally, this is my favorite new feature in this latest Stackato release.) This is possible (and safe) because each application instance runs in it's own para-virtualized container using LXC, providing more secure multi-tenancy.

In addition to OpenStack deployment and ssh support, Stackato has a new Management Console that replaces the Admin Dashboard with a new, improved user interface:

The Management Console offers deep visibility into the activity and events in a private cloud to help administrators better manage usage. This view includes showing activities of developers who deployed an application, number of instances deployed, memory usage, data services deployed, and languages used.

And there’s more: Additional updates in this latest release include improved application lifecycle management, improved Perl deployment speed, and new pre-staging setup hooks.

Take a test drive with Stackato on a microcloud (VM) on your own desktop or deploy directly to the Stackato Sandbox today!

<!–

–>

Last week, SD Times contributor Alex Handy published an interesting article titled The PaaS market accelerates. The piece was noteworthy not just for its insight into the burgeoning PaaS market space and introduction to the major players, but for some bizarrely unfounded assertions made by one of the article's subjects.

Really, Adam?

In his January 6^th SD Times article, Handy writes (emphases mine):

… Adam Wiggins, CTO of Heroku, said that PaaS is about the public cloud only. Although he used harsher language to describe the private cloud and PaaS market, he indicated that he and Heroku believe the private cloud to be entirely vapor.

"My opinion—and this is not a widely held opinion—is that private cloud is total BS. If you make it private it is no longer cloud. You destroy all the value you get from cloud by making it something you have to run yourself. Anyone pursuing that path is advocating a false trail," said Wiggins.

(At this point, the Enterprise IT Leads all do a spit take. I'm just curious to know what the unmentioned "harsher language" was.)

Really, Adam? "BS"? "Vapor"? Really? Is that the best you can do for FUD? Your comments are at best puerile, and at worst naïve. Let's focus on the naïve…

Save me, public cloud!

What Heroku's Wiggins is desperately trying to obscure is that in the real world, the enterprise demands private cloud and Private PaaS. Fortunately, he is correct in observing that his is "not a widely held opinion."

Wiggins suggests that customers derive value from the self-service nature of the public cloud, and that making the cloud "something that you have to run yourself" defeats its purpose. Hmm. Do I really want control over my own platform? My data is probably safe, I mean, it's in the cloud. And do I really want to deal with all that pesky provisioning and applications management and infrastructure administration and oh, I'm overwhelmed-Save me, simple public cloud!

If only it were that simple. As my colleague ActiveState VP of Engineering Jeff Hobbs noted in Handy's article, Public PaaS can lift the burden of cloud-management responsibility, lightening IT load. And public clouds also deliver benefits like shared-resource efficiencies, utility computing, and scalability. But let's be honest here: Every Public-PaaS developer promotes those same supposed advantages, and whee! The race towards commoditization is on. Indistinguishable public-cloud services foster desperate competition, which drives down price, which ultimately benefits the consumer…at least the one looking for the cheap solution.

Public PaaS: Slumlord of the cloud's low-rent district

In contrast with Private PaaS, the very nature of the Public PaaS' shared-services model enforces a lowest-common-denominator architectural approach. You're entering the low-rent district: You get what you pay for with public cloud ("First month free!"), and that means the same setup as the tenant sharing your virtual apartment building. ("Matching floor plans!") Does your company have unique data needs? Public PaaS may impose restrictive limitations on the way you manage your data. ("No pets!") Prefer a cookie-cutter approach, okay with limiting application portability, and don't mind sharing a multi-tenant virtual hallway? ("It's just like a hostel!") Then Public PaaS-at least as Adam Wiggins envisions it-might be right for you.

The real world demands Private PaaS

Recall those Enterprise IT leads? (I hope they've regained their composure.) They are the ones who are investing in private cloud and Private PaaS because it provides better security, better control, and-believe it or not, Adam-greater flexibility than public-cloud alternatives. Oh, and with the right technology, those Enterprise IT Leads can also realize those public-cloudy benefits like shared-resource efficiencies, utility computing, scalability, and even multi-tenancy.

But don't take my word for it. The U.S. Army recently invested a quarter billion dollars in private-cloud technology. Sure, you could argue that the military has special security needs that would never be well served by a public cloud. But so do very-much-not-like-the-military companies such as Federal Express. The package-delivery logistics giant has leveraged a private-cloud architecture in its newest data center, and even retrofitted an old data center to match. Even SD Times contributor Alex Handy noted in a November 2011 blog post that-despite Heroku's contention that private cloud is just "a myth and snake oil"-he could "entirely understand the need for an internal private cloud inside the major retailers [on Black Friday and Cyber Monday]."

Thanks Adam!

Don't get me wrong-I have long admired Heroku's accomplishments (particularly its carefully cultivated developer community), and Wiggins and team have produced some great technology. Let's face it, there will always be market segments that can settle for implementing a public-cloud solution. (Small business and municipal public sector are just two that immediately come to mind.) But ultimately, Wiggins' comments ignore (and even denigrate) the needs and priorities of the vast majority of the enterprise cloud market.

But you know what, the more I think about it, maybe that's not so bad. Who knows? Maybe Wiggins is right. Not that Heroku and Wiggins need my validation, but it's okay with me if they want to believe there's no future in private cloud and Private PaaS. Adam, you and Heroku can go play in your public-cloud-only sandbox. The rest of us will be just fine over here providing solutions for the real world.

###

About the Author

Originally published Tuesday, January 17, 2012 7:16 AM by David Marshallon VMBlog.

<!–

–>

Built on VMWare's Cloud Foundry Open Source project and hardened for the enterprise, ActiveState Stackato works on vSphere, Amazon EC2, and OpenStack, and will also support other infrastructure platforms.

While some PaaS offerings are content to rely on user-based Unix security, but ActiveState’s Stackato takes security a big step further, and is the first PaaS to bring lxc-based containerization to the HP Cloud, creating a more-trusted, commercial-grade level of security to private PaaS deployments.

So what’s the big deal about deploying a secure multi-tenant PaaS on HPCS, rather than on one that relies on Unix-based user level security, or one that shares resources amongst its tenants?

Why does multi-tenancy matter? (Hint: It’s more efficient.)

True multi-tenancy enables your organization to squeeze efficiencies out of a shared IaaS-level resource pool by running multiple applications on the same cloud servers. Multi-tenancy is what distinguishes PaaS offerings from Orchestration tools that do not manage multiple guest applications, but rather act as simple installers onto bare IaaS instances.

While multi-tenancy allows you to maximize usage of your allotment of cloud servers, having multiple applications running on the same server can be tricky (if not impractical) to secure.

Playing well with others in the PaaS playground

When you deploy a Platform as a Service, you provide a shared “playground” for a number of guest applications (tenants). The resource pool is presented as a single logical layer to those guest applications–The PaaS manages pools of OS resources and co-habits applications on shared instances.. Those guest apps should not have to be aware of the details of components that lie beneath the PaaS at the infrastructure level, nor should they have to be designed to play nicely with others. The PaaS layer should insulate both infrastructure and application from each other, and ensure that guest applications are not forced to be aware of other guest applications running on the same resource pool.

Some PaaS offerings manage the boundary between infrastructure and the application, But most do not, and PaaS solutions that offer only Unix user-level security leave openings for untrusted applications or users to exploit, and do not provide adequate security guarantees for the enterprise.

Securing the PaaS

Early on in the development of Stackato, ActiveState made the decision to extend Cloud Foundry, and take advantage of Linux technologies to provide better security. ActiveState's Stackato creates isolated, lightweight containers known as “lxc containers,”, Virtual Private Servers (VPS), or Jails. Conceptually, the containers are built atop several Linux technologies which provide each guest-application container with a segregated file system with web and ssh services.

The containers are protected from each other: a guest application in one container cannot read files owned by another container, or kill its tasks. With the Stackato-container approach, the outside world can still reach web servers and ssh servers on the containers and, more importantly, the hosting PaaS can protect its key files from guest applications and users in the containers.

The containers effectively allow Stackato to partition the resources into isolated groups to better balance conflicting demands on resource usage between the isolated groups. Stackato's innovative containerization approach has the dual benefits of running applications on a seemingly separate machine while still leveraging many of the underlying resources. As far as the guest application is aware, the PaaS has provided a private Playground. The architecture makes Private Cloud even more "private." For enterprises, the operational and security advantages of sharing these resources while isolating the guest applications can also lead to significantly lower overhead than true virtualization.

Implications for the HP Cloud Services platform

Adding support for HP Cloud Services to Stackato reinforces ActiveState’s commitment to providing the best-of-breed secure Private PaaS layer in the cloud that runs on HP's world-class hardware and software on OpenStack™ technology – giving your organization a clear alternative to Amazon Web Services.

The end result is a highly flexible, secure, multi-tenant PaaS layer that can be run in any environment (on-premises or in the cloud) helping your organization deliver a knock-out punch when if comes to securing your Cloud.

Stackato is a private PaaS that enables deployment, scaling, and management of Java, Python, Ruby, PHP, Perl, Node.js, Scala, and Clojure applications to any cloud. Stackato delivers the power of PaaS on-premise with the security, privacy, and control behind a corporate firewall. With Stackato, customers can deploy an application to either a private internal cloud (like one powered by VMWare vSphere™, Citrix XenServer, Linux KVM, or OpenStack™) or one hosted by a third-party cloud-hosting provider (such as Amazon or HP Cloud Services).

To see Stackato working with HP Cloud Services, we've provided this short screencast.
 

Developers and cloud administrators can download Stackato, currently in open beta including the free Stackato Micro Cloud, at: http://www.activestate.com/cloud.

<!–

–>

We had the opportunity to take HP Cloud Services (currently Beta) for a spin and get a Stackato cluster running there.

We were happy to take advantage of the computing horsepower HP made available to us to put Stackato through it’s paces and to try out a build running on Ubuntu 11.04, and we’re equally happy to report that it worked just as well as it did on EC2 and our internal VSphere testing cluster.

The video shows me deploying WordPress, as I did the recent Komodo video, but also talks a bit about how we got Bugzilla running in the cloud.

<!–

–>

I’ve been doing a lot of demos of Stackato lately and starting to incorporate the new Komodo IDE 7 client interface (currently RC). Without further ado, here’s the screencast!

It was tough to decide what to show. Debugging applications running on the micro cloud is pretty cool but it’s better demonstrated in a written tutorial (which I’ll be writing up soon). Showing off the DB Explorer is a little more visually interesting.

DB Explorer, dbshell, and tunnel

As the screencast shows, it’s extremely simple to connect to a locally running micro cloud from Komodo. That’s because Komodo has integration with the ‘stackato dbshell’ command, which has been part of the client for a while longer than the newer ‘tunnel’ command. Komodo will have better hooks into ‘tunnel’ in future releases, but for now, it’s fairly trivial to set up a connection to the database manually as shown.

What you’ll need to follow along

If you’d like to try this all out for yourself, you’ll need:

• Stackato micro cloud VM: There’s another screencast that shows how to get this set up. If your local system doesn’t have the resources to run a VM, feel free to use the Stackato Sandbox instead.
• Stackato client: Komodo uses this stand-alone executable in the background. You may need to do some minor client setup after download.
• Komodo IDE 7: Download an RC build (or a trial of the release, depending on when you’re reading this) and set the path to the ‘stackato’ executable in Komodo’s Stackato preferences if it’s not in your system $PATH.
• Sample apps: We’ve put together a Github repo full of sample applications in a variety of languages.

Let us know what you think!

As always, we would like to hear your feedback on how this works for you, and what you would like to see in future releases. You can do that in the Stackato Feedback and Komodo Beta forums.

<!–

–>

Date:     Wednesday, February 8, 2012
Time:     10:00 AM PST/ 1:00 PM EST

The ability to move applications–for instance, from a private to a public cloud, or from one hosting provider to another–is a holy grail of cloud computing. But with a little planning, application portability in the cloud can be attained.

Cloud-vendor solutions deliver cloud technologies, but at a price, and typically impose inflexible limitations, forcing IT leaders to commit to specific hypervisors, infrastructure, and APIs. That reliance on vendor-specific technology–coupled with application-specific code dependencies, configuration/provisioning requirements, and data-security mandates–can prevent organizations from realizing the idealized benefits of the cloud: reduced costs, greater efficiency, on-demand scaling, disaster recovery, and deployment flexibility.

So how do you ensure portability, avoid vendor-lock-in, and reap the rewards of the cloud?

The answer is clear: Implement a cloud strategy that includes a Platform-as-a-Service (PaaS) layer. PaaS provides the interoperability of application and infrastructure cloud services across cloud providers or hosting technology. And you realize the benefits of true cloud-application portability.

In this webinar, we’ll share insights and trends to help your organization avoid vendor lock-in and easily move applications from one cloud to another without major re-engineering of your application or deployment scripts.

Join us for a free and informative webinar where we will highlight:

• Why Portability Matters in Private & Public Clouds
• Understanding & Identifying Your Applications’ Cloud Dependencies
• Emerging Trends & Issues effecting Cloud Application Portability
• Creating a Cloud Strategy optimized for Portability
• Choosing a Technology Stack for Portability
• Implementing a Portable Cloud Strategy with PaaS

Sign up today to reserve your spot. Space is limited!

<!–

–>

For years now I’ve been working on tools that help build and debug Rails applications, but other than the Candidate Buzz demo application, I haven’t actually created many myself. I’ve started many apps as weekend projects over the years. I enjoy building tools to solve common problems I run into when working on these apps, and enjoy using those tools for future projects.

But when it came time to actually launch one, I would ask myself if it was worth the trouble of spending weekends and late nights trying to placate angry or confused users. Shouldn’t I spent my off-work time doing something other than… work? So yet another project would be consigned to a remote corner of my repository.

Bentframe: Visualizing Bicycle/Car Collision Data with Google Maps

In early December word got out that the indefatigable David Eaves had managed to get ICBC, the provincial automobile insurance company in British Columbia, to release a dataset of collision reports involving a car and bicycle for the years 2006-2010. Each record in the data contained a year, month, and location as a latitude, longitude pair.

I figured I could whip up a visualization of this data relatively quickly, seeing how one of those web sites I never launched was a Google Map visualization I wrote in late 2007 that showed a database of foreclosed houses for sale in the U.S. I was struck by how large the database was: about 70,000 properties scattered over all fifty states, but didn’t consider the economic implications. Had I not been focusing on the technical aspects of how to display tens of thousands of data points in a 600x300px window, and instead dipped into the financial press to learn about credit default swaps, short sales, and predicting the future, I might be writing this from a Caribbean island. But fortunately for fans of visualizations, I had focused on writing efficient clustering engines, and figured I could whip up something useful during the single-day hackathon.

Within the first hour I wrote a quick Ruby program to read in the data, fired up Komodo to create a new Rails app, copied over the relevant bits of the old “Closing Town” foreclosure code, and was back in Rails development (as opposed to deployment) mode.

Debugging the Initial App in Komodo

The 4-year-old foreclosure app didn’t need much changing to handle the new data set. Change mentions of “properties” to “collisions”. Change the default view from central Florida to Vancouver, BC.

By noon, I felt ready to go. I started the server, pointed Firefox at good old http://localhost:3000/spots, and found… an empty Google map of Vancouver. Komodo’s DB Explorer showed the database was populated with over 5,000 locations, as expected. Log statements on the server side showed that it was reading the locations from the database, and returning a non-empty JSON packet. On the front end, Firebug was showing that my Ajax callbacks were firing, and creating markers at the correct locations. The problem was, they weren’t showing up.

Meanwhile, my colleague for the day, Nathan Griffiths, had taken a different approach, pouring the raw CSV file into a new app at geocommons.com. While I was dealing with updating code from Rails 1.2 to 3, and pouring the usual combination of HTML, JavaScript, and CSS together into a coherent whole, Nathan had quickly mapped all the collisions, added city bike route data for the city of Vancouver to the mix, and quickly had built an impressive working app. The provincial minister for Labour, Citizens’ Services and Open Government, Margaret MacDiarmid, had spent most of that Saturday morning at the hackathon talking to people. When she came by to see what we were working on, Nathan’s app had locations marked as points, overlays, animations, and heat maps. I didn’t think she’d be as impressed that I could debug Rails on the server and JavaScript on the client side at the same time, with a log window open as well. The minister was a family physician before she entered politics, not a retired Cobol programmer. I stayed quiet while Nathan demoed his app, wondering if I was reinventing yet another wheel.

It turns out that the markers weren’t showing up because my code was using custom images provided by the authors of the Google Maps book I used in late 2007. Attempts to wget the images gave me silent 404s instead. I pulled out the code that was setting the custom images, reran the app, and my map filled with hundreds of the stock Google Map red pin images. Yay. Time to solve a few more problems, wondering if maybe I should take another look at using Geocommons instead.

But the rest went quickly. I found a source of images with digits so I could display the number of collisions in the marker, and used reverse geocoding to translate latitudes and longitudes back into street addresses.

I showed the app to a journalist, and she asked me what the top 10 spots were. I’m not displaying them yet, but I have the flexibility to add that. Vancouver added a couple of controversial bike lanes downtown during 2009 and 2010. Once I pour in the 2011 data, I hope to provide some charts that show how collision frequencies are dropping near those areas. That’s why in 2012 we’re still writing our own code — to create something new without
being constrained by existing tools.

Painless Deployment to Stackato

Unlike the foreclosure data app, word got out, and the aforementioned journalist was asking when my web site would be up. Four years ago there weren’t any single-button deployment solutions. Today there are a few, including Stackato by ActiveState. Deploying Rails apps used to be a major pain point, and the subject of many conference sessions. Now it took one evening after work to deploy the app on Stackato.

New features of Rails 3 makes deploying on a system like Stackato easier. For example, I added a data_loader.rb file to the config/initializers directory that checked to see if the database was empty. If it was, the single line Rake::Task['db:seed'].invoke would run db/seeds.rb, which was nothing more than the program I used during the hackathon to populate the database from the CSV file. I added the CSV file to git, and the next time the app was restarted it rebuilt the database automatically.

Now that I was actually deploying the code, I had to make sure the code behaved differently depending on whether I was testing the code in development or test mode, or deploying it. For example, I was juggling two Google Map API keys, one for localhost, and one for the target URL.
There was an easy fix for this, in config/environment.rb:

GOOGLE_MAPS_KEY = (Rails.env == "production" ?
                   [key for sandbox.activestate.com] :
                   [key for localhost]);

I pushed the app to the Stackato sandbox, tested it out manually and found it worked fine, notified the local cycling advocates, and watched the hits come in. The map helped me meet one of my goals, an increase in awareness that the designated routes were not 100% safe. In particular, one intersection, where the 10th Avenue bike route crossed Clark Drive, a 6-lane truck route through the city, was showing 16 collisions over the five-year period, easily one of the top ten intersections. Apparently people at city hall are discussing possible improvements.

Improving Performance

I was happy that I could release the app without any work at all, but was less pleased with the performance. When you zoomed out to a level that contained all of Vancouver, the server was taking up to 20 seconds to respond — it was spending most of that time converting over 2,000 locations into about 200 locations that Google Maps could quickly render. This hadn’t been as much of a factor with the foreclosure data, because there were few blocks that had more than one bank-owned house on them. But the ICBC data was revealing more than one collision along each block along designated routes and main streets. Quick profiling revealed that 10% of that time was spent converting rows from the data base into ActiveRecord objects, and most of the balance spent clustering those items.

I spent the next Saturday rewriting the clustering code in C. I had worked out the subtleties of the clustering algorithm in Ruby, so this was essentially a rewrite into C. I made sure the core code was independent of any scripting language API, and then wrote an interface layer between C and Ruby (so this library could be easily ported to any other language with a C interface). It went amazingly quickly, with benchmarking showing a drop in processing time of over 90%. The advantage of the C code, besides the speedup in executing machine code over interpreted Ruby bytecode, is that I knew the code’s memory requirements in advance, allocated it all in one shot, freed it at the end, and could structure the code to minimize sloshing bits around. With Ruby that was out of my control. And writing a layer of C code to interface with Ruby is actually surprisingly fun.

I built a gem for the C code, added an entry for it in the app’s Gemfile, ran bundle package (side note: use bundle install for gems registered with rubygems, bundle package if your app uses private gems), pushed the app to Stackato, and it worked much faster. As I zoomed out, the server was taking less than a second to process a large set of points. You can see the result at http://bentframe.sandbox.activestate.com.

Note that while I developed the C library on OSX, it didn’t matter that I was deploying to Linux. Stackato compiled the gems for me during staging.

Handling Cars

Just in time for Christmas, I received an Open Office Calc .xmlx file from ICBC containing about 110,000 rows of data representing car collisions at intersections in the province for the same years, 2006 through 2010. It took about a day to modify the bike app to handle the slight differences in the car data, and the clustering C module was performing fine. However the numbers were larger, compared to the 5500 rows for the bike data. If I zoomed out long enough, the server would take over 20 seconds to respond. Benchmarking showed that most of that time was spent simply reading data from the database (using a pure SQL call to bypass Rails’ ActiveRecord). But a call to determine how many rows matched a given query took no time, so I modified the JavaScript side to show a warning when there would be too much data to look at when you zoomed out too far, and left it at that. You can see the car crash map at http://carcrash.sandbox.activestate.com.

<!–

–>

As most people know, Java applications dominate the landscape in large enterprise. But as corporate IT departments are planning private PaaS implementations, what strategy are they taking regarding languages and data services?

Option 1: Java First

On the one hand, some are focusing first on Java to support the primary language amongst their developers. In these companies, Java is the primary language for legacy mission-critical applications. Often, other languages like Python and Perl were not used as much in legacy applications, or not viewed to be strategic languages for mission-critical languages. So in some large organizations, even if up to 25% of their applications use another language like Python or Perl, these organizations are implementing a PaaS for Java first with the idea of getting the majority at the first stage.

Option 2: Next-Gen Languages to Get Buy-In

However, some companies are taking a different strategy in order to get greater internal buy-in and adoption of their private PaaS. Even though most current applications may be written in Java, they feel that the younger generation of developers are using, or prefer to use, other languages like Python, Ruby, and PHP, and are demanding that IT support these options. Since the younger developers are also the ones that are more likely to try a new internal service like a PaaS for deploying and managing their applications, corporate IT groups are focusing on getting their adoption first. And the Java crowd will follow. Likewise, in addition to supporting traditional databases like Oracle and DB2, these corporate IT groups want to be able to offer next-generation data services like MongoDB to show the flexibility and options that an internal PaaS can provide. This puts corporate IT in a position of giving the customer (internal developers) what they want in order to satisfy their demands today, get early buy-in, and set-up for success tomorrow.

So which strategy are you taking, and why? Share your comments!

<!–

–>