Dear dotCloud Customers,

We are going open-source.

It has been a wild week for dotCloud. Of course as we prepared to open-source Docker, the container technology that powers the platform, we hoped it would be well received, like ZeroRPC and Hipache before it. But nothing could have prepared us to the magnitude of the response. Now, 6 days, 50,000 visits, 1000 github follows and 300 pull requests later… we think we get the message. You want an open-source dotCloud – and we’re going to give it to you.

Today, as the first step in our new open-source strategy, we are announcing an important change to our free Sandbox. In the coming weeks we will hand it over to the community as an open-source project which can be deployed and hosted anywhere. As part of this transition we will be sunsetting our free hosting tier – see below for details. The resources freed by this transition will be re-invested in our open-source roadmap.

I want to emphasize that this transition does not affect our Live and Enterprise flavors, and it does not change our business model. Our core competency is and will continue to be the operation and support of large-scale cloud services, for tens of millions of visitors, 24 hours a day, every day. We intend to continue expanding that business, and we believe the best way to do that is by embracing open-source.

1. Going open source

Our approach to open-source is simple: solve fundamental problems, one at a time, with the simplest possible tool. The result is a collection of components which can be used separately, or combined to solve increasingly large problems.

So far dotCloud’s open-source toolbox includes:

  • ZeroRPC, a communication layer for distributed services;
  • Hipache, a routing layer for HTTP and Websockets traffic;
  • Stack.io, a communication framework for real-time web applications
  • Docker, a runtime for linux containers.
  • Recipes for automatically deploying NodeJS, Django, Memcache and dozens of other software components as cloud services.

All these components are already available, and the open-source community is using them to build alternative implementations of dotCloud’s development sandbox. We want to make that even easier by open-sourcing the remaining proprietary components – including our uploader, build system, database components, application server configuration, and more.

To learn more about future open-source announcements, follow the Docker repository and join the Docker mailing list.

 

2. Sunsetting the hosted sandbox

In order to properly focus resources on our ongoing open-source effort, we will be phasing out the hosted version of the free Sandbox. Going forward, the recommended way to kick the tires on dotCloud will be to deploy a Live dotCloud application. For your existing Sandbox applications, we can provide an easy upgrade. If you don’t feel ready to pay us quite yet, take a look at what the community is building.

Below is a calendar of the sunset. As usual, our support and ops team will be happy to assist you in every way we can during the transition.

 

Date Change to Sandbox
April 8th (no change)
April 22nd All Sandbox applications will be unreachable via HTTP. You can still access them via SSH to download your code and data.
April 25th All Sandbox applications will be destroyed.

Note that we’ve pushed-out the sunset dates since first posting this blog. We’ve removed the ‘no push’ week of April 8 and extended HTTP access to the 22nd. 

How to Graduate from the Sandbox

We’ve made it easy for you to change your Sandbox application to a Live flavor if you want to keep it running on the dotCloud platform:

  1. add your billing information to your account and
  2. file a support ticket telling us which applications to migrate. Please use your account email address and give the full URLs to the applications.
  3. We’ll do the rest.

If you don’t want to move to a paid service, you can use several techniques to download your data and files before they are destroyed.

For those of you who have been using the Sandbox as staging for paid applications, we’re sorry for the inconvenience. We hope our hourly billing will help keep your staging and testing costs down, and that developing in a paid service will ease testing related to scaling.

Looking Back, Looking Forward

We want to thank you, our sandbox users, for trying out the dotCloud platform. We hope that you will enjoy experimenting with our open-source version, discovering the awesome features of our Live flavor, or both!

We look forward to helping you be the most awesome and productive developers out there.

Happy hacking!

/Solomon Hykes

CEO, dotCloud

A big thank you to everyone who participated at the DeveloperWeek Hackathon in SF February 1-3. With more than 150 participants and 21 total teams the event was a great success. We’ve got more photos of the weekend on our Facebook page.

 

Sponsor_slide_resized

As a partner sponsor, dotCloud offered an API prize to the best project deployed on dotCloud. Our head of Developer Support, Andy Rothfusz, evaluated projects based on functionality, novelty and usefulness.

Of the 11 projects submitted, Team FlyVerify was selected to win $500 cash, $1000 credit to take their app Live on dotCloud and a chance to present their project on the main stage at DeveloperWeek.

FlyVerify edged out their competitors by being highly functional and fulfilling a real need. We’re proud that dotCloud could help them focus on their project instead of on infrastructure. – Andy Rothfusz

About FlyVerify

 

Flyverify__never_verify_an_email_address_again

 

You can learn more about the project on its HackerLeague page, but the idea is simple…to eliminate the need for business’ to validate the email addresses of new users.

FlyVerify is a webservice that stores already verified email addresses using a REST API. If an email address isn’t in the FlyVerify system, they will handle the work of sending out an email confirmation and relaying to businesses that an address is verified using a simple callback mechanism. The system gets smarter over time.

Team Leader Ryan Gerard noted that “our team chose to use dotCloud because it was easy to use, had a wide variety of supported frameworks and languages, and offered unlimited support for sandbox development.”  

“We were able to get up to speed on dotCloud really fast. We had less than 3 days to turn our idea into a reality, and it was important that we focus on code. dotCloud provided us with a solid infrastructure that was easy to integrate and build on top of.” – Ryan Gerard, FlyVerify

The FlyVerify project uses a node.js and MongoDB stack. It is hosted on dotCloud, uses Sendgrid for email and Bootstrap for design.

What’s Next

Ryan Gerard and his teamate James Wildman plan to work on FlyVerify in conjunction with the work they do at the startup they co-founded, PickMoto.

They’ve just launched their app in Live mode on dotCloud with the domain www.FlyVerify.com.

They’ve also created a Google apps account so users can now email them at support@flyverify.com with any questions.

Next they will incorporate SSL support to integrate payment solutions, add logging/error reporting so that they know when something goes wrong and develop a marketing plan.

See them demo FlyVerify on Wednesday at DeveloperWeek!

My name is Ken and I’m a geek. Like most geeks I like to stay up to date with the latest technologies, and normally try to find some excuse to play with them. As a dotCloud engineer I take advantage of our once a month hackdays to do just this.

One of the technologies I have been hearing a lot about lately is a new programming language called Go.

On the Go website it says “Go is an open source programming environment that makes it easy to build simple, reliable, and efficient software.” At first pass that sounds like a lot of programming languages, so what makes it better than the rest, and why is this so special?

To answer those questions, I’m going to steal some sections from the Go FAQ.

Go is a concurrent, garbage-collected language with fast compilation. It was started at Google and was released to the public on November 10, 2009.

Features of Go

  • It is possible to compile a large Go program in a few seconds on a single computer.
  • Go provides a model for software construction that makes dependency analysis easy and avoids much of the overhead of C-style include files and libraries.
  • Go’s type system has no hierarchy, so no time is spent defining the relationships between types. Also, although Go has static types the language attempts to make types feel lighter weight than in typical OO languages.
  • Go is fully garbage-collected and provides fundamental support for concurrent execution and communication.
  • By its design, Go proposes an approach for the construction of system software on multicore machines.

Typical Go App

If you have never seen a Go program before, here is the typical Hello World application written in Go.

https://gist.github.com/4658856

Looking at the code it is pretty straightforward. If it wasn’t for the curly brackets it almost looks like Python.

If you want to learn more about Go, I recommend their excellent tutorial http://tour.golang.org. I also did a quick search on amazon and it looks like there are already a number of Go books. So pick one up today, and start learning.

Learning Go on dotCloud

We are starting to use Go on a few internal projects at dotCloud. I like what I see so far, but before I get too ahead of myself, I still have a lot to learn. I personally find the best way to learn is to have a small, real world, non-critical project that you can use as your guinea pig.

Reading books and going through tutorials is a good place to start, but it’s a lot like high school physics, when you ignore friction, wind resistance and gravity. Once you start doing real world problems which include those variables, things never go smooth the first time.

That is why it is always best to get your hands dirty on a real project as soon as possible. dotCloud’s free sandbox apps are perfect for this.

I’m not sure about you, but a lot of the projects that I work on lately are web-based. I was sad to find out that there were not many options for hosting Go applications. It seems you could run your own server (VPS, etc) or you could deploy to Google App Engine.

One of the many advantages of working for dotCloud is having access to this awesome deployment platform, where I’m like a kid in a candy shop. I can take these different services, write some code and easily deploy them to dotCloud, all without breaking a sweat.

Last Friday we had a hackday at dotCloud, where everyone takes the day off from their normal projects and we can work on anything we want. I took this time, to work on a Go on dotCloud custom service. It is still very early, but it allows you deploy your Go applications to dotCloud as easily as all of our other services.

Try it out on dotCloud

I have put the code up on github so that everyone can use it, and submit pull requests. It is still very early beta, but I welcome everyone to try it out, and please let me know what you think, good or bad. If you have a suggestion on how to make it better, feel free to submit pull requests and issues.

Using it is pretty straightforward. Assuming you already have a FREE dotCloud account, you just need to follow these steps.

  • Create a new application

$ dotcloud create -f sandbox mygoapp

  • Fork my github repo, and then clone it.

$  git clone git://github.com/kencochrane/golang-on-dotcloud.git

  • Add your code in the src directory.
  • Edit the dotcloud.yml
    • Change the build_package config variable, to the name of your GO package
    • Change the processes so that you can run the correct processes
    • Add or remove ports depending on your application needs.
  • Push the application

$ dotcloud push mygoapp .

Enjoy!

That should be about it. Best of all you can do all of this for FREE since dotCloud doesn’t charge for services in the Sandbox. So deploy as many times as you like, and let me know if you have any questions or problems.

Coming from a Python background, I am most interested in the fact that Go is a fast compiled language with concurrency built-in. I don’t think it will replace Python as my go-to language, but when I need something small and fast with concurrency, it is nice to know there is another option at my disposal.

I’m looking for new ideas for my next hackday, if you have any suggestions, please let me know.

Developerweek

Team dotCloud is joining forces with some of the Bay Area’s brightest minds to make DeveloperWeek 2013 a success.

We’re rolling up our sleeves and contributing to all facets of this inaugural event – as members of the 2013 Advisory Board, Hackathon Partner and speakers.

DeveloperWeek Hackathon

The week kicks off with a 400+ person Hackathon – and we can’t wait to see what fresh minds will deploy on dotCloud.

As the recommended web platform for the event, we’ll be offering two 30-minute coaching break-out sessions to get your application deployed to our free sandbox tier in no time.

Afterall, you have only two days to design, build and deploy your app. Why would you waste time with servers when dotCloud supports all the most popular languages and frameworks? Let us do the work.

Plus, deploying on dotCloud enters you for a chance to win our API prize and the chance to present your app on-stage at DeveloperWeek.

SF Beta :: DeveloperWeek Edition

On Monday night, you can find Jerome Petazzonni demoing dotCloud at SF::Beta’s DeveloperWeek mixer. We’re excited to share dotCloud with like-minded hackers, CTOs and engineers from across the technology spectrum.

DeveloperWeek Conference

Oh yes, the actual conference. We’re pitching in there too. This year, we are focusing on the Product & Design Day – Monday February 4th.

To start, dotCloud CEO, Solomon Hykes, will moderate the 10am panel discussion “Your User Interface Sucks: Case Studies in Why Design + UI Make a Huge Difference.” Panelists include UX Designers from companies we admire like Adobe, Quantcast and Bottle Rocket Apps.

Later in the afternoon Solomon will present “World of Services: The Software Architecture That is Eating the World.” Here he explores the vertical impact of cloud-native development on everything from the fundamental application architecture to optimal IT staffing.

Hungry for more? Ok, fine. On Wednesday at 12:40pm Solomon will rest his case as to why cloud-native app development is singularly changing how succesful technologies maintain their competitive edge.

Sill Need a Ticket?

Get in touch. We’ve got some free Pro Passes for the community.

Last June, dotCloud announced a brand new pricing model to encourage unlimited experimentation and elastic pricing for production applications.

Our new model defined applications in terms of flavors. You could create as many applications as you want and only pay for the resources you allocate for live apps.

You can see the high-level comparison on our documentation site.

  • The Sandbox flavor allows you to test the platform and develop simple applications for free.
  • The Live flavor is targeted at both development and production and is the preferred dotCloud flavor. It comes with all the platform features: Custom Domains (with SSL), Scaling and High Availability.
  • The Enterprise flavor is similar to the Live flavor but comes with premium support and custom application level monitoring.

In order to preserve the thousands of applications that you created before this pricing change, we created a temporary flavor called Legacy. Legacy apps are a lot like Sandbox apps, but we preserved your ability to attach (pre-existing) custom domains.

On December 14, 2012 at 11am PST we will migrate all remaining Legacy apps to the Sandbox flavor. We don’t anticipate any performance interruptions.

Not sure if your applications are Legacy?

Log into your new user dashboard. You will see a list of your applications and their associated Flavor.

 

Dotcloud_legacy_app

 

If your application is business-critical, please contact our support team immediately and we will take care of the transition to Live for you.

Important note: Once your application has been marked as Sandbox we will not convert it to Live for you. You can make it a Live application by creating a new application, pushing your code and restoring all your data to the new application.

Today, we are sharing a very simple demo of how to run the Autobahn WebSocket server-side Python implementation on dotCloud.

The real-time web movement is all around us. Mainstream tech media like Mashable, ars technica and the recently re-branded readwrite pubs have been talking it up for years.  At dotCloud, we have been sponsoring and speaking at Portland’s Keeping It Realtime Conf since its inception in 2011.

What is Autobahn?

To quote autobahn.ws:

Autobahn provides Open-Source client and server implementations of WebSocket and WAMP.WebSocket allows bidirectional real-time messaging between browsers (as well as other clients) and servers and WAMP provides asynchronous RPC and PubSub over WebSocket.

And since dotCloud is the only multi-lingual PaaS to natively support WebSockets, we’re an ideal platform to get started with Autobahn.

Why use Autobahn?

If you want to write real-time web apps in Python and leverage on the WebSocket protocol, Autobahn is a very solid option. It also provides WAMP, which will save you from re-inventing the wheel if you want to implement asynchronous RPC and PubSub. Autobahn provides not only a Python library for client and server code, but also implementations in JS (for browser based apps) and in Java (for Android).

Get Started Now!

Running Autobahn on dotCloud is easy, and free in Sandbox mode. This example will help you to get started in less than a minute:

  • go to the URL shown at the end of the push. Voilà!

The information in this posting is out-dated.

The Sandbox will no longer be available after April, 2013

Don’t stop using the sandbox just because you’ve gone to a ‘live’ flavor!

dotCloud’s sandbox is the perfect place to test your code before you push it to your live application.

There’s no reason to skimp on having a development environment — it’s free on dotCloud.

Using a sandbox for development and testing will help you get your deployment polished before you disturb your live app and your customers.

It is not sufficient to test the code on your local system!

There are just too many things that could be different, from the operating system to the framework, to some subtly-missing dependency. If you had to install something locally, for example, then you’re going to have to figure out how to also install it on your dotCloud service.

Dependencies

Your best bet for installing dependencies is to include them as part of a standard list of requirements.

The details of this list change depending on your development language, but Python has requirements.txt, PERL has Makefile.PL, Ruby has a Gemfile, Node.js has package.json, and so forth. The dotCloud build system will process each of these during your application’s build and install phase.

If your dependencies themselves require additional Ubuntu packages to be installed, sometimes the requirements system for your language will handle that. But if you must install other packages explicitly, we support a ‘systempackages‘ directive that allows you to install any standard Ubuntu 10.04LTS package.

This used to be available only as part of a ‘custom’ service type, but now you can use them in any code-based service type (e.g. type: python, perl, nodejs). With entries in the systempackages section of your dotcloud.yml, you can add the system libraries your app needs, even the ones that would normally require ‘sudo’ access. We don’t provide sudo access to you on the command line, so this is a powerful tool.

But what if you have even more complicated requirements? What if you have your own code that you need to build? Then you should look at our build hooks which allow you to execute your own build systems before and after the build, or even after the installation but before you start receiving traffic.

And the great thing is that the build system is exactly the same between ‘sandbox’ and ‘live’ flavors. You’ll be able to perfect all the building and installation you need before you push to live.

Keeping in Sync

In order to make your sandbox behavior predictive of what will happen on your live application, it is important to know what could make the two environments different, and how you can make them the same again.

Service Image Versions

One common source of differences occurs when the sandbox and live apps were created at different times, far enough apart that they have different versions of default services and libraries.

All those great updates we’re making and announcing here in the blog, they can cause your app to behave differently depending on when your production system hasn’t been updated (you control when this happens). We bundle those defaults into what we call the service image version. In CLI 0.4, we exposed this as image_version in the output of dotcloud info, and we will provide that information in the next minor update of CLI 0.9.

For now, the best practice would be to create the sandbox application and live application within minutes of each other to ensure that they get the same service image versions. If the ‘live’ app you want to test was created some time ago, then you can still test in a new sandbox.

The new sandbox may have a newer service image version, but once you have your app working in the sandbox, you can push to your live service with the –clean option, and this will upgrade your live app (and all its services, even the scaled ones) to the latest service image version, bringing it up to speed with the sandbox version where you were testing.

When you don’t use –clean, we preserve your service image version so that every time you get the same versions of things and you never get unexpected upgrades in the default versions, even when you scale horizontally.

If you need to move backward to an old service image version (as well as your old code), you can use dotcloud deploy previous and this will allow you to go backwards in time to the previous revision of your application and service image version (run dotcloud deploy –help for more information).

Spot problems early and roll back quickly: don’t keep pushing or you might not be able to get back to the old environment.

Dependency Versions

Even if two services have the same service image version, they can be different. They could, for instance, have different ‘systempackages’ installed, or different versions of dependencies. The service image version only refers to the default configuration you started with. Your own changes and installations are layered on top.

Imagine that you depend on package-x, version 2 or greater. You create a ‘live’ app and push, and your language satisfies the dependency with the latest version of package-x, version 2.1. Great! But now it is two weeks later and you want to test some new code in the sandbox. You create a new sandbox application and push with the same requirement: package-x version 2 or greater. This time the latest version is 2.9! So the dotcloud builder fetches version 2.9 of package-x for this new app (though you might not notice this unless you’re watching everything carefully).

Your testing goes great, so you push your code to your live app. The dotCloud builder starts from the previous build snapshot and checks: is the installed version of package-x greater than 2.0? Yes, 2.1>2.0, so no need to update. This could break your code, because your testing was on 2.9 but your live app is going to keep 2.1.

So it is important to keep strict control over the versions of all the libraries you depend on, either by freezing their versions (and sharing this frozen list between your live and sandbox environments) or by always testing on a new sandbox environment and pushing –clean to your live environment.

Data

As easy as it is to push the same code to both sandbox and live applications, data can be hard to move around, especially if you have a lot of it.

If you’re not making any changes to your data when testing then one option is to connect to your production (live) database from your sandbox application. To do that you’ll need to manually copy the connection information to the sandbox app.

But if your tests involve making changes to the data, or even migrating the database structure, you should test on a sandbox database. You should figure out what makes sense for test data: a sampling from your production database or some synthetic data generated just for testing.

This could, for example, be a great opportunity to test restoring from a backup file (you are backing up your production database, aren’t you?) But in all cases, you should not plan on moving a lot of data back and forth between the sandbox and live environments. Code is easy to move, but data is difficult.

Stability and Moving Forward

The dotCloud build system keeps things stable for you by preserving your service image version for as long as you keep that service (i.e. it only disappears if you dotcloud destroy that service or push –clean). Not only do we keep your service image version, but we also keep your previous build, incrementally building from where you left off (though always in a new container).

You don’t have to update or upgrade until you’re ready for it, and when you are, it is as simple as “push –clean”.

Testing on the latest and greatest service image version is as easy as creating a new sandbox application and pushing your code there. There’s every reason to keep moving forward, at your own pace, and its easy to do without introducing instabilities that will upset your customers.