Testing Go projects with Docker

Our CI server is getting a bit overloaded, and I didn’t want to install more packages on it to test our new go projects. Since we are also playing a lot with Docker lately, it felt natural to use it to run test suites and determine coverage of go projects. Just by installing docker on our private CI server, all our team is now able to add various projects, with different versions, without interfering with other projects or environments.

This first step was to define a common Go base image for our go projects :

# Prefer debian over ubuntu
FROM debian:wheezy
# Let's install go just like Docker (from source).
RUN apt-get update -q
RUN apt-get install -qy build-essential curl git mercurial bzr
RUN curl -s https://go.googlecode.com/files/go1.2.src.tar.gz | tar -v -C /usr/local -xz
RUN cd /usr/local/go/src && ./make.bash --no-clean 2>&1
ENV PATH /usr/local/go/bin:/go/bin:$PATH
ENV GOPATH /go
# we're using godep to save / restore dependancies
RUN go get github.com/kr/godep
# Cover is used in almost all projects too
RUN go get code.google.com/p/go.tools/cmd/cover
WORKDIR /go

Let’s create a Makefile along to this Dockerfile:

all: build

build:
        docker build -no-cache -rm -t docker:5000/go:1.2 .

run:
        docker run -rm docker:5000/go:1.2

We can argue on the no-cache parameter here, but since the second line is a “apt-get update”, we should not cache this. If we decide to modify th Dockerfile later, and add a new debian package with “apt-get install”, it will possibly fail because it won’t be available anymore for download.

Anyway, we now have our docker:5000/go:1.2 image ready to be used. We won’t use this image directly, but instead share it in other docker images.

Let’s Create our go project

Let’s create a new “go-example” project on our git server. The local path in $GOPATH will be something like “$GOPATH/src/gitlab.local/tech-angels/go-example”. Let’s create a Dockerfile for this project:

FROM docker:5000/go:1.2

ADD . /go/src/gitlab.local/tech-angels/go-example
# Fetch deps
WORKDIR /go/src/gitlab.local/tech-angels/go-example
RUN go get
# Allow to mount the current version in the container
VOLUME /go/src/gitlab.local/tech-angels/go-example

With the common Makefile:

all: build

build:
        docker build -no-cache -rm -t docker:5000/go-example .

We now have a full project environment, ready to compile and test, without installing anything else than Docker.

And now, configure the CI Server

We are using Gitlab-CI to test our projects, but the following will work with any CI server out there.

docker pull docker:5000/go-example
docker run -t -i -rm -v `pwd`:/go/src/gitlab.local/tech-angels/go-example docker:5000/go-example go test -cover

and Voilà!

Our CI will just start a container, run the tests, and get an exit code from the go command.

Go further

We didn’t add the save / restore deps step in this very simple example. It’s trivial to add it, and you’ll need a test script inside the container for that.

Also, you probably realize now how simple it is to test with different go versions!


Happy new year all! To celebrate 2014, we decided to refresh a bit our processes and tools. While Campfire was doing a great job, and was especially stable (ie: available), it was lacking a lot of features. The most annoying was the missing ability to edit a misspelled message. Our typos were frozen for eternity, what a shame!

We decided to give Flowdock a shot, and felt instantly in love with it. The notification are finally separated from the chat flow, they can be categorized, and have their own life (understand: a chat thread can be created a every notification!). The service is affordable (at least for a small team like us), and their mobile native apps are also a great way to keep in touch with co-workers. Especially, a mention “@user” will trigger a mobile notification if not read on the web, of the native app (windows/mac). We strongly encourage your team to try flowdock, you won’t regret it!


Tech-Angels is turning 4!

We are happy to celebrate our 4th birthday!

The last four years have been extremely challenging: we have improved the level of all our skills, and our team is very proud of all our realizations.

We would like to say an enormous THANK YOU to all our customers! We will continue to raise the bar, and provide a quality of service we’re proud of.

"Longue vie" Tech-Angels !


We’re using since this summer vagrant (with vmware fusion) to run virtual environments closer to our production servers. We discovered some slowness with the development environment, and especially when the number of assets is important, which is particularly true with our backbone apps. The network graph in chrome was showing a lot of network latency:

We first thought of ns lookup resolution, but it was rapidly clear that this couldn’t be the issue. There was really something odd about our VMs, since the dev env was running much faster outside the VM (on our laptops directly).

After some digging, we found this vagrant option: (http://docs.vagrantup.com/v2/synced-folders/nfs.html)

"In some cases the default shared folder implementations (such as VirtualBox shared folders) have high performance penalties. If you’re seeing less than ideal performance with synced folders, NFS can offer a solution."

Vagrant.configure("2") do |config|
  # ...

  config.vm.synced_folder ".", "/vagrant", nfs: true
end

Indeed, NFS is a solution. The access on the development unicorn is now much faster, without these big latencies.

Another good news, this change comes for “free” for Mac Os users (not tested on Linux, but it should act the same), the NFS exports on the host are managed by Vagrant. We just had to add this line to see a huge performance boost.


October News

We’ve been very busy this summer, and have some great news to share!

First of all, we spent a lot of time polishing our development environments. We were spending more and more time configuring envs, and fighting tiny differences between OS (Mostly Mac and Linux Ubuntu in dev, Linux Debian in production).

It was a big task, but we are now using Vagrant and Puppet to manage our development virtual machines. With Packer to the rescue, we were able to create standard images for both vmware and virtualbox. Puppet is in charge of the provisioning, and will use the same recipes as our production servers now. Every VM will be automatically synced with the dev puppet master so we don’t have to worry about the configuration of each computer any more. Everyone in the team has a running instance of the projects in minutes now. Having the exact same postgresql, redis, etc. versions is also a big win. The impact of upgrading one of these components is now extremely easy to determine.

We’d like to share our experience with Vagrant and managing software projects on the long term. That’s why we’ll be hosting a workshop session at dotRB next Oct. 19th in Paris.


Speaking of wvmare, we are also proud to announce that we have migrated all our infrastructure to 2 privates clouds powered by vmware vsphere. The 2 clouds are located in France and Canada, and every network appliance / server / san is fully redundant. We are now able to extend the capacity of our hosting without limits, with just a few mouse clicks. This migration was necessary since we are hosting more and more services for our customers, and for our own products. By the way, we will recruit sysadmins in the US very soon, contact us if you are interested.


First customer using our new infrastructure, EasyWelcome (http://easywelco.me and https://twitter.com/easywelcome ) is a platform that aims to take the hassle out of the short term rentals by providing services bookable directly online and operated by professionals and vetted local experts. The website was developed by our engineers, using Ruby on Rails and Backbone.js.


We’d like also to welcome Santech (http://www.santech.fr), which is now hosted on our servers. Santech is a new company created in 2012, specialized in online health products and services.


Tired of spending hours looking for what has changed in your dependencies?

Save your time! Gemnasium brings you 1-click changelogs!

For each of your project’s dependencies you can now quickly check what has changed. Just click on the Changelog viewer icon and start reading!

Gemnasium will automatically aggregate changelog data and scope it to what matters to you: the changes between your current version and the latest stable!

No changelog? We can still help you.

When package’s changelog is not available, Gemnasium still try to make it easier for you. It will indeed try to fall back to a nicely displayed list of commit messages.

At any time you can click on a commit from the list to see it on Github. You can also check the full compare view following the link provided in the header.

Pricing?

Our Changelog viewer is already included in all paid plans without any extra cost and is available to free users during their 30 days trial period.

For all existing free users, you’ll also be able to try it until june, 15th, enjoy!

We hope you’ll appreciate it and your feedback is welcome!

Cheers, Gemnasium Team


What’s new in Gemnasium?

We’ve released a bunch a small fixes and improvements to make Gemnasium better! Let’s review them.

Manual monitoring

You now can choose exactly which projects you want to monitor and which one you don’t care about.

Gemnasium will still try to automatically add/remove projects when syncing with Github, but as soon as you manually start or stop monitoring a project, it won’t override your choice anymore.

So you now can freely choose which projects you want to track, and drop the others.

To make it more clear, we’ve also cleaned up the profile view and you’ll now only see the monitored projects there. This will drastically reduce the signal to noise ratio so that you can focus on what matters to you.

You can retrieve all your profile’s projects in the Project’s management page in your settings. There you’ll be able to start/stop monitoring and edit projects settings to fine tune them.

Offline projects

First introduced with our Gemnasium Gem, the offline projects can now be totally managed on Gemnasium.

You can create projects from your profile settings and upload dependency files directly from the UI.

New platinum plan!

As required by our users who have a lot of private stuff, here comes the Platinum plan. It allows to track up to 125 private projects for $100/month!

Improved settings

We’ve changed a bit the settings to make account management easier.

The Settings home now displays the list of your Gemnasium profiles. Here you can manage them and add new ones.

The Github Permissions feature has been improved to give more granularity and now tells you how it affects your Gemnasium account.

Thanks for your support and your feedback that help us improving Gemnasium!

Cheers, Gemnasium Team


Rails 3.2.13 was released just a week ago. Since it fixes 4 important security breaches (CVE-2013-1854 for activerecord, CVE-2013-1855 for actionpack, CVE-2013-1856 for activesupport and CVE-2013-1857 for actionpack), you may want to upgrade it asap.
Yet, performance regressions and major bugs has been discovered in this new version of Rails.

What are the risks ?

Upgrading your application to Rails 3.2.13 will make bugs and performance issues popping out.

  • action_missing function will be broken
  • ActiveRecord chained scopes will cause bugs such as the one GitHub experienced lately
  • Assets and views loading time will drastically increase

How to remain safe without upgrading to 3.2.13?

We recommend you to not upgrade to Rails 3.2.13 and to wait for 3.2.14 to be released. But how to do so having in mind the 4 security breaches that still exists ?
Well, that’s simple, first of all, you might not be impacted by all of those issues and second, some monkey patches have been released to help you keep your application secure without upgrading.
So keep calm, create a temporary hotfix branch until 3.2.14 is out and apply the patches you need.

Fixing Symbol DoS vulnerability in Active Record (CVE-2013-1854)

Your application does not use params as a find value for a query? You’re safe!

Impacted code will look like User.where(:name => params[:name]). To fix this issue, you should call to_s method on params used as a find value. Basically, you’ll need to change code that looks like

User.where(:name => params[:name]) 

to:

User.where(:name => params[:name].to_s)

Patch provided by Rails team

Fixing XSS vulnerability in sanitize_css in Action Pack (CVE-2013-1855)

You don’t use sanitize_css method using user input as parameter? You’re safe!

Impacted code will look like sanitize_css(user_input). The following patch will fix the issue:

module HTML 
  class WhiteListSanitizer 
    # Sanitizes a block of css code. Used by #sanitize when it comes across a style attribute 
    def sanitize_css(style) 
      # disallow urls 
      style = style.to_s.gsub(/url\s*\(\s*[^\s)]+?\s*\)\s*/, ' ') 

      # gauntlet 
      if style !~ /\A([:,;#%.\sa-zA-Z0-9!]|\w-\w|\'[\s\w]+\'|\"[\s\w]+\"|\([\d,\s]+\))*\z/ || style !~ /\A(\s*[-\w]+\s*:\s*[^:;]*(;|$)\s*)*\z/ 
        return '' 
      end 

      clean = [] 
      style.scan(/([-\w]+)\s*:\s*([^:;]*)/) do |prop,val| 
        if allowed_css_properties.include?(prop.downcase) 
          clean <<  prop + ': ' + val + ';' 
        elsif shorthand_css_properties.include?(prop.split('-')[0].downcase) 
          unless val.split().any? do |keyword| 
            !allowed_css_keywords.include?(keyword) && 
            keyword !~ /\A(#[0-9a-f]+|rgb\(\d+%?,\d*%?,?\d*%?\)?|\d{0,2}\.?\d{0,2}(cm|em|ex|in|mm|pc|pt|px|%|,|\))?)\z/ 
          end 
          clean << prop + ': ' + val + ';' 
        end 
      end 
    end 
    clean.join(' ') 
  end 
end 

Patch provided by Rails team

Fixing XML Parsing Vulnerability affecting JRuby users (CVE-2013-1856)

Your application does not use JRuby? Your JRuby application does not use the JDOM backend? You’re safe!

To fix this issue, place this code in an application initializer:

ActiveSupport::XmlMini.backend="REXML"

Patch provided by Rails team

Fixing XSS Vulnerability in the sanitize helper (CVE-2013-1857)

Your app doesn’t use sanitize method helper with user input? You’re safe!

To fix this issue, place the following code into a file in your config/initializers folder.

module HTML
  class WhiteListSanitizer
    self.protocol_separator = /:|(&#0*58)|(&#x70)|(&#x0*3a)|(%|&#37;)3A/i

    def contains_bad_protocols?(attr_name, value)
      uri_attributes.include?(attr_name) &&
      (value =~ /(^[^\/:]*):|(&#0*58)|(&#x70)|(&#x0*3a)|(%|&#37;)3A/i &&
      !allowed_protocols.include?(value.split(protocol_separator).first.downcase.strip))
    end
  end
end

Patch provided by Rails team


Nothing is more important for us than your data safety, and we’d like to really thank our customers for their trust so far! Anyway, as we are very concerned with security, we find unacceptable to be forced asking you for a full access on your projects. We had a lot of complains regarding this problem, and we fully agree with you:** you should never grant access to your work.**

That’s why we’re proud to unveil today a new way to keep your projects in shape, without lowering your security requirements.

Why did we need a full access to your private repos?

Gemnasium is fully integrated with Github: it automatically fetches your repositories and gets notified about code updates (pushes). But for that, you need to authorize Gemnasium to access your code through Oauth.

Unfortunately, Github only provides a “Read + Write” access for private repos (the “repo” scope), which implies a full trust in the third party service. For freelancers, and agencies, it meant sharing their customer code with us, which is often forbidden by contracts or NDAs (and it’s a good thing!).

If you don’t trust this gem either, keep calm and relax, the source code is fully available: https://github.com/gemnasium/gemnasium-gem

How does it work?

Dependencies can be tracked now without authorizing any access to your application code, just by using that tiny gem!

Drop it in your project (or just install the gem), run install command, fill in the config file and you’re ready to go in a few seconds!

You can also use the provided post-commit Git hook to automatically push updates to Gemnasium when you commit changes on your dependency files.

Alternatively, you can use the “gemnasium” command, the rake task or even call directly Gemnasium’s gem classes from your code. See readme for more details.

Finally, using the Rake task, a CI server can be in charge of pushing the changes for you.

Behind the scene

When executed, Gemnasium gem will calculate SHAs for your dependency files (the same way Git does it) and send them to Gemnasium API to compare with remote ones. If anything changed in the files, SHAs will be different and then Gemnasium gem will upload updated files.

The after-commit Git hook we provide is even smarter as it will only fire SHAs check if your commit contains changes about your dependency files. It’s even faster than with our Github service hook!

One great thing with this gem is that you can check dependencies updates without pushing to the repository! Just update the dependency files, run “gemnasium push” and everything gets updated on gemnasium.com.

Gemnasium Gem is published under the MIT license. It’s compatible with all ruby projects, not only rails app. We’ve made it as light as possible and it doesn’t require any other dependency. Though, it’s only compatible with ruby >=1.9 for now (yes, that mean ruby 2.0 too!).

For Node.js projects owners, please note that this gem is also able to upload package.json and npm-shrinkwrap.json files, as long as you have a running ruby. For others don’t worry, a Npm module is coming!

Please also note that for now the gem only works with existing Gemnasium profiles. So you still need to link your Gemnasium account with Github after registration (with read-only public access at least) and order a plan for one of your Github profiles. You’ll then be able to add your offline projects to the profile of your choice. Offline private projects are taken into account for your plan’s private slots limit.

We’re also working on fully offline profiles to allow non-Github setup. This will come a bit later. Stay tuned!

How to migrate existing project?

If you have already setup your projects to be synchronized automatically with Github, you can switch them to use the gem. Just follow the readme to complete the setup and use the “gemnasium create —force” command. The “—force” option will override the existing setup on gemnasium.com, updating your projects origin attribute from “github” to “offline”. Warning: there is currently no easy way to switch back to Github origin (need to contact support). Use with caution.

You can then update your Github permissions in your settings to use the read-only public access.

No more excuses to not track your dependencies and keep your projects in shape with Gemnasium!

Cheers, Gemnasium Team