Gambling development has its own rules and if you have chance to face them, you may know how hard it can be to meet all the gambling regulatory authority requirements.  In one of our recent projects (horses racing betting website) we had to follow a lot of rules from the ARJEL - regulatory Authority for online gambling in France. The ARJEL provides french gambling operators with license, but only if they prove that they can protect players, prevent fraud, and, the most important, prove integrity of gambling activity. Which means for developers that each bet, and most of the users’ actions, must be tracked in secured locations in append-only mode.

In practice, we need to follow this workflow:

1. User sends a secure (https) request to the Collector, the request is intercepted before hitting the Gaming Platform (aka “GP”);

2. The Collector will hit the GP, but all params are saved from the initial request (so data can’t be altered by the GP);

3. The GP is answering to the Collector: if status is OK, then the process continues, if there’s error - the Collector respond to the user with the corresponding message and the process is stopped);

4. The Collector is sending data to the secure Vault; the Vault is answering to the Collector, if status is OK,  then the process continues, otherwise GP (which is important) must rollback the action;

5. the Collector sends the answer back from the GP to the User (if 1-5 went with OK status).

Such process is quite simple on the paper. But really hard to put in place in the real life. This article explains how we resolved this using only a few lines of Ruby.

The technical specs can be found (in French only) here and especially in this doc

Saving and protecting parameters

Parameters sent by the player must be saved locally in the Collector, to ensure their consistency when events are sent to the vault. They can’t be altered in any way by the Gaming Platform, so the best solution is to never trust it.

That was solved by using ruby closures, that make up most of collector’s logging. Closures will allow us to write code that can be passed like an object (to be called later) with all the variables in context saved.

A very simplified version of this code could be:

class Handler
  attr_reader :queue
  def initialize
    @queue = []
  end
  def enqueue(trace_type, options={}, &block)
    @queue << block
  end
end

which can be called like this:

collector = Collector.new.enqueue do
  Bet.create(params[:bet])
end

Performances

Next major problem was the performance - we need to take into account that Collector->Vault requests had to be as fast as possible because :

• while vaulting, the user is still waiting for an answer from the server (see schema above)
• for horse racing, most of the bets are placed in the last 2 minutes before the start and cannot be placed after the race start, so if the vaulting is slow, the number of the accepted bets might be low.

Unfortunately, by default, only one request can be sent to the Vault at once. And here comes Typhoeus, a Curl wrapper that can send send multiple requests in parallel, just what we need to improve performance (the vault has a simple HTTP API). Typhoeus gathers all the requests to send in a queue. This queue is called “hydra” and once fired, it sends all the requests with a nice concurrency. That way, we could hit the vault with dozens of requests (ie: bets) at once, in parallel.

Atomicity

As one of the ARJEL requirements, on the step 4 of the workflow it’s necessary to track all the events in the requests, so the Collector creates XML document with all the info about the request (some kind of a log entry) and sends it to the Vault in the request. The main issue here is that only one XML document can be sent per request.

Why is it an issue? Just imagine a typical signup request, which contains 2 XML documents (or they are also called “traces”) for 2 events: one straight for signup and one for accepting Terms of Services, or “TOS” (in France you must accept TOS to register). Those 2 traces must be sent together, but as mentioned above the Vault allows only one trace per request.

So, if the signup trace was not be successfully recorded (vaulted), the “Accept TOS” trace must not be sent to the Vault. That’s why we introduced a DFS algorythm (Deep first Search) in our CollectorTransaction model, where traces are stacked and can even be in a tree within one request: canceling a node will also cancel the following sub-nodes.

def cancel_dependent_transactions
    @stack = []
    dfs(self.on_complete)
    @stack.each(&:cancel)
    true
end

def dfs(ctransactions)
    ctransactions.each do |ct|
        @stack << ct
        dfs(ct.on_complete)
    end
end

(Surprisingly, it’s pretty rare to see recursive functions in Ruby.)

Remember that events are sent in parallel to the vault (using Typhoeus). If we send the 2 events in parallel, there’s a (small) chance that one of them fails: that’s why we force some events to be sent after others. This DFS implementation will help to cancel and clean dependent traces waiting to be sent.

Some Glue

In addition, it’s really important to be aware of requests’ state (e.g. in case of server down). That’s the role of the CollectorTransaction model, where state is maintained with State Machine, giving 2 advantages : 

• in case of crash, we know exactly what was the state of a request;
• we could match our data with the vault data (vault wasn’t directly accessible to us).

The state machine was quite simple:

state_machine :status, :initial => :pending do
  event :send_to_vault do
    transition :pending => :sent
  end

  event :cancel do
   transition :pending => :cancelled
  end

  event :rollback do
    transition :pending => :rollbacked
  end

  before_transition :on => :rollback, :do => :restore_model
  after_transition :on => :rollback, :do => :cancel_dependent_transactions
  before_transition :on => :send_to_vault, :do => :save_model_id
  after_transition :on => :send_to_vault, :do => :fire_model_callbacks

  state :sent do
    validates :response_code, :inclusion => {:in => [200]}
  end

  state :rollbacked do
    validates :response_code, :exclusion => {:in => [200]} # 0 = could not connect, 409 = no IDE header returned by vault
  end
end

Enqueuing events to be vaulted

So, here’s typical example of enqueued collector’s request :

bet = Bet.new(:combination => combination, :bet_type => bet_type, :stake => stake.to_i, :order_id => @order.id, :race_id => @race.id)
collector.enqueue(:PAHIMISE,
              :model => bet,
              :extra_params => {
    :race_name => [@race.to_s, @race.name, @race.place.name].join(' - '),
    :balance_before_checkout => current_user.balance,
     :balance_after_checkout => current_user.balance - bet.stake}) do
   bet.save!
 end

See how the closure is acting? We don’t actually save the bet in the controller, but we save the context of the bet inside a Collector::Handler instance. If the vaulting went ok, then we can execute that code, and the bet will be saved. Ruby has some magic inside sometimes, because the variable “bet” only live in this piece of code, and the closure block context. As long as the collector object is alive, the context is.

Note, that the model (ie: the bet) in the transaction is set before being changed, this is required because existing model will be used for the rollback in case of failure. Remember we should vault only when the GP returned a valid response. Literally, this means that the bet was committed to the DB, so we can’t use DB transactions to rollback in case of vaulting error (it would be too simple ;°) ).

New request from the the Collector queue calls enqueue method:

def enqueue(trace_type, options={}, &block)
  ct = CollectorTransaction.new do |ctransaction|
    ctransaction.trace_type = trace_type.to_s
    ctransaction.model = options[:model] if options[:model]
    ctransaction.extra_params = options[:extra_params].nil? ? {} : options[:extra_params]
  end
  ct.block = block
  @queue << ct
  if options[:after]
    options[:after].on_complete << ct
    ct.dont_enqueue = true
  end
  ct
end

block is executed when the collector will try to vault the whole transaction. Request parameters from the player are also saved in this model.

Fire!

Then, finally, vaulting is fired, which means “dequeuing” the handler:

ActiveRecord::Base.transaction do
  @queue.each do |ctransaction|
    ...
      # Transaction is created
      ctransaction.request = create_request(ctransaction)
    # And the process responses
    ctransaction.request.on_complete do |response|
      if response.success? and response.headers['IDE']
        ...
          Rails.logger.info "Vaulting successfull for transaction ##{ctransaction.id} (#{ctransaction.trace_type})"
        @queue.delete ctransaction if ctransaction.sent? # transactions left in the queue are failures
      else
        if ctransaction.max_attempts_tripped? # we have tried enough, it's time to give up
          Rails.logger.info "Vaulting failed for transaction ##{ctransaction.id} (#{ctransaction.trace_type}) with code=#{ctransaction.response_code}"
          ctransaction.rollback # Rollback the transaction, and cancel dependent ones (cf. CollectorTransaction model)
        else # let's retry!
          ctransaction.attempt += 1 # one more try
          @hydra.queue ctransaction.request
        end
      end
    end
  end
end

Conclusion

Thanks to ruby magic closures, we could face and solve a very complex problem in a short coding time. This system was audited and validated by the ARJEL in 2011. We did our best to ensure the maximum of features, but keeping simplicity in mind. The whole vaulting system can be seen as a queueing system, with specific transaction system for rollbacking. Less code means less bugs, and easy maintenance, always keep that in mind! The complete collector code is available is in this gist.