Implementing a comfortable (multi step) import process
As we did in one of our last projects, I want to show how we split up a CSV import process into several steps, giving the users the chance to preview and edit their data, before it was finally persisted into the database.
To communicate the result of background processes back to the user waiting in front of your application, we created a handy little resque based gem called StatefulJobs doing all the delegation/propagation work for you.
This post does not cover CSV handling and data import itself, but rather focuses on a concept how to manage the several steps in a responsive way to the user.
Specifying: The process
The desired import process looks like the following:
- Upload CSV
- Get a preview of the data
- Ability to change the data of invalid records
- Persist the records into the database
Where 2. and 3. are repeated until all records are valid to persist or marked to skip by the user.
Defining jobs: The model
Technically our import process splits up into the following jobs:
- extract (extracting the data from csv file, validating each record and temporarily store the results)
- update (merging modifications, validating and storing results again)
- persist (validating and persisting all the data into the database)
With StatefulJobs we can now wrap an ActiveRecord model around these three steps. The model handles the state of our process automatically and provides the context to store our intermediate results until the process is finished.
$ rails g model import current_job:string current_state:string csv_file:string intermediate_results:string
Besides the mandatory columns current_job/current_state to store the current job/state, there’s one to hold the csv file and another one to store our intermediate results to share between the several jobs processed.
Every job defined should return true or false to indicate it performed successfully or not. For example, the jobs return false, when validation of the data to import fails. In that case, we can render the import preview again for the user to decide whether the invalid records should be skipped or modified just in place.
# app/models/import.rb class Import < ActiveRecord::Base include StatefulJobs::Model stateful_job :extract, ImportExtractionJob stateful_job :update, ImportUpdateJob stateful_job :persist, ImportPersistanceJob end # app/jobs/import_extraction_job.rb class ImportExtractionJob < StatefulJobs::Job::Base def perform # extracting data from csv, validating and saving our intermediate results # we have access to our import model via @model end end # app/jobs/import_update_job.rb class ImportUpdateJob < StatefulJobs::Job::Base def perform # merging intermediate results with updates, validating # we have access to our import model via @model end end # app/jobs/import_persistance_job.rb class ImportPersistanceJob < StatefulJobs::Job::Base def perform # persist all records to the database # we have access to our import model via @model end end
Kicking it off: The controller
To provide the actions dealing with the state and state changes of each job, the controller has to know the import model. Besides the automatically defined state action which returns the current job/state of the according import process as json, we configure a state_changed action, which is invoked on every state change via ajax. This is the place to decide what happens next when a job has finished (e.g. rendering the preview after the data has been extracted).
Alternatively when omitting the ‘action’ option, we get a separate action for each state.
# app/controllers/imports_controller.rb class ImportsController < ApplicationController include StatefulJobs::Controller stateful_jobs :import, action: 'state_changed' def state_changed @import = Import.find[:id] # show preview / success / error based on the current state end end
As we wrapped an ActiveRecord Model around our jobs, accessing it is as simple as accessing any other RESTful resource. Each job implemented by our import model gets a ‘!’-method, enqueueing it directly on resque.
This happens the first time after a successful creation of a new import:
# app/controllers/imports_controller.rb class ImportsController < ApplicationController def create @import = Import.new params[:import] if @import.save @import.extract! redirect_to @import end end end
Tracking it: The view
To keep track of the current import’s state there is a helper which provides the view with a simple polling mechanism. It invokes our previously defined state_changed action remotely on every state change.
<%= stateful_job :imports, @import, :div, class: 'spinner', interval: 3000 do %> spinner <% end %>
Which turns into the following html, polling the server for a new status every 3 seconds. To indicate whether a job is running, the css class
running is applied to your spinner’s div while a job is processed.
Concluding: The use case
This is how we splitted up an interactional process in the frontend into several background jobs on the serverside.
StatefulJobs becomes handy, whenever:
- background jobs have to provide its state to the frontend again
- background jobs deal with user interaction
- background jobs are splitted into several steps and share process relational information