Software in context

Tags


A Healthy Gulp Setup for AngularJS Projects

27th February 2015

Note: If you want to get started with Gulp + Angular, the setup described below is available on GitHub.

Why did you switch from Grunt to Gulp?

Honestly, I could not personally answer this question for a while. I didn't really understand the benefit of a "streaming build system" until I spent a couple of days trying to write a Gulpfile that felt correct. But once I grokked some of the key ideas behind Gulp, it became refreshingly natural and quick to work with.

And since getting into Gulp, my sense from the community is that most Gulpers misunderstand the tool, and are missing out on what it does best: allow developers to quickly create custom automated build workflows—even quite complicated ones, as you'll see below.

What isn't Gulp?

One Gulp issue is a good example of this misunderstanding. The poster is concerned that the deprecation of the gulp.run() method limits flexibility in task creation. After all, Grunt has grunt.task.run. When creating complex build tasks, isn't it sensible to define simpler subtasks and piece them together? The poster correctly notes that now, "the only way to have a task run other tasks is by specifying them as dependencies." With Grunt, dependency-style task building looks like:

grunt.registerTask('dev', [  
  'htmlmin',
  'less',
  'jshint',
  'concat:dev',
  'copy'
]);

The dependencies are run in synchronous order, and I can run each of the constituent tasks independently if needed. Simple. Gulp has a similar-looking api:

gulp.task('mytask', ['array', 'of', 'task', 'names'], function() {  
  // Do stuff
});

But there's a big difference: Gulp will ensure that the dependent tasks complete before running 'mytask', however, they are run asynchronously. This doesn't help if you're trying to break a complex task down into sub-steps. If I need to run my scripts through jshint before concatenating and uglifying, do I have to wrap these up into one big task? But I also want to run jshint independently, so will I have duplicate code? Some taskrunner!

In the aforementioned GitHub thread, the contributors who respond actually argue that Gulp is not a taskrunner, but a "build system helper". Huh?

What is Gulp?

Gulp Logo

Gulp is an instance of Orchestrator that has a minimal task API connected to it, and a ton of plugins (some for doing build-ey stuff).

So what is Orchestrator? It's a simple and powerful node library to specify units of work, their dependencies, and then have work done in maximum concurrency. It's essentially a scheduling algorithm. In order to behave correctly, the units of work (really just functions), have to either:

  1. execute synchronously,
  2. accept a callback function,
  3. return a promise,
  4. or return a stream.

Streams are awesome. They can contain arbitrary objects, and can be connected with pipes, merged, split, filtered, etc. Streams are also leveraged heavily by Gulp. For example, the gulp.src method returns a stream of files:

gulp.src('client/templates/*.jade')  
  .pipe(jade())
  .pipe(minify())
  .pipe(gulp.dest('build/minified_templates'));

Here, the jade(), minify(), and gulp.dest() calls are operating (in order) on the streams coming through the pipes. Pretty cool, right?

There are only four Gulp methods. Gulp doesn't do a lot, aligning with their philosophy of "code over configuration". With a minimal API, Gulp has simply unlocked a powerful model for creating complex build tasks. You still have to write code, it'll just be more beautiful and concise code…

How to use Gulp

We could stick with specifying task dependencies for creating complex tasks. But this would get ugly very quickly. Try it! Declaring dependent units of work that are executed asynchronously isn't a great way to model a linear process, which is what most build tasks are. So what's the alternative? Pipes!!

Physical pipes are some of the most modular things humans have ever invented. Let's take advantage of Orchestrator's power and model our build tasks with streams. We'll formulate all tasks and subtasks as "pipe segments" that return streams of files in a certain condition. Then the work of writing a complex build task—like building and watching the entire application—becomes a matter of snapping together reusable pipe segments.

For example, here's a pipe segment that returns a stream of one file—our compiled application source app.min.js:

pipes.builtAppScriptsProd = function() {  
    var scriptedPartials = pipes.scriptedPartials();
    var validatedAppScripts = pipes.validatedAppScripts();

    return es.merge(scriptedPartials, validatedAppScripts)
        .pipe(pipes.orderedAppScripts())
        .pipe(plugins.sourcemaps.init())
            .pipe(plugins.concat('app.min.js'))
            .pipe(plugins.uglify())
        .pipe(plugins.sourcemaps.write())
        .pipe(gulp.dest(paths.distScriptsProd));
};

The "task" of building the production JavaScript does in fact need to wait for (1) the scripts to be validated, and (2) the partials to be converted to scripts. But Orchestrator absorbs the concept of temporality for us, remember? We just have to think about streams. Thus builtAppScriptsProd only needs to merge these two streams (defined in their own segments) before piping them into the segments that concat, uglify, create a sourcemap, etc.

This is so much easier to think about than waiting for dependent tasks to complete! If it's not clear yet, the full Gulpfile is covered in detail below.

A Gulp Angular Setup

The rest of this post discusses the Gulp setup I've settled on for AngularJS projects.

The techniques employed here are applicable for any web project using Gulp. The base setup is available as a project on GitHub.

Features

Combined with my recent switch to WebStorm editor (which auto-saves), it's the best web development experience I've ever had. Here are the top features of the setup:

  1. Development and production environments
  2. A development server that can serve either environment
  3. Watch tasks for both environments
  4. Live-reload capability—web page is auto-refreshed on change
  5. Uses Foundation/SASS
  6. Partials pre-loaded into the angular template cache
  7. Full hinting/concatenation/uglification/sourcemaps for all prod files
  8. Dev server automatically refreshed when source changes

Requirements

Before you can run any Gulp tasks:

  1. Check out the repository
  2. Ensure you have node installed
  3. Run npm install in the root directory (this will install bower dependencies too)
  4. For livereload functionality, install the livereload Chrome extension

Project Structure

The project ships with a directory structure like:

/healthy-gulp-angular
|
|---- package.json
|
|---- bower.json
|
|---- gulpfile.js
|
|---- /app
|     |
|     |---- index.html
|     |---- app.js
|     |
|     |---- /styles
|     |     |
|     |     |---- _settings.scss
|     |     |---- app.scss
|     |
|     |---- /components
|           |
|           ...
|
|---- server.js
|
|---- /devServer
|     |
|     |---- ...
|
|---- (/dist.dev)
|
|---- (/dist.prod)

Let's break this down..

package.json

Server-side (command-line) dependencies.

bower.json

Client-side (browser) dependencies.

gulpfile.js

Where all the Gulp streams and tasks are specified. Tasks are outlined below. This file is discussed in detail in the blog post.

/app

All first-party application source code lives here, including HTML, scripts, and styles of whatever flavor.

/app/index.html

The single page app "shell page". Adapted from Angular Seed. All sources are automatically wired in with gulp-inject.

/app/app.js

The app's main angular module is defined here. This file is always loaded first with gulp-angular-filesort.

/app/components

I like to group my angular scripts by comonent. Each sub-directory here typically contains a directive and a matching html partial.

/app/styles

Custom app styles (I use SASS) live here. There's also a foundation settings file.

server.js

This is the entrypoint for the ExpressJS development server. It respects the environment variable NODE_ENV, taking its value as the directory out of which to serve static resources. It defaults to dist.dev to serve development files, and also accepts dist.prod to serve the production files.

/devServer

The scripts for the development server. I'll typically put mock API responses in here.

Processed Sources

The gulp tasks listed below deal with taking sources from /app and "compiling" them for either development or production. *-dev tasks will output to /dist.dev, and *-prod will output to /dist.prod. Here's an overview of the directory structures for each:

/dist.dev

Sources built for development. Styles are compiled to CSS. Everything else from /app is validated and moved directly in here. Directory structure is preserved. Nothing is concatenated, uglified, or minified. Vendor scripts are moved in as well.

/dist.dev
|
|---- /bower_components
|
|---- /components
|     |
|     ...
|
|---- /styles
|     |
|     ...
|
|---- app.js
|
|---- index.html

/dist.prod

Sources built for production. Everything is validated, things are concatenated and uglified. HTML partials are pre-loaded into the angular template cache with gulp-ng-html2js.

/dist.prod
|
|---- /scripts
|     |
|     |---- app.min.js
|     |---- vendor.min.js
|
|---- /styles
|     |
|     |---- app.min.css
|
|---- index.html

Pretty self-explanatory.

The Gulpfile

Available here. Let's step through section by section.

1. Imports

var gulp = require('gulp');
var plugins = require('gulp-load-plugins')();
var del = require('del');
var es = require('event-stream');    
var bowerFiles = require('main-bower-files');
var print = require('gulp-print');
var Q = require('q');    
  • gulp-load-plugins — First off, notice that we're not loading every gulp plugin separately. The module gulp-load-plugins does this for us automatically. Now we can refer to any gulp plugin specified in our package.json with plugins.pluginName, without having to mess with the gulfile imports all the time. For example, gulp-minify-css is loaded at plugins.minifyCss. Note the conversion to camelCase.
  • del — A basic node module used to remove directories in clean tasks.
  • event-stream — A toolkit for working with streams, the objects processed by the Gulp API.
  • main-bower-files — Gets a list of all the project's main bower files. Great for discovering and injecting third-party dependencies automatically.
  • gulp-print — Gulp plugin to print what's in the pipe. Nice for debugging. Didn't work with gulp-load-plugins for some reason.
  • qPromises! You can have Gulp tasks return a promise, and it'll be respected when specified as a dependent task.

2. Common paths

These paths reflect the project structure detailed above. There are probably more of these that can be pulled up, these are the ones used most often.

var paths = {
    scripts: 'app/**/*.js',
    styles: ['./app/**/*.css', './app/**/*.scss'],
    index: './app/index.html',
    partials: ['app/**/*.html', '!app/index.html'],
    distDev: './dist.dev',
    distProd: './dist.prod',
    distScriptsProd: './dist.prod/scripts',
    scriptsDevServer: 'devServer/**/*.js'
};

3. Pipe segments

This is the real meat of the gulpfile, and the place where Gulp really shines. Notice that I've named all of the segments based on the stream they output, as opposed to the work they do. I think it's helpful to take time out of the equation when thinking about streams and pipes. We're building a plumbing system, not writing procedures.

All of our resuable "pipe segments" will live here:

var pipes = {};

3.1 Ordering scripts

These return streams that have scripts correctly ordered. gulp-angular-filesort is a neat plugin that reads angular scripts and figures out the correct loading order.

pipes.orderedVendorScripts = function() {  
    return plugins.order(['jquery.js', 'angular.js']);
};
pipes.orderedAppScripts = function() {  
    return plugins.angularFilesort();
};

3.2 Renaming files

Returns a stream that has renamed arbitrary files to have .min before the existing file extension.

pipes.minifiedFileName = function() {  
    return plugins.rename(function (path) {
        path.extname = '.min' + path.extname;
    });
};

It's worth noting that the pipe segments in 3.1 and 3.2 are like custom plugins in that they process the streams that are handed to them. All the remaining segments have the beginning of their streams built into them.

3.3 Building application scripts

Returns a stream of app scripts that check out with jshint.

pipes.validatedAppScripts = function() {  
    return gulp.src(paths.scripts)
        .pipe(plugins.jshint())
        .pipe(plugins.jshint.reporter('jshint-stylish'));
};

For development, validated scripts are simply moved to the dev environment. Returns a stream of the newly moved files.

pipes.builtAppScriptsDev = function() {  
    return pipes.validatedAppScripts()
        .pipe(gulp.dest(paths.distDev));
};

Returns a stream of one script called app.min.js that contains validated, correctly ordered, concatenated, and uglified application scripts. Also includes validated HTML partials that have been converted to JavaScript to pre-load the Angular template cache.

pipes.builtAppScriptsProd = function() {  
    var scriptedPartials = pipes.scriptedPartials();
    var validatedAppScripts = pipes.validatedAppScripts();

    return es.merge(scriptedPartials, validatedAppScripts)
        .pipe(pipes.orderedAppScripts())
        .pipe(plugins.sourcemaps.init())
            .pipe(plugins.concat('app.min.js'))
            .pipe(plugins.uglify())
        .pipe(plugins.sourcemaps.write())
        .pipe(gulp.dest(paths.distScriptsProd));
};

This one is interesting because we use es.merge, which combines the two dependent streams, scriptedPartials and validatedAppScripts, into a single stream. The downstream pipes.orderedAppScripts will block until this stream is complete, but makes no guarantees about the order of the events emmitted by the constituent streams. Also note that we're adding a sourcemap to the final script. The concat and uglify pipes have to be attached between sourcemaps.init() and sourcemaps.write().

3.4 Building vendor scripts

Returns a stream of third-party scripts in dist.dev/bower_components.

pipes.builtVendorScriptsDev = function() {  
    return gulp.src(bowerFiles())
        .pipe(gulp.dest('dist.dev/bower_components'));
};

For production, we order third-party scripts, then concatenate and uglify them into a single file. I chose not to create a sourcemap for this one, because it ended up being too large.

pipes.builtVendorScriptsProd = function() {  
    return gulp.src(bowerFiles())
        .pipe(pipes.orderVendorScripts())
        .pipe(plugins.concat('vendor.min.js'))
        .pipe(plugins.uglify())
        .pipe(gulp.dest(paths.distScriptsProd));
};

3.5 Building the dev server scripts

The development server JavaScript lives in, and runs from /devServer. This segment returns a stream of validated dev server scripts. We can use this later to watch changes to the server, if, for example we're modifying a mock API reponse.

pipes.validatedDevServerScripts = function() {  
    return gulp.src(paths.scriptsDevServer)
        .pipe(plugins.jshint())
        .pipe(plugins.jshint.reporter('jshint-stylish'));
};

3.6 Building application partials

Returns a stream of HTML files validated with htmlhint.

pipes.validatedPartials = function() {  
    return gulp.src(paths.partials)
        .pipe(plugins.htmlhint({'doctype-first': false}))
        .pipe(plugins.htmlhint.reporter());
};

For development, this segment returns validated partials in the dev environment.

pipes.builtPartialsDev = function() {  
    return pipes.validatedPartials()
        .pipe(gulp.dest(paths.distDev));
};

For production, we use ngHtml2js, which converts all partials to JavaScript and preloads them into the Angular template cache. This segment returns a stream of one JavaScript file that will execute the preloading. This stream is merged into pipes.builtAppScriptsProd in section 3.3. Note the moduleName value should be the name of the Angular app which uses the partials.

pipes.scriptedPartials = function() {  
    return pipes.validatedPartials()
        .pipe(plugins.htmlhint.failReporter())
        .pipe(plugins.htmlmin({collapseWhitespace: true, removeComments: true}))
        .pipe(plugins.ngHtml2js({
            moduleName: "healthyGulpAngularApp"
        }));
};

3.7 Building application styles

This project uses Zurb Foundation, which is based on SASS. For development, this segment returns a stream of CSS files that have been compiled from SASS. Because the app SASS references the Foundation SASS, Foundation itself is compiled and included here. All directory structures are preserved in the dev environment.

pipes.builtStylesDev = function() {  
    return gulp.src(paths.styles)
        .pipe(plugins.sass())
        .pipe(gulp.dest(paths.distDev));
};

For production, this segment returns a stream of a single, minified CSS file, including a sourcemap, in the production environment.

pipes.builtStylesProd = function() {  
    return gulp.src(paths.styles)
        .pipe(plugins.sourcemaps.init())
            .pipe(plugins.sass())
            .pipe(plugins.minifyCss())
        .pipe(plugins.sourcemaps.write())
        .pipe(pipes.minifiedFileName())
        .pipe(gulp.dest(paths.distProd));
};

3.8 Building the index

Because the index (shell page) pulls together many different streams, we have some special, more complicated pipe segments for it. This segment returns a stream of index.html, validated with htmlhint.

pipes.validatedIndex = function() {  
    return gulp.src(paths.index)
        .pipe(plugins.htmlhint())
        .pipe(plugins.htmlhint.reporter());
};

This stream outputs an index.html in the dev environment which references all the files built for development. Notice that there are three pipe segments that feed into the index stream. The gulp-inject plugin is used to write references into the index file in the places denoted.

pipes.builtIndexDev = function() {

    var orderedVendorScripts = pipes.builtVendorScriptsDev()
        .pipe(pipes.orderVendorScripts());

    var orderedAppScripts = pipes.builtAppScriptsDev()
        .pipe(pipes.orderAppScripts());

    var appStyles = pipes.builtStylesDev();

    return pipes.validatedIndex()
        .pipe(gulp.dest(paths.distDev)) // write first to get relative path for inject
        .pipe(plugins.inject(orderedVendorScripts, {relative: true, name: 'bower'}))
        .pipe(plugins.inject(orderedAppScripts, {relative: true}))
        .pipe(plugins.inject(appStyles, {relative: true}))
        .pipe(gulp.dest(paths.distDev));
};

The production stream is similar, except we use production versions of the built files. The index is also minified post-injection.

pipes.builtIndexProd = function() {

    var vendorScripts = pipes.builtVendorScriptsProd();
    var appScripts = pipes.builtAppScriptsProd();
    var appStyles = pipes.builtStylesProd();

    return pipes.validatedIndex()
        .pipe(gulp.dest(paths.distProd)) // write first to get relative path for inject
        .pipe(plugins.inject(vendorScripts, {relative: true, name: 'bower'}))
        .pipe(plugins.inject(appScripts, {relative: true}))
        .pipe(plugins.inject(appStyles, {relative: true}))
        .pipe(plugins.htmlmin({collapseWhitespace: true, removeComments: true}))
        .pipe(gulp.dest(paths.distProd));
};

3.9 Build everything

These segments output the entire client-side application stream for dev and prod, respectively. For development, we have to merge the index and partials streams because there's no direct reference to the partial files in index.html.

pipes.builtAppDev = function() {  
    return es.merge(pipes.builtIndexDev(), pipes.builtPartialsDev());
};

For production, we simply forward the stream from pipes.builtIndexProd because the partials are included in the app scripts.

pipes.builtAppProd = function() {  
    return pipes.builtIndexProd();
};

4. Gulp tasks

The bulk of the build-ey work was done with streams and pipe segments, so most Gulp tasks end up simply tapping existing streams and wiring them to the command line. A nice side-effect of this is that we don't have specify inter-task dependencies so much.

This task removes the development environment. Because of the asynchronous nature of the del function, this task returns a promise, so that any task that needs a clean environment doesn't start early. There's a similar one for prod:

// removes all compiled dev files
gulp.task('clean-dev', function() {  
    var deferred = Q.defer();
    del(paths.distDev, function() {
        deferred.resolve();
    });
    return deferred.promise;
});

For brevity, most of the tasks have been ommitted, but here's a small sampling:

// checks html source files for syntax errors
gulp.task('validate-partials', pipes.validatedPartials);  
// builds a complete prod environment
gulp.task('build-app-prod', pipes.builtAppProd);  
// cleans and builds a complete prod environment
gulp.task('clean-build-app-prod', ['clean-prod'], pipes.builtAppProd);  
// default task builds for prod
gulp.task('default', ['clean-build-app-prod']);  

The following task will clean the development environment, build the complete app for dev, start the dev server, and watch for any and all changes. nodemon will watch and reload the dev server. And with gulp-livereload, the actual web-page is refreshed automatically whenever a change completes:

// clean, build, and watch live changes to the dev environment
gulp.task('watch-dev', ['clean-build-app-dev'], function() {

    // start nodemon to auto-reload the dev server
    plugins.nodemon({ script: 'server.js', ext: 'js', watch: ['devServer/'], env: {NODE_ENV : 'development'} })
        .on('change', ['jshint-devserver'])
        .on('restart', function () {
            console.log('[nodemon] restarted dev server');
        });

    // start live-reload server
    plugins.livereload.listen({ start: true });

    // watch index
    gulp.watch(paths.index, function() {
        return pipes.builtIndexDev()
            .pipe(plugins.livereload());
    });

    // watch app scripts
    gulp.watch(paths.scripts, function() {
        return pipes.builtAppScriptsDev()
            .pipe(plugins.livereload());
    });

    // watch html partials
    gulp.watch(paths.partials, function() {
        return pipes.builtPartialsDev()
            .pipe(plugins.livereload());
    });

    // watch styles
    gulp.watch(paths.styles, function() {
        return pipes.builtStylesDev()
            .pipe(plugins.livereload());
    });

});

There's another version of watch for production.

Conclusion

First, some possible improvements:

  • The Foundation SASS files are recompiled by gulp-sass every time the application styles are edited. This doesn't take a long time. It'd be nice to eliminate this, and other similar redundancies if possible.
  • If an application source file is removed or renamed, it stays in the dev or prod environment until the next clean happens. Maybe there's a way to watch for this and re-build as necessary.
  • I haven't worked in unit or end-to-end testing yet. That's obviously a critical part of a healthy build scheme, and my next task. I'll update the GitHub project once this is complete.
  • Many of the Gulp pipe segments could be made more pluggable by removing the stream sources into their own segments. Then most of the pipe segments would essentially be first-party plugins that bundle a series of third-party plugins, to be reused between projects.

That's it!

I've found that it was much easier—and even a pleasure—to work on this Gulpfile after I understood Gulp's raison d'etre. Feel free to track the GitHub project to stay updated as improvements are made.

Caleb Sotelo
AUTHOR

Caleb Sotelo

I'm a Software Engineer and Director of OpenX Labs. I try to write about software in a way that helps people truly understand it. Follow me @calebds.

View Comments