Bending Webpack Into Rails Engines

Why Rails Engines ?

First, let me define my problem. I want to write a rails engine for my customer, this engine is to provide several rails type features such as helpers, components etc… But, it also needs to package its assets in a modern webpack type way.

However, unlike sprockets, webpacker knows nothing about rails engines so your application assets live in app/javascripts and thats it. So, how do I write my engine in such a way that the application can easily use its assets ?

The 2 options I thought of are :-

1. Have the engine’s source code as both a rubygem and a node module so it can be included into the application as a gem for the rails features and as a node module for the assets.
2. Configure the engine to have webpack installed and produce its own packs that can be used by the application.

I liked option 1 for a while but when I started to use it, I was hitting issues when requiring jsx files from the gem as this was not going through webpacker first.

So, this article is going to focus on option 2

Where To Start

My plan is to setup a rails engine with a dummy app for testing. I will configure this dummy app so it can easily be used in development mode as well as in the test environment. This gives me a ‘rails like environment’.

After that, I will follow the instructions in https://github.com/rails/webpacker/blob/master/docs/engines.md to install webpacker.

Creating The Engine

Looking at the help for the ‘rails plugin’ command, I have decided on the following command line to generate the engine.

rails plugin new my_fancy_engine --database=sqlite3 --skip-action-mailer --skip-action-mailbox --skip-action-text --skip-active-record --skip-active-storage --skip-action-cable --skip-sprockets --skip-spring --skip-turbolinks --skip-bootsnap --full

This produced a load of files with no errors – so looks good so far. Im now going to commit to my github repo so that I can see my steps again later.

Running A Development Server

Ok, lets give this a go – run a rails server like this

./bin/rails s

Boom – we get a massive error saying

The gemspec at /home/garytaylor/projects/my_fancy_engine/my_fancy_engine.gemspec is not valid. Please fix this gemspec. (Gem::InvalidSpecificationException

Ok, lets fix this – editing the gemspec file putting in some valid values and then try again

garytaylor2@garyslaptop:~/projects/my_fancy_engine$ ./bin/rails s
 => Booting WEBrick
 => Rails 6.0.2.2 application starting in development http://localhost:3000
 => Run rails server --help for more startup options
 [2020-04-23 07:06:33] INFO  WEBrick 1.4.2
 [2020-04-23 07:06:33] INFO  ruby 2.5.1 (2018-03-29) [x86_64-linux]
 [2020-04-23 07:06:33] INFO  WEBrick::HTTPServer#start: pid=7370 port=3000

Perfect, we now have a web server running. If I go to http://localhost:3000 – I get the normal welcome to rails page.

Installing Webpacker

Following the instructions in https://github.com/rails/webpacker/blob/master/docs/engines.md starting at step 2 (as we have already created the engine), I add 8 files as follows

config/webpacker.yml

# Note: You must restart bin/webpack-dev-server for changes to take effect

default: &default
  source_path: app/javascript
  source_entry_path: packs
  public_root_path: public
  public_output_path: packs
  cache_path: tmp/cache/webpacker
  check_yarn_integrity: false
  webpack_compile_output: true

  # Additional paths webpack should lookup modules
  # ['app/assets', 'engine/foo/app/assets']
  resolved_paths: []

  # Reload manifest.json on all requests so we reload latest compiled packs
  cache_manifest: false

  # Extract and emit a css file
  extract_css: false

  static_assets_extensions:
    - .jpg
    - .jpeg
    - .png
    - .gif
    - .tiff
    - .ico
    - .svg
    - .eot
    - .otf
    - .ttf
    - .woff
    - .woff2

  extensions:
    - .jsx
    - .mjs
    - .js
    - .sass
    - .scss
    - .css
    - .module.sass
    - .module.scss
    - .module.css
    - .png
    - .svg
    - .gif
    - .jpeg
    - .jpg

development:
  <<: *default
  compile: true

  # Verifies that correct packages and versions are installed by inspecting package.json, yarn.lock, and node_modules
  check_yarn_integrity: true

  # Reference: https://webpack.js.org/configuration/dev-server/
  dev_server:
    https: false
    host: localhost
    port: 3035
    public: localhost:3035
    hmr: false
    # Inline should be set to true if using HMR
    inline: true
    overlay: true
    compress: true
    disable_host_check: true
    use_local_ip: false
    quiet: false
    pretty: false
    headers:
      'Access-Control-Allow-Origin': '*'
    watch_options:
      ignored: '**/node_modules/**'


test:
  <<: *default
  compile: true

  # Compile test packs to a separate directory
  public_output_path: packs-test

production:
  <<: *default

  # Production depends on precompilation of packs prior to booting for performance.
  compile: false

  # Extract and emit a css file
  extract_css: true

  # Cache manifest.json for performance
  cache_manifest: true

One thing I noticed (literally as I pasted it into this blog) are these lines

# Additional paths webpack should lookup modules
  # ['app/assets', 'engine/foo/app/assets']
  resolved_paths: []

I wish I had noticed them before when I was experimenting with option 1 – it might have solved my problem. However, I much prefer the route I am following as it will give me a typical rails engine experience where I just add it to my gemfile and everything is available – I dont want to force the projects using this gem to have to modify the webpack config etc..
When you think about it further, taking this approach doesnt even force the application to use webpack – just anything that can require javascript modules really.

config/webpack/environment.js

const { environment } = require('@rails/webpacker')

module.exports = environment

config/webpack/development.js

process.env.NODE_ENV = process.env.NODE_ENV || 'development'

const environment = require('./environment')

module.exports = environment.toWebpackConfig()

config/webpack/production.js

process.env.NODE_ENV = process.env.NODE_ENV || 'production'

const environment = require('./environment')

module.exports = environment.toWebpackConfig()

config/webpack/test.js

process.env.NODE_ENV = process.env.NODE_ENV || 'development'

const environment = require('./environment')

module.exports = environment.toWebpackConfig()

bin/webpack

#!/usr/bin/env ruby

ENV["RAILS_ENV"] ||= ENV["RACK_ENV"] || "development"
ENV["NODE_ENV"]  ||= "development"

require "pathname"
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile",
  Pathname.new(__FILE__).realpath)

require "bundler/setup"

require "webpacker"
require "webpacker/webpack_runner"

APP_ROOT = File.expand_path("..", __dir__)
Dir.chdir(APP_ROOT) do
  Webpacker::WebpackRunner.run(ARGV)
end

bin/webpack-dev-server

#!/usr/bin/env ruby

ENV["RAILS_ENV"] ||= ENV["RACK_ENV"] || "development"
ENV["NODE_ENV"]  ||= "development"

require "pathname"
ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../../Gemfile",
  Pathname.new(__FILE__).realpath)

require "bundler/setup"

require "webpacker"
require "webpacker/dev_server_runner"

APP_ROOT = File.expand_path("..", __dir__)
Dir.chdir(APP_ROOT) do
  Webpacker::DevServerRunner.run(ARGV)
end

package.json

{
  "name": "my_fancy_engine",
  "private": true,
  "dependencies": {
    "@babel/preset-react": "^7.9.4",
    "@rails/actioncable": "^6.0.0",
    "@rails/activestorage": "^6.0.0",
    "@rails/ujs": "^6.0.0",
    "@rails/webpacker": "4.2.2",
    "babel-plugin-transform-react-remove-prop-types": "^0.4.24",
    "prop-types": "^15.7.2",
    "react": "^16.13.1",
    "react-dom": "^16.13.1",
    "turbolinks": "^5.2.0"
  },
  "version": "0.1.0",
  "devDependencies": {
    "webpack-dev-server": "^3.10.3"
  }
}

Now make sure the bins are executable – Im doing this in linux – the same command also applies for mac OSX but I havent touched windows for 10 years so I cannot remember the command for windows.

chmod +x bin/webpack
chmod +x bin/webpack-dev-server

I am probably not the most up to date with front end tools as the majority of my work for the last 2 years has been backend work – but I seem to find that anyone using react (which I will be doing) uses ‘yarn’ instead of ‘npm’ – I dont pretend to know why, but I am going to use yarn to install everything now.

yarn install

and I get back

success Saved lockfile.

Done in 9.86s.

Cool – everything installed – as I have completed a logical step I will commit to git repo after adding node_modules to my .gitignore file

Now onto steps 3 to 6 in https://github.com/rails/webpacker/blob/master/docs/engines.md – not forgetting to replace MyEngine with the name of your engine – and in the rake task, ‘my_engine’ needs replacing with the underscored name of your engine.

Step 7 gives us some decisions – it talks about putting the engine’s assets into the public dir of the main application OR using a separate middleware.

I have chosen not to use a separate middleware as I would think this would mean that the packs that I produce in the gem cannot be bundled together with the application’s packs – meaning more http requests for the browser. I would much prefer that the application require’s the module inside one of its own packs and webpacker will automatically pack the engine’s packs into the applications. We will see how that works out, I might be re visiting this bit.

At the same time putting the engine’s assets into the public dir of the main application means knowing the application’s public dir path relative to the engine’s gem – which of course when we are done will be where the rest of the gems are kept on your system – it wont have a chance of knowing where the application is – and this is all just yaml so we cant put any ruby code in there.

So, where do we go from here.

Well, when the engine is finished, I will setup a build pipeline to build all of the packs and publish them to npm meaning the application can do something like

require("my_fancy_engine")

after adding my_fancy_engine to the package.json of course

But, this is going away from the simple ‘add the gem and all done’ approach that we expect from engines.

But, saying that – it must be the best way as this is how rails does it with the actioncable stuff for example. You dont see their assets coming straight from the gem, I guess the reason is we are trying to use decent well established front end tools rather than defining a ‘rails way’ of doing things like we had with sprockets.

So, thats it – decision made – I will publish to npm.

But, what about development ? we dont want to be having to publish to npm every time we make a change to the gem do we ?

This is where npm link comes in, we can tell npm to use a local version of the module whilst developing. We wont go into that here, but that will be our strategy hopefully.

The next worry is css and images – how do they get handled ? After doing a bit of research, I believe that when webpacker sees something like this

import "./styles.css";

It works from within the main app, but if we were to change this to

import "my_fancy_engine/style.css";

Then this should work

Also, images seem to be supported – all of this theory needs trying though – in part 2.