ruby - issue deploying rails 5 application to AWS using Elastic Beanstalk due to rb-readline

Question

I am trying to deploy my code to AWS using elastic beanstalk but keep running into errors when deploying.

I added a gem 'rb-readline' and when I check the logs I see that there is an error with this gem in deployment. I can't remember what error adding the gem 'rb-readline' fixed but I don't know if removing the gem would fix this problem. Here is the link to the gem: https://github.com/ConnorAtherton/rb-readline

here is the error log from the deployment.

Application update failed at 2017-10-24T19:27:36Z with exit status 1 and error: Hook /opt/elasticbeanstalk/hooks/appdeploy/pre/11_asset_compilation.sh failed.

++ /opt/elasticbeanstalk/bin/get-config container -k script_dir
+ EB_SCRIPT_DIR=/opt/elasticbeanstalk/support/scripts
++ /opt/elasticbeanstalk/bin/get-config container -k app_staging_dir
+ EB_APP_STAGING_DIR=/var/app/ondeck
++ /opt/elasticbeanstalk/bin/get-config container -k app_user
+ EB_APP_USER=webapp
++ /opt/elasticbeanstalk/bin/get-config container -k support_dir
+ EB_SUPPORT_DIR=/opt/elasticbeanstalk/support
+ . /opt/elasticbeanstalk/support/envvars
++ export RACK_ENV=production
++ RACK_ENV=production
++ export RAILS_SKIP_ASSET_COMPILATION=false
++ RAILS_SKIP_ASSET_COMPILATION=false
++ export RAILS_SKIP_MIGRATIONS=false
++ RAILS_SKIP_MIGRATIONS=false
++ export BUNDLE_WITHOUT=test:development
++ BUNDLE_WITHOUT=test:development
++ export RDS_PORT=5432
++ RDS_PORT=5432
++ export RDS_HOSTNAME=aa16lalntiz6vr0.cghvpjf0tzo5.us-east-1.rds.amazonaws.com
++ RDS_HOSTNAME=aa16lalntiz6vr0.cghvpjf0tzo5.us-east-1.rds.amazonaws.com
++ export RDS_USERNAME=theasteve
++ RDS_USERNAME=theasteve
++ export RDS_DB_NAME=ebdb
++ RDS_DB_NAME=ebdb
++ export RDS_PASSWORD=Colombia1
++ RDS_PASSWORD=Colombia1
+ RAKE_TASK=assets:precompile
+ . /opt/elasticbeanstalk/support/scripts/use-app-ruby.sh
++ . /usr/local/share/chruby/chruby.sh
+++ CHRUBY_VERSION=0.3.9
+++ RUBIES=()
+++ for dir in '"$PREFIX/opt/rubies"' '"$HOME/.rubies"'
+++ [[ -d /opt/rubies ]]
++++ ls -A /opt/rubies
+++ [[ -n ruby-1.9.3-p551
ruby-2.0.0-p648
ruby-2.1.10
ruby-2.2.8
ruby-2.3.5
ruby-2.4.2
ruby-current ]]
+++ RUBIES+=("$dir"/*)
+++ for dir in '"$PREFIX/opt/rubies"' '"$HOME/.rubies"'
+++ [[ -d /.rubies ]]
+++ unset dir
+++ cat /etc/elasticbeanstalk/.ruby_version
++ chruby 2.3.5
++ case "$1" in
++ local dir match
++ for dir in '"${RUBIES[@]}"'
++ dir=/opt/rubies/ruby-1.9.3-p551
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[@]}"'
++ dir=/opt/rubies/ruby-2.0.0-p648
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[@]}"'
++ dir=/opt/rubies/ruby-2.1.10
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[@]}"'
++ dir=/opt/rubies/ruby-2.2.8
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[@]}"'
++ dir=/opt/rubies/ruby-2.3.5
++ case "${dir##*/}" in
++ match=/opt/rubies/ruby-2.3.5
++ for dir in '"${RUBIES[@]}"'
++ dir=/opt/rubies/ruby-2.4.2
++ case "${dir##*/}" in
++ for dir in '"${RUBIES[@]}"'
++ dir=/opt/rubies/ruby-current
++ case "${dir##*/}" in
++ [[ -z /opt/rubies/ruby-2.3.5 ]]
++ shift
++ chruby_use /opt/rubies/ruby-2.3.5 ''
++ [[ ! -x /opt/rubies/ruby-2.3.5/bin/ruby ]]
++ [[ -n '' ]]
++ export RUBY_ROOT=/opt/rubies/ruby-2.3.5
++ RUBY_ROOT=/opt/rubies/ruby-2.3.5
++ export RUBYOPT=
++ RUBYOPT=
++ export PATH=/opt/rubies/ruby-2.3.5/bin:/opt/elasticbeanstalk/lib/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
++ PATH=/opt/rubies/ruby-2.3.5/bin:/opt/elasticbeanstalk/lib/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin
+++ /opt/rubies/ruby-2.3.5/bin/ruby -
++ eval 'export RUBY_ENGINE=ruby;
export RUBY_VERSION=2.3.5;
export GEM_ROOT="/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0";'
+++ export RUBY_ENGINE=ruby
+++ RUBY_ENGINE=ruby
+++ export RUBY_VERSION=2.3.5
+++ RUBY_VERSION=2.3.5
+++ export GEM_ROOT=/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0
+++ GEM_ROOT=/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0
++ (( 0 != 0 ))
+ cd /var/app/ondeck
+ su -s /bin/bash -c 'bundle exec /opt/elasticbeanstalk/support/scripts/check-for-rake-task.rb assets:precompile' webapp
`/home/webapp` is not a directory.
Bundler will use `/tmp/bundler/home/webapp' as your home directory temporarily.
+ '[' false == true ']'
+ su -s /bin/bash -c 'bundle exec rake assets:precompile' webapp
`/home/webapp` is not a directory.
Bundler will use `/tmp/bundler/home/webapp' as your home directory temporarily.
rake aborted!
Bundler::GemRequireError: There was an error while trying to load the gem 'rb-readline'.
Gem Load Error is: HOME environment variable (or HOMEDRIVE and HOMEPATH) must be set and point to a directory
Backtrace for gem load error is:
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:1097:in `<module:RbReadline>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.4/lib/rbreadline.rb:17:in `<top (required)>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.4/lib/readline.rb:10:in `require'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.4/lib/readline.rb:10:in `<module:Readline>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.4/lib/readline.rb:8:in `<top (required)>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.4/lib/rb-readline.rb:16:in `require'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rb-readline-0.5.4/lib/rb-readline.rb:16:in `<top (required)>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/runtime.rb:82:in `require'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/runtime.rb:82:in `block (2 levels) in require'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/runtime.rb:77:in `each'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/runtime.rb:77:in `block in require'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/runtime.rb:66:in `each'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/runtime.rb:66:in `require'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler.rb:108:in `require'
/var/app/ondeck/config/application.rb:17:in `<top (required)>'
/var/app/ondeck/Rakefile:4:in `require_relative'
/var/app/ondeck/Rakefile:4:in `<top (required)>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/rake_module.rb:28:in `load_rakefile'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/application.rb:687:in `raw_load_rakefile'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/application.rb:96:in `block in load_rakefile'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/application.rb:95:in `load_rakefile'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/application.rb:79:in `block in run'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rubies/ruby-2.3.5/bin/rake:23:in `load'
/opt/rubies/ruby-2.3.5/bin/rake:23:in `<top (required)>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/cli/exec.rb:74:in `load'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/cli/exec.rb:27:in `run'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/cli.rb:360:in `exec'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor.rb:369:in `dispatch'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/cli.rb:20:in `dispatch'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/vendor/thor/lib/thor/base.rb:444:in `start'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/cli.rb:10:in `start'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/exe/bundle:35:in `block in <top (required)>'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/lib/bundler/friendly_errors.rb:121:in `with_friendly_errors'
/opt/rubies/ruby-2.3.5/lib/ruby/gems/2.3.0/gems/bundler-1.15.0/exe/bundle:27:in `<top (required)>'
/opt/rubies/ruby-2.3.5/bin/bundle:23:in `load'
/opt/rubies/ruby-2.3.5/bin/bundle:23:in `<main>'
Bundler Error Backtrace:
/var/app/ondeck/config/application.rb:17:in `<top (required)>'
/var/app/ondeck/Rakefile:4:in `require_relative'
/var/app/ondeck/Rakefile:4:in `<top (required)>'
/opt/rubies/ruby-2.3.5/bin/bundle:23:in `load'
/opt/rubies/ruby-2.3.5/bin/bundle:23:in `<main>'
HOME environment variable (or HOMEDRIVE and HOMEPATH) must be set and point to a directory
/var/app/ondeck/config/application.rb:17:in `<top (required)>'
/var/app/ondeck/Rakefile:4:in `require_relative'
/var/app/ondeck/Rakefile:4:in `<top (required)>'
/opt/rubies/ruby-2.3.5/bin/bundle:23:in `load'
/opt/rubies/ruby-2.3.5/bin/bundle:23:in `<main>'
(See full trace by running task with --trace).
Incorrect application version "app-cd09-171024_152653" (deployment 3). Expected version "app-cd09-171024_150253" (deployment 2).
100 % of CPU is in use.

Answer 1

I had a similar error, but with dot-env rails gem. It looks like it might be same root cause.

It appeared to be because $HOME was not set by AWS during the deploy, and a call to File.expand_path dies when it's not set.

Possible fixes:

1. Remove rbreadline
2. Add a HOME environment variable into your EB configuration or eb setenv (and redeploy)
3. Add a config file that overwrites the existing deploy script & adds a $HOME variable before the bundle install command is run

I successfully tried the first and last option, and settled with using a forked version of the Gem that handled a missing $HOME correctly.

I suspect the 2nd version is the more correct way to do it, but to be honest it's unclear why Amazon didn't set a $HOME to begin with, and what it should actually be.

The app will exist in either /var/app/ondeck or /var/app/current so either one of those two folders, or /home/webapp (per the error) all might be a good value to start with.

This may not be 100% the root cause because I've had varying degrees of luck with replicating this locally with different versions of Ruby. However, the expand_path docs do say HOME must be set correctly so this may be the cause.

https://ruby-doc.org/core-2.4.2/File.html#method-c-expand_path

More detail on working solutions:

I found two solutions that worked (try one or the other)

At one point I did submit a PR with dotenv to fix this issue. https://github.com/bkeepers/dotenv/pull/314. As far as I recall, this fixes it, and doesn't break anything else. The issue was marked as wontfix by the automated bot, but has recently been re-opened by the new maintainers

I ultimately switched back to the ebextensions approach after it was clear my PR wouldn't be merged, but last I recall it was functioning as expected (and I think the better approach)

The other solution I used was to create a new config file that replaced the /opt/elasticbeanstalk/hooks/appdeploy/pre/10_bundle_install.sh file.

• Get the current version of that file from an existing application server
• Create an ebextension config that creates that file (it will overwrite the original) but include a line to set the $HOME variable before the bundle install command is run.
• Set the $HOME value to whatever it would be for the default user

The documentation for creating such a file using ebextensions are here:

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html#linux-files