bash - Install everything from a `conda list` output

Question

I have a file, conda_env.txt, resulting from saving the stdout of conda list. It has the following structure.

conda_env.txt

# packages in environment at /scratch/hello/anaconda3:
#
# Name                    Version                   Build  Channel
_ipyw_jlab_nb_ext_conf    0.1.0                    py37_0
absl-py                   0.7.1                    pypi_0    pypi
alabaster                 0.7.12                   py37_0
anaconda-client           1.7.2                    py37_0
anaconda-navigator        1.9.7                    py37_0
anaconda-project          0.8.2                    py37_0
annoy                     1.14.0                   pypi_0    pypi

I would like to conda install all the packages listed in this file, with version specified.

My idea is to for loop over all lines (starting from the 4th line), get the value from the first and second field, and save them into PACKAGE and VERSION, and install the package by calling conda install $PACKAGE=$VERSION (for example, conda install pandas=0.20.3

I would like to write a shell script to do this. I have written a demo file just to read the ith line and it's second field:

#!/bin/bash
LINES=$(wc -l < conda_env.txt)
for i in {4...10}
do
awk -F '|' 'NR==${i}{print $2}' conda_env.txt
done

and it gives me error: awk: bailing out at source line 1.

Any idea what is going wrong here and how to do it correctly?

Answer 1

Convert to YAML

I wrote an AWK script to convert the output of conda list to YAML that is compatible with conda env create -f env.yaml. This should be strongly preferred over a sequence of conda install calls, which results in running a solve for each individual package.

AWK Script

list_to_yaml.awk

#!/usr/bin/env awk -f
#' Author: Mervin Fansler
#' GitHub: @mfansler
#' License: MIT

#' Usage
#' $ conda list | awk -f list_to_yaml.awk

{
  # skip header
  if ($1 ~ /^#/) { next }

  if ($4 ~ /pypi/) {  # pypi packages
    pip=1;
    pypi[i++]="    - "$1"=="$2" ";
  } else {  # conda packages
    if ($1 ~ /pip/) {
      pip=1;
    } else {
      conda[j++]="  - "$1"="$2" ";
    }
    
    # include channels
    if (!seen[$4]) {
      if (length($4) == 0) {
        channels[k++]="  - defaults ";
      } else {
        channels[k++]="  - "$4" ";
      }
      ++seen[$4];
    }
  }
}
END {
  # emit channel info
  print "channels: ";
  for (k in channels) print channels[k];
  
  # emit conda pkg info                                                                                                               
  print "dependencies: ";
  for (j in conda) print conda[j];

  # emit PyPI pkg info
  if (pip) print "  - pip ";
  if (length(pypi) > 0) {
    print "  - pip: ";
    for (i in pypi) print pypi[i];
  }
}

Usage

Suppose we have an environment foo that someone dumped a conda list for (foo_list.txt), we can create a corresponding YAML file (foo.yaml), and then use that to generate a new environment, bar.

# dump a `conda list` for environment `foo`
conda list -n foo > foo_list.txt

# convert to YAML
awk -f list_to_yaml.awk foo_list.txt > foo.yaml

# edit the foo.yaml channels section manually!
# usually should be `conda-forge` at the top, `defaults` at the bottom
# emacs foo.yaml

# create the new environment
conda env create -n bar -f foo.yaml

Alternatively, if you just want to test it, piping also works:

conda list -n foo | awk -f list_to_yaml.awk

Advantages

• automatically handles packages from PyPI
• handles channels (sort of; the order may not be optimal)
• creates a single file for installation
• doesn't repeatedly call conda install, which leads to a ton of unnecessary solves