.. index:: westurner/tools

.. index:: Tools

.. _tools:

================

Tools

================

.. index:: Packages

.. _packages:

Packages

==========

| Wikipedia: `<https://en.wikipedia.org/wiki/Package_(package_management_system)>`__

A software package is an archive of files

with a manifest that lists the files included.

Often, the manifest contains file checksums

and a *signature*.

Many packaging tools make a distinction between source

and/or binary packages.

Some packaging tools provide configuration options for:

* Scripts to run when packaging

* Scripts to run at install time

* Scripts to run at uninstal time

* Patches to apply to the "*vanilla*" source tree,

as might be obtained from a version control repository

There is a package maintainer whose responsibilities include:

* Testing new *upstream* releases

* *Vetting* changes from release to release

* *Repackaging* upstream releases

* *Signing* new package releases

*Packaging lag* refers to how long it takes a package maintainer

to repackage upstream releases for the target platform(s).

.. index:: Anaconda

.. _anaconda:

Anaconda

~~~~~~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Anaconda_(Python_distribution)>`__

| Homepage: https://store.continuum.io/cshop/anaconda/

Anaconda is a maintained distribution of :ref:`Conda`

packages for many languages; especially :ref:`Python`.

.. note:: `<https://en.wikipedia.org/wiki/Anaconda_(installer)>`__ (1999)

is the installer for :ref:`RPM`-based :ref:`Linux` distributions; which is

also written in :ref:`Python` (and :ref:`C`).

.. index:: APT

.. _apt:

APT

~~~~~~~~~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Advanced_Packaging_Tool>`_

| Homepage: https://alioth.debian.org/projects/apt

| Docs: https://wiki.debian.org/Apt

| Docs: https://www.debian.org/doc/manuals/debian-reference/ch02.en.html

| Docs: https://www.debian.org/doc/manuals/apt-howto/

| Docs: https://wiki.debian.org/SecureApt

APT ("Advanced Packaging Tool") is the core of :ref:`Debian`

package management.

* An APT package repository can be accessed from a local filesystem

or over a network protocol ("apt transports") like HTTP, HTTPS, RSYNC, FTP,

and BitTorrent (`debtorrent`).

An example of APT usage

(e.g. to maintain an updated :ref:`Ubuntu` :ref:`Linux` system):

.. code-block:: bash

apt-get update

apt-get upgrade

apt-get dist-upgrade

apt-cache show bash

apt-get install bash

apt-get --help

man apt-get

man sources.list

.. index:: AUR

.. _aur:

AUR

~~~~

| Wikipedia: https://en.wikipedia.org/wiki/Arch_Linux#Arch_User_Repository_.28AUR.29

AUR (:ref:`Arch` *User Repository*) contains :ref:`PKGBUILD`

packages which can be installed by :ref:`pacman`.

.. index:: Bower

.. _bower:

Bower

~~~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Bower_(software)>`__

| Homepage: https://www.bower.io/

Bower is "a package manager for the web" (:ref:`Javascript` packages)

built on :ref:`NPM`.

.. index:: BUILD

.. _build:

BUILD

~~~~~~

| Docs: https://pantsbuild.github.io/build_files.html

| Docs: https://pantsbuild.github.io/build_dictionary.html

| Docs: https://pantsbuild.github.io/options_reference.html

A ``BUILD`` file describes a :ref:`Pants Build` build.

.. index:: Cabal

.. _cabal:

Cabal

~~~~~~

| Homepage: https://www.haskell.org/cabal/

| Docs: https://www.haskell.org/cabal/users-guide/

| Docs: https://www.haskell.org/cabal/release/cabal-latest/doc/API/Cabal/

Cabal is a package manager for :ref:`Haskell` packages.

Hackage is the community Cabal package index: https://hackage.haskell.org/

.. index:: Conda Package

.. index:: Conda

.. _conda:

Conda

| PyPI: https://pypi.python.org/pypi/conda

Conda is a package build, environment, and distribution system

written in :ref:`Python`

to install packages written in any language.

* Conda was originally created for the Anaconda Python Distribution,

which installs packages written in :ref:`Python`,

R,

:ref:`Javascript`,

:ref:`Ruby`,

:ref:`C`,

:ref:`Fortran`

* Conda packages are basically tar archives with build, and optional

link/install and uninstall scripts.

* ``conda-build`` generates conda packages from conda recipes

with a ``meta.yaml``, a ``build.sh``, and/or a ``build.bat``.

* Conda recipes reference and build from

a source package URI

* ``conda skeleton``-generated recipes can be updated

with additional metadata, scripts, and source URIs

(as separate patches or consecutive branch commits

of e.g. a conda-recipes repository

in order to get a diff of the skeleton recipe and the current recipe).

* Conda (and :ref:`Anaconda`) packages are hosted by

`<https://binstar.org>`__,

which hosts free public and paid private Conda packages.

* Anaconda Server is an internal

"Private, Secure Package Repository"

that

"supports over 100 different repositories,

including PyPI, CRAN, conda, and the Anaconda repository."

To create a fresh conda env:

.. code:: bash

# Python 2.7

# Python 3.X

Work on a conda env:

.. code:: bash

source activate exmpl2

conda list

source deactivate

files which list conda and :ref:`pip` packages.

Work with conda envs and ``environment.yml`` files:

.. code:: bash

# Install conda-env globally (in the "root" conda environment)

conda install -n root conda-env

# Create a conda environment with ``conda-create`` and install conda-env

# Install some things with conda (and envs/science/bin/pip)

# https://github.com/westurner/notebooks/blob/gh-pages/install.sh

conda search pandas; conda info pandas

conda install blaze dask bokeh odo \

sqlalchemy hdf5 h5py \

scikit-learn statsmodels \

beautiful-soup lxml html5lib pandas qgrid \

ipython-notebook

pip install -e git+https://github.com/rdflib/rdflib@master#egg=rdflib

pip install arrow sarge structlog

# Export an environment.yml

#source deactivate

conda env export -n science | tee environment.yml

# Create an environment from an environment.yml

conda env create -n projectname -f ./environment.yml

To install a conda package from a custom channel:

- https://github.com/conda/conda-recipes/tree/master/cookiecutter

- https://binstar.org/pydanny/cookiecutter

.. code:: bash

conda install -c pydanny cookiecutter # OR pip install cookiecutter

The conda-forge custom channel packages are built with :ref:`continuous

integration` on multiple platforms:

* https://conda-forge.github.io/

Sources:

* https://github.com/conda

* https://github.com/conda/conda -- conda

* https://github.com/ContinuumIO/pycosat -- pycosat SAT solver

* https://github.com/conda/conda-env -- conda-env

(the ``conda env`` command)

* https://github.com/conda/conda-build -- conda-build

(the ``conda build`` command)

* https://github.com/conda/conda-recipes -- Community-maintained

conda recipes (which users may build and

:ref:`maintain <packages>` in https://binstar.org

package repositories)

See also: :ref:`Anaconda`, :ref:`conda-forge` (:ref:`conda-smithy`)

.. index:: conda-forge

.. _conda-forge:

conda-forge

~~~~~~~~~~~~~~~~~

| Homepage: https://conda-forge.github.io/

| Src: https://github.com/conda-forge

| Src: https://github.com/conda-forge/feedstocks

| Src: https://github.com/conda-forge/feedstocks/tree/master/feedstocks

| Src: https://github.com/conda-forge/staged-recipes

| DockerHub: https://hub.docker.com/r/condaforge/linux-anvil

| Docs: https://conda-forge.github.io/docs/

| Docs: https://conda-forge.github.io/docs/recipe.html

* https://conda-forge.github.io/#add_recipe

* A. fork: https://github.com/conda-forge/staged-recipes

* conda-forge.yml

* run_docker_build.sh

https://github.com/conda-forge/staged-recipes/blob/master/scripts/run_docker_build.sh

* bootstrap-obvious-ci-and-miniconda.py

https://github.com/conda-forge/conda-smithy/blob/master/bootstrap-obvious-ci-and-miniconda.py

.. code:: bash

# create a conda package recipe from a pypi package

cd $VIRTUAL_ENV/src

conda skeleton pypi jupyterthemes

ls -ld jupyterthemes/

edit jupyterthemes/meta.yaml

# - git repo tags || pypi releases

# create a conda-forge feedstock from a conda recipe

## https://github.com/conda-forge/conda-smithy#making-a-new-feedstock

cd $VIRTUAL_ENV/src

ls -ld jupyterthemes

conda-smithy init jupyterhemes

ls jupyterthemes-feedstock/

# build a conda-forge feedstock with docker

# FROM condaforge/linux-anvil

cat ./scripts/run_docker_build.sh

./scripts/run_docker_build.sh

./ci_support/run_docker_build.sh

.. index:: conda-smithy

.. _conda-smithy:

conda-smithy

~~~~~~~~~~~~~~

| Src: https://github.com/conda-forge/conda-smithy

| Docs: https://github.com/conda-forge/conda-smithy#making-a-new-feedstock

.. index:: DEB

.. _deb:

DEB

~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Deb_(file_format)>`__

DEB packages are built with :ref:`dpkg` and often hosted in an :ref:`APT`

package repository.

.. index:: Dpkg

.. _dpkg:

Dpkg

~~~~~~~~~~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Dpkg>`_

| Homepage: https://wiki.debian.org/Teams/Dpkg

| Docs: `<https://en.wikipedia.org/wiki/Debian_build_toolchain>`_

| Docs: https://www.debian.org/doc/manuals/debian-faq/ch-pkg_basics.en.html

| Docs: https://www.debian.org/doc/manuals/debian-faq/ch-pkgtools.en.html

| Docs:

Dpkg is a collection of tools for creating and working with

:ref:`DEB` packages.

.. index:: dnf

.. _dnf:

dnf

~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/DNF_(software)>`__

| Docs: https://github.com/rpm-software-management/dnf/wiki

| Docs: https://rpm-software-management.github.io/dnf-plugins-core/

dnf is a an open source package manager written in :ref:`Python`.

* dnf was introduced in :ref:`Fedora` 18.

* dnf is the default package manager in :ref:`Fedora` 22;

replacing :ref:`yum`.

* [ ] ``yum`` errors if TODO package is installed (* :ref:`salt`

provider)

* [ ] ``repoquery`` redirects with an error to ``dnf repoquery``

* See ``dnf help`` (and ``man dnf``)

* dnf integrates with the Anaconda system installer.

* :ref:`dnf` supports **Delta** RPM packages (DRPM),

which often significantly reduce the required amount of network

transfer required to regularly retrieve and upgrade to

the latest repository packages.

.. index:: ebuild

.. _ebuild:

ebuild

~~~~~~~~

| Wikipedia: https://en.wikipedia.org/wiki/Ebuild

| Homepage: https://wiki.gentoo.org/wiki/Ebuild

| Docs: https://devmanual.gentoo.org/quickstart/

| Docs: https://devmanual.gentoo.org/ebuild-writing/

| Docs: https://devmanual.gentoo.org/ebuild-writing/file-format/

| Docs: https://devmanual.gentoo.org/ebuild-writing/variables/

| Docs: https://devmanual.gentoo.org/ebuild-writing/use-conditional-code/

| Docs: https://wiki.gentoo.org/wiki/Submitting_ebuilds

ebuild is a software package definition format.

* ebuilds are like special :ref:`bash` scripts.

* ebuilds have ``USE`` flags for specifying build features.

* :ref:`Gentoo` is built from ebuild package definitions

* :ref:`Portage` packages are built from ebuilds.

* The :ref:`emerge` :ref:`Portage` command installs ebuilds.

.. index:: emerge

.. _emerge:

emerge

~~~~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Portage_(software)#Emerge>`__

| Homepage: https://wiki.gentoo.org/wiki/Portage#emerge

| Docs: https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification

| Docs: https://projects.gentoo.org/pms/6/pms.html

``emerge`` is the primary CLI tool used for installing

.. index:: fpm

.. _fpm:

fpm

~~~~~

| Wikipedia:

| Src: https://github.com/jordansissel/fpm

| Docs: https://github.com/jordansissel/fpm/wiki/

| Docs: https://github.com/jordansissel/fpm/wiki/PackageMakeInstall

fpm (*effing package management*) is a tool for building many types

of software packages from many other types of software packages

(e.g. :ref:`DEB`. :ref:`RPM`, :ref:`Python Packages`);

often more easily than working with the actual package manager.

* fpm package source types include: dir rpm gem python empty tar deb cpan npm osxpkg pear pkgin virtualenv zip.

* fpm target package types include: rpm deb solaris puppet dir osxpkg p5p puppet sh tar zip

.. index:: Brew

.. index:: Homebrew

.. _homebrew:

Homebrew

~~~~~~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Homebrew_(package_management_software)>`__

Homebrew is a package manager (``brew``) for :ref:`OSX`.

.. index:: NPM

.. index:: Node Package Manager

.. _npm:

NPM

~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Npm_(software)>`__

| Homepage: https://www.npmjs.org/

NPM is a :ref:`Javascript` package manager created for :ref:`Node.js`.

* an NPM package is defined by a ``package.json`` :ref:`JSON` file.

* NPM packages are installed with the ``npm`` CLI utility.

* :ref:`Bower` builds upon NPM.

.. index:: NuGet

.. _nuget:

NuGet

~~~~~~

| Wikipedia: https://en.wikipedia.org/wiki/NuGet

| Homepage: https://www.nuget.org/

NuGet is an open source package manager for :ref:`Windows`.

* Chocolatey maintains variously updated packages

for various windows programs:

https://chocolatey.org/

+ An example list of Chocolatey NuGet packages as a :ref:`PowerShell` script:

https://gist.github.com/westurner/10950476

.. index:: pacman

.. _pacman:

pacman

~~~~~~~~

| Wikipedia: https://en.wikipedia.org/wiki/Arch_Linux#Pacman

| Homepage: https://www.archlinux.org/pacman/

| Docs: https://wiki.archlinux.org/index.php/Pacman

Pacman is an open source package manager which installs

``.pkg.tar.xz`` files for :ref:`Arch` Linux.

.. index:: PEX

.. _pex:

PEX

~~~~~

PEX (*Python Executable*) is a :ref:`ZIP`-based software package archive

format with an executable header.

* :ref:`Pants` creates PEX packages.

.. index:: PKGBUILD

.. _pkgbuild:

PKGBUILD

~~~~~~~~~~

| Homepage: https://wiki.archlinux.org/index.php/PKGBUILD

| Docs: https://www.archlinux.org/pacman/PKGBUILD.5.html

| Docs: https://wiki.archlinux.org/index.php/Makepkg

| Docs: https://wiki.archlinux.org/index.php/Creating_packages

PKGBUILD is a shell script containing the build information

.. index:: Portage

.. _portage:

Portage

~~~~~~~~~

| Wikipedia: `<https://en.wikipedia.org/wiki/Portage_(software)>`__

| Docs: https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification

| Docs: https://projects.gentoo.org/pms/6/pms.html

Portage is a package management and repository system

written in :ref:`Python` initially just for :ref:`Gentoo` :ref:`Linux`.

.. index:: Ports

.. _ports:

Ports

~~~~~~~

| Wikipedia: https://en.wikipedia.org/wiki/Ports_collection

| Homepage: https://www.freebsd.org/ports/

A Ports collection contains *Sources* (e.g. archived releases and patch

sets)

and :ref:`Makefiles <make>`

designed to compile software :ref:`packages`

for particular :ref:`operating systems <operating system>`

distributions' kernel and standard libraries

usually for a particular platform.

.. index:: RPM

.. _rpm:

RPM

~~~~~

| Wikipedia: https://en.wikipedia.org/wiki/RPM_Package_Manager

RPM (*RPM Package Manager*, :ref:`RedHat` *Package Manager*)

is a :ref:`package` format and a set of commandline utilities

written in :ref:`C` and :ref:`Perl`.

* RPM packages can be installed with ``rpm``,

:ref:`yum`, :ref:`dnf`.

* RPM pacage can be built with tools like ``rpmbuild`` and ``fpm``

* Python packages can be built into RPM packages with

:ref:`setuptools' <setuptools>` ``bdist_rpm``, ``fpm``

* List contents of RPM packages (archives) with e.g. ``less`` and

``lesspipe``::

less ~/path/to/local.rpm # requires lesspipe to be configured

* RPM Packages are served by and retrieved from

repositories by tools like :ref:`yum` and :ref:`dnf`:

* :ref:`dnf` supports **Delta** RPM packages (DRPM),

which often significantly reduce the required amount of network

transfer required to regularly retrieve and upgrade to

the latest repository packages.

.. note:: There's not yet a :ref:`debtorrent` for

:ref:`RPM`, :ref:`YUM`, :ref:`DNF`.

.. index:: Egg

.. index:: Python Egg

.. index:: Python Packages