Distribution packages, package management, and deploying applications

Just as the “big” Python, MicroPython supports creation of “third party” packages, distributing them, and easily installing them in each user’s environment. This chapter discusses how these actions are achieved. Some familiarity with Python packaging is recommended.

Overview

Steps below represent a high-level workflow when creating and consuming packages:

  1. Python modules and packages are turned into distribution package archives, and published at the Python Package Index (PyPI).
  2. upip package manager can be used to install a distribution package on a MicroPython port with networking capabilities (for example, on the Unix port).
  3. For ports without networking capabilities, an “installation image” can be prepared on the Unix port, and transferred to a device by suitable means.
  4. For low-memory ports, the installation image can be frozen as the bytecode into MicroPython executable, thus minimizing the memory storage overheads.

The sections below describe this process in details.

Distribution packages

Python modules and packages can be packaged into archives suitable for transfer between systems, storing at the well-known location (PyPI), and downloading on demand for deployment. These archives are known as distribution packages (to differentiate them from Python packages (means to organize Python source code)).

The MicroPython distribution package format is a well-known tar.gz format, with some adaptations however. The Gzip compressor, used as an external wrapper for TAR archives, by default uses 32KB dictionary size, which means that to uncompress a compressed stream, 32KB of contiguous memory needs to be allocated. This requirement may be not satisfiable on low-memory devices, which may have total memory available less than that amount, and even if not, a contiguous block like that may be hard to allocate due to memory fragmentation. To accommodate these constraints, MicroPython distribution packages use Gzip compression with the dictionary size of 4K, which should be a suitable compromise with still achieving some compression while being able to uncompressed even by the smallest devices.

Besides the small compression dictionary size, MicroPython distribution packages also have other optimizations, like removing any files from the archive which aren’t used by the installation process. In particular, upip package manager doesn’t execute setup.py during installation (see below), and thus that file is not included in the archive.

At the same time, these optimizations make MicroPython distribution packages not compatible with CPython‘s package manager, pip. This isn’t considered a big problem, because:

  1. Packages can be installed with upip, and then can be used with CPython (if they are compatible with it).
  2. In the other direction, majority of CPython packages would be incompatible with MicroPython by various reasons, first of all, the reliance on features not implemented by MicroPython.

Summing up, the MicroPython distribution package archives are highly optimized for MicroPython’s target environments, which are highly resource constrained devices.

upip package manager

MicroPython distribution packages are intended to be installed using the upip package manager. upip is a Python application which is usually distributed (as frozen bytecode) with network-enabled MicroPython ports. At the very least, upip is available in the MicroPython Unix port.

On any MicroPython port providing upip, it can be accessed as following:

import upip
upip.help()
upip.install(package_or_package_list, [path])

Where package_or_package_list is the name of a distribution package to install, or a list of such names to install multiple packages. Optional path parameter specifies filesystem location to install under and defaults to the standard library location (see below).

An example of installing a specific package and then using it:

>>> import upip
>>> upip.install("micropython-pystone_lowmem")
[...]
>>> import pystone_lowmem
>>> pystone_lowmem.main()

Note that the name of Python package and the name of distribution package for it in general don’t have to match, and oftentimes they don’t. This is because PyPI provides a central package repository for all different Python implementations and versions, and thus distribution package names may need to be namespaced for a particular implementation. For example, all packages from micropython-lib follow this naming convention: for a Python module or package named foo, the distribution package name is micropython-foo.

For the ports which run MicroPython executable from the OS command prompts (like the Unix port), upip can be (and indeed, usually is) run from the command line instead of MicroPython’s own REPL. The commands which corresponds to the example above are:

micropython -m upip -h
micropython -m upip install [-p <path>] <packages>...
micropython -m upip install micropython-pystone_lowmem

[TODO: Describe installation path.]

Cross-installing packages

For MicroPython ports without native networking capabilities, the recommend process is “cross-installing” them into a “directory image” using the MicroPython Unix port, and then transferring this image to a device by suitable means.

Installing to a directory image involves using -p switch to upip:

micropython -m upip install -p install_dir micropython-pystone_lowmem

After this command, the package content (and contents of every dependency packages) will be available in the install_dir/ subdirectory. You would need to transfer contents of this directory (without the install_dir/ prefix) to the device, at the suitable location, where it can be found by the Python import statement (see discussion of the upip installation path above).

Cross-installing packages with freezing

For the low-memory MicroPython ports, the process described in the previous section does not provide the most efficient resource usage,because the packages are installed in the source form, so need to be compiled to the bytecome on each import. This compilation requires RAM, and the resulting bytecode is also stored in RAM, reducing its amount available for storing application data. Moreover, the process above requires presence of the filesystem on a device, and the most resource-constrained devices may not even have it.

The bytecode freezing is a process which resolves all the issues mentioned above:

  • The source code is pre-compiled into bytecode and store as such.
  • The bytecode is stored in ROM, not RAM.
  • Filesystem is not required for frozen packages.

Using frozen bytecode requires building the executable (firmware) for a given MicroPython port from the C source code. Consequently, the process is:

  1. Follow the instructions for a particular port on setting up a toolchain and building the port. For example, for ESP8266 port, study instructions in ports/esp8266/README.md and follow them. Make sure you can build the port and deploy the resulting executable/firmware successfully before proceeding to the next steps.
  2. Build MicroPython Unix port and make sure it is in your PATH and you can execute micropython.
  3. Change to port’s directory (e.g. ports/esp8266/ for ESP8266).
  4. Run make clean-frozen. This step cleans up any previous modules which were installed for freezing (consequently, you need to skip this step to add additional modules, instead of starting from scratch).
  5. Run micropython -m upip install -p modules <packages>... to install packages you want to freeze.
  6. Run make clean.
  7. Run make.

After this, you should have the executable/firmware with modules as the bytecode inside, which you can deploy the usual way.

Few notes:

  1. Step 5 in the sequence above assumes that the distribution package is available from PyPI. If that is not the case, you would need to copy Python source files manually to modules/ subdirectory of the port port directory. (Note that upip does not support installing from e.g. version control repositories).
  2. The firmware for baremetal devices usually has size restrictions, so adding too many frozen modules may overflow it. Usually, you would get a linking error if this happens. However, in some cases, an image may be produced, which is not runnable on a device. Such cases are in general bugs, and should be reported and further investigated. If you face such a situation, as an initial step, you may want to decrease the amount of frozen modules included.

Creating distribution packages

Distribution packages for MicroPython are created in the same manner as for CPython or any other Python implementation, see references at the end of chapter. Setuptools (instead of distutils) should be used, because distutils do not support dependencies and other features. “Source distribution” (sdist) format is used for packaging. The post-processing discussed above, (and pre-processing discussed in the following section) is achieved by using custom sdist command for setuptools. Thus, packaging steps remain the same as for the standard setuptools, the user just needs to override sdist command implementation by passing the appropriate argument to setup() call:

from setuptools import setup
import sdist_upip

setup(
    ...,
    cmdclass={'sdist': sdist_upip.sdist}
)

The sdist_upip.py module as referenced above can be found in micropython-lib: https://github.com/micropython/micropython-lib/blob/master/sdist_upip.py

Application resources

A complete application, besides the source code, oftentimes also consists of data files, e.g. web page templates, game images, etc. It’s clear how to deal with those when application is installed manually - you just put those data files in the filesystem at some location and use the normal file access functions.

The situation is different when deploying applications from packages - this is more advanced, streamlined and flexible way, but also requires more advanced approach to accessing data files. This approach is treating the data files as “resources”, and abstracting away access to them.

Python supports resource access using its “setuptools” library, using pkg_resources module. MicroPython, following its usual approach, implements subset of the functionality of that module, specifically pkg_resources.resource_stream(package, resource) function. The idea is that an application calls this function, passing a resource identifier, which is a relative path to data file within the specified package (usually top-level application package). It returns a stream object which can be used to access resource contents. Thus, the resource_stream() emulates interface of the standard open() function.

Implementation-wise, resource_stream() uses file operations underlyingly, if distribution package is install in the filesystem. However, it also supports functioning without the underlying filesystem, e.g. if the package is frozen as the bytecode. This however requires an extra intermediate step when packaging application - creation of “Python resource module”.

The idea of this module is to convert binary data to a Python bytes object, and put it into the dictionary, indexed by the resource name. This conversion is done automatically using overridden sdist command described in the previous section.

Let’s trace the complete process using the following example. Suppose your application has the following structure:

my_app/
    __main__.py
    utils.py
    data/
        page.html
        image.png

__main__.py and utils.py should access resources using the following calls:

import pkg_resources

pkg_resources.resource_stream(__name__, "data/page.html")
pkg_resources.resource_stream(__name__, "data/image.png")

You can develop and debug using the MicroPython Unix port as usual. When time comes to make a distribution package out of it, just use overridden “sdist” command from sdist_upip.py module as described in the previous section.

This will create a Python resource module named R.py, based on the files declared in MANIFEST or MANIFEST.in files (any non-.py file will be considered a resource and added to R.py) - before proceeding with the normal packaging steps.

Prepared like this, your application will work both when deployed to filesystem and as frozen bytecode.

If you would like to debug R.py creation, you can run:

python3 setup.py sdist --manifest-only

Alternatively, you can use tools/mpy_bin2res.py script from the MicroPython distribution, in which can you will need to pass paths to all resource files:

mpy_bin2res.py data/page.html data/image.png

References