Distribution packages, package management, and deploying applications¶
Just as the “big” Python, MicroPython supports creation of “third party” packages, distributing them, and easily installing them in each user’s environment. This chapter discusses how these actions are achieved. Some familiarity with Python packaging is recommended.
Steps below represent a high-level workflow when creating and consuming packages:
- Python modules and packages are turned into distribution package archives, and published at the Python Package Index (PyPI).
upippackage manager can be used to install a distribution package on a
MicroPython portwith networking capabilities (for example, on the Unix port).
- For ports without networking capabilities, an “installation image” can be prepared on the Unix port, and transferred to a device by suitable means.
- For low-memory ports, the installation image can be frozen as the bytecode into MicroPython executable, thus minimizing the memory storage overheads.
The sections below describe this process in details.
Python modules and packages can be packaged into archives suitable for transfer between systems, storing at the well-known location (PyPI), and downloading on demand for deployment. These archives are known as distribution packages (to differentiate them from Python packages (means to organize Python source code)).
The MicroPython distribution package format is a well-known tar.gz format, with some adaptations however. The Gzip compressor, used as an external wrapper for TAR archives, by default uses 32KB dictionary size, which means that to uncompress a compressed stream, 32KB of contguous memory needs to be allocated. This requirement may be not satisfiable on low-memory devices, which may have total memory available less than that amount, and even if not, a contiguous block like that may be hard to allocate due to memory fragmentation. To accommodate these constraints, MicroPython distribution packages use Gzip compression with the dictionary size of 4K, which should be a suitable compromise with still achieving some compression while being able to uncompressed even by the smallest devices.
Besides the small compression dictionary size, MicroPython distribution
packages also have other optimizations, like removing any files from
the archive which aren’t used by the installation process. In particular,
upip package manager doesn’t execute
setup.py during installation
(see below), and thus that file is not included in the archive.
At the same time, these optimizations make MicroPython distribution
packages not compatible with
CPython’s package manager,
This isn’t considered a big problem, because:
- Packages can be installed with
upip, and then can be used with CPython (if they are compatible with it).
- In the other direction, majority of CPython packages would be incompatible with MicroPython by various reasons, first of all, the reliance on features not implemented by MicroPython.
Summing up, the MicroPython distribution package archives are highly optimized for MicroPython’s target environments, which are highly resource constrained devices.
upip package manager¶
MicroPython distribution packages are intended to be installed using
upip package manager.
upip is a Python application which is
usually distributed (as frozen bytecode) with network-enabled
MicroPython ports. At the very least,
upip is available in the
MicroPython Unix port.
Where package_or_package_list is the name of a distribution package to install, or a list of such names to install multiple packages. Optional path parameter specifies filesystem location to install under and defaults to the standard library location (see below).
An example of installing a specific package and then using it:
>>> import upip
>>> import pystone_lowmem
Note that the name of Python package and the name of distribution
package for it in general don’t have to match, and oftentimes they
don’t. This is because PyPI provides a central package repository
for all different Python implementations and versions, and thus
distribution package names may need to be namespaced for a particular
implementation. For example, all packages from
follow this naming convention: for a Python module or package named
foo, the distribution package name is
For the ports which run MicroPython executable from the OS command
prompts (like the Unix port),
upip can be (and indeed, usually is)
run from the command line instead of MicroPython’s own REPL. The
commands which corresponds to the example above are:
micropython -m upip -h
micropython -m upip install [-p <path>] <packages>...
micropython -m upip install micropython-pystone_lowmem
[TODO: Describe installation path.]
MicroPython ports without native networking
capabilities, the recommend process is “cross-installing” them into a
“directory image” using the
MicroPython Unix port, and then
transferring this image to a device by suitable means.
Installing to a directory image involves using
-p switch to
micropython -m upip install -p install_dir micropython-pystone_lowmem
After this command, the package content (and contents of every depenency
packages) will be available in the
install_dir/ subdirectory. You
would need to transfer contents of this directory (without the
install_dir/ prefix) to the device, at the suitable location, where
it can be found by the Python
import statement (see discussion of
upip installation path above).
Cross-installing packages with freezing¶
For the low-memory
MicroPython ports, the process
described in the previous section does not provide the most efficient
resource usage,because the packages are installed in the source form,
so need to be compiled to the bytecome on each import. This compilation
requires RAM, and the resulting bytecode is also stored in RAM, reducing
its amount available for storing application data. Moreover, the process
above requires presence of the filesystem on a device, and the most
resource-constrained devices may not even have it.
The bytecode freezing is a process which resolves all the issues mentioned above:
- The source code is pre-compiled into bytecode and store as such.
- The bytecode is stored in ROM, not RAM.
- Filesystem is not required for frozen packages.
Using frozen bytecode requires building the executable (firmware)
for a given
MicroPython port from the C source code. Consequently,
the process is:
- Follow the instructions for a particular port on setting up a
toolchain and building the port. For example, for ESP8266 port,
study instructions in
ports/esp8266/README.mdand follow them. Make sure you can build the port and deploy the resulting executable/firmware successfully before proceeding to the next steps.
MicroPython Unix portand make sure it is in your PATH and you can execute
- Change to port’s directory (e.g.
make clean-frozen. This step cleans up any previous modules which were installed for freezing (consequently, you need to skip this step to add additional modules, instead of starting from scratch).
micropython -m upip install -p modules <packages>...to install packages you want to freeze.
After this, you should have the executable/firmware with modules as the bytecode inside, which you can deploy the usual way.
- Step 5 in the sequence above assumes that the distribution package
is available from PyPI. If that is not the case, you would need
to copy Python source files manually to
modules/subdirectory of the port port directory. (Note that upip does not support installing from e.g. version control repositories).
- The firmware for baremetal devices usually has size restrictions, so adding too many frozen modules may overflow it. Usually, you would get a linking error if this happens. However, in some cases, an image may be produced, which is not runnable on a device. Such cases are in general bugs, and should be reported and further investigated. If you face such a situation, as an initial step, you may want to decrease the amount of frozen modules included.
Creating distribution packages¶
Distribution packages for MicroPython are created in the same manner
as for CPython or any other Python implementation, see references at
the end of chapter. Setuptools (instead of distutils) should be used,
because distutils do not support dependencies and other features. “Source
sdist) format is used for packaging. The post-processing
discussed above, (and pre-processing discussed in the following section)
is achieved by using custom
sdist command for setuptools. Thus, packaging
steps remain the same as for the standard setuptools, the user just
needs to override
sdist command implementation by passing the
appropriate argument to
from setuptools import setup
The sdist_upip.py module as referenced above can be found in
A complete application, besides the source code, oftentimes also consists of data files, e.g. web page templates, game images, etc. It’s clear how to deal with those when application is installed manually - you just put those data files in the filesystem at some location and use the normal file access functions.
The situation is different when deploying applications from packages - this is more advanced, streamlined and flexible way, but also requires more advanced approach to accessing data files. This approach is treating the data files as “resources”, and abstracting away access to them.
Python supports resource access using its “setuptools” library, using
pkg_resources module. MicroPython, following its usual approach,
implements subset of the functionality of that module, specifically
pkg_resources.resource_stream(package, resource) function.
The idea is that an application calls this function, passing a
resource identifier, which is a relative path to data file within
the specified package (usually top-level application package). It
returns a stream object which can be used to access resource contents.
resource_stream() emulates interface of the standard
resource_stream() uses file operations
underlyingly, if distribution package is install in the filesystem.
However, it also supports functioning without the underlying filesystem,
e.g. if the package is frozen as the bytecode. This however requires
an extra intermediate step when packaging application - creation of
“Python resource module”.
The idea of this module is to convert binary data to a Python bytes
object, and put it into the dictionary, indexed by the resource name.
This conversion is done automatically using overridden
described in the previous section.
Let’s trace the complete process using the following example. Suppose your application has the following structure:
utils.py should access resources using the
You can develop and debug using the
MicroPython Unix port as usual.
When time comes to make a distribution package out of it, just use
overridden “sdist” command from sdist_upip.py module as described in
the previous section.
This will create a Python resource module named
R.py, based on the
files declared in
MANIFEST.in files (any non-
file will be considered a resource and added to
R.py) - before
proceeding with the normal packaging steps.
Prepared like this, your application will work both when deployed to filesystem and as frozen bytecode.
If you would like to debug
R.py creation, you can run:
python3 setup.py sdist --manifest-only
Alternatively, you can use tools/mpy_bin2res.py script from the MicroPython distribution, in which can you will need to pass paths to all resource files:
mpy_bin2res.py data/page.html data/image.png