How to create python package

How to Build Your Very First Python Package

A few months ago, I decided to release Caer, a Computer Vision package available in Python. I found the process to be excruciatingly painful. You can probably guess why — little (and confusing) documentation, lack of good tutorials, and so on.

So I decided to write this article in the hope that it’ll help people who are struggling to do this. We’re going to build a very simple module and make it available to anyone around the world.

The contents of this module follow a very basic structure. There are, in total, four Python files, each of which has a single method within it. We’re going to keep this real simple for now.

Bringing out the __init__s

Something that you’ll always find in every Python package is an __init__.py file. This file will tell Python to treat directories as modules (or sub-modules).

Very simply, it will hold the names of all the methods in all the Python files that are in its immediate directory.

A typical __init__.py file has the following format:

When building packages in Python, you are required to add an __init__.py file in every sub-directory in your package. These sub-directories are the sub-modules of your package.

and we’re going to do the same for the extras folder, like this:

Once that’s done, we’re pretty much halfway through the process!

How to set up setup.py

Within the base-verysimplemodule folder (and in the same directory as our module verysimplemodule ), we need to add a setup.py file. This file is essential if you intend to build the actual module in question.

Note: Feel free to name the setup.py file as you wish. This file is not name-specific as our __init__.py file is.

The setup.py file will contain information about your package, specifically the name of the package, its version, platform-dependencies and a whole lot more.

For our purposes, we’re not going to require advanced meta information, so the following code should suit most packages you build:

With that done, all we have to do next is run the following command in the same directory as base-verysimplemodule :

This will build all the necessary packages that Python will require. The sdist and bdist_wheel commands will create a source distribution and a wheel that you can later upload to PyPi.

PyPi — here we come!

PyPi is the official Python repository where all Python packages are stored. You can think of it as the Github for Python Packages.

To make your Python package available to people around the world, you’ll need to have an account with PyPi.

How to upload your package to PyPi

Assuming you have twine installed, go ahead and run:

Now, if you’ve followed this tutorial to the T, you might get an error along the lines of repository already exists.

This is usually because there is a name clash between the name of your package and a package that already exists. In other words, change the name of your package — somebody else has already taken that name.

To proudly pip install your module, fire up a terminal and run:

Watch how Python neatly installs your package from the binaries that were generated earlier.

Open up a Python interactive shell and try importing your package:

It’s as simple as that.

Congratulations! You’ve just built your first Python package. Albeit very simple, your package is now available to be downloaded by anyone around the world (so long as they have Python, of course).

What’s next?

Test PyPi

The package that we used in this tutorial was an extremely simple module — basic mathematical operations of addition, subtraction, multiplication and division. It doesn’t make sense to upload them directly to PyPi especially since you’re trying this out for the first time.

Lucky for us, there is Test PyPi, a separate instance of PyPi where you can test out and experiment on your package (you will need to sign up for a separate account on the platform).

The process that you follow to upload to Test PyPi is pretty much the same with a few minor changes.

To download projects from Test PyPi:

Advanced Meta Information

The meta information we used in the setup.py file was very basic. You can add additional information such as multiple maintainers (if any), author email, license information and a whole host of other data.

This article will prove particularly helpful if you intend to do so.

Look at other repositories

Looking at how other repositories have built their packages can prove to be super useful to you.

When building Caer, I would constantly look at how Numpy and Sonnet set up their packages. I would recommend taking a look at Caer’s, Numpy’s, and Tensorflow’s repositories if you plan on building slightly more advanced packages.

How to write your own Python Package and publish it on PyPi

Why PyPi?

The Python Package Index (PyPI) is a repository of software for the Python programming language. PyPI helps you find and install software developed and shared by the Python community. If you use pip command, you are already using PyPi.

When your package is published to PyPi, everyone can install and use it with familiar simple command:

So, what do you need to do to publish your own package to PyPi? Here is a short list of steps:

Step #1. Make your code publish-ready

This tutorial will take my package called elastictools as a real example. You can found it here on PyPi, source code is here on Github.

Some rules of thumb:

Create a python package

Package in Python is simply a folder with name of your package. This folder contains files (modules) and other sub-folders (sub-packages).

And you need to put a file __init__.py (two underscores before and after init ) to mark this folder as a Python package. Inside this __init__.py file you can specify which classes you want the user to access through the package interface.

Sample __init.py__ file:

Add files needed for PyPi

PyPi needs following file in order to work:

Sample project structure:

The setup.py file

Most of the options are self-explainable, you can just copy the content of setup.py above and modify it as your need. Please remember to list all dependencies of your package in install_requires list, so that this requirements can be installed automatically while your package is being installed.

Include data files

To include data files (config, env, templates….) to the package, at first, we need to add this line to setup.py :

And then create a MANIFEST.in file next to setup.py file, and list all the file needed to include as following:

Step #2. Create a PyPi account

If you already have a PyPi account (and still remember your username/password, of course), you can skip this step. Otherwise, please go to PyPi homepage and register new account instantly (for free, of course).

Step #3. Generate distribution archives and upload to PyPi

Generating distribution archives

These are archives that are uploaded to the Package Index and can be installed by pip.

Make sure you have the latest versions of setuptools and wheel installed:

Now run this command from the same directory where setup.py is located:

This command should output a lot of text and once completed should generate two files in the dist directory, among others in build and *.egg-info folder:

Uploading the distribution archives

Then upload all the archives to PyPi:

After successful uploading, go to PyPi website, under your project, you can found your published package.

Step #4. Install your own package using pip

Now everyone can install your package with familiar pip install command:

How to write a Python module/package?

I’ve been making Python scripts for simple tasks at work and never really bothered packaging them for others to use. Now I have been assigned to make a Python wrapper for a REST API. I have absolutely no idea on how to start and I need help.

What I have:

Here’s the current directory tree

What I wanted to do:

If possible, I want a general step by step process on writing Python modules.

6 Answers 6

Trending sort

Trending sort is based off of the default sorting method — by highest score — but it boosts votes that have happened recently, helping to surface more up-to-date answers.

It falls back to sorting by highest score if no posts are trending.

Switch to Trending sort

create hello.py then write the following function as its content:

Then you can import hello :

You can go about with the import statement on your module the usual way.

For more information, see 6.4. Packages.

Found the accepted answer useful, yet wished to expand on several points for the benefit of others based on my own experiences.

Module Example: Assume we have a single python script in the current directory, here I am calling it mymodule.py

The file mymodule.py contains the following code:

If we run the python3 interpreter from the current directory, we can import and run the function myfunc in the following different ways (you would typically just choose one of the following):

Ok, so that was easy enough.

Now assume you have the need to put this module into its own dedicated folder to provide a module namespace, instead of just running it ad-hoc from the current working directory. This is where it is worth explaining the concept of a package.

Package: Packages are a way of structuring Python’s module namespace by using “dotted module names”. For example, the module name A.B designates a submodule named B in a package named A. Just like the use of modules saves the authors of different modules from having to worry about each other’s global variable names, the use of dotted module names saves the authors of multi-module packages like NumPy or the Python Imaging Library from having to worry about each other’s module names.

Package Example: Let’s now assume we have the following folder and files. Here, mymodule.py is identical to before, and __init__.py is an empty file:

The __init__.py files are required to make Python treat the directories as containing packages. For further information, please see the Modules documentation link provided later on.

Our current working directory is one level above the ordinary folder called mypackage

If we run the python3 interpreter now, we can import and run the module mymodule.py containing the required function myfunc in the following different ways (you would typically just choose one of the following):

Assuming Python 3, there is excellent documentation at: Modules

Modules should have short, all-lowercase names. Underscores can be used in the module name if it improves readability. Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.

Create Python Packages

Why should you care about creating packages?

Library vs Package vs Module:

Packaging python code is quite easy, you require a single setup.py script to package in several distribution formats.

Topics

1. Packaging Setup

Let’s use the folder structure from an earlier post, and create a virtual environment in it:

Create a setup.py file in the root directory that we will use to define the way we’d like to package our code, containing the following code:

You can now call this script to package your code in several ways:

Run the first one in the list above. When it succeeds, you will be able to import your code as follows:

Note: if you are receiving “No module named demo…», you’ll need to add an empty __init__.py file in all folders you want to import from. In our example, that only includes the demo folder. You can read more about these __init__.py files here.

Now that we were able to install the project, we should take a closer look at the arguments that we pass to the setuptools.setup function:

2. Documentation

Run the quickstart to generate the source directory for your documentation:

Now you can start populating the docs/index.rst file with your documentation. Learn more about automating this process on sphinx-doc.org.

3. Linting and Testing

As part of your packaging process, you’d want to apply some static code analyses, linting, and testing.

Preferably, you’d run a command to verify the code style, run some tests and checks before you push your commits to the remote repository, and cause a build pipeline to fail if the tests don’t pass.

4. Makefile

As we’re rapidly introducing new commands that are part of packaging our specific project, it is useful to record common commands. Most build automation tools (such as Gradle or npm) provide this feature by default.

Make is a tool to organize code compilation, traditionally used in c-oriented projects. But it can be used to run any other command.

5. Installing wheel

Having the wheel, you can create a new folder, somewhere on your desktop, copy the wheel into it, and install it using:

Print out the installed packages:

6. Include Config Files

It’s also possible to add data to your package by adding the following lines to your setup.py script:

Note: This may not work on distributed systems (such as Databricks).

You’ll then be able to read the file by using this function:

If you create a wheel again, and install it in a virtual environment in a new folder, without copying the data file, you should be able to access the data by executing the function above.

7. DevOps

As part of our packaging process, we want to integrate changes from many contributors, and automate as many repetitive processes that are required to successfully release a new version.

For this example we’ll use Azure DevOps, where the following pipeline will be triggered on git tags as well as the master branch.

Have a look, and we’ll discuss the various stages and tasks afterwards:

In the Test stage we install the project in the pipeline container, without creating a virtual environment. We then run the make lint and make test commands, just like you would on your machine.

In the Build stage we will try to extract the package version from a git tag, and we construct a fallback package version. We run the python setup.py bdist_wheel command to build a wheel, knowing that our package version environment variable is set. Finally, we publish the artifact to Azure DevOps artifacts, and (optionally) to our feed.

On how to install packages from a private feed, have a look at this post.

How To Open-Source Your First Python Package

A beginner-friendly step-by-step guide to Python packaging

Why a Python package?

Python is an intuitive general-purpose language with an easy-to-understand syntax. It makes prototyping, web development, data science and more, much easier than in any other programming languages. This is due to its large and supportive community and its numerous software libraries.

You have ever found yourself writing mutliple times the same lines of code in different projects? Or perhaps, you have already written some useful library that you would like to make available to others? Or maybe, you are just interested in Python Packaging or in contributing to the Python community?

When I built my first package in Python, I found the process somewhat tedious. That motivated me to write this article which will put you through the basics of how to build a Python package and share it with the rest of the world.

Requirements

If you have a GitHub account and Python installed you can jump ahead to the Project Overview.

Git Repository

To be part of the open-source community, you must publicly host a project on a version control hosting service. The most common choice is to use a Git repository host which holds a virtual storage of your project. It allows you to coordinate work among programmers and save versions of your code, which you can access when needed. The most used hosting service is GitHub. If you don’t have an account yet, the first step is to create one right here.

Python

We will use Python 3. Any version from Python 3.5 can be used for the project. If you don’t have Python, you can download it from python.org’s download page. I will be using Python 3.7 on Windows, yet it doesn’t matter what version or OS you have for this project. If you are new to Python, I could suggest you to go through some exercises of this tutorial.

An Integrated Development Environment (IDE) is a source code editor that provides you with debugging and automation tools. It doesn’t matter what code editor or IDE you use as long as you like it. I will personnaly use a light yet powerful code editor called Visual Studio Code.

Project Overview

Let’s now build a simple package dealing with prime numbers. If you already have some code you would like to use instead, feel free to skip this part after structuring your code to resemble the project.

Below is how the project structure will look like. You can either create the files alongside me or directly download the whole repository on GitHub.

This may look like a lot of files but don’t worry, we are going to make and explain them one after the other.

Build The Project Step by Step

Initiate and Clone the GitHub Repository

You should now have your project copied on your computer. Feel free to open the primelib folder in your favorite code editor. You should have the following structure.

Python Code

In the primelib/ folder, you can now add a new folder also called primelib that will contain all your code. In this new folder, create 2 Python files called prime_numbers.py and __init__.py (with two leading and two trailing underscores).

Let’s start with prime_numbers.py which contains the Python code we will be using. You can just copy and paste this code into your prime_numbers.py file.

The Python code contains two functions:

Tests

Alright now is time to create some tests to make sure our functions work as expected. You can now create a tests folder within your project directory and add a test_prime.py file in it.

You can simply copy the following tests.

Then you can use the Pytest library which is part of Python Standard Library, so it comes with Python, no need to install it. Pytest will automatically check your assert statements in any test_*.py file. In your terminal, you can just run the following command.

You should now see your two tests passed with success. This means you are ready to build the package and share your library with the community.

Build the Library

One last file is required to build the package. You must add a setup.py file at the root of your directory. This tells setuptools how to actually build the package. Let’s add this last file to obtain the final tree structure.

Here is the setup.py file. You can copy and paste it while replacing YOUR-USERNAME with your name to make the library name unique.

The three mandatory parameters are name, version and packages. You can now build the library with the following.

Upload on PyPi

Now that you have a working library, you can upload it on Python Package Index (PyPi) to share it with the Python community. For the purpose of this tutorial we are going to use TestPyPi, it works exactly like PyPi but it is made for testing packages in order to keep the real servers as clean as possible.

First you have to register an account on TestPypi and verify your email. You can finally upload your package as archives using the following line.

If you encounter any issues with packaging, please refer to the Python Packaging tutorial.

Your library is now available within your Python environment. I strongly recommend to use virtualenv to manage different environments.

Conclusion

Congrats! You have learnt how to create a Python package to wrap some code of yours and then how to share your library with the community so that anyone can use your work.

I hope you now feel ready to contribute to the Python community and share your great code with the rest of the world!

If you’re interested in going further, I would recommend these important, yet much more advanced topics: