Python invoke task automation tool

1.What can invoke do?

Invoke is a spin-off from the well-known remote deployment tool Fabric, and together with paramiko are the two most core foundational components of the Fabric.

In addition to being a command-line tool, it focuses on "task execution", which allows you to label and organize tasks and execute them through the CLI (command-line interface) and shell commands.

Also a task automation tool, invoke has a different focus than tox/nox, which we've covered before：

• tox/nox is mainly for automation in packaging, testing, continuous integration, etc. (of course they can do more than that)

• invoke is more general and can be used in any scenario where you need to "perform a task", either as an unrelated group of tasks or as a step-by-step workflow with sequential dependencies

invoke has 2.7K stars on Github and is very popular, so let's see how it works?

2.How do I use invoke?

First, the installation is simple: pip install invoke.

Secondly, the following elements are available for simple use：

• Task files. Create a tasks.py file.

• @task decorator. Adding a @task decorator to a function marks the function as a task to be managed by invoke scheduling.

• Context argument. Add a context argument to the decorated function, noting that it must be the first argument, and that the naming convention can be c or ctx or context.

• Command line execution. Execute invoke --list on the command line to see all tasks, and run invoke xxx to execute the task named xxx. The word "invoke" on the command line can be abbreviated to "inv".

Here is a simple example：

# File name: tasks.py
from invoke import task

@task
def hello(c):
    print("Hello world!")

@task
def greet(c, name):
    c.run(f"echo {name}!")

In the above code, we define two tasks：

• The "hello" task calls Python's built-in print function, which prints a string "Hello world!"

• The "greet" task calls the run() method with contextual arguments and can execute shell commands, and in this case can take one argument. In shell commands, echo can be interpreted as print, so this is also a print task that prints out "Go xxx! (xxx is the parameter we pass)

The above code is written in tasks.py file, first import the decorator from invoke import task, @task decorator can be without arguments or with arguments (see next section), the function decorated by it is a task.

The context argument (i.e. "c" in the above example) must be explicitly specified, and if it is missing, the execution will throw an exception: "TypeError: Tasks must have an initial Context argument! "

Then, in the same directory as tasks.py, open a command line window and execute the command. If the task file is not found in the execution location, an error will be thrown: "Can't find any collection named 'tasks'!"

Under normal circumstances, a list of all tasks (sorted alphabetically) can be seen by executing inv --l or inv -l at

>>> inv -l
Available tasks:

  greet
  hello

We perform these two tasks in turn, where the passing of parameters can be done by default by location parameters or by specifying keywords. The result is.

>>> inv hello
Hello world!
>>> inv greet 007
>>> inv greet --name="007"

3.How to use invoke well?

After introducing the simple use of invoke, we know a few of the elements it requires and a general idea of the steps to use it.

3.1 Add help information

In the above example, "inv -l" can only see the task name, without the necessary auxiliary information, in order to enhance readability, we can write it like this：

@task(help={'name': 'A param for test'})
def greet(c, name):
    """
    A test for shell command.
    Second line.
    """
    c.run(f"echo {name}加油!")

The first line of the document string is displayed as an excerpt in the query result of "inv -l", and the full content and the help content of @task are displayed in "inv --help"：

>>> inv -l
Available tasks:

  greet   A test for shell command.
>>> inv --help greet
Usage: inv[oke] [--core-opts] greet [--options] [other tasks here ...]

Docstring:
  A test for shell command.
  Second line.

Options:
  -n STRING, --name=STRING   A param for test

3.2 Decomposition and combination of tasks

Usually a large task can be decomposed into a set of small tasks, and in turn, a series of small tasks may be linked together into a large task. When decomposing, abstracting and combining tasks, here are two ideas：

• Decompose internally, unify externally: define only one @task task as the overall task entry, the actual processing logic can be abstracted into multiple methods, but they are not perceived externally

• Multi-point presentation, single-point aggregation: define multiple @task tasks that can be sensed and invoked externally, and combine related tasks so that when a task is invoked, other related tasks are also executed

The first idea is easy to understand, simple to implement and use, but has the disadvantage of lacking flexibility and making it difficult to execute one/some of the subtasks individually. It is suitable for relatively independent single tasks and usually can be done without invoke (the advantage of using invoke is that it has command line support).

The second idea is more flexible, facilitating both the execution of a single task and the execution of a combination of multiple tasks. In fact, this is the scenario where invoke is most valuable.

So, how does invoke implement the combination of step-by-step tasks? It can be specified in the "pre" and "post" parameters of the @task decorator, which represent the pre-task and post-task respectively：

@task
def clean(c):
    c.run("echo clean")

@task
def message(c):
    c.run("echo message")

@task(pre=[clean], post=[message])
def build(c):
    c.run("echo build")

The clean and message tasks are subtasks that can be called individually or combined as a pre and post task of the build task：

>>> inv clean
clean
>>> inv message
message
>>> inv build
clean
build
message

These two parameters are list types, i.e. multiple tasks can be set. In addition, by default, the position parameter of the @task decorator is treated as a predecessor task. Following the above code, we write a：

@task(clean, message)
def test(c):
    c.run("echo test")

Then execute it and you will see that both parameters are considered as predecessors：

>>> inv test
clean
message
test

3.3 Module splitting and integration

If you want to manage many large, relatively independent tasks, or if you need multiple teams to maintain their own tasks, then it is necessary to split and consolidate tasks.py.

For example, if you have multiple copies of tasks.py, which are relatively complete and independent task modules, and it is not convenient to put everything in a single file, how can you effectively manage them together?

invoke provides support for this. First, only one file named "tasks.py" can be kept, second, other renamed task files can be imported into that file, and finally, invoke's Collection class can be used to associate them.

Let's rename the first example file in this article to task1.py and create a new tasks.py file with the following contents

：

# File name：tasks.py
from invoke import Collection, task
import task1

@task
def deploy(c):
    c.run("echo deploy")

namespace = Collection(task1, deploy)

Each py file has its own namespace, and here we use a collection to create a new namespace to manage all the tasks in one place. The result is as follows：

>>> inv -l
Available tasks:

  deploy
  task1.greet
  task1.hello
>>> inv deploy
deploy
>>> inv task1.hello
Hello world!
>>> inv task1.greet 007

3.4 Interactive operation

Some tasks may require interactive input, such as asking for a "y" and pressing enter before proceeding. The ability to automate tasks is greatly diminished if human involvement is required during task execution.

invoke provides the ability to monitor the program during runtime, listen to stdout and stderr, and support inputting the necessary information in stdin.

For example, suppose a task (excitable-program) is executed with the prompt "Are you ready? [y/n]", and only if you enter "y" and press enter will the subsequent action be executed.

Then, by specifying the contents of the responses parameter in the code, the program will automatically perform the corresponding action as soon as it listens for a match：

responses = {r"Are you ready? \[y/n\] ": "y\n"}
ctx.run("excitable-program", responses=responses)

The responses are dictionary types, and the key-value pairs are the listener content and its response content, respectively. Note that the keys are treated as regular expressions, so the square brackets, as in this example, have to be escaped first.

3.5 As a command line tool library

There are many good command-line tool libraries in Python, such as argparse from the standard library, click from Flask and fire from Google, etc. Invoke can also be used as a command-line tool library.

(PS: A Prodesire student wrote a series of articles called "Python Command Line Tour", which describes the usage of several other command line tool libraries in detail. (If you are interested, you can check the history of the article.)

In fact, the Fabric project originally separated invoke into a separate library because it wanted to take on the task of parsing the command line and executing subcommands. So, in addition to being an automated task management tool, invoke can also be used to develop command line tools.

An example is given in the official documentation to get a basic idea of how it is used.

Suppose we want to develop a tester tool that allows users to pip install tester installations, and this tool provides two execution commands: tester unit and tester intergration.

These two subcommands need to be defined in the tasks.py file

：

# tasks.py
from invoke import task

@task
def unit(c):
    print("Running unit tests!")

@task
def integration(c):
    print("Running integration tests!")

Then introduce it in the program entry file：

# main.py
from invoke import Collection, Program
from tester import tasks

program = Program(namespace=Collection.from_module(tasks), version='0.1.0')

Finally, declare the entry function in the package file：

# setup.py
setup(
    name='tester',
    version='0.1.0',
    packages=['tester'],
    install_requires=['invoke'],
    entry_points={
        'console_scripts': ['tester = tester.main:program.run']
    }
)

The library so packaged and distributed is a full-featured command-line tool：

$ tester --version
Tester 0.1.0
$ tester --help
Usage: tester [--core-opts] <subcommand> [--subcommand-opts] ...

Core options:
  ... core options here, minus task-related ones ...

Subcommands:
  unit
  integration

$ tester --list
No idea what '--list' is!
$ tester unit
Running unit tests!

Easy to get started and ready to use out of the box, invoke is a good command line tool library to consider. For more detailed usage, please refer to the documentation.