Ansible - Show output of long running command

• 2024 June 11
• ansible
• command
• print
• output
• rsync
• plugin
• python

Problem #

There's no built in solution in Ansible to be able to see the output of a long running command as it exeecutes.

You can register a task's output and then print its output with the debug module after it has finished, but not while it is running:

- name: Echo command
  command: echo "hello"
  register: hello

- name: Print output
  debug:
    msg: "{{ hello.stdout }}"

Async task #

There's an official-ish solution: you can mark a task async and poll it. Setting the poll parameter to 0 tells Ansible to start the task and move on to the next one, making this task run in the background asynchronously. Registering this asnyc task and using the async_status module you can wait for it to complete and get "pings" whether it is still executing and not hanging.

---
- hosts: localhost
  tasks:
  - name: Simulate long running op, allow to run for 45 sec, fire and forget
    ansible.builtin.shell: |
      /bin/sleep 15
      echo test
      /bin/sleep 15
    async: 45
    poll: 0
    register: sleeper

  - name: Check on async task
    async_status:
      jid: "{{ sleeper.ansible_job_id }}"
    register: res
    until: res.finished
    retries: 100
    delay: 10

But this still does not show the output of the command that you are running, and for example when downloading large files using rsync it can be useful to see the progress and download rate.

Solution #

So the solution we came up with is to

1. Start the long running command
   • mark it async,
   • set poll to 0,
   • register the Ansible task's output,
   • and most importantly pipe its standard output into a file
     (this file will be created on the node on which the command is being executed on)
2. Using a custom Ansible action plugin written in python you can poll the file and print it essentially seeing the output of the command.

The 2 Ansible tasks described above look like this:

---
- name: Start rsync for folder {{ dir_to_sync }}
  become: true
  ansible.builtin.shell:
    executable: /bin/bash
    cmd: |
      set -o pipefail; /usr/bin/rsync --info=progress2 --info=name0 --delay-updates --compress --archive --update --partial \
      --rsh='/usr/bin/ssh -S none -i /home/debian/.ssh/id_ed25519 -o Port=6363 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' \
      --out-format='<<CHANGED>>%i %f%L' dev-read@{{ template_provider_host }}:/data/{{ dir_to_sync }} /data/{{ dir_to_sync }} | tee {{ rsync_log_file }}
  async: 86400
  poll: 0
  register: rsync_out
  changed_when: true

- name: Follow download status ({{ dir_to_sync }})
  become: true
  print_download_status:
    status_file: "{{ rsync_log_file }}"
    async_job: "{{ rsync_out }}"

Notes:

• the pipefail option ensures that tasks fail as expected if the first command fails
• rsync's --rsh= parameter is used to specify the remote shell command, so you can specify additional parameters for ssh, like which private key, port to use
• | tee is used to write to both stdout and to a file (but also > {{ rsync_log_file }} should be enough, which just writes into a file)
• the custom Ansible action is called using the python file's name (print_download_status), the file's path and the async job's register has to be passed as parameters to it

Custom Ansible action plugin #

You can write your own python code which then can be called by Ansible creating your own kind of tasks.
These can be either modules or plugins. The main difference between them is that modules run on target machines, while plugins run on the control node (that starts the ansible-playbook command).

You have to specify in an ansible.cfg file where Ansible should look for plugins by assigning the folder's path to the action_plugins key, which is under the [defaults] section.

The print_download_status plugin #

You have to create a python file with the name you want to call your own kind of ansible tasks with. We called ours print_download_status.py.
You have to create an ActionModule class which extends the ActionBase class for action plugins.
From the custom plugins code you can call built in ansible modules (self._execute_module method), which will execute on the remote target host.

Using the command module we run stat on the long running command's stdout file, this way we get its size. We use this information to only read the number of bytes we haven't read since our last read out (head -q -c {stat_reported_bytes} {status_file} | tail -q -c +{bytes_read_so_far}). We print this text, and repeat these steps in a loop until the async job finishes, which can be checked by calling the async_status module and passing the asynchrounous job's id to it.

You can try it out by running the provided example, async_task.yml:

ansible-playbook async_task.yml

• Previous post: HTB Business CTF 2024 - pwn - regularity
• Next post: From disk image to offline windows AD account login

Want to message us? Contact us: blog@ukatemi.com