Upload MUB file fails with ansible-for-nsxt

While trying to upgrade the NSX-T enviroment via ansible we stumbled upon the issue that we couldn’t upload the mub file to the NSX Manager.

Ansible Task:

- name: Upload upgrade file to NSX-T manager coordinator node from file
  vmware.ansible_for_nsxt.nsxt_upgrade_upload_mub:
    hostname: "{{ coordinator }}"
    username: "{{ username }}"
    password: "{{ password }}"
    validate_certs: False
    file: "{{ nsx_upgrade_mub_file }}"

This gave us the following error message:

fatal: [127.0.0.1]: FAILED! => {
    "changed": true,
    "invocation": {
        "module_args": {
            "file": "/root/hypervisor/upgrade_bundle/VMware-NSX-upgrade-bundle-3.2.1.1.0.20115686.mub",
            "hostname": "manager1",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "port": 443,
            "timeout": 9000,
            "url": null,
            "username": "admin",
            "validate_certs": false
        }
    },
    "msg": "Error: string longer than 2147483647 bytes"

So it looks like the upload can’t handle files over 2GB.

Honoustly my python skills are a bit rusty, so i asked one of the developers in our team to help me out and see if we could get this fixed.

The 2GB+ filesize is the issue. You can find multiple references to the error, usually referring to the httplib, urllib or ssl..
One solution is to use streaming upload.

This is what we did to make the upload work.

Install request-toolbelt package

Edit nsxt_upgrade_upload_mub.py
NOTE: This will break the URL upload!
Add:

import requests
from requests_toolbelt.multipart import encoder
from requests.auth import HTTPBasicAuth 

Replace line 140 – 174 with:

        session = requests.Session()
        with open(file_path, 'rb') as src_file:
             body = encoder.MultipartEncoder({
                 "file": (src_file.name, src_file, "application/octet-stream")
             })
             headers = {"Prefer": "respond-async", "Content-Type": body.content_type}
             resp = session.post(mgr_url + endpoint, auth=HTTPBasicAuth(mgr_username, mgr_password), timeout=None, verify=False, data=body, headers=headers)
             bundle_id = 'latest'#resp['bundle_id']
             headers = dict(Accept="application/json")
             headers['Content-Type'] = 'application/json'
             try:
                 wait_for_operation_to_execute(mgr_url,
                     '/upgrade/bundles/%s/upload-status'% bundle_id,
                     mgr_username, mgr_password, validate_certs,
                     ['status'], ['SUCCESS'], ['FAILED'])
             except Exception as err:
                 module.fail_json(msg='Error while uploading upgrade bundle. Error [%s]' % to_native(err))
             module.exit_json(changed=True, ip_address=ip_address,
             message='The upgrade bundle %s got uploaded successfully.' % module.params[mub_type])
        session.close()

NOTE: This will break the URL upload!

The response will show:

changed: [127.0.0.1] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "file": "/upgrade_bundle/VMware-NSX-upgrade-bundle-3.2.1.1.0.20115686.mub",
            "hostname": "nsxmanager",
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER",
            "port": 443,
            "timeout": 9000,
            "url": null,
            "username": "admin",
            "validate_certs": false
        }
    },
    "ip_address": "<ip>",
    "message": "The upgrade bundle /upgrade_bundle/VMware-NSX-upgrade-bundle-3.2.1.1.0.20115686.mub got uploaded successfully."
}

Solution is also added to a github BUG report:

https://github.com/vmware/ansible-for-nsxt/issues/416

All Kudos to my colleague for fixing the issue!

READ MORE