text:
Add support for Iris 3.0

This (draft) PR adds support to IMPROVER for Iris 3.0 at the expense of supporting Iris 2.*. I have marked this PR as a draft so it can be used as a place for further discussion.
Here is a summary of the main changes:

removed usage of CubeList.extract_strict and use of strict keyword argument to CubeList.extract and replaced them with CubeList.extract_cube (for more, see here in the Iris docs changelog)
removed unavailable import from iris._cube_coord_common import LimitedAttributeDict which was introduced for type annotations
added a Conda environment file environment_py37_iris30.yml which has the same environment as environment_py37_iris24.yml except for replacing Iris 2.4 for 3.0
added this environment to GitHub Actions and removed Iris 2.* environments
updated the Iris version in setup.cfg
updated the readthedocs environment file
updated some docs

Further issues to be addressed:

[x] this PR only tests Python 3.7, whereas Iris 3.0 supports Python 3.6 and 3.8 as well (note: I could not build a Conda environment with the exactly pinned versions as environment_py37_iris24.yml with Iris 3.0 and Python 3.8)
[x] we may need to changed pinned (or want to unpin) package versions of cftime and scipy
[x] patching Iris for Nimrod support is no longer required (see this PR for more)

Addresses #1437, #1517
Testing:

[x] Ran tests and they passed OK

Currently the unit tests are passing in the py37_iris30 environment. However, I marked one test Test_lat_lon_determine in improver_tests/utilities/test_spatial.py as xfail. I did some digging but could not find why it was failing.

Although the unit tests are passing, 36 acceptance tests are failing with errors like:
_______________________ test_normal_point_by_point_sites _______________________
[gw4] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0')

    def test_normal_point_by_point_sites(tmp_path):
        """Test using a normal distribution when coefficients have been calculated
        independently at each site (initial guess and minimisation)."""
        kgo_dir = acc.kgo_root() / "apply-emos-coefficients/sites/point_by_point"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / ".." / "input.nc"
        emos_est_path = kgo_dir / "coefficients.nc"
        output_path = tmp_path / "output.nc"
        args = [
            input_path,
            emos_est_path,
            "--random-seed",
            "0",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path, atol=LOOSE_TOLERANCE)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_apply_emos_coefficients.py:99: AssertionError
----------------------------- Captured stdout call -----------------------------
improver apply-emos-coefficients /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/../input.nc /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/coefficients.nc --random-seed 0 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0/output.nc

It seems some datasets are missing attributes, mostly units, e.g.
name is '/wmo_id'
desired is {'units': '1', 'long_name': 'wmo_id'}
actual is {'long_name': 'wmo_id'}

Here is the full output from pytest -n 16 -k 'acc':

============================= test session starts ==============================
platform linux -- Python 3.7.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/dmentipl/repos/improver, configfile: pytest.ini, testpaths: improver_tests
plugins: cov-2.12.0, forked-1.3.0, xdist-2.2.1
gw0 I / gw1 I / gw2 I / gw3 I / gw4 I / gw5 I / gw6 I / gw7 I / gw8 I / gw9 I / gw10 I / gw11 I / gw12 I / gw13 I / gw14 I / gw15 I
gw0 [347] / gw1 [347] / gw2 [347] / gw3 [347] / gw4 [347] / gw5 [347] / gw6 [347] / gw7 [347] / gw8 [347] / gw9 [347] / gw10 [347] / gw11 [347] / gw12 [347] / gw13 [347] / gw14 [347] / gw15 [347]

..............F................F.....s.....s............................ [ 20%]
.........................F......F..F.....F.F....F..F..............F..... [ 41%]
.........F...FF........F.FF..F..F..FF....FFF..FF..FF..FssF........F..... [ 62%]
..........................x.......x..................................... [ 82%]
..........................F...........FF.....F...FF........              [100%]
=================================== FAILURES ===================================
__________________________________ test_basic __________________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_basic0')

    def test_basic(tmp_path):
        """Test basic create grid with halo"""
        kgo_dir = acc.kgo_root() / "create-grid-with-halo/basic"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / "source_grid.nc"
        output_path = tmp_path / "output.nc"
        args = [input_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /grid_with_halo - ['units']

improver_tests/acceptance/test_create_grid_with_halo.py:50: AssertionError
----------------------------- Captured stdout call -----------------------------
improver create-grid-with-halo /mnt/data/dmentipl/improver_tests/create-grid-with-halo/basic/source_grid.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_basic0/output.nc
_______________________ test_normal_point_by_point_sites _______________________
[gw4] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0')

    def test_normal_point_by_point_sites(tmp_path):
        """Test using a normal distribution when coefficients have been calculated
        independently at each site (initial guess and minimisation)."""
        kgo_dir = acc.kgo_root() / "apply-emos-coefficients/sites/point_by_point"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / ".." / "input.nc"
        emos_est_path = kgo_dir / "coefficients.nc"
        output_path = tmp_path / "output.nc"
        args = [
            input_path,
            emos_est_path,
            "--random-seed",
            "0",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path, atol=LOOSE_TOLERANCE)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_apply_emos_coefficients.py:99: AssertionError
----------------------------- Captured stdout call -----------------------------
improver apply-emos-coefficients /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/../input.nc /mnt/data/dmentipl/improver_tests/apply-emos-coefficients/sites/point_by_point/coefficients.nc --random-seed 0 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw4/test_normal_point_by_point_sit0/output.nc
____________________________ test_nearest[uk-ukvx] _____________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_uk_ukvx_0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest(tmp_path, domain, model):
        """Test basic neighbour finding"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:59: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_uk_ukvx_0/output.nc
_________________________ test_nearest[global-global] __________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_global_global_0')
domain = 'global', model = 'global'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest(tmp_path, domain, model):
        """Test basic neighbour finding"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:59: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_global_global_0/output.nc
________________________________ test_halo_size ________________________________
[gw2] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw2/test_halo_size0')

    def test_halo_size(tmp_path):
        """Test basic create grid with halo"""
        kgo_dir = acc.kgo_root() / "create-grid-with-halo/halo_size"
        kgo_path = kgo_dir / "kgo.nc"
        input_path = kgo_dir / "../basic/source_grid.nc"
        output_path = tmp_path / "output.nc"
        args = [input_path, "--halo-radius", "75000", "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /grid_with_halo - ['units']

improver_tests/acceptance/test_create_grid_with_halo.py:61: AssertionError
----------------------------- Captured stdout call -----------------------------
improver create-grid-with-halo /mnt/data/dmentipl/improver_tests/create-grid-with-halo/halo_size/../basic/source_grid.nc --halo-radius 75000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw2/test_halo_size0/output.nc
______________________________ test_invalid_site _______________________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_invalid_site0')

    def test_invalid_site(tmp_path):
        """Test invalid global site coordinates"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / "outputs/nearest_global_invalid_site_kgo.nc"
        sites_path = kgo_dir / "inputs/global_sites_invalid.json"
        orography_path = kgo_dir / "inputs/global_orography.nc"
        landmask_path = kgo_dir / "inputs/global_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        with pytest.warns(UserWarning, match=".*outside the grid.*"):
            run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:223: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites_invalid.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_invalid_site0/output.nc
___________________________ test_coord_beyond_bounds ___________________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_coord_beyond_bounds0')

    def test_coord_beyond_bounds(tmp_path):
        """Test coordinates up to 360 degrees"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / "outputs/nearest_global_kgo.nc"
        sites_path = kgo_dir / "inputs/global_sites_360.json"
        orography_path = kgo_dir / "inputs/global_orography.nc"
        landmask_path = kgo_dir / "inputs/global_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [orography_path, landmask_path, sites_path, "--output", output_path]
        run_cli(args)
>       acc.compare(output_path, kgo_path, exclude_vars=["longitude"])
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:236: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites_360.json --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_coord_beyond_bounds0/output.nc
______________________ test_unset_wmo_ids_with_unique_ids ______________________
[gw8] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_unset_wmo_ids_with_unique0')

    def test_unset_wmo_ids_with_unique_ids(tmp_path):
        """Test sites with missing WMO ID numbers, but with valid unique IDs for
        all sites"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / "outputs/nearest_uk_kgo_some_unset_wmo_ids_unique_ids.nc"
        sites_path = kgo_dir / "inputs/uk_sites_missing_wmo_ids_with_unique_ids.json"
        orography_path = kgo_dir / "inputs/ukvx_orography.nc"
        landmask_path = kgo_dir / "inputs/ukvx_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--unique-site-id-key",
            "met_office_site_id",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:258: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites_missing_wmo_ids_with_unique_ids.json --unique-site-id-key met_office_site_id --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw8/test_unset_wmo_ids_with_unique0/output.nc
__________________________ test_nearest_land[uk-ukvx] __________________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_uk_ukvx_0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land(tmp_path, domain, model):
        """Test neighbour finding with a land constraint"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:80: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --land-constraint --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_uk_ukvx_0/output.nc
_______________________ test_nearest_land[global-global] _______________________
[gw7] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_global_globa0')
domain = 'global', model = 'global'

    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land(tmp_path, domain, model):
        """Test neighbour finding with a land constraint"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:80: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --land-constraint --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw7/test_nearest_land_global_globa0/output.nc
____________ test_normal_point_by_point_default_initial_guess_sites ____________
[gw13] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw13/test_normal_point_by_point_def0')

    @pytest.mark.slow
    def test_normal_point_by_point_default_initial_guess_sites(tmp_path):
        """
        Test estimate-emos-coefficients for diagnostic with assumed
        normal distribution where coefficients are computed independently at each
        site location (minimisation only).
        """
        kgo_dir = acc.kgo_root() / "estimate-emos-coefficients/normal/sites"
        kgo_path = kgo_dir / "point_by_point_default_initial_guess" / "kgo.nc"
        history_path = kgo_dir / "history/*.nc"
        truth_path = kgo_dir / "truth/*.nc"
        output_path = tmp_path / "output.nc"
        est_emos_tol = str(0.01)
        compare_emos_tolerance = 0.1
        args = [
            history_path,
            truth_path,
            "--distribution",
            "norm",
            "--truth-attribute",
            "mosg__model_configuration=uk_det",
            "--tolerance",
            est_emos_tol,
            "--point-by-point",
            "--use-default-initial-guess",
            "--output",
            output_path,
        ]
        run_cli(args)
        acc.compare(
>           output_path, kgo_path, atol=compare_emos_tolerance, rtol=compare_emos_tolerance
        )
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_estimate_emos_coefficients.py:366: AssertionError
----------------------------- Captured stdout call -----------------------------
improver estimate-emos-coefficients '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/history/*.nc' '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/truth/*.nc' --distribution norm --truth-attribute mosg__model_configuration=uk_det --tolerance 0.01 --point-by-point --use-default-initial-guess --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw13/test_normal_point_by_point_def0/output.nc
_______________________ test_nearest_minimum_dz[uk-ukvx] _______________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_uk_ukv0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with minimum altitude difference"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:104: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_uk_ukv0/output.nc
__________________________ test_all_methods[uk-ukvx] ___________________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_uk_ukvx_0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_all_methods(tmp_path, domain, model):
        """Test neighbour finding using all methods"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/all_methods_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--all-methods",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /neighbour_selection_method_name - ['units']

improver_tests/acceptance/test_neighbour_finding.py:153: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --all-methods --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_uk_ukvx_0/output.nc
__ test_nearest_uk[all_methods_uk.nc-extra_args0-nearest_uk_temperatures.nc] ___
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk0')
neighbour_cube = 'all_methods_uk.nc', extra_args = []
kgo_file = 'nearest_uk_temperatures.nc'

    @pytest.mark.parametrize(
        "neighbour_cube,extra_args,kgo_file",
        (
            ("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
            ("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
            ("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
        ),
    )
    def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
        """Test spot extraction using nearest location"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / f"outputs/{kgo_file}"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            *extra_args,
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk0/output.nc --new-title 'IMPROVER UK Spot Values'
___ test_nearest_uk[all_methods_uk.nc-extra_args1-mindz_uk_temperatures.nc] ____
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk1')
neighbour_cube = 'all_methods_uk.nc', extra_args = ['--similar-altitude']
kgo_file = 'mindz_uk_temperatures.nc'

    @pytest.mark.parametrize(
        "neighbour_cube,extra_args,kgo_file",
        (
            ("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
            ("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
            ("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
        ),
    )
    def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
        """Test spot extraction using nearest location"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / f"outputs/{kgo_file}"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            *extra_args,
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk1/output.nc --similar-altitude --new-title 'IMPROVER UK Spot Values'
_________________________ test_lapse_rate_adjusted_uk __________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_lapse_rate_adjusted_uk0')

    def test_lapse_rate_adjusted_uk(tmp_path):
        """Test spot extraction with lapse rate adjustment"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        lapse_path = kgo_dir / "inputs/ukvx_lapse_rate.nc"
        kgo_path = kgo_dir / "outputs/lapse_rate_adjusted_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            lapse_path,
            "--output",
            output_path,
            "--apply-lapse-rate-correction",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:94: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_lapse_rate.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_lapse_rate_adjusted_uk0/output.nc --apply-lapse-rate-correction --new-title 'IMPROVER UK Spot Values'
_ test_nearest_uk[all_methods_uk_unique_ids.nc-extra_args2-nearest_uk_temperatures_unique_ids.nc] _
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk2')
neighbour_cube = 'all_methods_uk_unique_ids.nc', extra_args = []
kgo_file = 'nearest_uk_temperatures_unique_ids.nc'

    @pytest.mark.parametrize(
        "neighbour_cube,extra_args,kgo_file",
        (
            ("all_methods_uk.nc", [], "nearest_uk_temperatures.nc"),
            ("all_methods_uk.nc", ["--similar-altitude"], "mindz_uk_temperatures.nc"),
            ("all_methods_uk_unique_ids.nc", [], "nearest_uk_temperatures_unique_ids.nc"),
        ),
    )
    def test_nearest_uk(tmp_path, neighbour_cube, extra_args, kgo_file):
        """Test spot extraction using nearest location"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / f"inputs/{neighbour_cube}"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / f"outputs/{kgo_file}"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            *extra_args,
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:72: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk_unique_ids.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_nearest_uk_all_methods_uk2/output.nc --new-title 'IMPROVER UK Spot Values'
_____________________________ test_new_spot_title ______________________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_new_spot_title0')

    def test_new_spot_title(tmp_path):
        """Test spot extraction with external JSON metadata"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures_amended_metadata.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--new-title",
            UK_SPOT_TITLE,
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:179: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --new-title 'IMPROVER UK Spot Values' --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_new_spot_title0/output.nc
___________________________ test_lapse_rate_mismatch ___________________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_lapse_rate_mismatch0')

    def test_lapse_rate_mismatch(tmp_path):
        """Test lapse rate adjustment mismatch between datasets"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        lapse_path = kgo_dir / "inputs/ukvx_lapse_rate_2m.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            lapse_path,
            "--output",
            output_path,
            "--apply-lapse-rate-correction",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        with pytest.warns(UserWarning, match=".*height.*not adjusted.*"):
            run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:139: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_lapse_rate_2m.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_lapse_rate_mismatch0/output.nc --apply-lapse-rate-correction --new-title 'IMPROVER UK Spot Values'
_______________________ test_percentile_percentile_input _______________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_percentile_inp0')

    def test_percentile_percentile_input(tmp_path):
        """Test extracting percentiles from percentile input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_percentiles.nc"
        kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:261: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_percentiles.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_percentile_inp0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
________________________ test_percentile_deterministic _________________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic0')

    def test_percentile_deterministic(tmp_path):
        """Test extracting percentiles from deterministic input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        with pytest.warns(
            UserWarning, match="Diagnostic cube is not a known probabilistic type."
        ):
            run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:303: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
__________________________________ test_spot ___________________________________
[gw5] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw5/test_spot0')

    def test_spot(tmp_path):
        """Test extraction from spot file"""
        kgo_dir = acc.kgo_root() / "extract/sites"
        kgo_path = kgo_dir / "kgo_spot.nc"
        input_path = kgo_dir / "input_spot.nc"
        output_path = tmp_path / "output.nc"
        args = [
            input_path,
            "--constraints",
            "wmo_id=['700', '845', '996', '3346', '3382']",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_extract.py:168: AssertionError
----------------------------- Captured stdout call -----------------------------
improver extract /mnt/data/dmentipl/improver_tests/extract/sites/input_spot.nc --constraints 'wmo_id=['"'"'700'"'"', '"'"'845'"'"', '"'"'996'"'"', '"'"'3346'"'"', '"'"'3382'"'"']' --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw5/test_spot0/output.nc
__________________ test_multiple_percentile_thresholded_input __________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_thres0')

    def test_multiple_percentile_thresholded_input(tmp_path):
        """Test extracting multiple percentiles from thresholded input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_thresholds.nc"
        kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "25, 50, 75",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:354: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_thres0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
_____________________ test_percentile_deterministic_quiet ______________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic_0')

    def test_percentile_deterministic_quiet(tmp_path):
        """Test extracting percentiles from deterministic input. In this case the
        --suppress-warnings flag is enabled. This excludes the warning raised when
        spot-extract is set to extract percentiles and used with deterministic data.
        This is intended to reduce output in logs when this warning is expected and
        thus not useful."""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/nearest_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
            "--suppress-warnings",
        ]
        with pytest.warns(None) as collected_warns:
            run_cli(args)

        msg = "Diagnostic cube is not a known probabilistic type."
        assert all([msg not in str(warning.message) for warning in collected_warns])
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:333: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_deterministic_0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values' --suppress-warnings
__________________ test_multiple_percentile_percentile_input ___________________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_perce0')

    def test_multiple_percentile_percentile_input(tmp_path):
        """Test extracting multiple percentiles from percentile input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_percentiles.nc"
        kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "25, 50, 75",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:375: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_percentiles.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_multiple_percentile_perce0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
__________________________ test_multiple_constraints ___________________________
[gw9] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_multiple_constraints0')

    def test_multiple_constraints(tmp_path):
        """Test use of multiple constraints"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/ukvx_temperature.nc"
        kgo_path = kgo_dir / "outputs/mindz_land_constraint_uk_temperatures.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--similar-altitude",
            "--land-constraint",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:219: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/ukvx_temperature.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_multiple_constraints0/output.nc --similar-altitude --land-constraint --new-title 'IMPROVER UK Spot Values'
______________________ test_percentile_realization_input _______________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_realization_in0')

    def test_percentile_realization_input(tmp_path):
        """Test extracting percentiles from realization input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        realization_path = kgo_dir / "inputs/enukx_temperature_realizations.nc"
        kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            realization_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:396: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_realizations.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_percentile_realization_in0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
______________________ test_percentile_thresholded_input _______________________
[gw9] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_percentile_thresholded_in0')

    def test_percentile_thresholded_input(tmp_path):
        """Test extracting percentiles from thresholded input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        threshold_path = kgo_dir / "inputs/enukx_temperature_thresholds.nc"
        kgo_path = kgo_dir / "outputs/extract_percentile_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            threshold_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "50",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:240: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw9/test_percentile_thresholded_in0/output.nc --extract-percentiles 50 --new-title 'IMPROVER UK Spot Values'
__________________ test_multiple_percentile_realization_input __________________
[gw14] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_multiple_percentile_reali0')

    def test_multiple_percentile_realization_input(tmp_path):
        """Test extracting multiple percentiles from realization input"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        realization_path = kgo_dir / "inputs/enukx_temperature_realizations.nc"
        kgo_path = kgo_dir / "outputs/extract_multiple_percentiles_kgo.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            realization_path,
            "--output",
            output_path,
            "--extract-percentiles",
            "25, 50, 75",
            "--new-title",
            UK_SPOT_TITLE,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:417: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_temperature_realizations.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw14/test_multiple_percentile_reali0/output.nc --extract-percentiles '25, 50, 75' --new-title 'IMPROVER UK Spot Values'
_______________ test_percentile_from_threshold_with_realizations _______________
[gw1] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_from_threshold0')

    def test_percentile_from_threshold_with_realizations(tmp_path):
        """Test requesting a percentile from a cube with thresholds where the realizations
        need collapsing first"""
        kgo_dir = acc.kgo_root() / "spot-extract"
        neighbour_path = kgo_dir / "inputs/all_methods_uk.nc"
        diag_path = kgo_dir / "inputs/enukx_preciprate_realizations_thresholds.nc"
        kgo_path = kgo_dir / "outputs/with_realization_collapse.nc"
        output_path = tmp_path / "output.nc"
        args = [
            neighbour_path,
            diag_path,
            "--output",
            output_path,
            "--realization-collapse",
            "--extract-percentiles",
            "50",
            "--new-title",
            "MOGREPS-UK Spot Values",
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_spot_extract.py:480: AssertionError
----------------------------- Captured stdout call -----------------------------
improver spot-extract /mnt/data/dmentipl/improver_tests/spot-extract/inputs/all_methods_uk.nc /mnt/data/dmentipl/improver_tests/spot-extract/inputs/enukx_preciprate_realizations_thresholds.nc --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw1/test_percentile_from_threshold0/output.nc --realization-collapse --extract-percentiles 50 --new-title 'MOGREPS-UK Spot Values'
_______________________ test_normal_point_by_point_sites _______________________
[gw10] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw10/test_normal_point_by_point_sit0')

    @pytest.mark.slow
    def test_normal_point_by_point_sites(tmp_path):
        """
        Test estimate-emos-coefficients for diagnostic with assumed
        normal distribution where coefficients are computed independently at each
        site location (initial guess and minimisation).
        """
        kgo_dir = acc.kgo_root() / "estimate-emos-coefficients/normal/sites"
        kgo_path = kgo_dir / "point_by_point" / "kgo.nc"
        history_path = kgo_dir / "history/*.nc"
        truth_path = kgo_dir / "truth/*.nc"
        output_path = tmp_path / "output.nc"
        est_emos_tol = str(0.01)
        compare_emos_tolerance = 0.1
        args = [
            history_path,
            truth_path,
            "--distribution",
            "norm",
            "--truth-attribute",
            "mosg__model_configuration=uk_det",
            "--tolerance",
            est_emos_tol,
            "--point-by-point",
            "--output",
            output_path,
        ]
        run_cli(args)
        acc.compare(
>           output_path, kgo_path, atol=compare_emos_tolerance, rtol=compare_emos_tolerance
        )
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_estimate_emos_coefficients.py:296: AssertionError
----------------------------- Captured stdout call -----------------------------
improver estimate-emos-coefficients '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/history/*.nc' '/mnt/data/dmentipl/improver_tests/estimate-emos-coefficients/normal/sites/truth/*.nc' --distribution norm --truth-attribute mosg__model_configuration=uk_det --tolerance 0.01 --point-by-point --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw10/test_normal_point_by_point_sit0/output.nc
____________________ test_nearest_minimum_dz[global-global] ____________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_global0')
domain = 'global', model = 'global'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with minimum altitude difference"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:104: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_minimum_dz_global0/output.nc
____________________ test_nearest_land_minimum_dz[uk-ukvx] _____________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_u0')
domain = 'uk', model = 'ukvx'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with land constraint plus min altitude diff"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_constraint_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        if domain == "uk":
            args += ["--node-limit", "100"]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:131: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/ukvx_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/uk_sites.json --land-constraint --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_u0/output.nc --node-limit 100
_________________ test_nearest_land_minimum_dz[global-global] __________________
[gw12] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_g0')
domain = 'global', model = 'global'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_nearest_land_minimum_dz(tmp_path, domain, model):
        """Test neighbour finding with land constraint plus min altitude diff"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/nearest_land_constraint_minimum_dz_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--land-constraint",
            "--similar-altitude",
            "--search-radius",
            "50000",
            "--output",
            output_path,
        ]
        if domain == "uk":
            args += ["--node-limit", "100"]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /wmo_id - ['units']

improver_tests/acceptance/test_neighbour_finding.py:131: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --land-constraint --similar-altitude --search-radius 50000 --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw12/test_nearest_land_minimum_dz_g0/output.nc
_______________________ test_all_methods[global-global] ________________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_global_global0')
domain = 'global', model = 'global'

    @pytest.mark.slow
    @pytest.mark.parametrize("domain,model", UK_GLOBAL)
    def test_all_methods(tmp_path, domain, model):
        """Test neighbour finding using all methods"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        kgo_path = kgo_dir / f"outputs/all_methods_{domain}_kgo.nc"
        sites_path = kgo_dir / f"inputs/{domain}_sites.json"
        orography_path = kgo_dir / f"inputs/{model}_orography.nc"
        landmask_path = kgo_dir / f"inputs/{model}_landmask.nc"
        output_path = tmp_path / "output.nc"
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--all-methods",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /neighbour_selection_method_name - ['units']

improver_tests/acceptance/test_neighbour_finding.py:153: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_sites.json --all-methods --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_all_methods_global_global0/output.nc
______________________ test_alternative_coordinate_system ______________________
[gw0] linux -- Python 3.7.10 /home/dmentipl/conda/envs/improver_py37_iris30/bin/python

tmp_path = PosixPath('/tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_alternative_coordinate_sy0')

    def test_alternative_coordinate_system(tmp_path):
        """Test use of an alternative coordinate system"""
        kgo_dir = acc.kgo_root() / "neighbour-finding"
        # same KGO as test_nearest global domain
        kgo_path = kgo_dir / "outputs/nearest_global_kgo.nc"
        sites_path = kgo_dir / "inputs/LAEA_grid_sites.json"
        orography_path = kgo_dir / "inputs/global_orography.nc"
        landmask_path = kgo_dir / "inputs/global_landmask.nc"
        output_path = tmp_path / "output.nc"
        coord_opts = {
            "central_latitude": 54.9,
            "central_longitude": -2.5,
            "false_easting": 0.0,
            "false_northing": 0.0,
            "globe": {"semimajor_axis": 6378137.0, "semiminor_axis": 6356752.314140356},
        }
        coord_opts_json = json.dumps(coord_opts)
        args = [
            orography_path,
            landmask_path,
            sites_path,
            "--site-coordinate-system",
            "LambertAzimuthalEqualArea",
            "--site-coordinate-options",
            coord_opts_json,
            "--site-x-coordinate",
            "projection_x_coordinate",
            "--site-y-coordinate",
            "projection_y_coordinate",
            "--output",
            output_path,
        ]
        run_cli(args)
>       acc.compare(output_path, kgo_path)
E       AssertionError: different attributes of /neighbour_selection_method_name - ['units']

improver_tests/acceptance/test_neighbour_finding.py:189: AssertionError
----------------------------- Captured stdout call -----------------------------
improver neighbour-finding /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_orography.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/global_landmask.nc /mnt/data/dmentipl/improver_tests/neighbour-finding/inputs/LAEA_grid_sites.json --site-coordinate-system LambertAzimuthalEqualArea --site-coordinate-options '{"central_latitude": 54.9, "central_longitude": -2.5, "false_easting": 0.0, "false_northing": 0.0, "globe": {"semimajor_axis": 6378137.0, "semiminor_axis": 6356752.314140356}}' --site-x-coordinate projection_x_coordinate --site-y-coordinate projection_y_coordinate --output /tmp/pytest-of-dmentipl/pytest-46/popen-gw0/test_alternative_coordinate_sy0/output.nc
=========================== short test summary info ============================
FAILED improver_tests/acceptance/test_create_grid_with_halo.py::test_basic - ...
FAILED improver_tests/acceptance/test_apply_emos_coefficients.py::test_normal_point_by_point_sites
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest[global-global]
FAILED improver_tests/acceptance/test_create_grid_with_halo.py::test_halo_size
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_invalid_site
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_coord_beyond_bounds
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_unset_wmo_ids_with_unique_ids
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land[global-global]
FAILED improver_tests/acceptance/test_estimate_emos_coefficients.py::test_normal_point_by_point_default_initial_guess_sites
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_minimum_dz[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_all_methods[uk-ukvx]
FAILED improver_tests/acceptance/test_spot_extract.py::test_nearest_uk[all_methods_uk.nc-extra_args0-nearest_uk_temperatures.nc]
FAILED improver_tests/acceptance/test_spot_extract.py::test_nearest_uk[all_methods_uk.nc-extra_args1-mindz_uk_temperatures.nc]
FAILED improver_tests/acceptance/test_spot_extract.py::test_lapse_rate_adjusted_uk
FAILED improver_tests/acceptance/test_spot_extract.py::test_nearest_uk[all_methods_uk_unique_ids.nc-extra_args2-nearest_uk_temperatures_unique_ids.nc]
FAILED improver_tests/acceptance/test_spot_extract.py::test_new_spot_title - ...
FAILED improver_tests/acceptance/test_spot_extract.py::test_lapse_rate_mismatch
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_percentile_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_deterministic
FAILED improver_tests/acceptance/test_extract.py::test_spot - AssertionError:...
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_thresholded_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_deterministic_quiet
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_percentile_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_constraints
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_realization_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_thresholded_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_multiple_percentile_realization_input
FAILED improver_tests/acceptance/test_spot_extract.py::test_percentile_from_threshold_with_realizations
FAILED improver_tests/acceptance/test_estimate_emos_coefficients.py::test_normal_point_by_point_sites
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_minimum_dz[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land_minimum_dz[uk-ukvx]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_nearest_land_minimum_dz[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_all_methods[global-global]
FAILED improver_tests/acceptance/test_neighbour_finding.py::test_alternative_coordinate_system
= 36 failed, 305 passed, 4 skipped, 2 xfailed, 492 warnings in 137.60s (0:02:17) =

Thanks @dmentipl. This looks like good progress on support for Iris 3 and the changes needed look a lot less difficult than I was worried might be needed.
I think it'd be good to add tests/support for Python 3.8, but tackle that in a separate follow up PR to this Iris 3 work. I expect that the main constraint there will be library builds available on conda-forge rather than any breaking changes in the Python language.
I think the main outstanding questions are:

Is a hard changeover in Iris version (no overlap period with support for both Iris 2 and Iris 3) OK for Met Office usage?
The 36 failing acceptance tests all relate to the 'units' attribute. I'm not too worried by these differences as the NetCDF variables where the 'units' attribute is no longer present all seem to be coordinate variables where units aren't well defined. For example, there aren't really any physical units for the WMO ID number of a site, or the neighbour selection method. Is this metadata change OK for Met Office usage?
Changes to the pinned scipy library version will change some acceptance test outputs. See #1398 and #1397 for details. I think the change to Iris 3 is a good opportunity to move to a newer scipy at the same time. Are the Met Office team happy to move to a newer scipy or are there other python stack limitations?
Test that the Nimrod changes in Iris 3 cover the existing monkeypatches in IMPROVER and it's safe to remove iris_nimrod_patcher. We don't use this at BOM and the relevant tests get skipped by pytest if this library isn't available, so Nimrod handling will likely need some testing from the MO team.

I have added a Python 3.8 environment. It is passing the unit tests. It fails the same acceptance tests as the Python 3.7 environment. The changes from the 3.7 env (other than Python version) are:

unpin cftime
use cf-units rather than cf_units

Hard changeover to Iris 3 - in principle fine, as so far it looks like migrating to it is mandatory this year.
wmo_id units - what are the changes exactly? You are right that units just don't make sense for this, and as far as I can tell they are optional in CF for this kind of variable anyway.
scipy version - we'd expect to move to ~1.6.2 with Iris 3 at the moment?

Hard changeover to Iris 3 - in principle fine

That's good to know.

what are the changes exactly?

For '/wmo_id', and some others, the desired attribute dictionary is
{'units': '1', 'long_name': 'wmo_id'}

whereas the actual is
{'long_name': 'wmo_id'}

So it's just missing the units. I'm not sure why this is the case.

scipy version - we'd expect to move to ~1.6.2 with Iris 3

I've updated the scipy version in both environments. It leads to a few (8) more acceptance test failures (as @tjtg said above).

I've now removed nimrod patching. Unit tests still pass, and the acceptance test failures are the same as prior.

@dmentipl I have (scandalously) force pushed over your branch after rebasing and making some changes. The changes I have made are as follows:

Updated the version of pysteps to match our HPC environment.

This required a small tweak to the code to set the interpolation mode
the environments were updated too.

Updated the checksums for the acceptance test data.

Tom can hopefully give you access to the new data in branch im1517 of the acceptance test data repo.
There are details about what changed here https://github.com/metoppv/improver/issues/1517 where most were the unit change you noted, but we are also jumping forward to newer scipy in our HPC env, which affects interpolation as @tjtg noted above.

I have removed an IMPROVER function called equalise_cube_attributes as there is now functionality to do this within Iris which I have adopted.
Removed the iris2 environments from /envs as these will no longer work with IMPROVER, and added pytest-xdist to the envs as well to ensure acceptance tests can be run in parallel (lest any user grows old waiting for them to run sequentially).

I've run all the tests with both iris3 environments and everything passes as expected, as well as with our local environment. I still need to do a bit more testing on the HPC before this can all be merged at our end, but hopefully none of the changes I've made cause you any problems.
I've promoted this PR to ready for review. Could you check that what I've done is okay for you guys. I'll finish up the HPC changes and get someone to look at this from our end as well. Then when it gets merged we will finally leave iris 2 behind for good.

Thanks for making those changes @bayliffe 👍
They look good, in principle, from my perspective. However, even though the tests are passing in GitHub actions, I'm getting four unit test failures locally (using the conda environment files) in nbhood/circular_kernel. I need to investigate further. I suspect there is an unpinned package causing problems.

Tom can hopefully give you access to the new data in branch im1517 of the acceptance test data repo.

Yes, I've got access, and I'm in the process of testing these.
Here are the details of the unit-test failures referenced above (for the Python 3.8 env; the errors are the same for the Python 3.7 env):

================================================ FAILURES =================================================
___________________________ Test_pad_and_unpad_cube.test_single_point_on_corner ___________________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_pad_and_unpad_cube testMethod=test_single_point_on_corner>

    def test_single_point_on_corner(self):
        """Test behaviour for a single non-zero grid cell on the corner."""

        expected = np.array(
            [
                [
                    [0.0, 0.4, 1.0, 1.0, 1.0],
                    [0.4, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [0.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                ],
            ]
        )

        self.cube.data[0, 0] = 0

        # Point is right on the corner.
        percentiles = np.array([10, 50, 90])
        kernel = np.array([[0.0, 1.0, 0.0], [1.0, 1.0, 1.0], [0.0, 1.0, 0.0]])
        result = GeneratePercentilesFromACircularNeighbourhood(
            percentiles=percentiles
        ).pad_and_unpad_cube(self.cube, kernel)
>       self.assertArrayAlmostEqual(result.data, expected)

improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:347:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_pad_and_unpad_cube testMethod=test_single_point_on_corner>
a = array([[[0. , 0.4, 1. , 1. , 1. ],
        [0.4, 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. ,...      [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[0. , 0.4, 1. , 1. , 1. ],
        [0.4, 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. ,..., 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 1 / 75 (1.33%)
E       Max absolute difference: 1.
E       Max relative difference: 0.00000001
E        x: array([[[0. , 0.4, 1. , 1. , 1. ],
E               [0.4, 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. ],...
E        y: array([[[0. , 0.4, 1. , 1. , 1. ],
E               [0.4, 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. ],...

../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
_______________________ Test_run.test_number_of_percentiles_equals_number_of_points _______________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_percentiles_equals_number_of_points>

    def test_number_of_percentiles_equals_number_of_points(self):
        """Test when the number of percentiles is equal to the number of points
        used to construct the percentiles."""

        expected = np.array(
            [
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.2, 0.2, 0.2, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.4, 0.4, 0.4, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.6, 0.6, 0.6, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.8, 0.8, 0.8, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
            ]
        )

        data = np.ones((7, 7), dtype=np.float32)
        data[3, 3] = 0
        cube = set_up_variable_cube(data, spatial_grid="equalarea",)
        percentiles = np.array([5, 10, 15, 20, 25])
        radius = 2000.0
        result = GeneratePercentilesFromACircularNeighbourhood(
            percentiles=percentiles
        ).run(cube, radius)
>       self.assertArrayAlmostEqual(result.data, expected)

improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:715:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_percentiles_equals_number_of_points>
a = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.2,..., 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.2,..., 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 1 / 245 (0.408%)
E       Max absolute difference: 0.6
E       Max relative difference: 1.5
E        x: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 0.2, 1. , 1. , 1. ],...
E        y: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 0.2, 1. , 1. , 1. ],...

../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
______________________ Test_run.test_number_of_points_half_of_number_of_percentiles _______________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_points_half_of_number_of_percentiles>

    def test_number_of_points_half_of_number_of_percentiles(self):
        """Test when the number of points is half the number of percentiles."""

        expected = np.array(
            [
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.1, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.1, 0.1, 0.1, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.1, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.2, 0.2, 0.2, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.2, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.3, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.3, 0.3, 0.3, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.3, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.4, 0.4, 0.4, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.4, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.5, 0.5, 0.5, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.5, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.6, 0.6, 0.6, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.6, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.7, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.7, 0.7, 0.7, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.7, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.8, 0.8, 0.8, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.8, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.9, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.9, 0.9, 0.9, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 0.9, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
                ],
            ]
        )

        data = np.ones((7, 7), dtype=np.float32)
        data[3, 3] = 0
        cube = set_up_variable_cube(data, spatial_grid="equalarea",)

        percentiles = np.array([2.5, 5, 7.5, 10, 12.5, 15, 17.5, 20, 22.5, 25])
        radius = 2000.0
        result = GeneratePercentilesFromACircularNeighbourhood(
            percentiles=percentiles
        ).run(cube, radius)
>       self.assertArrayAlmostEqual(result.data, expected)

improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:824:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_number_of_points_half_of_number_of_percentiles>
a = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.1,..., 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 0.1,..., 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 9 / 490 (1.84%)
E       Max absolute difference: 0.8
E       Max relative difference: 4.
E        x: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 0.1, 1. , 1. , 1. ],...
E        y: array([[[1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 1. , 0.1, 1. , 1. , 1. ],...

../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
_______________________________ Test_run.test_single_point_low_percentiles ________________________________
[gw0] linux -- Python 3.8.10 /home/dmentipl/conda/envs/improver_py38_iris30/bin/python

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_single_point_low_percentiles>

    def test_single_point_low_percentiles(self):
        """Test behaviour with low percentiles."""

        expected = np.array(
            [
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.2, 1.0, 1.0],
                    [1.0, 0.2, 0.2, 0.2, 1.0],
                    [1.0, 1.0, 0.2, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.4, 1.0, 1.0],
                    [1.0, 0.4, 0.4, 0.4, 1.0],
                    [1.0, 1.0, 0.4, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                ],
                [
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                    [1.0, 1.0, 0.8, 1.0, 1.0],
                    [1.0, 0.8, 0.8, 0.8, 1.0],
                    [1.0, 1.0, 0.8, 1.0, 1.0],
                    [1.0, 1.0, 1.0, 1.0, 1.0],
                ],
            ]
        )
        self.cube.data[2, 2] = 0

        percentiles = np.array([5, 10, 20])
        radius = 2000.0
        result = GeneratePercentilesFromACircularNeighbourhood(
            percentiles=percentiles
        ).run(self.cube, radius)
>       self.assertArrayAlmostEqual(result.data, expected)

improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py:613:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <improver_tests.nbhood.circular_kernel.test_GeneratePercentilesFromACircularNeighbourhood.Test_run testMethod=test_single_point_low_percentiles>
a = array([[[1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 0.2, 1. , 1. ],
        [1. , 0.2, 0.2, 0.2, 1. ],
        [1. ,...      [1. , 0.8, 0.8, 0.8, 1. ],
        [1. , 1. , 0.8, 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]], dtype=float32)
b = array([[[1. , 1. , 1. , 1. , 1. ],
        [1. , 1. , 0.2, 1. , 1. ],
        [1. , 0.2, 0.2, 0.2, 1. ],
        [1. ,..., 1. , 1. ],
        [1. , 0.8, 0.8, 0.8, 1. ],
        [1. , 1. , 0.8, 1. , 1. ],
        [1. , 1. , 1. , 1. , 1. ]]])
decimal = 6

    def assertArrayAlmostEqual(self, a, b, decimal=6):
>       np.testing.assert_array_almost_equal(a, b, decimal=decimal)
E       AssertionError:
E       Arrays are not almost equal to 6 decimals
E
E       Mismatched elements: 2 / 75 (2.67%)
E       Max absolute difference: 0.6
E       Max relative difference: 1.5
E        x: array([[[1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 0.2, 1. , 1. ],
E               [1. , 0.2, 0.2, 0.2, 1. ],...
E        y: array([[[1. , 1. , 1. , 1. , 1. ],
E               [1. , 1. , 0.2, 1. , 1. ],
E               [1. , 0.2, 0.2, 0.2, 1. ],...

../../conda/envs/improver_py38_iris30/lib/python3.8/site-packages/iris/tests/__init__.py:670: AssertionError
========================================= short test summary info =========================================
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_pad_and_unpad_cube::test_single_point_on_corner
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_number_of_percentiles_equals_number_of_points
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_number_of_points_half_of_number_of_percentiles
FAILED improver_tests/nbhood/circular_kernel/test_GeneratePercentilesFromACircularNeighbourhood.py::Test_run::test_single_point_low_percentiles
================= 4 failed, 2428 passed, 319 skipped, 1 xfailed, 3568 warnings in 48.88s ==================

I've determined that the errors in the unit tests reported above are due to the version of NumPy. In my local testing environment I am using 1.21.0, whereas the version in the conda environment on GitHub is 1.20.3. See below for a diff of the conda environments.
We can pin numpy to 1.20.* if that satisfies Met Office requirements. Or else, dig a bit more to see what the issue is with circular kernel neighbourhood-ing. @bayliffe please let me know what works for the Met Office.

diff --git a/home/dmentipl/tmp/improver_py38_iris30-github.txt b/home/dmentipl/tmp/improver_py38_iris30-local.txt
index 76989d40..53f60e2c 100644
--- a/home/dmentipl/tmp/improver_py38_iris30-github.txt
+++ b/home/dmentipl/tmp/improver_py38_iris30-local.txt
@@ -1,4 +1,4 @@
-# packages in environment at /usr/share/miniconda/envs/improver_py38_iris30:
+# packages in environment at /home/dmentipl/conda/envs/improver_py38_iris30:
 #
 # Name                    Version                   Build  Channel
 _libgcc_mutex             0.1                 conda_forge    conda-forge
@@ -7,7 +7,7 @@ alabaster                 0.7.12                     py_0    conda-forge
 alsa-lib                  1.2.3                h516909a_0    conda-forge
 antlr-python-runtime      4.7.2           py38h578d9bd_1002    conda-forge
 appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
-astroid                   2.5.8            py38h578d9bd_0    conda-forge
+astroid                   2.6.0            py38h578d9bd_0    conda-forge
 attrs                     19.3.0                     py_0    conda-forge
 babel                     2.9.1              pyh44b312d_0    conda-forge
 bandit                    1.7.0            py38h578d9bd_0    conda-forge
@@ -34,10 +34,10 @@ cryptography              3.4.7            py38ha5dfef3_0    conda-forge
 curl                      7.77.0               hea6ffbf_0    conda-forge
 cycler                    0.10.0                     py_2    conda-forge
 cytoolz                   0.11.0           py38h497a2fe_3    conda-forge
-dask                      2021.6.1           pyhd8ed1ab_0    conda-forge
-dask-core                 2021.6.1           pyhd8ed1ab_0    conda-forge
-dbus                      1.13.18              hb2f20db_0
-distributed               2021.6.1         py38h578d9bd_0    conda-forge
+dask                      2021.6.2           pyhd8ed1ab_0    conda-forge
+dask-core                 2021.6.2           pyhd8ed1ab_0    conda-forge
+dbus                      1.13.6               h48d8840_2    conda-forge
+distributed               2021.6.2         py38h578d9bd_0    conda-forge
 docutils                  0.17.1           py38h578d9bd_0    conda-forge
 dparse                    0.5.1                      py_0    conda-forge
 execnet                   1.9.0              pyhd8ed1ab_0    conda-forge
@@ -49,14 +49,14 @@ fontconfig                2.13.1            hba837de_1005    conda-forge
 freetype                  2.10.4               h0708190_1    conda-forge
 fsspec                    2021.6.0           pyhd8ed1ab_0    conda-forge
 geos                      3.9.1                h9c3ff4c_2    conda-forge
-gettext                   0.21.0               hf68c758_0
+gettext                   0.19.8.1          h0b5b191_1005    conda-forge
 gitdb                     4.0.7              pyhd8ed1ab_0    conda-forge
 gitpython                 3.1.18             pyhd8ed1ab_0    conda-forge
 glib                      2.68.3               h9c3ff4c_0    conda-forge
 glib-tools                2.68.3               h9c3ff4c_0    conda-forge
 gmp                       6.2.1                h58526e2_0    conda-forge
-gnutls                    3.6.15               he1e5248_0
-graphite2                 1.3.14               h23475e2_0
+gnutls                    3.6.13               h85f3911_1    conda-forge
+graphite2                 1.3.13            he1b5a44_1001    conda-forge
 gst-plugins-base          1.18.4               hf529b03_2    conda-forge
 gstreamer                 1.18.4               h76c114f_2    conda-forge
 harfbuzz                  2.8.1                h83ec7ef_0    conda-forge
@@ -90,7 +90,7 @@ libcblas                  3.9.0                9_openblas    conda-forge
 libclang                  11.1.0          default_ha53f305_1    conda-forge
 libcurl                   7.77.0               h2574ce0_0    conda-forge
 libdeflate                1.7                  h7f98852_5    conda-forge
-libedit                   3.1.20210216         h27cfd23_1
+libedit                   3.1.20191231         he28a2e2_2    conda-forge
 libev                     4.33                 h516909a_1    conda-forge
 libevent                  2.1.10               hcdb4288_3    conda-forge
 libffi                    3.3                  h58526e2_2    conda-forge
@@ -100,14 +100,13 @@ libgfortran5              9.3.0               hff62375_19    conda-forge
 libglib                   2.68.3               h3e27bee_0    conda-forge
 libgomp                   9.3.0               h2828fa1_19    conda-forge
 libiconv                  1.16                 h516909a_0    conda-forge
-libidn2                   2.3.1                h7f98852_0    conda-forge
 liblapack                 3.9.0                9_openblas    conda-forge
 liblapacke                3.9.0                9_openblas    conda-forge
 libllvm10                 10.0.1               he513fc3_3    conda-forge
 libllvm11                 11.1.0               hf817b99_2    conda-forge
 libnetcdf                 4.8.0           nompi_hcd642e3_103    conda-forge
 libnghttp2                1.43.0               h812cca2_0    conda-forge
-libogg                    1.3.5                h27cfd23_1
+libogg                    1.3.4                h7f98852_1    conda-forge
 libopenblas               0.3.15          pthreads_h8fe5266_1    conda-forge
 libopencv                 4.5.2            py38hcdf9bf1_0    conda-forge
 libopus                   1.3.1                h7f98852_1    conda-forge
@@ -116,18 +115,16 @@ libpq                     13.3                 hd57d9b9_0    conda-forge
 libprotobuf               3.15.8               h780b84a_0    conda-forge
 libssh2                   1.9.0                ha56f1ee_6    conda-forge
 libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
-libtasn1                  4.16.0               h27cfd23_0
 libtiff                   4.3.0                hf544144_1    conda-forge
-libunistring              0.9.10               h14c3975_0    conda-forge
 libuuid                   2.32.1            h14c3975_1000    conda-forge
 libvorbis                 1.3.7                he1b5a44_0    conda-forge
 libwebp-base              1.2.0                h7f98852_2    conda-forge
-libxcb                    1.14                 h7b6447c_0
+libxcb                    1.13              h7f98852_1003    conda-forge
 libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
 libxml2                   2.9.12               h72842e0_0    conda-forge
 libzip                    1.8.0                h4de3113_0    conda-forge
 llvmlite                  0.36.0           py38h4630a5e_0    conda-forge
-locket                    0.2.1            py38h06a4308_1
+locket                    0.2.0                      py_2    conda-forge
 lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
 markupsafe                2.0.1            py38h497a2fe_0    conda-forge
 matplotlib-base           3.4.2            py38hcc49a3a_0    conda-forge
@@ -135,17 +132,17 @@ mccabe                    0.6.1                      py_1    conda-forge
 mock                      4.0.3            py38h578d9bd_1    conda-forge
 more-itertools            8.8.0              pyhd8ed1ab_0    conda-forge
 msgpack-python            1.0.2            py38h1fd1430_1    conda-forge
-mypy                      0.902            py38h497a2fe_0    conda-forge
+mypy                      0.910            py38h497a2fe_0    conda-forge
 mypy_extensions           0.4.3            py38h578d9bd_3    conda-forge
 mysql-common              8.0.25               ha770c72_2    conda-forge
 mysql-libs                8.0.25               hfa10184_2    conda-forge
 ncurses                   6.2                  h58526e2_4    conda-forge
 netcdf4                   1.5.7           nompi_py38h5e9db54_100    conda-forge
-nettle                    3.7.3                hbbd107a_1
+nettle                    3.6                  he412f7d_0    conda-forge
 nspr                      4.30                 h9c3ff4c_0    conda-forge
 nss                       3.67                 hb5efdd6_0    conda-forge
 numba                     0.53.1           py38h8b71fd7_1    conda-forge
-numpy                     1.20.3           py38h9894fe3_1    conda-forge
+numpy                     1.21.0           py38h9894fe3_0    conda-forge
 od                        1.0                        py_0    conda-forge
 olefile                   0.46               pyh9f0ad1d_1    conda-forge
 opencv                    4.5.2            py38h578d9bd_0    conda-forge
@@ -153,7 +150,7 @@ openh264                  2.1.1                h780b84a_0    conda-forge
 openjpeg                  2.4.0                hb52868f_1    conda-forge
 openssl                   1.1.1k               h7f98852_0    conda-forge
 packaging                 20.9               pyh44b312d_0    conda-forge
-pandas                    1.2.4            py38h1abd341_0    conda-forge
+pandas                    1.2.5            py38h1abd341_0    conda-forge
 partd                     1.2.0              pyhd8ed1ab_0    conda-forge
 pathspec                  0.8.1              pyhd3deb0d_0    conda-forge
 patsy                     0.5.1                      py_0    conda-forge
@@ -165,6 +162,7 @@ pixman                    0.40.0               h36c2ea0_0    conda-forge
 pluggy                    0.13.1           py38h578d9bd_4    conda-forge
 proj                      7.2.0                h277dcde_2    conda-forge
 psutil                    5.8.0            py38h497a2fe_1    conda-forge
+pthread-stubs             0.4               h36c2ea0_1001    conda-forge
 py                        1.10.0             pyhd3deb0d_0    conda-forge
 py-opencv                 4.5.2            py38hd0cf306_0    conda-forge
 pycodestyle               2.7.0              pyhd8ed1ab_0    conda-forge
@@ -196,7 +194,7 @@ regex                     2021.4.4         py38h497a2fe_0    conda-forge
 requests                  2.25.1             pyhd3deb0d_0    conda-forge
 safety                    1.10.3             pyhd8ed1ab_0    conda-forge
 scipy                     1.6.3            py38h7b17777_0    conda-forge
-setuptools                52.0.0           py38h06a4308_0
+setuptools                49.6.0           py38h578d9bd_3    conda-forge
 shapely                   1.7.1            py38haeee4fe_5    conda-forge
 sigtools                  2.0.2            py38h578d9bd_3    conda-forge
 six                       1.16.0             pyh6c4a22f_0    conda-forge
@@ -231,6 +229,8 @@ xorg-kbproto              1.0.7             h14c3975_1002    conda-forge
 xorg-libice               1.0.10               h516909a_0    conda-forge
 xorg-libsm                1.2.3             hd9c2040_1000    conda-forge
 xorg-libx11               1.7.2                h7f98852_0    conda-forge
+xorg-libxau               1.0.9                h14c3975_0    conda-forge
+xorg-libxdmcp             1.1.3                h516909a_0    conda-forge
 xorg-libxext              1.3.4                h7f98852_1    conda-forge
 xorg-libxrender           0.9.10            h7f98852_1003    conda-forge
 xorg-renderproto          0.11.1            h14c3975_1002    conda-forge

Further digging shows in test_single_point_low_percentiles, for example, a difference between the expected:
[[[1.  1.  1.  1.  1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  0.2 0.2 0.2 1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  1.  1.  1. ]
  [1.  1.  0.4 1.  1. ]
  [1.  0.4 0.4 0.4 1. ]
  [1.  1.  0.4 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  1.  1.  1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  0.8 0.8 0.8 1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  1.  1.  1.  1. ]]]

and the result (note the misplaced 0.8 at [1, 0, 2] and 1.0 at [1, 1, 2]):
[[[1.  1.  1.  1.  1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  0.2 0.2 0.2 1. ]
  [1.  1.  0.2 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  0.8 1.  1. ]
  [1.  1.  1.  1.  1. ]
  [1.  0.4 0.4 0.4 1. ]
  [1.  1.  0.4 1.  1. ]
  [1.  1.  1.  1.  1. ]]

 [[1.  1.  1.  1.  1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  0.8 0.8 0.8 1. ]
  [1.  1.  0.8 1.  1. ]
  [1.  1.  1.  1.  1. ]]]

@dmentipl thanks for looking into this. Let's pin the version of numpy in the envs for now and move this work forward. If you could then create a ticket in this repo explaining what you found (or linking to the comments above), we can address this issue in slower time before we are compelled to move to numpy 1.21.0. Sound okay?

@bayliffe no worries.
I've pinned numpy to 1.20.*, and raised an issue for dealing with moving to 1.21 in the future (#1522).
Also, I've pinned cf-units to 2.1.5. Version 3.0.0 was released overnight and leads to 48 unit test failures. I've raised an issue for this too (#1523).

We (Mostly @MoseleyS) have found a problem between Iris 3 and  wind_components.py: line 212.
Iris adds status_flag to the coord name so it is no longer wind_speed.
This then throws the error ValueError: Wind speed and direction cubes have unmatched coordinates.
This is now being looked into so it can be replicated with acceptance tests.

@tjtg @dmentipl Would it be possible to update improver branch iris3 to include changes in master (in particular I am interested in including #1528)? We are using improver/iris3 branch in raincal CIs but need #1528 - thanks.
cc @benowen-bom

@lucyleeow I've rebased against master and force pushed.

I have pinned cftime to < 1.5 as compatibility with cf-units 2.1.5 breaks.

authors: bayliffe benfitzpatrick dmentipl lucyleeow neilCrosswaite tjtg