text:
Istio stores duplicate certificates as seen in the /certs endpoint

Bug Description
Issue 1: When istio proxy comes up for the first time, and it is given certificates via SDS, it was observed that it stores duplicate entries of the certificate as seen in the /certs endpoint.
Issue 2: Whenever the new certificate has a different serial number, then envoy is serving on expired certificate, even though it has the new certificate in its memory as seen in the /certs endpoint.
Version
$ istioctl version
1.13.4

$ kubectl version
1.21 (both)

Additional Information
Some parsed data from the /certs endpoint:
╰─$ cat ratings-v1-b6994bb9-vjhtq.certs.json| grep cert_chain | wc -l
      29
╭─aaeron@macos-C02G72ZBMD6T ~/workspace/setup/istio/releases/istio-1.13.4
╰─$ cat ratings-v1-b6994bb9-vjhtq.certs.json| grep cert_chain -A 3 | grep serial | wc -l
      29
╭─aaeron@macos-C02G72ZBMD6T ~/workspace/setup/istio/releases/istio-1.13.4
╰─$ cat ratings-v1-b6994bb9-vjhtq.certs.json| grep cert_chain -A 3 | grep serial | sort -u
     "serial_number": "26931ad6c653ea029212660e6edb781c",

The above command show that there are 29 instances of cert_chain, even though the serial number of all the certificates are the same. Which means that /certs endpoint is storing duplicate entries.
Data from /certs endpoint in cluster created via kind.
{
 "certificates": [
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "e2110e139cc6cc7a",
     "subject_alt_names": [],
     "days_until_expiration": "34863",
     "valid_from": "2018-01-24T19:15:51Z",
     "expiration_time": "2117-12-31T19:15:51Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "26931ad6c653ea029212660e6edb781c",
     "subject_alt_names": [
      {
       "uri": "spiffe://cluster.local/ns/bookinfo/sa/bookinfo-ratings"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T05:18:54Z",
     "expiration_time": "2022-07-20T05:20:54Z"
    }
   ]
  }
 ]
}

can you show the config_dump

Duplicate certs in /certs endpoint is expected if the certs are used across many clusters and listeners. That shows certs of each SSL context that Envoy creates.

Duplicate certs in /certs endpoint is expected if the certs are used across many clusters and listeners. That shows certs of each SSL context that Envoy creates.

Got it, thanks for the clarification. But then regarding Issue 2, mentioned in the description, why is envoy still serving on stale certificates?

Whenever the new certificate has a different serial number, then envoy is serving on expired certificate, even though it has the new certificate in its memory as seen in the /certs endpoint.

That is because the connections that used old certificate during handshake will continue to use them till they are  destroyed.

But this is causing MTLS to fail, when a client is connecting to an upstream, which is serving on expired certificates.

Yes it is expected behaviour if the other side is not on Mesh and validating expiry.
See https://github.com/istio/istio/issues/7630.
https://github.com/istio/api/pull/2341
you will have to set this https://github.com/istio/api/blob/master/networking/v1alpha3/destination_rule.proto#L707 so that the connection terminates before cert expires.

I see this even for new connections.

It should not if certs are updated correctly. Config dump will help.

This (https://github.com/istio/api/blob/87dfc8c8c07ae884e54b40246dc179ca97e58a7e/networking/v1alpha3/destination_rule.proto#L707) should only be applicable to long running connections right?

Every connection is a long running. Envoy uses pool of connections that gets terminated only on errors or via some config above.

I see this even for new connections.

It should not if certs are updated correctly. Config dump will help.

This (https://github.com/istio/api/blob/87dfc8c8c07ae884e54b40246dc179ca97e58a7e/networking/v1alpha3/destination_rule.proto#L707) should only be applicable to long running connections right?

Every connection is a long running. Envoy uses pool of connections that gets terminated only on errors or via some config above.

I always see that envoy is able to load the latest certificates in memory (as seen in /certs endpoint), but for some reason it does not flush out the old certificates, and continues to serve on them, even after a day.

can you show the config_dump

Just the secrets section?

This is the dynamic secret part of the config dump:
{
  "@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump",
  "dynamic_active_secrets": [
    {
      "name": "default",
      "last_updated": "2022-07-19T19:32:30.735Z",
      "secret": {
        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
        "name": "default",
        "tls_certificate": {
          "certificate_chain": {
            "inline_bytes": "xx"
          },
          "private_key": {
            "inline_bytes": "xx"
          }
        }
      }
    },
    {
      "name": "file-cert:/etc/certs/cert-chain.pem~/etc/certs/key.pem",
      "last_updated": "2022-07-19T21:16:41.635Z",
      "secret": {
        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
        "name": "file-cert:/etc/certs/cert-chain.pem~/etc/certs/key.pem",
        "tls_certificate": {
          "certificate_chain": {
            "inline_bytes": "xx"
          },
          "private_key": {
            "inline_bytes": "xx"
          }
        }
      }
    },
    {
      "name": "ROOTCA",
      "last_updated": "2022-07-19T19:32:30.589Z",
      "secret": {
        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
        "name": "ROOTCA",
        "validation_context": {
          "trusted_ca": {
            "inline_bytes": "xx"
          }
        }
      }
    },
    {
      "name": "file-root:/etc/certs/root-cert.pem",
      "last_updated": "2022-07-19T21:16:41.526Z",
      "secret": {
        "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
        "name": "file-root:/etc/certs/root-cert.pem",
        "validation_context": {
          "trusted_ca": {
            "inline_bytes": "xx"
          }
        }
      }
    }
  ]
}

And this is what is returned by /certs endpoint:
{
 "certificates": [
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "fcfdfe1ae4ae4a0712bc106bb5cc47f4",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "224",
     "valid_from": "2021-09-01T00:00:00Z",
     "expiration_time": "2023-02-28T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "07142d2cdd619df63affc07ee27b8e6b",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T21:21:47Z",
     "expiration_time": "2022-07-19T21:51:47Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  },
  {
   "ca_cert": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "b02ec8ae47967e56a69b3749f8b41617",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "408",
     "valid_from": "2022-03-01T00:00:00Z",
     "expiration_time": "2023-08-31T23:59:59Z"
    }
   ],
   "cert_chain": [
    {
     "path": "\u003cinline\u003e",
     "serial_number": "a3a7d2f7e604808ad4ba33e0f8912430",
     "subject_alt_names": [
      {
       "uri": "spiffe:"
      }
     ],
     "days_until_expiration": "0",
     "valid_from": "2022-07-19T19:32:30Z",
     "expiration_time": "2022-07-19T20:02:30Z"
    }
   ]
  }
 ]
}

The /certs endpoint shows that it still has certificates which were valid from 19:21, but new certs were added at 21:15 (which it shows in the /certs endpoint along with old ones)

I see this even for new connections.

It should not if certs are updated correctly. Config dump will help.

This (https://github.com/istio/api/blob/87dfc8c8c07ae884e54b40246dc179ca97e58a7e/networking/v1alpha3/destination_rule.proto#L707) should only be applicable to long running connections right?

Every connection is a long running. Envoy uses pool of connections that gets terminated only on errors or via some config above.

Added the config dump and o/p from /certs endpoint above.

@howardjohn @ramaraochavali - any help will be appreciated.

Where is the config dump?

The full config_dump (I pasted the dynamic secret section). Let me grab the full config_dump.

Is there a particular section of the config_dump I can give? As I want to remove company specific details, it will take a little longer.

@howardjohn - added config_dump: https://gist.github.com/nirvanagit/20aa807f4a6e4c87cf1b17737952a5be

@howardjohn @ramaraochavali - any help will be appreciated. We are consistently seeing that envoy is listening on the first certificate which were loaded when it came up.

what version of Istio are you seeing this on?

It looks like some of the clusters are using the new secret and some or not? Is that right? if yes can you share which clusters has this problem of using old certs?

Looking at @shankgan - 1.13.5
I see there is a difference in dynamic active secrets configuration between one which is working and not working
Working:

Not working:

The one working, just has two items in the array. There names are:

default
ROOTCA
The one which is not working, has multiple items in the array. There names are:
default
file-cert:/etc/certs/cert-chain.pem~/etc/certs/key.pem
ROOTCA
file-root:/etc/certs/root-cert.pem

In the second one, it seems you are setting up some DRs with MUTUAL/SIMPLE as TLS Mode. In that case it generates names like file-cert:/etc/certs/cert-chain.pem~/etc/certs/key.pem. default and ROOTCA are used for ISTIO_MUTUAL.
From config dump can you identify which clusters are using old certs and share that config?

@howardjohn @ramaraochavali - found the issue.
It occurs whenever we have a DR with mode: MUTUAL, and path to the three files defined.
On changing the mode to ISTIO_MUTUAL, and removing the paths to the three files, envoy is able to load the new certificates fine.
This must be a bug.

It could be that Istio is not able to add watchers for those fine. Can you share proxy logs?

Yes. That's true.
This function - https://github.com/istio/istio/blob/c6abaf5e5fd6488d4b654ac7b6a8d4cb44a96316/security/pkg/nodeagent/cache/secretcache.go#L343 is responsible for adding file watch events, and then eventually resulting in an SDS push. It checks if the file is already being watched, if so, it does not re-add it. The file-root and ROOTCA point to the same file, and ROOTCA’s watch is getting added first, hence there is not watch for file-root. Similarly for file-cert.

The confusion I have is, that listening certificates are pushed as ROOTCA and default, which are getting updated correctly.
Whereas certificates pushes as file-root and file-cert are not getting updated because there is no watcher for them. But why is file-root and file-cert resulting in envoy listening on them, even though these are configured as client certificates in DestinationRules.

@ramaraochavali - some useful logs:

As seen in the logs, the watch event for the file-root/file-cert were logged just once, at pod startup time.

I think there is a race condition, sometimes file-cert is picked up for file watch events, and sometimes, default is picked up for watch events.
But one thing is guaranteed, that either client certificate will expire as they are not getting updated, OR serving certificates will expire as they are not getting updated.

It is because the file paths are the same, but there is only one inotify event, when the file is rotated. Hence only one will get pushed via SDS.

This is the possible fix: https://github.com/istio/istio/pull/40137. I tested it in my environment where I was consistently able to recreate the issue. It doesn't have any tests yet

authors: howardjohn nirvanagit ramaraochavali shankgan


text:
gateway api status not reflected when running multiple istio version

Is this the right place to submit this?

[X] This is not a security vulnerability or a crashing bug
[X] This is not a question about how to use Istio

Bug Description
We use istio for gateway api.
We would like to employ canary versioning to validate istio behavior and run multiple versions of istio for a period of time.
we can deploy multiple istio envoys using the istio.io/rev label in Gateway resource, but only one revision of the istio is reflected in the Gateway resource status.
Therefore, it is currently not possible to have multiple versions of istio running in parallel for gateway api.
use istio version
$ istioctl tag list
TAG    REVISION NAMESPACES
stable 1-20-8
canary 1-22-2

gateway status
NAMESPACE       NAME              CLASS   ADDRESS         PROGRAMMED   AGE
istio-ingress   default-gateway   istio                   Unknown      49m
istio-ingress   specify-gateway   istio   10.195.47.182   True         49m

# default-gateway
Status:
  Conditions:
    Last Transition Time:  1970-01-01T00:00:00Z
    Message:               Waiting for controller
    Reason:                Pending
    Status:                Unknown
    Type:                  Accepted
    Last Transition Time:  1970-01-01T00:00:00Z
    Message:               Waiting for controller
    Reason:                Pending
    Status:                Unknown
    Type:                  Programmed

# specify-gateway
Status:
  Addresses:
    Type:   IPAddress
    Value:  10.195.47.182
  Conditions:
    Last Transition Time:  2024-07-17T04:50:04Z
    Message:               Resource accepted
    Observed Generation:   1
    Reason:                Accepted
    Status:                True
    Type:                  Accepted
    Last Transition Time:  2024-07-17T04:51:41Z
    Message:               Resource programmed, assigned to service(s) specify-gateway-istio.istio-ingress.svc.cluster.local:80
    Observed Generation:   1
    Reason:                Programmed
    Status:                True
    Type:                  Programmed

Gateway manifest
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: default-gateway
  namespace: istio-ingress
  labels:
    istio.io/rev: stable
spec:
  gatewayClassName: istio
  listeners:
  - name: default-gw
    hostname: "*.example.com"
    port: 80
    protocol: HTTP
    allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            kubernetes.io/metadata.name: default-gw

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: specify-gateway
  namespace: istio-ingress
  labels:
    istio.io/rev: canary
spec:
  gatewayClassName: istio
  listeners:
  - name: specify-gw
    hostname: "*.example.com"
    port: 80
    protocol: HTTP
    allowedRoutes:
      kinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      namespaces:
        from: Selector
        selector:
          matchLabels:
            kubernetes.io/metadata.name: specify-gw

I followed the istiod logs and it seemed that only certain revisions of the pod were getting the istio-gateway-status-leader as the leader election.
version: can update gateway status
2024-07-17T05:00:54.504507Z     info    klog    successfully acquired lease istio-system/istio-gateway-status-leader
2024-07-17T05:00:54.504766Z     info    leader election lock obtained: istio-gateway-status-leader
2024-07-17T05:00:54.504779Z     info    Starting gateway status writer
2024-07-17T05:00:54.505320Z     info    klog    successfully acquired lease istio-system/istio-gateway-deployment-1-22-2

version: cannot update gateway status
2024-07-17T04:47:25.322462Z     info    klog    attempting to acquire leader lease istio-system/istio-gateway-status-leader...
2024-07-17T04:47:25.322510Z     info    klog    attempting to acquire leader lease istio-system/istio-gateway-deployment-1-20-8...

Version
$ istioctl version
client version: 1.22.2
istiod version: 1.20.8
istiod version: 1.22.2
data plane version: 1.20.8 (1 proxies), 1.22.2 (1 proxies)
$ kubectl version
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4

Additional Information
bug-report.tar.gz

maybe I think...
gateway deployment controller use NewPerRevisionLeaderElection ,  so can deploy every revision without any problem.
https://github.com/istio/istio/blob/master/pilot/pkg/bootstrap/configcontroller.go#L169
https://github.com/istio/istio/blob/master/pilot/pkg/leaderelection/leaderelection.go#L225-L228
but, gateway status writer use NewLeaderElection, so can not reflect gateway status.
https://github.com/istio/istio/blob/master/pilot/pkg/bootstrap/configcontroller.go#L149
Is there any reason why NewPerRevisionLeaderElection cannot be used in gateway status writer?

@howardjohn
Hi, the following PR shows that the deployment controller uses NewPerRevisionLeaderElection.
https://github.com/istio/istio/pull/44746
Is there any reason why not using it in status writer?
If you know of any, please let me know when you have time.

This is the same root cause as https://github.com/istio/istio/issues/52438  fwiw

source: gharchive/issue

repo: istio/istio

license_type: permissive

license_source: github-api


text:
kubernetes 1.5.x

Hi,
I wanted to ask if istio is compatible with k8s 1.5.x? As I saw in the getting started docs, it assumes k8s 1.7.x is present.
My use case revolves around just using envoy as an edge router to ingress to a grpc service and also do in cluster load balancing (service to service) for the same service.

0.1.6 is compatible with k8s 1.5
0.2.x requires at least 1.7

authors: arunkjn ldemailly