question

Taylor-3961 avatar image
1 Vote"
Taylor-3961 asked Taylor-3961 commented

Azure IoT Edge: Could not load cert with ID "aziot-edged-trust-bundle"

I've been trying to get IoT Edge (v1.2.1) which includes installing aziot-identity-service (v1.2.0) up and running on my CentOS 7 installation, but I keep running into this problem.
I have followed all the instructions for installing the MS repo/GPG key, the aziot-identity-service, and the aziot-edge packages as per how-to-install-iot-edge and grabbing the correct rpm packages from the release page for 1.2.1.

I am attempting to use symmetric key authentication and have tried two different methods for setting up the config.toml file:

  1. Copying /etc/aziot/config.toml.edge.template to /etc/aziot/config.toml, and then configuring only the values for symmetric key authentication using the connection string obtained from the device registered in IoT Hub.

  2. Creating a new file by running 'sudo iotedge config mp -c "<conn_string>"'

Neither option got me any further. I'm not trying to do anything in particular using TPM or anything, so I'm not sure what the roadblock is here. As a last resort, I did try to install the softhsm package as per softhsm.html, however this did not help.

Here are some outputs:

 > sudo iotedge check
    
 Configuration checks (aziot-identity-service)
 ---------------------------------------------
 √ keyd configuration is well-formed - OK
 √ certd configuration is well-formed - OK
 √ tpmd configuration is well-formed - OK
 √ identityd configuration is well-formed - OK
 √ daemon configurations up-to-date with config.toml - OK
 √ identityd config toml file specifies a valid hostname - OK
 √ aziot-identity-service package is up-to-date - OK
 √ host time is close to reference time - OK
 √ preloaded certificates are valid - OK
 √ keyd is running - OK
 √ certd is running - OK
 √ identityd is running - OK
 × read all preloaded certificates from the Certificates Service - Error
     could not load cert with ID "aziot-edged-trust-bundle"
    
     Caused by:
         parameter "id" has an invalid value
         caused by: not found
 √ read all preloaded key pairs from the Keys Service - OK
 √ ensure all preloaded certificates match preloaded private keys with the same ID - OK
    
 Connectivity checks (aziot-identity-service)
 --------------------------------------------
 √ host can connect to and perform TLS handshake with iothub AMQP port - OK
 √ host can connect to and perform TLS handshake with iothub HTTPS / WebSockets port - OK
 √ host can connect to and perform TLS handshake with iothub MQTT port - OK
    
 Configuration checks
 --------------------
 √ aziot-edged configuration is well-formed - OK
 √ configuration up-to-date with config.toml - OK
 √ container engine is installed and functional - OK
 × configuration has correct URIs for daemon mgmt endpoint - Error
     One or more errors occurred. (Connection refused /var/lib/iotedge/mgmt.sock)
 √ aziot-edge package is up-to-date - OK
 √ container time is close to host time - OK
 ‼ DNS server - Warning
     Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
     Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
     You can ignore this warning if you are setting DNS server per module in the Edge deployment.
 √ production readiness: container engine - OK
 ‼ production readiness: logs policy - Warning
     Container engine is not configured to rotate module logs which may cause it run out of disk space.
     Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
     You can ignore this warning if you are setting log policy per module in the Edge deployment.
 × production readiness: Edge Agent's storage directory is persisted on the host filesystem - Error
     Could not check current state of edgeAgent container
 × production readiness: Edge Hub's storage directory is persisted on the host filesystem - Error
     Could not check current state of edgeHub container
 √ Agent image is valid and can be pulled from upstream - OK
    
 Connectivity checks
 -------------------
 √ container on the default network can connect to upstream  AMQP port - OK
 √ container on the default network can connect to upstream HTTPS / WebSockets port - OK
 √ container on the default network can connect to upstream MQTT port - OK
 √ container on the IoT Edge module network can connect to upstream AMQP port - OK
 √ container on the IoT Edge module network can connect to upstream HTTPS / WebSockets port - OK
 √ container on the IoT Edge module network can connect to upstream MQTT port - OK
 30 check(s) succeeded.
 2 check(s) raised warnings. Re-run with --verbose for more details.
 4 check(s) raised errors. Re-run with --verbose for more details.

 > sudo iotedge config apply
 > sudo iotedge system logs | tail -n50
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Starting Azure IoT Edge Module Runtime
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Version - 1.2.1
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Initializing the module runtime...
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Initializing module runtime...
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Using runtime network id azure-iot-edge
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Successfully initialized module runtime
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Finished initializing the module runtime.
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [INFO] - Obtaining edge device provisioning data...
 Jun 14 14:06:55 host systemd[1]: Started Azure IoT Identity Service.
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - Starting service...
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - Version - 1.2.0
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - Provisioning starting. Reason: Startup
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - Provisioning complete.
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - Identity reconciliation started. Reason: Startup
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - Identity reconciliation complete.
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - Starting server...
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "2f72756e2f617a696f742f6964656e74697479642e736f636b:0", "content-length": "16"}
 Jun 14 14:06:55 host systemd[1]: Started Azure IoT Keys Service.
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [INFO] - Starting service...
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [INFO] - Version - 1.2.0
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [INFO] - Loaded libaziot-keys with version 0x02000000
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [INFO] - Starting server...
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - !!! Key client error
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - !!! caused by: internal error
 Jun 14 14:06:55 host aziot-identityd[17719]: 2021-06-14T14:06:55Z [INFO] - --> 404 {"content-type": "application/json"}
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [INFO] - <-- GET /key/device-id?api-version=2020-09-01 {"host": "keyd.sock"}
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [ERR!] - Permission denied (os error 13)
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [ERR!] - !!! internal error
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [ERR!] - !!! caused by: could not load key
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [ERR!] - !!! caused by: could not load key: AZIOT_KEYS_RC_ERR_EXTERNAL
 Jun 14 14:06:55 host aziot-keyd[17727]: 2021-06-14T14:06:55Z [INFO] - --> 500 {"content-type": "application/json"}
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [WARN] - The daemon could not start up successfully: Could not retrieve device information
 Jun 14 14:06:55 host aziot-edged[17709]: 2021-06-14T14:06:55Z [WARN] -         caused by: HTTP response error: [404 Not Found] {"message":"Key client error\ncaused by: internal error"}


 > sudo iotedge system status
 System services:
     aziot-edged             Running
     aziot-identityd         Running
     aziot-keyd              Running
     aziot-certd             Ready
     aziot-tpmd              Ready


I did notice the os permission denied error in the system logs and ensured as best I could that any necessary folders/files were given the correct permissions, but I can't be 100% on that, nor have I really found any documentation requiring that I make any permissions modifications.

tldr; after following all the normal documentation for installation and configuration of this on CentOS 7, I'm still struggling to get this working correctly (it obviously doesn't connect to the Azure IoT Hub yet as a result of this).

If anyone has any thoughts/ideas - that would be great!


azure-iot-edge
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I'm also running into this exact issue. I've been trying to do a most basic installation of Azure IoT Edge V1.2.3 with symmetric keys, No TPM, No certs. There is not much troubleshooting guide available for this issue.

My environment is Ubuntu server 18.04, aziot-edge V1.2.3, aziot-identity-service V1.2.2

0 Votes 0 ·
SatishBoddu-MSFT avatar image SatishBoddu-MSFT VasanthBalakrishnan-9826 ·

Hello @VasanthBalakrishnan-9826 , below is the resolution posted by @ Taylor-3961 on this thread. Please let me know if you need to check any un-wnated packages on your system causing this issue? please do comment in the below section.

the problem is somehow caused by some other package I have installed on the system. I'm not sure exactly what the package is doing to cause this problem, but having removed that package, I was able to get everything working as expected - I do need that other package so I'll have to figure out what's going on there, but thankfully this is not a problem with the IoT Edge software.

0 Votes 0 ·
Taylor-3961 avatar image Taylor-3961 SatishBoddu-MSFT ·

Yea, I unfortunately am unsure what package was causing the problem, it was likely one of our own custom packages we had installed on the system, but I don't know which one was the problem one. Moreover, I no longer work for that employer and don't have access to the hardware that was hosting iot edge, so I can't find out any more details about it.
For me, I had looked at file permissions and everything I could find about that topic and made sure everything was the was it was supposed to be, but it still didn't help. I don't know if there was some SELinux policy or something getting in the way, I'm not sure. Wish I could be of more help!

1 Vote 1 ·

1 Answer

SatishBoddu-MSFT avatar image
0 Votes"
SatishBoddu-MSFT answered SatishBoddu-MSFT edited

Hello @Taylor-3961
In most cases, this is caused by a file permission error. Make sure that the iotedge user has read permissions on the directory and certificates inside.

Could you please point out the documentation(s) you have used or referred to for this scenario?

Resolution posted by @ Tylor

the problem is somehow caused by some other package I have installed on the system. I'm not sure exactly what the package is doing to cause this problem, but having removed that package, I was able to get everything working as expected - I do need that other package so I'll have to figure out what's going on there, but thankfully this is not a problem with the IoT Edge software.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yea, I've read the same thing about the permissions, which is why I looked into it in the first place.

Honestly, I don't have a great piece of documentation I could refer to here, the documentation is severely lacking with regards to the 1.2 version of IoT Edge. Many things still refer to the previous architecture set up that had different users/groups for the different services. The best I could do was verify that things like /etc/aziot/keyd were owned by the aziotks user, and similarly for the other service folders.

In addition, I'm not using any certificates or anything, at least not explicitly. I've done as basic an install as I possibly could, so if there happens to be any os permissions issues, then I honestly don't know what file/folder it might be looking at. Are there certain folders I should be checking for their permissions?

1 Vote 1 ·