@Aaron Althaus , Thanks for the question.
Just to confirm, I believe you have the admin or co-admin access on the Azure subscription? If it's MySql in-app, phpMyAdmin is enabled by default with the feature. You can access it through https://<sitename>.scm.azurewebsites.net/phpMyAdmin/
.
Since MySql is only started with the main site, do make sure that the main site is running (simplest way is to turn on AlwaysOn) before using phpMyAdmin. Unlike phpMyAdmin from SiteExtenions gallery, this phpMyAdmin is aware of MySql credentials and will connect automatically.
Also, the connection string is stored at 'D:\home\data\mysql\MYSQLCONNSTR_localdb.txt'. Your WordPress app reads from this file for what database, username and password to use.