question

83066584 avatar image
0 Votes"
83066584 asked ·

VMSS instance failes to connect to DB on first startup

Hello,

I've got an image for scale set, it's Win Server with defined system service, which starts Spring Boot application. This application uses Hibernate to connect to SQL Server DB (it's in the same resource group).

Issue i got - when VMSS scale out and creates new instance - spring boot service fails on startup. Issue with secure connection to database

 WARN: HHH000342: Could not obtain connection to query metadata : The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Unexpected rethrowing".
 Dec 09, 2019 9:51:23 AM org.hibernate.dialect.Dialect 
 INFO: HHH000400: Using dialect: org.hibernate.dialect.SQLServerDialect
 Dec 09, 2019 9:51:24 AM org.hibernate.engine.jdbc.spi.SqlExceptionHelper logExceptions
 WARN: SQL Error: 1038, SQLState: S0004
 Dec 09, 2019 9:51:24 AM org.hibernate.engine.jdbc.spi.SqlExceptionHelper logExceptions
 ERROR: An object or column name is missing or empty. For SELECT INTO statements, verify each column has a name. For other statements, look for empty alias names. Aliases defined as "" or [] are not allowed. Change the alias to a valid name.
 [main] WARN org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [com/kmslh/spot/db/config/SpotDBConfig.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.exception.SQLGrammarException: Error accessing table metadata
 Dec 09, 2019 9:51:24 AM org.apache.catalina.core.StandardService stopInternal
 INFO: Stopping service [Tomcat]
 [main] INFO org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLoggingListener - 
    
 Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
 org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [com/kmslh/spot/db/config/SpotDBConfig.class]: Invocation of init method failed; nested exception is javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.exception.SQLGrammarException: Error accessing table metadata
     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778)
     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
     at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
     at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1105)
     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
     at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140)
     at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:742)
     at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:389)
     at org.springframework.boot.SpringApplication.run(SpringApplication.java:311)
     at org.springframework.boot.SpringApplication.run(SpringApplication.java:1213)
     at org.springframework.boot.SpringApplication.run(SpringApplication.java:1202)
     at com.kmslh.spot.htmlconverter.HtmlConverter.main(HtmlConverter.java:11)
     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
     at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:47)
     at org.springframework.boot.loader.Launcher.launch(Launcher.java:86)
     at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
     at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
 Caused by: javax.persistence.PersistenceException: [PersistenceUnit: default] Unable to build Hibernate SessionFactory; nested exception is org.hibernate.exception.SQLGrammarException: Error accessing table metadata
     at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:402)
     at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.afterPropertiesSet(AbstractEntityManagerFactoryBean.java:377)
     at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.afterPropertiesSet(LocalContainerEntityManagerFactoryBean.java:341)
     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1837)
     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1774)
     ... 24 more
 Caused by: org.hibernate.exception.SQLGrammarException: Error accessing table metadata
     at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:106)
     at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
     at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
     at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
     at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.convertSQLException(InformationExtractorJdbcDatabaseMetaDataImpl.java:98)
     at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.getTables(InformationExtractorJdbcDatabaseMetaDataImpl.java:341)
     at org.hibernate.tool.schema.extract.internal.DatabaseInformationImpl.getTablesInformation(DatabaseInformationImpl.java:120)
     at org.hibernate.tool.schema.internal.GroupedSchemaMigratorImpl.performTablesMigration(GroupedSchemaMigratorImpl.java:65)
     at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.performMigration(AbstractSchemaMigrator.java:207)
     at org.hibernate.tool.schema.internal.AbstractSchemaMigrator.doMigration(AbstractSchemaMigrator.java:114)
     at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.performDatabaseAction(SchemaManagementToolCoordinator.java:184)
     at org.hibernate.tool.schema.spi.SchemaManagementToolCoordinator.process(SchemaManagementToolCoordinator.java:73)
     at org.hibernate.internal.SessionFactoryImpl.(SessionFactoryImpl.java:309)
     at org.hibernate.boot.internal.SessionFactoryBuilderImpl.build(SessionFactoryBuilderImpl.java:462)
     at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:935)
     at org.hibernate.jpa.HibernatePersistenceProvider.createContainerEntityManagerFactory(HibernatePersistenceProvider.java:141)
     at org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean.createNativeEntityManagerFactory(LocalContainerEntityManagerFactoryBean.java:365)
     at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.buildNativeEntityManagerFactory(AbstractEntityManagerFactoryBean.java:390)
     ... 28 more
 Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: An object or column name is missing or empty. For SELECT INTO statements, verify each column has a name. For other statements, look for empty alias names. Aliases defined as "" or [] are not allowed. Change the alias to a valid name.
     at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:259)
     at com.microsoft.sqlserver.jdbc.TDSTokenHandler.onEOF(tdsparser.java:256)
     at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:108)
     at com.microsoft.sqlserver.jdbc.TDSParser.parse(tdsparser.java:28)
     at com.microsoft.sqlserver.jdbc.SQLServerConnection$1ConnectionCommand.doExecute(SQLServerConnection.java:2754)
     at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7344)
     at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2713)
     at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectionCommand(SQLServerConnection.java:2759)
     at com.microsoft.sqlserver.jdbc.SQLServerConnection.setCatalog(SQLServerConnection.java:3076)
     at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.switchCatalogs(SQLServerDatabaseMetaData.java:345)
     at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetFromStoredProc(SQLServerDatabaseMetaData.java:297)
     at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getResultSetWithProvidedColumnNames(SQLServerDatabaseMetaData.java:321)
     at com.microsoft.sqlserver.jdbc.SQLServerDatabaseMetaData.getTables(SQLServerDatabaseMetaData.java:493)
     at org.hibernate.tool.schema.extract.internal.InformationExtractorJdbcDatabaseMetaDataImpl.getTables(InformationExtractorJdbcDatabaseMetaDataImpl.java:329)
     ... 40 more

But if i'm just restarting my service- it works fine.
The vnet of VMSS is added to DB firewall rules.
More over - i added option to system service Recovery to restart service on failure. And again, the second run is successful.

Could it be an issue, that my VM in scale set starts a little bit earlier than it's IP have been added to vnet?

azure-virtual-machinesazure-virtual-machines-scale-set
· 3
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Can you add ssl protocol to the jdbc connection string as shown in the below github comment and try
https://github.com/Microsoft/mssql-jdbc/issues/879#issuecomment-438825486


TLS 1.2 is recommended though.

0 Votes 0 · ·

Thanks for advice!
Unforunatelly this change didn't help.
Also i forgot to mention, that i got 3 more Spring Boot apps, running as App Services. This apps are using same DB connection module to communicate with this DB. And scaling out for these apps works well without any connection issues. That's why i'm thinking, that problem is in VMSS.

0 Votes 0 · ·

Thanks for the input.


I will check with the team for more info

0 Votes 0 · ·

0 Answers