Difference between schema.history and database.history in mysql connector

Eddy Tran 40

I set up kafka, mysql and debezium in my computer successfully. However, i have still not understood about the difference between database.history.kafka.topic and schema.history.internal.kafka.topic. Below is my connector: User's image

Please help me explaining the difference between database.history.kafka.topic and schema.history.internal.kafka.topic! Why when i run kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092, the result is schemahistory.dbserver1; what about dbhistory.classicmodels, where can i find topic dbhistory.classicmodels?
Many thanks!

PRADEEPCHEEKATLA-MSFT 77,086 Reputation points Microsoft Employee

2024-01-17T03:32:10.2566667+00:00

@Eddy Tran - Thanks for the question and using MS Q&A platform.

Could you please help us understand which Azure service are you using?
Eddy Tran 40 Reputation points

2024-01-17T03:47:01.6066667+00:00

I'm using Azure Event Hub. Thanks!
PRADEEPCHEEKATLA-MSFT 77,086 Reputation points Microsoft Employee

2024-01-17T04:48:51.08+00:00

@Eddy Tran - Thanks for the details.

The database.history.kafka.topic and schema.history.kafka.topic are two different topics used by Debezium to store the history of changes in the database schema and data respectively.

The database.history.kafka.topic is used to store the history of changes in the database schema. This topic is used to store the DDL statements that are executed on the database. This topic is used to store the history of changes in the database schema. This topic is used to store the DDL statements that are executed on the database.

The schema.history.kafka.topic is used to store the history of changes in the database data. This topic is used to store the changes in the data that are captured by Debezium.

When you run the command kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092, the result is schemahistory.dbserver1 because this is the default name of the topic used by Debezium to store the history of changes in the database schema. Regarding the dbhistory.classicmodels topic, it is not created by default. You can configure Debezium to use a different topic name for the database.history.kafka.topic property. If you have configured Debezium to use dbhistory.classicmodels as the database.history.kafka.topic, you can find this topic by running the same command kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092.

Hope this helps.
Eddy Tran 40 Reputation points

2024-01-18T03:24:23.7333333+00:00

Actually, when i configure the connector with schema.history.kafka.topic and database.history.kafka.topic successfully. When i run kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092, I still not see the database.history.kafka.topic.

Thanks!
PRADEEPCHEEKATLA-MSFT 77,086 Reputation points Microsoft Employee

2024-01-18T05:10:17.69+00:00

@Eddy Tran - If you have configured the database.history.kafka.topic property in your connector configuration, but you are not seeing the topic when you run the kafka-topics.bat --list --bootstrap-server 127.0.0.1:9092 command, there could be a few reasons for this:

The topic has not been created yet: Debezium will automatically create the database.history.kafka.topic topic when it starts up and detects that the topic does not exist. If you have not yet started the connector, or if the connector has not yet executed any DDL statements, the topic may not have been created yet. You can check if the topic has been created by running the kafka-topics.bat --describe --bootstrap-server 127.0.0.1:9092 --topic database.history.kafka.topic command.

The topic has been created with a different name: If you have specified a custom name for the database.history.kafka.topic property in your connector configuration, the topic may have been created with a different name. You can check the actual name of the topic by running the kafka-topics.bat --describe --bootstrap-server 127.0.0.1:9092 --topic <actual-topic-name> command, replacing <actual-topic-name> with the name of the topic that you specified in your configuration.

There is an issue with the configuration: If there is an issue with the configuration of your connector, Debezium may not be able to create the database.history.kafka.topic topic. You can check the logs of the connector to see if there are any errors related to the topic creation.

I hope this helps! Let me know if you have any further questions.
PRADEEPCHEEKATLA-MSFT 77,086 Reputation points Microsoft Employee

2024-01-25T04:42:15.9933333+00:00

@Eddy Tran - We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.