DevOps Interview : Part 1. Kotaro Ogino on Continuous System Testing

As a DevOps evangelist in Japan, I'd like to introduce some cool DevOps guys in Japan.
This is not only Microsoft technologies, but also non-Microsoft technologies.

Rakuten DevOps practitioners on transitioning to DevOps

I asked about the transition into DevOps on Rakuten. Rakuten is one of the biggest electronic commerce and Internet company based in Tokyo, Japan. Its B2B2C e-commerce platform Rakuten Ichiba is the largest e-commerce site in Japan and among the world's largest by sales.
They are also known for "Englishnization" which means they use English as the company's official language from 2012.

Today's guests

Kotaro Ogino: Testing and Automation professional. He helps to adopt testing automation for over 20 projects in Rakuten.

Hirobumi Takahashi: Testing and Automation professional. He helps Kotaro and works with him.

Yasunobu Kawaguchi: Agile Coach in Rakuten. He is one of the most influential people in the agile community in Japan.

Eiji Ienaga: Agile coach and TDD and Testing professional in Eiwa system management.

Tsuyoshi Ushio: Senior Techinical Evagelist – DevOps Microsoft Japan

Part 1. Kotaro Ogino on Continuous System Testing

 Tsuyoshi: Thank you for coming to this interview. Could you introduce yourself?
 Kotaro: I'm part of the Development support division in Rakuten. I help to adopt automated testing in projects. It is over 20 projects now. In addition, I develop a platform for testing, deployment and automation of operations.
 Hirobumi: I lived in Spain for 3 months. Rakuten has a lot of subsidiaries in the world. This allowed me do the job in English. Now I help Kotaro.
 Tsuyoshi: How about the overseas job? Did they do a good job?

*Notice : In Japan, not many people work internationally so they are afraid to use English. Now they are about to change to use English, so they don't know much about the working environment in foreign company and are really interested in it. We just admire the overseas job and we want to know the actual status.

 Hirobumi: Yes. The engineering culture is different in Japan and Spain.
 Tsuyoshi: Could you explain more detail about this?
 Hirobumi: This can be my personal experience, but I felt that the responsibility of individual engineers is clear in Japan, so they focus on their responsibility area. In Spain, the responsibility boundary is not clear and the team tries to achieve the goal.
 Tsuyoshi: I think DevOps has three elements. People, Process and Product. People refers to mindset and Product refers to technologies and tools. First of all, could you tell me about "Product", please?
 Kotaro:  When I automate something, I always consider about the Value stream. For example, testing is often considered as a dedicated process from Dev or Ops but it should be considered as a part in a flow from requirement analysis to release. Test automation can provide the values to both developers and operators such as speedy and reliable release.
 Tsuyoshi: Any challenges for automation?
 Kotaro: We have 2 groups. One is Devs division and the other is Ops division. When a development team needed a server, they needed to ask Ops division to set up and run it which took some times.
 Tsuyoshi: Could you tell me about the servers that you look after?
 Kotaro: Web / API servers. We use Java, PHP and Ruby.
 Tsuyoshi: Could you tell me about the toolset of the automation.
 Kotaro: Chef, Docker and OpenStack for provisioning servers, Capistrano and Fabric for deployment and JUnit for unit testing and so on.
 Tsuyoshi: It is very common that developers don't write test code properly. Do they write these well?
 Kotaro: Well, it's difficult... However, I think the development team should decide the policy. I just tell them how to do it.
 Tsuyoshi: Could you tell me about non-unit testing?
 Kotaro: Ok. I love this term called "Continuous System Testing" which I coined.
 Tsuyoshi: Tell me more about it?
 Kotaro: Continuous integration is about automating integration and testing continuously. Continuous System Testing is about automating system tests and then doing it continuously.
 Tsuyoshi: I think it is one of the most difficult practices in DevOps.
 Kotaro: Yes.
 Tsuyoshi: What is the outcome?
 Kotaro: Before the adoption of the continuous system testing, it took 5 days for a system test which includes manual testing and deployment. Now it takes 2 days even if it has some problems. Jenkins fires the automated system testing every night, so we can see the results the next day. Which means it takes 2 days maximum even if we had some problems.
 Tsuyoshi: How long does it take?
 Kotaro: 2 hours. It will take 50 hours unless it is executed concurrently. We use Jenkins with 20-30 concurrent tasks.
 Tsuyoshi: Cool. In DevOps world, it is said that we have a system which enables "10 deployments per day", however it does not include deep automated system testing to make it easy to notice their own problem and roll it back. I'm impressed because yours includes deep automated system testing.
 Kotaro: Thanks. In addition, the Jenkins system has 100 slaves.
 Everyone: Cool!
 Tsuyoshi: This is the famous Mary Poppendeck's question. "How long would it take your organization to deploy a change that involves just one single line of code? Do you do this on repeatable, reliable basis?"
 Kotaro: It depends on how much lines of code it has against a production line of code.
 Eiji: Hmm. The code smells. Something wrong happens in your code.
 Kotaro: No. For example, we have a search engine. It requires us to change 10 thousand lines of code, if we change one line of code.
 Tsuyoshi: Ok. It must be a special case. Please tell me the normal case, please?
 Kotaro: It will take 3 hours for us to deploy an application into production.
 Tsuyoshi: Cool. Especially if you are enterprise company. By the way, could you tell me the definition of "System testing"?
 Kotaro: End to end functional testing and non-functional testing which includes performance, data-consistency, availability, search-quality, redundancy and index testing for distributed environments. It includes Recovery testing and Operability testing as well.
 Tsuyoshi: Perfect! Tell me more about the tools for it.
 Kotaro: We use Jenkins. We use JUnit/Selenium/CucumberJVM for web-based acceptance testing.
 Tsuyoshi: When does the team write acceptance tests?
 Kotaro: It depends. A scrum team writes it during a sprint. However, another writes it in the endgame.
 Tsuyoshi: Do you use Azure for automation?
 Kotaro: Now we need to write system tests for other regions in the world. Azure has a lot of data centers in the world. We are going to use Azure for performance testing across multiple regions.
 Tsuyoshi: Could you tell me about another toolset besides acceptance testing.
 Kotaro: It is totally same as acceptance testing. JUnit / Selenium / CucumberJVM.
 Tsuyoshi: By the way, where is a bottle of soy sauce?
 Yasunobu: I will show you. This is the highest-ranking pun. Because it will require to understand both Japanese and English.

* Note: Soy sauce is called "Shou yu" in Japan. Which is very similar to "Show you".


They successfully implement "Continuous System Testing". They automate a lot of system testing for keeping quality. It is quite the Japanese style to improve the software lifecycle and keeping up high quality. Kotaro presented at JaSST (Japan Software testing symposium). You can refer to his presentation or his blog

To be continued.