What is AI Kiritan?

This time, I will introduce AI Kiritan. AI Kiritan is an AI virtual singer that uses AI for singing voice synthesis technology.

Her singing voice has a reputation for being very similar to how humans sing, using the latest AI technology. In this article, we will introduce the development process and technology of AI Kiritan in detail.

What is AI Kiritan?

AI Kiritan applies the latest AI technology to give a realistic singing voice to the character “Tohoku Kiritan”.

What is Tohoku Kiritan?

Tohoku Kiritan is one of the Tohoku support characters developed by SSS LLC. SSS LLC provides Tohoku support characters as “moe characters” that can be used by companies and local governments in the Tohoku region without prior copyright application.

Tohoku Kiritan is the younger sister of the company’s most famous character, Tohoku Zunko. It looks like a kiritanpo, a specialty of Tohoku, and carries a kiritanpo cannon on its back.

What is text-to-speech software?

Even before AI Kiritan was developed, there were several attempts to give Tohoku Kiritan a singing voice. A typical example is the freeware “Singing Voice Synthesis Tool UTAU” released in 2008 by creator Ameya/Ayame.

This is the Tohoku Kiritan official demo song created by creator higeoex using UTAU.

You can create Kiritan’s vocal sound source by specifying the lyrics, pitch, length of sound, and singing style with speech synthesis software. By combining the vocal sound source and the sound source of the performance, one song can be created.

How to use AI Kiritan

Many people would like to actually use AI Kiritan by themselves. Therefore, here we will introduce how to use AI Kiritan and the points to note when using it step by step.

1. Download synthesis software
2. Prepare sheet music file
3. Create singing data
4. Notes on tone
5. Is commercial use possible?

I will explain each.

Download synthesis software

First, download the speech synthesizer software. The synthesis software that can use Kiritan is NEUTRINO and CeVIO AI.

Download the software below. There are files for macOS and windows, so please choose according to your PC.

First, I will introduce how to use NEUTRINO.

NEUTRINO, released in 2020, is speech synthesis software developed by creator SHACHI.

By inputting the score and lyrics, a natural singing voice is automatically generated by a neural network. In other words, AI automatically estimates the utterance timing, pitch, voice quality, and hoarseness of the voice from the score and lyrics.

In addition to Tohoku Kiritan, NEUTRINO has released the sound sources of seven other artists, each with their recommended range, BPM, and genre.

This is a characteristic that comes from the database from which the sound source was created. For example, if you create a sound source using a database with a lot of ballad music, the sound source will be suitable for creating ballad-style music.

Tohoku Kiritan’s recommended songs are as follows.

Kiritan is included with NEUTRINO. Also, the file downloaded with NEUTRINO is a zip file, so unzip it in an appropriate location.

Next is CeVIO AI.

CeVIO AI, released in 2021, is speech synthesis software developed by Techno Speech, a venture company from Nagoya Institute of Technology.

CeVIO is a new project that creates entertainment that has never existed, born to support UGC (User Generated Contents) provided by Techno Speech.

CeVIO AI was born as part of this project.

“CeVIO AI Tohoku Kiritan Song Voice” is a song voice that runs on CeVIO AI.

Using the AI ​​technology developed by the CeVIO project, it is possible to create sound sources with realistic singing styles. In addition, the CeVIO AI Song Editor has an easy-to-use GUI that enables smooth editing.

If you download this as well as NEUTRINO, you will be able to use AI Kiritan, and you can also use cv Himika Akaneya’s dedicated song voice for CeVIO AI.

Prepare sheet music file

Next, prepare a sheet music file for singing.

Also, the .musicxml format is recommended for lyrics files. .musicxml files are created by MuseScore.

Create singing data

After preparing the score file, create the singing data.

How to write singing data is explained in the following article.

Notes on tone

Here are some points to keep in mind when actually tuning.

When making AI sing, the vowel “ai” such as “ja nai” will sound unnatural if you input it normally.

Therefore, this article offers the following solutions:

If you want to sing “Sky”, (omitted) “A” “E” “I” is repeated. Certainly, in the process of changing from “a” to “i”, it passes through the mouth of “e”.

In the same way, going from “a” to “u” seems natural if you pass “o”.

If you pay attention to the above points when tuning, you will be successful.

Is commercial use possible?

Finally, I will introduce whether the songs created using AI Kiritan can be used commercially.

On the official website of AHS Co., Ltd., commercial use is explained as follows.

If a corporation uses it for commercial purposes or business use, a separate corporate contract is required regardless of whether it is for a fee or not. In addition, even if you are an individual or doujin circle, please be sure to contact us in advance when distributing for a fee for commercial purposes.

Compensation (printing fee, disk fee, shipping fee, transportation fee, etc.) for works and secondary creations created by individuals and doujin circles using our characters and our VOICEROID products at doujin events, etc. It refers to the case of obtaining and distributing for a fee. Paid distribution within this range will be handled in the same way as free distribution. Profit-making distribution that generates more than the consideration is considered commercial use.

In other words, there seems to be no particular problem when using it for individual doujin activities. If you are unsure if there is a problem with your planned usage, you can inquire on the official website.

AI Kiritan Technology

Next, I will introduce the technology of AI Kiritan. What kind of innovation has AI-based speech synthesis technology achieved compared to conventional speech synthesis technology?

Also, when you use AI in speech synthesis software, what kind of technology are you specifically introducing?

Speech synthesis software using deep learning

AI Kiritan’s singing technology uses deep learning. In February 2020, AI Kiritan’s speech synthesis software “NEUTRINO” was released, and its technology became a hot topic.

NEUTRINO was developed based on the singing database of AI Kiritan created by Associate Professor Masamasa Morise of Meiji University’s Faculty of Interdisciplinary Mathematics.

AI Kiritan’s singing database

From October 2018 to March 2022, Mr. Masamasa Morise is a research project of the Japan Science and Technology Agency Strategic Creative Research Promotion Project (PRESTO) called “Development of Human-in-the-loop type singing design”. Submitted a theme.

With conventional speech synthesis technology, very complicated adjustments had to be made to eliminate the unnaturalness of singing and speaking, and a great deal of time and effort was required to create singing voice data.

In this research theme, we aim to build a foundation that automatically adjusts parameters to create a natural singing voice.

As part of this research, the “Tohoku Kiritan Singing Database” was made public. This database contains the singing data sung a cappella by Tohoku Kiritan and data such as phoneme labels necessary for singing voice synthesis.

Singing data by voice actor Himika Akaneya

The character voice (CV) of Tohoku Kiritan is handled by voice actor Himika Akaneya. The singing data included in this database are 50 music songs of the idol group that Akaneya belongs to, sung in Tohoku Kiritan’s voice.

Deep learning based on the sound source of this singing data, the MIDI data created based on it, and the text data of the characteristics of the sound makes it possible to create a singing voice synthesis system.

AI Kiritan’s work

In this section, we introduce some songs created using AI Kiritan. All of these songs are available as sample songs on CeVIO AI’s official website.

[CeVIO AI Tohoku Kiritan Official Demo Song] Wagamama My Way!! [Original Song]

[kawaii future disco] Okome Disco/CeVIO AI Tohoku Kiritan Official Demo Song -OSTER project

CeVIO AI Kiritan Official Demo Song / Meido Destruction

The Future of AI Kiritan

AI Kiritan is expected to enrich its content as more works are produced and posted.

Besides that, AI Kiritan is expected to have a considerable impact on the future of speech synthesis software.

The future that speech synthesis software aims for

As pointed out earlier, speech synthesis software has a feature that depends on the singing database. For example, a sound source created by deep learning a database containing pop songs is still suitable for creating pop songs.

In this way, creating songs in various genres and vocal ranges requires a huge vocal database. With AI Kiritan becoming a hot topic, it is expected that there will be more moves to create other singing voice databases.

in conclusion

In this article, we introduced the development history, technology, and music of AI Kiritan. Many original songs and cover songs created by AI Kiritan are posted on video posting sites such as youtube.

Among them, there are many interesting experiments, such as a cover of a VOCALOID song that has been booming since the 2000s, and a Japanese translation of a famous overseas folk song by AI Kiritan. I have.