During this period, I have been working on some details of iOS development. First, I followed up the Audio part (hereinafter referred to as Audio), which is mainly divided into the following parts:

  1. Architecture and framework of Audio
  2. Codec/file encapsulation format
  3. Play system sound/vibration/prompt sound
  4. Comprehensive demo
  5. Using AVFoundation framework for Chinese and English speech recognition

Speaking of Audio in iOS, the most familiar is AVFoundation. After all, it is an all-purpose framework. However, AVFoundation’s current status can be compared to that of JavaScript, which even has a hand in embedded development 🙂.

But these so-called all-rounders, with their broad repertoire of skills, lack some foundation. It was during this period that I discovered that there were some excellent audio processing frameworks, such as openAL for 3D sound, AudioToolBox for codec process, etc. The key point is that these frameworks are provided in the iOS SDK itself.

According to online information, the position of each frame of audio processing in iOS is sorted out as follows.

Top service

AVAudioPlayer

** Basic operations: ** Play, pause, stop, loop and some basic audio playback functions.

** Control: ** Can play the audio at any time and position; Schedule control.

** Others: ** Can play sounds from files or buffers; Get key audio and video parameters, such as audio title, author, power, and so on.

If we don’t want to implement things like 3D stereo sound, precise synchronization of audio lyrics, etc., then the API provided by this framework is perfectly adequate, but if we want to capture something like audio stream, then we need to do some streaming protocols like RTSP and RTMP. Or some RAC, PCM or PCM to MP3 and other audio transcoding way processing, that the framework is very catch chicken. 🙂 but it is very easy to carry out simple audio operations, as shown above basic operations, controls, etc.

AudioQueue

It’s much more powerful than AVAudioPlayer! Not only can it play audio and record audio, it can also get the raw information of the audio through the AudioQueue, think about it! We can get the audio of the original information, that can do such as arbitrary encoding and decoding, some special effects such as change sound and so on SAO operation! We can also carry out any application layer encapsulation, such as encapsulation for RTMP, RTSP streaming protocol processing.

With Audio Queue, there are only three steps:

  1. Initialize the Audio Queue. Add some playback sources, audio formats, etc.
  2. Manage callback methods. In the callback method we can get the raw data of the audio.
  3. Instantiate the Audio Queue. Use AudioQueueOutput to complete the final playback of the audio.

openAL

OpenGL is mainly used to process some 3D images or changes. OpenAL is mainly used to set between the sound source object, the sound buffer and the listener to achieve 3D effects, such as setting the direction, speed and state of the sound source. So we can hear this 3D effect of sound from far to near.

In general, openAL has three main aspects,

  1. Sound source setting;
  2. Receiver control;
  3. Setting sound source mode. For example, if the sound source is moving from far to near, or from near to far, we can also set the sound source in a 3D space.

AudioFile

To read the information of an AudioFile (note that the AudioFile is not encoded or decoded), read the information of an AudioFile through the relevant API of AudioFile framework, mainly through the following steps:

  1. AudioFileOpenURL. First we open the audio file through a URL.

  2. AudioFileGetPropertyInfo. Gets the type of audio file information we want to read.

  3. AudioFileGetProperty. I’m just going to get the NSLog of the relevant audio.

  4. AudioFileClose. Close the audio file. (Open the file to close the file 🙂)

We can see from the above that they are basically classified as Get methods, but AudioFile also provides a rich set method, which can modify the corresponding audio information in real time.

For example 🌰🍐!!

We first have to introduce # import < AudioToolbox/AudioToolbox. H > framework, begin from Xcode 7, we won’t need to manually introduced framework, because when we introduce the corresponding framework of the iOS SDK. H file, Xcode will automatically import the corresponding framework for us.

    // First extract the audio file path from the application sandbox
    NSString *audioPath = [[NSBundle mainBundle] pathForResource:@"test" ofType:@"mp3"];
    // transpose to URL
    NSURL *audioURL = [NSURL fileURLWithPath:audioPath];
    // Open audio
    // Set the audio file identifier
    AudioFileID audioFile;
    // Use the transposed audio file URL to open the obtained audio file
    // kAudioFileReadPermission: open an audio file in read-only mode; (__bridge CFURLRef) : only C-style type variables are accepted, so we use a strong bridge type to go back
    AudioFileOpenURL((__bridge CFURLRef)audioURL, kAudioFileReadPermission, 0, &audioFile);
    / / read
    UInt32 dictionarySize = 0;
    AudioFileGetPropertyInfo(audioFile, kAudioFilePropertyInfoDictionary, &dictionarySize, 0);
    CFDictionaryRef dictionary;
    AudioFileGetProperty(audioFile, kAudioFilePropertyInfoDictionary, &dictionarySize, &dictionary);
    // After the above two steps, we get the relevant information of the corresponding audio. Then forcibly switch the bridge type back.
    NSDictionary *audioDic = (__bridge NSDictionary *)dictionary;
    for (int i = 0; i < [audioDic allKeys].count; i++) {
        NSString *key = [[audioDic allKeys] objectAtIndex:i];
        NSString *value = [audioDic valueForKey:key];
        NSLog(@ % @ - % @ "", key, value);
    }
    CFRelease(dictionary);
    AudioFileClose(audioFile);
Copy the code

After running the project, you can see the corresponding log,

Frameworks associated with iOS Audio include:

framework Name uses
MediaPlayer.framework VC, provides some control class ViewController, relatively simple to use, fatal shortcomings: single function, the underlying API is highly encapsulated, highly integrated, not conducive to customization
AudioIUnit.framework Low-level, provides core audio processing plug-ins, such as audio unit types, audio component interfaces, and audio input/output units, for controlling the low-level interaction of audio
OpenAL.framework 3D, provides 3D audio effects
AVFoundation.framework Versatile, audio recording, playback and post-processing (based on C)
AudioToolbox.framework Codec, audio codec format conversion

To sum up, in our daily development, we should also focus on the high-level service framework in the iOS audio architecture, which is the part of the daily development of the code, and at the framework level, we should focus on AVFoundation, although it is a C-based framework. 🙂, but it can carry out subtle control of audio. When we use AVFoundation to record and play, we can get the original PCM decoded data of audio, which can carry out special processing of audio. If we were making an audio player class, we would use mediaPlayer. framework a lot.

In mid-level Services, if you do some streaming of Audio, such as RTMP, RTSP, etc., you may want to use Audio Convert Services (I feel I can’t use 😂). For example, when we use RTMP for voice broadcast, the data collected by the microphone may be the original PCM data, but if we want to use AAC format for playback, we need to Convert THE PCM to AAC, and then use Audio Convert Services, the intermediate layer service.

When we want to do some Audio encryption algorithms or Audio encryption sound waves, we may use the Audio Unit Services of the middle layer, which can carry out some fine control of the hardware layer. Audio File Services is the encapsulation and change of Audio files. Therefore, in addition to the relevant framework of the low-level services, the middle-level and high-level services are what we (especially myself 🙂) need to focus on.

Audio SystemSound

The SystemSound framework is used to play system sounds, such as special beeps, vibrations, etc. If you want to use this framework to play custom sounds, the corresponding audio should be encoded as PCM’s original audio, which is usually no longer than 30 seconds (you can’t do more than that, but 🙂 is not recommended).

When we used the framework to invoke vibration, it only worked on the iPhone series, not the iPod or iPad series, because only the iPhone series was thick enough to allow the vibration module to be plugged in (and that was the improved Tapic Engine). When we use this frame to play the system music effect, the mute situation is invalid; When playing prompt music effect, it is effective whether mute or not.

SystemSound is therefore more useful for playing prompts and special short sound effects in games.

An 🌰 🍐!

    NSString *deviceType = [[UIDevice currentDevice] model];
    if ([deviceType isEqualToString:@"iPhone"]) {
        // Call normal vibration module, mute invalid
        AudioServicesPlaySystemSound(kSystemSoundID_Vibrate);
    } else {
        UIAlertController *alertVC = [UIAlertController alertControllerWithTitle:@ "note" message:@" Your device does not support vibration" preferredStyle:UIAlertControllerStyleAlert];
        [self presentViewController:alertVC animated:true completion:^{
            
        }];
    }
Copy the code

The above is our test code for calling the vibration module. It has been stated above that the effect can only be reflected in iPhone series devices, so we had better add the device type judgment (of course, you can not add 🙂). The framework is also based on C (directly operating the underlying hardware), and the code style tends to BE C. Actually this sentence AudioServicesPlaySystemSound (kSystemSoundID_Vibrate); You can find other SystemSoundids in this article. If the sound provided by the system doesn’t work for us, we can load custom sounds.

    NSURL *systemSoundURL = [NSURL fileURLWithPath:[[NSBundle mainBundle] pathForResource:@"test" ofType:@"mp3"]]./ / create the ID
    SystemSoundID systemSoundID;
    AudioServicesCreateSystemSoundID((CFURLRef)CFBridgingRetain(systemSoundURL), &systemSoundID);
    / / register the callBack
    AudioServicesAddSystemSoundCompletion(systemSoundID, nil.nil, soundFinishPlaying, nil);
    // Play the sound
    AudioServicesPlaySystemSound(systemSoundID);
Copy the code

Analysis of the above test code found an interesting phenomenon, even custom sound is also going through AudioServicesPlaySystemSound load audio file identifier, so you can bold speculation! The reason iOS takes up so much storage space is that there is a significant portion of the audio resources for system sound. The unused sound effects cannot be deleted, probably because other apps will use them. 🙂

Audio parameters (not much to know, first record a wave)

Sampling rate:

44100 CD 44100 CD 44100 There are some other 32 kilohertz. The higher the sampling frequency, the higher the frequency of the sound wave that can be depicted.

Quantitative accuracy

Precision is a measure of how accurate something is. A unit of quantization that divides an analog signal into multiple levels. The more accurate the quantization, the closer the amplitude of the sound is to the original sound. Because we usually hear music or sound are analog signals, and computer processing is digital signals, analog signals into digital signals this process is called quantization. And quantization, we need a certain signal to approach it, and the process of approximation, that is the process of quantization, the precision of approximation, is also called quantization precision. So no matter how we approximate it, it’s only approximate, it’s still a little bit different from the original simulation information. The higher the precision, the more nuanced it sounds

Bit rate

The semaphore transmitted by a digital signal per second.

Take a comprehensive example at 🌰🍐

By using the < AVFoundation/AVFoundation. H > and < AudioToolbox/AudioToolbox. H > frame to complete this case, in this instance, speak read an audio file, to play, pause, stop operation, such as In addition, you can set whether to mute, play times, adjust volume and time, and see the current audio playing progress.

The interface is very simple to build, we can customize, just need to drag out the corresponding related control properties and methods.

// Play button click events
- (IBAction)playerBtnClick:(id)sender {
    // Set the audio resource path
    NSString *playMusicPath = [[NSBundle mainBundle] pathForResource:@"test" ofType:@"mp3"];
    if (playMusicPath) {
        // Start the Audio session instance
        [[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayback error:nil];
        NSURL *musicURL = [NSURL fileURLWithPath:playMusicPath];
        audioPlayer = [[AVAudioPlayer alloc] initWithContentsOfURL:musicURL error:nil];
        audioPlayer.delegate = self;
        audioPlayer.meteringEnabled = true;
        // Set timer to refresh the corresponding file information of audio every 0.1 seconds (disguised as real-time 🙂)
        timer = [NSTimer scheduledTimerWithTimeInterval:0.1 target:self selector:@selector(monitor) userInfo:nil repeats:true]; [audioPlayer play]; }}Copy the code
// Timer task
- (void)monitor {
    // numberOfChannels
    NSUInteger channels = audioPlayer.numberOfChannels;
    NSTimeInterval duration = audioPlayer.duration;
    [audioPlayer updateMeters];
    NSString *peakValue = [NSString stringWithFormat:@"%f, %f\n channels=%lu duration=%lu\n currentTime=%f", [audioPlayer peakPowerForChannel:0], [audioPlayer peakPowerForChannel:1], (unsigned long)channels, (unsigned long)duration, audioPlayer.currentTime];
    self.audioInfo.text = peakValue;
    self.musicProgress.progress = audioPlayer.currentTime / audioPlayer.duration;
}
Copy the code
// Pause button click events
- (IBAction)pauseBtnClick:(id)sender {
    // Click pause again to play
    if ([audioPlayer isPlaying]) {
        [audioPlayer pause];
    } else{ [audioPlayer play]; }}Copy the code
// Stop button click event
- (IBAction)stopBtnClick:(id)sender {
    self.volSlider.value = 0;
    self.timeSlider.value = 0;
    [audioPlayer stop];
}
Copy the code
// Mute button click method
- (IBAction)muteSwitchClick:(id)sender {
    // In fact, the volume is 0, which means mute
    // It happens that this is a Switch
    audioPlayer.volume = [sender isOn];
}
Copy the code
// Adjust the audio time method (UIProgress)
- (IBAction)timeSliderClick:(id)sender {
    [audioPlayer pause];
    // Prevent normalization (Xcode defaults to 0~1, converted to actual values)
    [audioPlayer setCurrentTime:(NSTimeInterval)self.timeSlider.value * audioPlayer.duration];
    [audioPlayer play];
}
Copy the code
UIStepper click event (audio loop)
- (IBAction)cycBtnClick:(id)sender {
    audioPlayer.numberOfLoops = self.cyc.value;
}
Copy the code

Speech recognition

With iOS 7, AVFoundation provides speech recognition, which is very easy to use,

// Voice recognition controller
AVSpeechSynthesizer* speechManager = [[AVSpeechSynthesizer alloc] init];
speechManager.delegate = self;
// Speech recognition unit
AVSpeechUtterance* uts = [[AVSpeechUtterance alloc] initWithString:@ "23333"];
uts.rate = 0.5;
[speechManager speakUtterance:uts];
Copy the code

Note that if the native system language is set to English, it can not recognize Chinese oh!

See the Demo here.

The original link: pjhubs.com