Lifelong Learning with you, this is Android programmer

This article covers some of the basics of Android development. Here’s what you’ll learn:

Treble mechanism on Android O ii. Framework update of Camera HAL3 III. Core concept: Request

Treble on Android O

In Android O, when the system starts up, it launches a CameraProvider service, which is separated from the Cameraserver process. [email protected] as a separate process is used to control camera HAL, cameraserver through HIDL mechanism in the camera to communicate with the provider. HIDL is derived from the Treble mechanism added in Android O. Its main function is to isolate service from HAL so that HAL can be upgraded independently. It is similar to the communication mechanism between APP and Framework with Binder. Decoupling of the different levels (from Local call to Remote call) was achieved by introducing an interprocess communication mechanism. The diagram below:

　　　　　　

The call logic of cameraserver and Provider is as follows:

　　　　　　

Second, Camera HAL3 frame update

• Application framework:

The Camera API2, used to provide apps with access to Hardware, uses binder to access Camera Service.

• AIDL:

Binder-based interface for App FW code to access Natice FW code. Actually exists in the following path: frameworks/av/camera/aidl/android/hardware. Among them:

(1) ICameraService is the interface of camera service. Used to request connections, add listeners, and so on. (2) ICameraDeviceUser is the interface of the particular camera device that has been opened. Application frameworks can access specific devices through it. ICameraServiceListener and ICameraDeviceCallbacks are callbacks to the application framework from CameraService and CameraDevice, respectively.

• Natice framework:

Frameworks/av /. Provides ICameraService, ICameraDeviceUser, ICameraDeviceCallbacks, ICameraServiceListener and other AIDL interface implementation. And camera Server’s main function.

• Binder IPC interface:

Provides an interface for interprocess communication, APP and CameraService communication, CameraService and HAL communication. Among them, AIDL and HIDL are implemented based on Binder.

• Camera Service:

Frameworks/av/services/camera /. Services that interact with APP and HAL serve as a link between the preceding and the following.

• HAL:

HAL defines a standard interface that Camera Services can access. For vendors, these interfaces must be implemented.

##2.1 Camera HAL3 build a routing process

As shown below (red dotted line is the up route, black dotted line is the down route) :

2.2 Call Flow from App to CameraService

The connection from Application to CameraService involves three layers in the Android architecture: the App layer, the Framework layer, and the Runtime layer. The App layer calls the methods encapsulated by the Framework layer directly, while the Framework layer calls the CameraService functions in Runtime remotely through Binder. The main function call logic for this part is shown below:

In the App, the API to open the camera needs to be called as follows:

• CameraCharacteristics:

To describe the various camera features, we can obtain the CameraManager getCameraCharacteristics(@nonnull String cameraId) method.

• CameraDevice:

Describe a system Camera, similar to the early Camera.

• CameraCaptureSession:

Session class. When you need to take photos, preview, etc., you need to create an instance of this class and use methods in the instance to control it (for example, capture photos ()).

• CaptureRequest:

The CaptureRequest parameter is passed in for all CameraRequest operations. The specific parameter control is also set by CameraRequest member variables.

• CaptureResult:

Describe the result after the photo is taken.

For example, Java code for opening a camera:

mCameraManager.openCamera(currentCameraId, stateCallback, backgroundHandler);
Copy the code

The Camera2 photographing process is as follows:

　　

2.2.1 Framework CameraManager

/frameworks/base/core/java/android/hardware/camera2/CameraManager.java

The initial entry point is the openCamera method of CameraManager, but as you can see from the code, it simply calls the openCameraForUid method.

@RequiresPermission(android.Manifest.permission.CAMERA) public void openCamera(@NonNull String cameraId,
        @NonNull final CameraDevice.StateCallback callback, @Nullable Handler handler) throws CameraAccessException {

    openCameraForUid(cameraId, callback, handler, USE_CALLING_UID);
}
Copy the code

The following code to check the related operations, overlooked some parameters finally main call openCameraDeviceUserAsync method.

public void openCameraForUid(@NonNull String cameraId,
        @NonNull final CameraDevice.StateCallback callback, @Nullable Handler handler, int clientUid) throws CameraAccessException { /* Do something in*/ ...... /* Do something out*/ openCameraDeviceUserAsync(cameraId, callback, handler, clientUid);
}
Copy the code

Refer to the following notes for analysis:

private CameraDevice openCameraDeviceUserAsync(String cameraId, CameraDevice.StateCallback callback, Handler handler, final int uid) throws CameraAccessException { CameraCharacteristics characteristics = getCameraCharacteristics(cameraId); CameraDevice device = null; synchronized (mLock) { ICameraDeviceUser cameraUser = null; Android. Hardware. Camera2. Impl. CameraDeviceImpl deviceImpl = / / instantiate a CameraDeviceImpl. Construct was introduced into the CameraDevice StateCallback and Handler. new android.hardware.camera2.impl.CameraDeviceImpl( cameraId, callback, handler, characteristics, mContext.getApplicationInfo().targetSdkVersion); ICameraDeviceCallbacks callbacks = deviceImpl.getCallbacks(); // Get the CameraDeviceCallback instance, which is the interface provided for remote connections to The CameraDeviceImpl. Try {if (supportsCamera2ApiLocked(cameraId)) {//HAL3 takes this logic and obtains the local interface of the CameraService from the CameraManagerGlobal. It remotely calls (with Binder mechanism) the connectDevice method to connect to the camera device. // Note that the returned cameraUser actually points to the local interface of the remote CameraDeviceClient. Use Cameraservice's CameraDeviceclient implementation for HAL3.2+ Devices ICameraService Cameraservice = CameraManagerGlobal.get().getCameraService(); if (cameraService == null) { throw new ServiceSpecificException( ICameraService.ERROR_DISCONNECTED, "Camera service is currently unavailable"); } cameraUser = cameraService.connectDevice(callbacks, cameraId, mContext.getOpPackageName(), uid); } else { // Use legacy camera implementation for HAL1 devices int id; try { id = Integer.parseInt(cameraId); } catch (NumberFormatException e) { throw new IllegalArgumentException("Expected cameraId to be numeric, but it was: " + cameraId); } Log.i(TAG, "Using legacy camera HAL."); cameraUser = CameraDeviceUserShim.connectBinderShim(callbacks, id); } } catch (ServiceSpecificException e) { /* Do something in */ ...... /* Do something out */ } // TODO: factor out callback to be non-nested, then move setter to constructor // For now, calling setRemoteDevice will fire initial // onOpened/onUnconfigured callbacks. // This function call may post onDisconnected and throw CAMERA_DISCONNECTED if // cameraUser dies during setup. deviceImpl.setRemoteDevice(cameraUser);  // Set CameraDeviceClient to CameraDeviceImpl for management. device = deviceImpl; } return device; }Copy the code

2.2.2 CameraDeviceImpl /frameworks/base/core/java/android/hardware/camera2/Impl/CameraDeviceImpl.java

Before moving on to the camera opening process, take a quick look at the setRemoteDevice method called in CameraDeviceImpl, which saves the retrieved remote device:

/** * Set remote device, which triggers initial onOpened/onUnconfigured callbacks * * <p>This function may post onDisconnected and throw CAMERA_DISCONNECTED if remoteDevice dies * during setup.</p> * */ public void setRemoteDevice(ICameraDeviceUser remoteDevice) throws CameraAccessException { synchronized(mInterfaceLock) { // TODO: Move from decorator to direct binder-mediated exceptions // If setRemoteFailure already called, do nothing if (mInError) return; mRemoteDevice = new ICameraDeviceUserWrapper(remoteDevice); // Wrap the remote device instance with ICameraDeviceUserWrapper. IBinder remoteDeviceBinder = remoteDevice.asBinder(); // Some basic Settings for using the Binder mechanism. // For legacy camera device, remoteDevice is in the same process, and // asBinder returns NULL. if (remoteDeviceBinder ! = null) { try { remoteDeviceBinder.linkToDeath(this, /*flag*/ 0); // If the binder disappears, register a receiver for the flag message. } catch (RemoteException e) { CameraDeviceImpl.this.mDeviceHandler.post(mCallOnDisconnected); throw new CameraAccessException(CameraAccessException.CAMERA_DISCONNECTED, "The camera device has encountered a serious error"); } } mDeviceHandler.post(mCallOnOpened); // This is where the onOpened and onUnconfigured callbacks are triggered. Each callback is invoked using mDeviceHandler to enable a new thread. mDeviceHandler.post(mCallOnUnconfigured); }}Copy the code

2.2.3 the Runtime

With Binder, we remotely call the connectDevice method (called a function in C++, but more conveniently called a method), which is implemented in the CameraService class.

2.2.4 CameraService

/frameworks/av/services/camera/libcameraservice/CameraService.cpp

Status CameraService::connectDevice( const sp<hardware::camera2::ICameraDeviceCallbacks>& cameraCb, const String16& cameraId, const String16& clientPackageName, int clientUid, /*out*/ sp<hardware::camera2::ICameraDeviceUser>* device) { ATRACE_CALL(); Status ret = Status::ok(); String8 id = String8(cameraId); sp<CameraDeviceClient> client = nullptr; // The connectHelper method called here actually implements the connection logic (this method is also eventually called at HAL1). Note that the template types are ICameraDeviceCallbacks and CameraDeviceClient. ret = connectHelper<hardware::camera2::ICameraDeviceCallbacks,CameraDeviceClient>(cameraCb, id, CAMERA_HAL_API_VERSION_UNSPECIFIED, clientPackageName, clientUid, USE_CALLING_PID, API_2, /*legacyMode*/ false, /*shimUpdateOnly*/ false, /*out*/client); if(! ret.isOk()) { logRejected(id, getCallingPid(), String8(clientPackageName), ret.toString8()); return ret; } *device = client; // The client points to type CameraDeviceClient, and the example is the final return result. return ret; }Copy the code

ConnectHelper has a lot of content, but ignores the analysis that we don’t need to focus on yet:

template<class CALLBACK, class CLIENT> Status CameraService::connectHelper(const sp<CALLBACK>& cameraCb, const String8& cameraId, int halVersion, const String16& clientPackageName, int clientUid, int clientPid, apiLevel effectiveApiLevel, bool legacyMode, bool shimUpdateOnly, /*out*/sp<CLIENT>& device) { binder::Status ret = binder::Status::ok(); String8 clientName8(clientPackageName); /* Do something in */ ...... /* Do something out */ sp<BasicClient> tmp = nullptr; // Call makeClient to generate the CameraDeviceClient instance. if(! (ret = makeClient(this, cameraCb, clientPackageName, cameraId, facing, clientPid, clientUid, getpid(), legacyMode, halVersion, deviceVersion, effectiveApiLevel, /*out*/&tmp)).isOk()) { return ret; } // Initialize the CLIENT instance. Note that the template type CLIENT is CameraDeviceClient and the parameter mCameraProviderManager passed in is HAL Service. client = static_cast<CLIENT*>(tmp.get()); LOG_ALWAYS_FATAL_IF(client.get() == nullptr, "%s: CameraService in invalid state", __FUNCTION__); err = client->initialize(mCameraProviderManager); /* Do something in */ ...... /* Do something out */ // Important: release the mutex here so the client can call back into the service from its // destructor (can be at the end of the call) device = client; return ret; }Copy the code

MakeClient selects a specific Client instance based on the API version and HAL version. The Client is then returned to the CameraDeviceImpl instance along the path analyzed above and stored in mRemoteDevice.

Status CameraService::makeClient(const sp<CameraService>& cameraService, const sp<IInterface>& cameraCb, const String16& packageName, const String8& cameraId, int facing, int clientPid, uid_t clientUid, int servicePid, bool legacyMode, int halVersion, int deviceVersion, apiLevel effectiveApiLevel, /*out*/sp<BasicClient>* client) { if (halVersion < 0 || halVersion == deviceVersion) { // Default path: HAL version is unspecified by caller, create CameraClient // based on device version reported by the HAL. switch(deviceVersion) { case CAMERA_DEVICE_API_VERSION_1_0: /* Do something in */ ...... /* Do something out */ case CAMERA_DEVICE_API_VERSION_3_0: case CAMERA_DEVICE_API_VERSION_3_1: case CAMERA_DEVICE_API_VERSION_3_2: case CAMERA_DEVICE_API_VERSION_3_3: case CAMERA_DEVICE_API_VERSION_3_4: if (effectiveApiLevel == API_1) { // Camera1 API route sp<ICameraClient> tmp = static_cast<ICameraClient*>(cameraCb.get()); *client = new Camera2Client(cameraService, tmp, packageName, cameraIdToInt(cameraId), facing, clientPid, clientUid, servicePid, legacyMode); } else { // Camera2 API route : Instantiate the CameraDeviceClient class as a Client (note that the construct passed ICameraDeviceCallbacks, This is connected to the distal CameraDeviceImpl callback) sp < hardware: : camera2: : ICameraDeviceCallbacks > TMP = static_cast<hardware::camera2::ICameraDeviceCallbacks*>(cameraCb.get()); *client = new CameraDeviceClient(cameraService, tmp, packageName, cameraId, facing, clientPid, clientUid, servicePid); } break; default: // Should not be reachable ALOGE("Unknown camera device HAL version: %d", deviceVersion); return STATUS_ERROR_FMT(ERROR_INVALID_OPERATION, "Camera device \"%s\" has unknown HAL version %d", cameraId.string(), deviceVersion); } } else { /* Do something in */ ...... /* Do something out */ } return Status::ok(); }Copy the code

At this point, in the camera opening process, the call logic from App to CameraService is basically complete.

Summary of schematic diagram:

　　

Ps:

• Cameraman Global is the real implementation layer that creates a connection with the JAVA layer’s CameraService to create a connection to the camera.
• CameraDeviceImpl acts as a run context, replacing the JNI layer prior to Android N.

2.3 From CameraService to HAL Service

Since Android O has added Treble, the CameraServer side is CameraService, which will look for existing Provider services. It is added to the internal CameraProviderManager for management through remote calls. The main body of the Provider service is the CameraProvider, which is connected to the Camera HAL implementation of LibHardware at initialization and managed by the CameraModule. After the process is started, the “carrier” for connection is completed (note that QCamera3HWI has not been created yet), which can be simply shown in the following figure:

When the camera is turned on, the full link of this layer will be created, and the main call logic is shown as follows:

　　

Talked about last time in CameraService: : makeClient, instantiate a CameraDeviceClient. Now let’s continue exploring the process of opening the camera, starting with its constructor. This part is mainly carried out at the Runtime layer, which is divided into CameraService and HAL Service for analysis.

2.3.1 CameraDeviceClient

frameworks\av\services\camera\libcameraservice\api2\CameraDeviceClient.cpp

CameraDeviceClient::CameraDeviceClient(const sp<CameraService>& cameraService, const sp<hardware::camera2::ICameraDeviceCallbacks>& remoteCallback, const String16& clientPackageName, const String8& cameraId, int cameraFacing, int clientPid, uid_t clientUid, int servicePid) : Camera2ClientBase(cameraService, remoteCallback, clientPackageName, cameraId, cameraFacing, clientPid, clientUid, ServicePid), // Inherit its parent Camera2ClientBase mInputStream(), mStreamingRequestId(REQUEST_ID_NONE), mRequestIdCounter(0), mPrivilegedClient(false) { char value[PROPERTY_VALUE_MAX]; property_get("persist.camera.privapp.list", value, ""); String16 packagelist(value); if (packagelist.contains(clientPackageName.string())) { mPrivilegedClient = true; } ATRACE_CALL(); ALOGI("CameraDeviceClient %s: Opened", cameraId.string()); }Copy the code

CameraService calls its initialization function after creating CameraDeviceClient:

// Provide the initialization function interface initialize. status_t CameraDeviceClient::initialize(sp<CameraProviderManager> manager) { return initializeImpl(manager); } // The template TProviderPtr is in this case the CameraProviderManager class. Template <typename TProviderPtr> // Initialize the parent class first, and note that the CameraProviderManager is passed in. status_t CameraDeviceClient::initializeImpl(TProviderPtr providerPtr) { ATRACE_CALL(); status_t res; res = Camera2ClientBase::initialize(providerPtr); if (res ! = OK) { return res; } String8 threadName; mFrameProcessor = new FrameProcessorBase(mDevice); threadName = String8::format("CDU-%s-FrameProc", mCameraIdStr.string()); mFrameProcessor->run(threadName.string()); mFrameProcessor->registerListener(FRAME_PROCESSOR_LISTENER_MIN_ID, FRAME_PROCESSOR_LISTENER_MAX_ID, /*listener*/this, /*sendPartials*/true); return OK; }Copy the code

2.3.2 Camera2ClientBase

frameworks\av\services\camera\libcameraservice\common\Camera2ClientBase.cpp**

Template <typename TClientBase> Is specified as CameraDeviceClientBase when CameraDeviceClient inherits From Camera2ClientBase. Camera2ClientBase<TClientBase>::Camera2ClientBase(// constructs related parameters, and the initialization list, Note that TCamCallbacks are specified as ICameraDeviceCallbacks in CameraDeviceClientBase. const sp<CameraService>& cameraService, const sp<TCamCallbacks>& remoteCallback, const String16& clientPackageName, const String8& cameraId, int cameraFacing, int clientPid, uid_t clientUid, int servicePid): TClientBase(cameraService, remoteCallback, clientPackageName, cameraId, cameraFacing, clientPid, clientUid, servicePid), mSharedCameraCallbacks(remoteCallback), mDeviceVersion(cameraService->getDeviceVersion(TClientBase::mCameraIdStr)), mDeviceActive(false) { ALOGI("Camera %s: Opened. Client: %s (PID %d, UID %d)", cameraId.string(), String8(clientPackageName).string(), clientPid, clientUid); mInitialClientPid = clientPid; mDevice = new Camera3Device(cameraId); // Create a Camera3Device. LOG_ALWAYS_FATAL_IF(mDevice == 0, "Device should never be NULL here."); }Copy the code

Go back to the initialization function:

Template <typename TClientBase> // initializes the function interface, the actual implementation part is in initializeImpl. status_t Camera2ClientBase<TClientBase>::initialize(sp<CameraProviderManager> manager) { return initializeImpl(manager);  } //TClientBase corresponds to CameraDeviceClientBase, and TProviderPtr corresponds to CameraProviderManager. template <typename TClientBase> template <typename TProviderPtr> status_t Camera2ClientBase<TClientBase>::initializeImpl(TProviderPtr providerPtr) { ATRACE_CALL(); ALOGV("%s: Initializing client for camera %s", __FUNCTION__, TClientBase::mCameraIdStr.string()); status_t res; // Verify ops permissions res = TClientBase::startCameraOps(); // Call the startCameraOps method of CameraDeviceClientBase to check the ops permissions. if (res ! = OK) { return res; } if (mDevice == NULL) { ALOGE("%s: Camera %s: No device connected", __FUNCTION__, TClientBase::mCameraIdStr.string()); return NO_INIT; } res = mDevice->initialize(providerPtr); // Initialize the instance of Camera3Device, noting that CameraProviderManager is passed in. if (res ! = OK) { ALOGE("%s: Camera %s: unable to initialize device: %s (%d)", __FUNCTION__, TClientBase::mCameraIdStr.string(), strerror(-res), res); return res; } // Set the Notify callback in the Camera3Device instance. wp<CameraDeviceBase::NotificationListener> weakThis(this); res = mDevice->setNotifyCallback(weakThis); return OK; }Copy the code

2.3.3 Camera3Device

frameworks\av\services\camera\libcameraservice\device3\Camera3Device.cpp**

Camera3Device::Camera3Device(const String8 &id): mId(id), mOperatingMode(NO_MODE), mIsConstrainedHighSpeedConfiguration(false), mStatus(STATUS_UNINITIALIZED), mStatusWaiters(0), mUsePartialResult(false), mNumPartialResults(1), mTimestampOffset(0), mNextResultFrameNumber(0), mNextReprocessResultFrameNumber(0), mNextShutterFrameNumber(0), mNextReprocessShutterFrameNumber(0), mListener(NULL), MVendorTagId (CAMERA_METADATA_INVALID_VENDOR_ID) {ATRACE_CALL (); Camera3_callback_ops ::notify = &snotify; camera3_callback_ops::notify = &snotify; camera3_callback_ops::notify = &snotify; camera3_callback_ops::process_capture_result = &sProcessCaptureResult; ALOGV("%s: Created device for camera %s", __FUNCTION__, mId.string()); }Copy the code

The initialization function is long and the operations related to RequestMetadataQueue are omitted.

status_t Camera3Device::initialize(sp<CameraProviderManager> manager) { ATRACE_CALL(); Mutex::Autolock il(mInterfaceLock); Mutex::Autolock l(mLock); ALOGV("%s: Initializing HIDL device for camera %s", __FUNCTION__, mId.string()); if (mStatus ! = STATUS_UNINITIALIZED) { CLOGE("Already initialized!" ); return INVALID_OPERATION; } if (manager == nullptr) return INVALID_OPERATION; sp<ICameraDeviceSession> session; ATRACE_BEGIN("CameraHal::openSession"); Status_t res = manager->openSession(mid.string (), this, CameraProviderManager openSession(), ** */ *out*/ &session); ATRACE_END(); if (res ! = OK) { SET_ERR_L("Could not open camera session: %s (%d)", strerror(-res), res); return res; } /* Do something in */ ...... /* Do something out */ return initializeCommonLocked(); }Copy the code

2.3.4 CameraProviderManager

frameworks\av\services\camera\libcameraservice\common\CameraProviderManager.cpp

status_t CameraProviderManager::openSession(const std::string &id, const sp<hardware::camera::device::V3_2::ICameraDeviceCallback>& callback, /*out*/ sp<hardware::camera::device::V3_2::ICameraDeviceSession> *session) { std::lock_guard<std::mutex> lock(mInterfaceMutex); Auto deviceInfo = findDeviceInfoLocked(id, findDeviceInfoLocked, DeviceInfo3 /*minVersion*/ {3,0}, /*maxVersion*/ {4,0}); if (deviceInfo == nullptr) return NAME_NOT_FOUND; auto *deviceInfo3 = static_cast<ProviderInfo::DeviceInfo3*>(deviceInfo); Status status; hardware::Return<void> ret; // Create an instance of CameraDeviceSession by remotely calling the open method of CameraDevice and return its local call interface through the input session. ret = deviceInfo3->mInterface->open(callback, [&status, &session] (Status s, const sp<device::V3_2::ICameraDeviceSession>& cameraSession) { status = s; if (status == Status::OK) { *session = cameraSession; }}); if (! ret.isOk()) { ALOGE("%s: Transaction error opening a session for camera device %s: %s", __FUNCTION__, id.c_str(), ret.description().c_str()); return DEAD_OBJECT; } return mapToStatusT(status); }Copy the code

2.3.5 CameraDevice

The hardware \ interfaces \ camera \ device \ \ default \ CameraDevice 3.2 CPP

An instance of CameraDevice actually exists after HAL Service is initialized. Call the remote CameraDevice instance’s Open method via the deviceInfo interface in CameraProviderManager.

Return<void> CameraDevice::open(const sp<ICameraDeviceCallback>& callback, open_cb _hidl_cb) { Status status = initStatus(); sp<CameraDeviceSession> session = nullptr; if (callback == nullptr) { ALOGE("%s: cannot open camera %s. callback is null!" , __FUNCTION__, mCameraId.c_str()); _hidl_cb(Status::ILLEGAL_ARGUMENT, nullptr); return Void(); } if (status ! = Status::OK) { /* Do something in */ ...... /* Do something out */ } else { mLock.lock(); /* Do something in */ ...... /* Do something out */ /** Open HAL device */ status_t res; camera3_device_t *device; ATRACE_BEGIN("camera3->open"); Res = mModule->open(McAmeraid.c_str (), // Note that mModule is configured when HAL Service is initialized. It encapsulates the Camera HAL interface loaded from libHardware. From here you go all the way down to the QCamera3HWI construction process. reinterpret_cast<hw_device_t**>(&device)); ATRACE_END(); /* Do something in */ ...... /* create a session and let the internal member mSession hold it. session = createSession( device, info.static_camera_characteristics, callback); /* Do something in */ ...... /* Do something out */ mSession = session; IF_ALOGV() { session->getInterface()->interfaceChain([]( ::android::hardware::hidl_vec<::android::hardware::hidl_string>  interfaceChain) { ALOGV("Session interface chain:"); for (auto iface : interfaceChain) { ALOGV(" %s", iface.c_str()); }}); } mLock.unlock(); } _hidl_cb(status, session->getInterface()); return Void(); }Copy the code

A CameraDeviceSession is created directly in createSession. Of course, the internal initialization function will be called in its constructor, and then enter HAL interface layer QCamera3HWI initialization process, so that the camera opening process from the CameraService to HAL Service is basically complete. Summary of schematic diagram:

　　

2.4 From HAL Service to Camera HAL

In HAL3, Camera HAL interface conversion layer (layer as well as stream parsing) by QCamera3HardwareInterface bear, and the interface layer and implementation layer with no difference in HAL1 basic, Both are in mm_camerA_interface. C and mm_camera.c. When was the instance of the interface transformation layer created, how was it initialized, and how did it interact with the interface layer and implementation layer when created? The main call flow is shown below:

　　

Against 2.4.1 CameraModule (HAL Service)

The hardware \ interfaces \ camera \ common \ \ default \ CameraModule 1.0 CPP

CameraDevice:: Open: mModule->open: CameraModule::open: mModule->common. Methods ->open: CameraModule::open: mModule->common. To move to the next level of the process. Open is a pointer to the camera_device_open method of QCamera2Factory. This brings us back to the HAL Service startup initialization process.

int CameraModule::open(const char* id, struct hw_device_t** device) { int res;
    ATRACE_BEGIN("camera_module->open");
    res = filterOpenErrorCode(mModule->common.methods->open(&mModule->common, id, device));
    ATRACE_END(); return res;
}
Copy the code

2.4.2 QCamera2Factory (Camera HAL)

/*=========================================================================== * FUNCTION : camera_device_open * * DESCRIPTION: static function to open a camera device by its ID * * PARAMETERS : * @camera_id : camera ID * @hw_device : ptr to struct storing camera hardware device info * * RETURN : int32_t type of status * NO_ERROR -- success * none-zero failure code *==========================================================================*/ int QCamera2Factory::camera_device_open( const struct hw_module_t *module, const char *id, struct hw_device_t **hw_device) { /* Do something in */ ...... /* Do something out */ #ifdef QCAMERA_HAL1_SUPPORT // note that HAL1 compatibility is added via macro definition. You're actually calling cameraDeviceOpen for the next step. if(gQCameraMuxer) rc = gQCameraMuxer->camera_device_open(module, id, hw_device); else #endif rc = gQCamera2Factory->cameraDeviceOpen(atoi(id), hw_device); return rc; } struct hw_module_methods_t QCamera2Factory::mModuleMethods = { .open = QCamera2Factory::camera_device_open, // Specify the open pointer as camerA_device_open. };Copy the code

CameraDeviceOpen works:

/*=========================================================================== * FUNCTION : cameraDeviceOpen * * DESCRIPTION: open a camera device with its ID * * PARAMETERS : * @camera_id : camera ID * @hw_device : ptr to struct storing camera hardware device info * * RETURN : int32_t type of status * NO_ERROR -- success * none-zero failure code *==========================================================================*/ int QCamera2Factory::cameraDeviceOpen(int camera_id, struct hw_device_t **hw_device) { /* Do something in */ ...... /* Do something out */ if ( mHalDescriptors[camera_id].device_version == CAMERA_DEVICE_API_VERSION_3_0 ) { QCamera3HardwareInterface *hw = new QCamera3HardwareInterface(mHalDescriptors[camera_id].cameraId, / / created QCamera3HardwareInterface instance in the first place. mCallbacks); if (! hw) { LOGE("Allocation of hardware interface failed"); return NO_MEMORY; } rc = hw->openCamera(hw_device); // Call the openCamera method of the instance. if (rc ! = 0) { delete hw; } } /* Do something in */ ...... /* Do something out */ return rc; }Copy the code

2.4.3 QCamera3HardwareInterface

 hardware\qcom\camera\qcamera2\hal3\QCamera3HWI.cpp

The first thing to notice is the definition of internal members mCameraOps. When constructing an instance, there is McAmeradevice.ops = &mCameraOps; (Key points)

camera3_device_ops_t QCamera3HardwareInterface::mCameraOps = {
    .initialize = QCamera3HardwareInterface::initialize,
    .configure_streams = QCamera3HardwareInterface::configure_streams,
    .register_stream_buffers = NULL,
    .construct_default_request_settings = QCamera3HardwareInterface::construct_default_request_settings,
    .process_capture_request = QCamera3HardwareInterface::process_capture_request,
    .get_metadata_vendor_tag_ops = NULL,
    .dump = QCamera3HardwareInterface::dump,
    .flush = QCamera3HardwareInterface::flush,
    .reserved = {0},
};
Copy the code

Let’s continue with the openCamera implementation:

int QCamera3HardwareInterface::openCamera(struct hw_device_t **hw_device) { /* Do something in */ ...... /* Do something out */ rc = openCamera(); // Call another openCamera method, which is the implementation part. if (rc == 0) { *hw_device = &mCameraDevice.common; // After the camera is successfully opened, the common part of the device structure is returned through the dual pointer hw_device. } else *hw_device = NULL; /* Do something in */ ...... /* Do something out */ return rc; } int QCamera3HardwareInterface::openCamera() { /* Do something in */ ...... /* Do something out */ rc = camera_open((uint8_t)mCameraId, &mCameraHandle); // The camerA_open interface of the interface layer is called. Note that mCameraHandle is available here. /* Do something in */...... /* Do something out */ rc = mCameraHandle->ops->register_event_notify(mCameraHandle->camera_handle, // Notice that we passed camEvtHandle, (void *)this); /* Do something in */ ...... /* Do something out */ rc = mCameraHandle->ops->get_session_id(mCameraHandle->camera_handle, &sessionId[mCameraId]); /* Do something in */ ...... /* Do something out */ return NO_ERROR; }Copy the code

This is the interface transformation layer, the part about openCamera, so let’s look at its initialization function. As discussed above, creating an instance of CameraDeviceSession calls its internal initialization methods, which include the initialize method that calls QCamera3HWI

int QCamera3HardwareInterface::initialize(const struct camera3_device *device, const camera3_callback_ops_t *callback_ops) { LOGD("E"); QCamera3HardwareInterface *hw = reinterpret_cast<QCamera3HardwareInterface *>(device->priv); if (! hw) { LOGE("NULL camera device"); return -ENODEV; } int rc = hw->initialize(callback_ops); // call the function LOGD("X") that actually implements the initialization logic; return rc; } int QCamera3HardwareInterface::initialize( const struct camera3_callback_ops *callback_ops) { ATRACE_CALL(); int rc; LOGI("E :mCameraId = %d mState = %d", mCameraId, mState); pthread_mutex_lock(&mMutex); // Validate current state switch (mState) { case OPENED: /* valid state */ break; default: LOGE("Invalid state %d", mState); rc = -ENODEV; goto err1; } rc = initParameters(); // mParameters is initialized. Note that the parameters here are different from the CameraParameter, which is the structure of the metadatA_buffer related parameters. if (rc < 0) { LOGE("initParamters failed %d", rc); goto err1; } mCallbackOps = callback_ops; Camera3_call_back_ops is associated with mCallbackOps. MChannelHandle = mCameraHandle->ops->add_channel The method called is actually mm_camerA_intF_ADD_channel in mm_camerA_interface.c. mCameraHandle->camera_handle, NULL, NULL, this); if (mChannelHandle == 0) { LOGE("add_channel failed"); rc = -ENOMEM; pthread_mutex_unlock(&mMutex); return rc; } pthread_mutex_unlock(&mMutex); mCameraInitialized = true; mState = INITIALIZED; LOGI("X"); return 0; err1: pthread_mutex_unlock(&mMutex); return rc; }Copy the code

2.4.4 mm_camerA_interface.c (Interface Layer)

hardware\qcom\camera\qcamera2\stack\mm-camera-interface\src\mm_camera_interface.c

Camera_open doesn’t do much either, omitting the memory allocation and initialization for cam_obj. The actual operation of opening the camera device is actually performed by calling mm_CamerA_open in the implementation layer, and the various information about the device is filled into the CAM_obj structure.

int32_t camera_open(uint8_t camera_idx, mm_camera_vtbl_t **camera_vtbl)
{
    int32_t rc = 0;
    mm_camera_obj_t *cam_obj = NULL; /* Do something in */ ...... /* Do something out */ 
    rc = mm_camera_open(cam_obj); /* Do something in */ ...... /* Do something out */ 
}
Copy the code

The mm_camerA_INTF_add_channel code for initialization is as follows:

static uint32_t mm_camera_intf_add_channel(uint32_t camera_handle, mm_camera_channel_attr_t *attr, mm_camera_buf_notify_t channel_cb, void *userdata) { uint32_t ch_id = 0; mm_camera_obj_t * my_obj = NULL; LOGD("E camera_handler = %d", camera_handle); pthread_mutex_lock(&g_intf_lock); my_obj = mm_camera_util_get_camera_by_handler(camera_handle); if(my_obj) { pthread_mutex_lock(&my_obj->cam_lock); pthread_mutex_unlock(&g_intf_lock); ch_id = mm_camera_add_channel(my_obj, attr, channel_cb, userdata); // Get a channel ID, its handle, by calling the implementation layer's mm_camerA_add_channel. } else { pthread_mutex_unlock(&g_intf_lock); } LOGD("X ch_id = %d", ch_id); return ch_id; }Copy the code

2.4.5mm_camera.c (Implementation layer)

hardware\qcom\camera\qcamera2\stack\mm-camera-interface\src\mm_camera.c

Finally, we get to the lowest level implementation, where mm_camerA_open basically populates my_obj and starts and initializes some thread-related stuff, which I’ve omitted here.

int32_t mm_camera_open(mm_camera_obj_t *my_obj) { char dev_name[MM_CAMERA_DEV_NAME_LEN]; int32_t rc = 0; int8_t n_try=MM_CAMERA_DEV_OPEN_TRIES; uint8_t sleep_msec=MM_CAMERA_DEV_OPEN_RETRY_SLEEP; int cam_idx = 0; const char *dev_name_value = NULL; int l_errno = 0; pthread_condattr_t cond_attr; LOGD("begin\n"); if (NULL == my_obj) { goto on_error; } dev_name_value = mm_camera_util_get_dev_name(my_obj->my_hdl); // The function called here is to get a handle to my_obj. if (NULL == dev_name_value) { goto on_error; } snprintf(dev_name, sizeof(dev_name), "/dev/%s", dev_name_value); sscanf(dev_name, "/dev/video%d", &cam_idx); LOGD("dev name = %s, cam_idx = %d", dev_name, cam_idx); do{ n_try--; errno = 0; my_obj->ctrl_fd = open(dev_name, O_RDWR | O_NONBLOCK); My_obj ->ctrl_fd = my_obj->ctrl_fd l_errno = errno; LOGD("ctrl_fd = %d, errno == %d", my_obj->ctrl_fd, l_errno); if((my_obj->ctrl_fd >= 0) || (errno ! = EIO && errno ! = ETIMEDOUT) || (n_try <= 0 )) { break; } LOGE("Failed with %s error, retrying after %d milli-seconds", strerror(errno), sleep_msec); usleep(sleep_msec * 1000U); }while (n_try > 0); if (my_obj->ctrl_fd < 0) { LOGE("cannot open control fd of '%s' (%s)\n", dev_name, strerror(l_errno)); if (l_errno == EBUSY) rc = -EUSERS; else rc = -1; goto on_error; } else { mm_camera_get_session_id(my_obj, &my_obj->sessionid); // Get the session id after successfully retrieving the file descriptor. LOGH("Camera Opened id = %d sessionid = %d", cam_idx, my_obj->sessionid); } /* Do something in */ ...... /* Do something out */ /* unlock cam_lock, we need release global intf_lock in camera_open(), * in order not block operation of other Camera in dual camera use case.*/ pthread_mutex_unlock(&my_obj->cam_lock); return rc; }Copy the code

For the relevant part of the initialization, the mm_camerA_add_channel code is as follows:

uint32_t mm_camera_add_channel(mm_camera_obj_t *my_obj, mm_camera_channel_attr_t *attr, mm_camera_buf_notify_t channel_cb, void *userdata) { mm_channel_t *ch_obj = NULL; uint8_t ch_idx = 0; uint32_t ch_hdl = 0; For (ch_idx = 0; ch_idx = 0; ch_idx < MM_CAMERA_CHANNEL_MAX; ch_idx++) { if (MM_CHANNEL_STATE_NOTUSED == my_obj->ch[ch_idx].state) { ch_obj = &my_obj->ch[ch_idx]; break; }} /* Initialize the ch_obj structure. Call mm_camerA_util_GENERate_handler to generate a handle (which is also the return value of this function), * then set the status to STOPPED, notice that this also holds a pointer to my_obj and its session ID, Finally, the Channel is initialized by calling mm_channel_init. */ if (NULL ! = ch_obj) { /* initialize channel obj */ memset(ch_obj, 0, sizeof(mm_channel_t)); ch_hdl = mm_camera_util_generate_handler(ch_idx); ch_obj->my_hdl = ch_hdl; ch_obj->state = MM_CHANNEL_STATE_STOPPED; ch_obj->cam_obj = my_obj; pthread_mutex_init(&ch_obj->ch_lock, NULL); ch_obj->sessionid = my_obj->sessionid; mm_channel_init(ch_obj, attr, channel_cb, userdata); } pthread_mutex_unlock(&my_obj->cam_lock); return ch_hdl; }Copy the code

Summary of schematic diagram:

In a word, after the above operation, the whole connection of the camera from top to bottom has been opened. Then the APP should send the Preview Request according to the process and start to obtain Preview data.

Iii. Core Concept: Request

Request is the most important concept throughout the camera2 data processing process. The application framework sends request to the Camera subsystem to obtain the result it wants. Request has the following important characteristics:

• A request can correspond to a series of results
• The request should contain all necessary configuration information and be stored in metadata. Such as resolution and pixel format; Control information of sensor, lens, flash, etc. 3A operation mode; RAW to YUV processing controls; And the generation of statistics.
• Request needs to carry the corresponding surface (that is, the stream inside the frame) to receive the returned image.
• Multiple requests can be in in-flight state at the same time, and submit requests are blocked. If the previous request is not completed, submit a new request.
• Requests in queues are always processed in a FIFO.
• Requests from Snapshot have a higher priority than requests from Preview.

3.1 Overall processing flow chart of Request

　　

• Open process (black arrow line)

  CameraManager registers the AvailabilityCallback callback to receive notifications of changes in the availability status of the camera device. CameraManager gets the currently available camera ID by calling getCameraIdList() and the features of the specified camera device by using getCameraCharacteristcs(). CameraManager calls openCamera() to open the specified CameraDevice and returns a CameraDevice object, which is then used to manipulate the specific CameraDevice. Create a session using the createCaptureSession() of the CameraDevice object. Data requests (previews, photos, etc.) are made through the session. When you create a session, you need to provide the Surface as a parameter to receive the returned image.

• Configure Stream flow (blue arrow lines)

  Apply the Surface, such as the OUTPUT STREAMS DESTINATIONS box above, to receive the image returned by the session as a parameter when the session is created. After the session is created, the Surface is configured as a stream for the framework. In the frame, stream defines the size and format of the image. Each request needs to carry a Target surface that specifies which stream the returned image belongs to.

• Request processing flow (orange arrow line)

  The CameraDevice object creates requests through createCaptureRequest(). Each ReqeUST requires a surface and Settings. All configuration information contained in request is contained in metadata. Requests can be sent to the framework using the Session’s Capture (), captureBurst(), setStreamingRequest(), setStreamingBurst() apis. The previewed request, sent via setStreamingRequest(), setStreamingBurst(), is called only once. Add the request to the Repeating Request List. Copy requests from repeating List to pending Queue as long as there are no requests in the pending Request queue. Capture (), captureBurst() and is called every time a photo needs to be taken. The pending Request queue is added to the pending Request queue each time the request is triggered, so the photographed request has a higher priority than the previewed request. The in-Progress queue represents the queue of pending requests. After each request is processed, a new request is removed from the pending Queue and placed there.

• Data return flow (purple arrow line)

  The data returned by the hardware layer will be returned as a result and will be returned via the Session’s Capture callback.

##3.2 Request processing in HAL

• 1. The framework sends asynchronous requests to Hal.
• Hal must process requests sequentially, returning timestamp (shutter), metadata, and image buffers for each request.
• 3. For each class of Steam referenced by Request, result must be returned as FIFO. For example, result ID 9 must be returned before result ID 10 for a stream preview. However, the stream being photographed can currently only return result ID 7, because the stream being photographed is not the same as the stream being previewed.
• 4. Information required by Hal is received through the metadata carried by request, and information required by Hal is returned through the metadata carried by result.

The overall process of HAL processing request is shown below.

• Request processing flow (black arrow line)

  The framework asynchronously submits the request to Hal, which processes it in turn and returns result. Every request submitted to Hal must carry a stream. The stream is divided into input stream and output stream: The buffer corresponding to the input stream is the buffer of the existing image data, and Hal reprocesses these buffers. The corresponding buffer of the output stream is the empty buffer, which Hal fills with the generated image data.

• Input Stream processing flow (image input Stream 1**) **

  Request carries the input Stream and input buffer to Hal. Hal does the reprocess, and the new image data is repopulated into buffer and returned to the framework.

• Output stream 1**… N) * *

  Request carries the output stream and output buffer to Hal. Hal writes the image data to buffer and returns it to frameowork through a series of modules.

Original link: www.cnblogs.com/blogs-of-lx…

Friendly recommendation: Android dry goods to share

At this point, this has ended, if there is wrong place, welcome your suggestion and correction. Meanwhile, I look forward to your attention. Thank you for reading. Thank you!