The architecture of Camera is consistent with the overall architecture of Android system, as shown in the figure below. This paper mainly explains it from the following four aspects.

  1. Framework: Camera. Java
  2. Android Runtime: android_hardware_Camera. CPP
  3. Library: Camera Client and Camera Service
  4. HAL: CameraHardwareInterface

 

Framework: Camera.java

Camera is a class directly used by application layer software, covering all the interfaces of Camera operation such as starting, preview, shooting and closing. Camera. Java in the Android source code path for: framework/base/core/Java/Android/hardware. In order to explain the architecture of the whole Camera system, the functions of Camera. Java are not analyzed horizontally here, and the open() method is used to start:

public static Camera open() {
    int numberOfCameras = getNumberOfCameras();
    CameraInfo cameraInfo = new CameraInfo();
    for (int i = 0; i < numberOfCameras; i++) {
        getCameraInfo(i, cameraInfo);
        if (cameraInfo.facing == CameraInfo.CAMERA_FACING_BACK) {
            return new Camera(i);
        }
    }
    return null;
}Copy the code

The open() method needs to be aware of the following:

  • Getnumberofcamerasnative method, implemented in Android_hardware_camera.cpp;
  • CameraInfo isa static internal class defined by Camera, including Facing, Orientation and canDisableShutterSound.
  • GetCameraInfo internally calls native method _getCameraInfo to obtain camera information.
  • Open () starts the rear camera (CAMERA_FACING_BACK) by default.
/** used by Camera#open, Camera#open(int) */ Camera(int cameraId) { int err = cameraInitNormal(cameraId); if (checkInitErrors(err)) { switch(err) { case EACCESS: throw new RuntimeException("Fail to connect to camera service"); case ENODEV: throw new RuntimeException("Camera initialization failed"); default: // Should never hit this. throw new RuntimeException("Unknown camera error"); }}}Copy the code

The core implementation of the Camera constructor is in cameraInitNormal, which calls cameraInitVersion, The parameters cameraId and CAMERA_HAL_API_VERSION_NORMAL_CONNECT are passed, the latter representing the version of HAL.

private int cameraInitVersion(int cameraId, int halVersion) {
Copy the code
...Copy the code
    String packageName = ActivityThread.currentPackageName();
    return native_setup(new WeakReference<Camera>(this), cameraId, halVersion, packageName);
}Copy the code

CameraInitNormal calls the local method native_setup(), which enters android_hardware_camera.cpp with the following signature:

private native final int native_setup(Object camera_this, int cameraId, int halVersion, String packageName);Copy the code

 

Android Runtime: android_hardware_camera.cpp

Native_setup () is dynamically registered with JNI, through which the android_hardware_Camera_native_setup() method is called.

Static JNINativeMethod camMethods[] = {...... { "native_setup", "(Ljava/lang/Object; ILjava/lang/String;) V "(void *) android_hardware_Camera_native_setup}... };Copy the code

JNI focuses on the implementation of the android_hardware_Camera_native_setup() method:

// connect to camera service static jint android_hardware_Camera_native_setup(JNIEnv *env, jobject thiz, jobject weak_this, jint cameraId, jint halVersion, jstring clientPackageName) { // Convert jstring to String16 const char16_t *rawClientName = env->GetStringChars(clientPackageName, NULL); jsize rawClientNameLen = env->GetStringLength(clientPackageName); String16 clientName(rawClientName, rawClientNameLen); env->ReleaseStringChars(clientPackageName, rawClientName); sp<Camera> camera; if (halVersion == CAMERA_HAL_API_VERSION_NORMAL_CONNECT) { // Default path: hal version is don't care, do normal camera connect. camera = Camera::connect(cameraId, clientName, Camera::USE_CALLING_UID); } else { jint status = Camera::connectLegacy(cameraId, halVersion, clientName, Camera::USE_CALLING_UID, camera); if (status ! = NO_ERROR) { return status; } } if (camera == NULL) { return -EACCES; } // make sure camera hardware is alive if (camera->getStatus() ! = NO_ERROR) { return NO_INIT; } jclass clazz = env->GetObjectClass(thiz); if (clazz == NULL) { // This should never happen jniThrowRuntimeException(env, "Can't find android/hardware/Camera"); return INVALID_OPERATION; } // We use a weak reference so the Camera object can be garbage collected. // The reference is only used as a proxy for  callbacks. sp<JNICameraContext> context = new JNICameraContext(env, weak_this, clazz, camera); context->incStrong((void*)android_hardware_Camera_native_setup); camera->setListener(context); // save context in opaque field env->SetLongField(thiz, fields.context, (jlong)context.get()); return NO_ERROR; }Copy the code

The android_hardware_Camera_native_setup() method requests a connection to the CameraService service by calling the Camera::connect() method. Into the reference:

  • ClientName is obtained by converting clientPackageName from JString to String16 format;
  • Camera::USE_CALLING_UID is an enumeration type defined in camera.h with a value of ICameraService::USE_CALLING_UID (also an enumeration type with a value of -1).

Camera::connect() is in camera.cpp, which leads into the Library layer.

 

Library: Camera Client and Camera Service

H, ICameraClient. H and ICamera.h classes define the interface and architecture of Camera. ICameraService. The specific function of Camera is realized by calling the hardware related interface in the lower layer. Camera. H is the interface of Camera system to the upper layer.

Specifically, the Camera class inherits from the CameraBase template class, where Camera:: Connect () calls the connect() method in camerabase.cpp.

sp<Camera> Camera::connect(int cameraId, const String16& clientPackageName,
        int clientUid)
{
    return CameraBaseT::connect(cameraId, clientPackageName, clientUid);
}Copy the code

CameraBase actually inherits IBinder’s DeathRecipient inner class, which virtually inherits from RefBase. RefBase is the reference counting base class in Android, which defines pointer manipulation functions involving SP/WP such as incStrong, decStrong, incWeak, and decWeak.

template <typename TCam> struct CameraTraits { }; template <typename TCam, typename TCamTraits = CameraTraits<TCam> > class CameraBase : public IBinder::DeathRecipient { public: static sp<TCam> connect(int cameraId, const String16& clientPackageName, int clientUid); ... }Copy the code
class DeathRecipient : public virtual RefBase
{
public:
    virtual void binderDied(const wp<IBinder>& who) = 0;
};Copy the code

Back to the implementation of Camera:: Connect (), where new TCam(cameraId) generates the BnCameraClient object, which is defined in the icameraclient.h file and inherits from the template class BnInterface. The getCameraService() method returns the service proxy for CameraService, BpCameraService, which also inherits from the template class BnInterface. The Binder then communicates with BnCameraService to send the CONNECT command. When BnCameraService receives the CONNECT command, it calls the CONNECT () member function of CameraService to do the corresponding processing.

template <typename TCam, typename TCamTraits> sp<TCam> CameraBase<TCam, TCamTraits>::connect(int cameraId, const String16& clientPackageName, int clientUid) { ALOGV("%s: connect", __FUNCTION__); sp<TCam> c = new TCam(cameraId); // BnCameraClient sp<TCamCallbacks> cl = c; status_t status = NO_ERROR; const sp<ICameraService>& cs = getCameraService(); // BpCameraService if (cs ! = 0) { TCamConnectService fnConnectService = TCamTraits::fnConnectService; status = (cs.get()->*fnConnectService)(cl, cameraId, clientPackageName, clientUid, /*out*/ c->mCamera); } if (status == OK && c->mCamera ! = 0) { c->mCamera->asBinder()->linkToDeath(c); c->mStatus = NO_ERROR; } else { ALOGW("An error occurred while connecting to camera: %d", cameraId); c.clear(); } return c; }Copy the code
class BnCameraClient: public BnInterface<ICameraClient>
{
public:
    virtual status_t    onTransact( uint32_t code,
                                    const Parcel& data,
                                    Parcel* reply,
                                    uint32_t flags = 0);
};Copy the code
class BpCameraService: public BpInterface<ICameraService> { public: BpCameraService(const sp<IBinder>& impl) : BpInterface < ICameraService > (impl) {}... }Copy the code

Note: The connect() function is declared as a pure virtual function in BpCameraService and BnCameraService’s parent ICameraService, and is implemented in BpCameraService and BnCameraService respectively. As a proxy class, BpCameraService provides interface to the client, which is really implemented in BnCameraService subclass CameraService.

In BpCameraService, the connect() function is implemented as follows:

// connect to camera service (android.hardware.Camera) virtual status_t connect(const sp<ICameraClient>& cameraClient, int cameraId, const String16 &clientPackageName, int clientUid, /*out*/ sp<ICamera>& device) { Parcel data, reply; data.writeInterfaceToken(ICameraService::getInterfaceDescriptor()); data.writeStrongBinder(cameraClient->asBinder()); data.writeInt32(cameraId); data.writeString16(clientPackageName); data.writeInt32(clientUid); remote()->transact(BnCameraService::CONNECT, data, &reply); // BpBinder's transact() function sends a message to the IPCThreadState instance notifying it of a message to be sent to the Binder driverCopy the code
if (readExceptionCode(reply)) return -EPROTO; status_t status = reply.readInt32(); if (reply.readInt32() ! = 0) { device = interface_cast<ICamera>(reply.readStrongBinder()); Bind} return status; }Copy the code

The Camera object cameraClient is first converted to type IBinder, the parameters are written to the Parcel (the conduit Binder communicates with), and a message is sent via BpBinder’s Transact () function. BnCameraService then responds to the connection and waits for the server to return, generating a BpCamera instance if successful.

The real server-side response is implemented in BnCameraService’s onTransact() function, which is responsible for unpacking the received Parcel and executing the client-side request.

status_t BnCameraService::onTransact(
    uint32_t code, const Parcel& data, Parcel* reply, uint32_t flags)
{
    switch(code) {
    Copy the code
... case CONNECT: { CHECK_INTERFACE(ICameraService, data, reply); sp<ICameraClient> cameraClient = interface_cast<ICameraClient>(data.readStrongBinder()); Cameraclient int32_t cameraId = data.readInt32(); cameraclient int32_t cameraId = data.readInt32(); const String16 clientName = data.readString16(); int32_t clientUid = data.readInt32(); sp<ICamera> camera; status_t status = connect(cameraClient, cameraId, clientName, clientUid, /*out*/camera); // Pass the generated BpCameraClient object as an argument to CameraService's connect() function reply->writeNoException(); reply->writeInt32(status); // Packing BpCamera objects into a Parcel as IBinder returns if (camera! = NULL) { reply->writeInt32(1); reply->writeStrongBinder(camera->asBinder()); } else { reply->writeInt32(0); } return NO_ERROR; } break; ... }}Copy the code

The main treatments include:

  1. Generate Camera client agent BpCameraClient instance through Binder object of Camera in data.
  2. The generated BpCameraClient object is passed as a parameter to the CameraService (/ frameworks/av/services/camera / libcameraservice CameraService. CPP) of the connect () function, the function returns a BpCamera instance;
  3. Returns the above instance object as an IBinder packaged into a Parcel.

Finally, the BpCamera instance is returned via the CameraService::connect() function. The core of the CameraService:: Connect () implementation is to call connectHelperLocked() to create different client instances for different versions of HAL apis. But the function is basically similar).

status_t CameraService::connectHelperLocked( /*out*/ sp<Client>& client, /*in*/ const sp<ICameraClient>& cameraClient, int cameraId, const String16& clientPackageName, int clientUid, int callingPid, int halVersion, bool legacyMode) { int facing = -1; int deviceVersion = getDeviceVersion(cameraId, &facing); if (halVersion < 0 || halVersion == deviceVersion) { // Default path: HAL version is unspecified by caller, create CameraClient // based on device version reported by the HAL. switch(deviceVersion) { case CAMERA_DEVICE_API_VERSION_1_0: client = new CameraClient(this, cameraClient, clientPackageName, cameraId, facing, callingPid, clientUid, getpid(), legacyMode); break; case CAMERA_DEVICE_API_VERSION_2_0: case CAMERA_DEVICE_API_VERSION_2_1: case CAMERA_DEVICE_API_VERSION_3_0: case CAMERA_DEVICE_API_VERSION_3_1: case CAMERA_DEVICE_API_VERSION_3_2: client = new Camera2Client(this, cameraClient, clientPackageName, cameraId, facing, callingPid, clientUid, getpid(), legacyMode); break; case -1: ALOGE("Invalid camera id %d", cameraId); return BAD_VALUE; default: ALOGE("Unknown camera device HAL version: %d", deviceVersion); return INVALID_OPERATION; } } else { // A particular HAL version is requested by caller. Create CameraClient // based on the requested HAL version. if (deviceVersion > CAMERA_DEVICE_API_VERSION_1_0 && halVersion == CAMERA_DEVICE_API_VERSION_1_0) { // Only Support higher HAL Version device Opened as HAL1.0 device. client = new CameraClient(this, CameraClient, clientPackageName, cameraId, facing, callingPid, clientUid, getpid(), legacyMode); } else { // Other combinations (e.g. HAL3.x open as HAL2.x) are not supported yet. ALOGE("Invalid camera HAL version %x:  HAL %x device can only be" " opened as HAL %x device", halVersion, deviceVersion, CAMERA_DEVICE_API_VERSION_1_0); return INVALID_OPERATION; } } status_t status = connectFinishUnsafe(client, client->getRemote()); if (status ! = OK) { // this is probably not recoverable.. maybe the client can try again return status; } mClient[cameraId] = client; LOG1("CameraService::connect X (id %d, this pid is %d)", cameraId, getpid()); return OK; }Copy the code

As you can see, CameraClient is instantiated before CAMERA_DEVICE_API_VERSION_2_0 and after that Camera2Client is instantiated. In the case of CameraClient, the initialize() function looks like this:

status_t CameraClient::initialize(camera_module_t *module) { int callingPid = getCallingPid(); status_t res; LOG1("CameraClient::initialize E (pid %d, id %d)", callingPid, mCameraId); // Verify ops permissions res = startCameraOps(); if (res ! = OK) { return res; } char camera_device_name[10]; snprintf(camera_device_name, sizeof(camera_device_name), "%d", mCameraId);mHardware = new CameraHardwareInterface(camera_device_name);res = mHardware->initialize(&module->common); if (res ! = OK) { ALOGE("%s: Camera %d: unable to initialize device: %s (%d)", __FUNCTION__, mCameraId, strerror(-res), res); mHardware.clear(); return res; } mHardware->setCallbacks(notifyCallback, dataCallback, dataCallbackTimestamp, (void *)(uintptr_t)mCameraId); // Enable zoom, error, focus, and metadata messages by default enableMsgType(CAMERA_MSG_ERROR | CAMERA_MSG_ZOOM | CAMERA_MSG_FOCUS | CAMERA_MSG_PREVIEW_METADATA | CAMERA_MSG_FOCUS_MOVE); LOG1("CameraClient::initialize X (pid %d, id %d)", callingPid, mCameraId); return OK; }Copy the code

In the above functions, pay attention to the following flows:

  1. The bold code CameraHardwareInterface creates a new Camera hardware interface. Of course, camerA_device_name is the Camera device name.
  2. MHardware ->initialize(& Module ->common) calls the initialization method of the underlying hardware;
  3. MHardware ->setCallbacks registers callbacks at CamerService with HAL.

The CameraHardwareInterface defines the hardware abstraction characteristics of the Camera, which goes into HAL.

 

HAL: CameraHardwareInterface

CameraHardwareInterface is used to link Camera Server to V4L2. By implementing CameraHardwareInterface, different drivers can be shielded from Camera Server. The CameraHardwareInterface also inherits virtually from RefBase.

class CameraHardwareInterface : public virtual RefBase {
public:
    CameraHardwareInterface(const char *name)
    {
        mDevice = 0;
        mName = name;
    }
    ……
}Copy the code

The CameraHardwareInterface contains a control channel and a data channel. The control channel handles the start/stop of preview and video capture, taking photos, and auto focus functions. The data channel uses callback functions to get preview, video recording, and auto focus data. When you need to support new hardware, you need to inherit from the CameraHardwareInterface to implement the corresponding functionality. The CameraHardwareInterface provides the following public methods:

In the previous section, the initialize () function call mHardware – > initialize and mHardware – > setCallbacks, see below CameraHardwareInterface. H to its implementation.

status_t initialize(hw_module_t *module) { ALOGI("Opening camera %s", mName.string()); camera_module_t *cameraModule = reinterpret_cast<camera_module_t *>(module); camera_info info; status_t res = cameraModule->get_camera_info(atoi(mName.string()), &info); if (res ! = OK) return res; int rc = OK; if (module->module_api_version >= CAMERA_MODULE_API_VERSION_2_3 && info.device_version > CAMERA_DEVICE_API_VERSION_1_0) {// Open higher version camera device as HAL1.0 device. rc = cameraModule-> Open_legacy (module, mname.string (), CAMERA_DEVICE_API_VERSION_1_0, (hw_device_t **)&mDevice); } else { rc = CameraService::filterOpenErrorCode(module->methods->open( module, mName.string(), (hw_device_t **)&mDevice)); } if (rc ! = OK) { ALOGE("Could not open camera %s: %d", mName.string(), rc); return rc; } initHalPreviewWindow(); return rc; }Copy the code

In the initialize() method, open the cameraModule via cameraModule->open_legacy. InitHalPreviewWindow () is used to initialize the related stream opspreview_stream_ops for Preview, Initialize the Hal preview window.

void initHalPreviewWindow()
    {
        mHalPreviewWindow.nw.cancel_buffer = __cancel_buffer;
        mHalPreviewWindow.nw.lock_buffer = __lock_buffer;
        mHalPreviewWindow.nw.dequeue_buffer = __dequeue_buffer;
        mHalPreviewWindow.nw.enqueue_buffer = __enqueue_buffer;
        mHalPreviewWindow.nw.set_buffer_count = __set_buffer_count;
        mHalPreviewWindow.nw.set_buffers_geometry = __set_buffers_geometry;
        mHalPreviewWindow.nw.set_crop = __set_crop;
        mHalPreviewWindow.nw.set_timestamp = __set_timestamp;
        mHalPreviewWindow.nw.set_usage = __set_usage;
        mHalPreviewWindow.nw.set_swap_interval = __set_swap_interval;

        mHalPreviewWindow.nw.get_min_undequeued_buffer_count =
                __get_min_undequeued_buffer_count;
    }Copy the code
/** Set the notification and data callbacks */ void setCallbacks(notify_callback notify_cb, data_callback data_cb, data_callback_timestamp data_cb_timestamp, void* user) { mNotifyCb = notify_cb; mDataCb = data_cb; mDataCbTimestamp = data_cb_timestamp; mCbUser = user; ALOGV("%s(%s)", __FUNCTION__, mName.string()); if (mDevice->ops->set_callbacks) { mDevice->ops->set_callbacks(mDevice, __notify_cb, __data_cb, __data_cb_timestamp, __get_memory, this); }}Copy the code

 

The above brief analysis of the response of the application layer after calling Camera.open() in the Framework, ART, Library and HAL layers explains the overall architecture of the Camera system in Android, hoping to be of some help to readers. In the future, we will understand the basis of the overall architecture of Camera. Stay tuned for more efficient Preview!