在 Android 上使用 Firebase ML 標記映像

您可以使用 Firebase ML 來標記影像中識別的物件。有關此 API 功能的信息,請參閱概述

在你開始之前

  1. 如果您尚未將 Firebase 新增至您的 Android 專案中,請將其新增至您的 Android 專案中。
  2. 模組(應用程式等級)Gradle 檔案(通常<project>/<app-module>/build.gradle.kts<project>/<app-module>/build.gradle )中,新增 Firebase ML 的依賴項Android視覺庫。我們建議使用Firebase Android BoM來控制函式庫版本控制。
    dependencies {
        // Import the BoM for the Firebase platform
        implementation(platform("com.google.firebase:firebase-bom:32.8.0"))
    
        // Add the dependency for the Firebase ML Vision library
        // When using the BoM, you don't specify versions in Firebase library dependencies
        implementation 'com.google.firebase:firebase-ml-vision'
    }
    

    透過使用Firebase Android BoM ,您的應用程式將始終使用 Firebase Android 程式庫的相容版本。

    (替代方法)在不使用 BoM 的情況下新增 Firebase 庫依賴項

    如果您選擇不使用 Firebase BoM,則必須在其依賴項行中指定每個 Firebase 庫版本。

    請注意,如果您在應用程式中使用多個Firebase 程式庫,我們強烈建議使用 BoM 來管理程式庫版本,這可確保所有版本相容。

    dependencies {
        // Add the dependency for the Firebase ML Vision library
        // When NOT using the BoM, you must specify versions in Firebase library dependencies
        implementation 'com.google.firebase:firebase-ml-vision:24.1.0'
    }
    
    正在尋找 Kotlin 特定的庫模組?2023 年 10 月(Firebase BoM 32.5.0)開始,Kotlin 和 Java 開發人員都可以依賴主庫模組(有關詳細信息,請參閱有關此計劃的常見問題解答)。
  3. 如果您尚未為您的專案啟用基於雲端的 API,請立即執行此操作:

    1. 開啟 Firebase 控制台的Firebase ML API 頁面
    2. 如果您尚未將項目升級到 Blaze 定價計劃,請按一下升​​級來執行此操作。 (只有當您的專案不在 Blaze 計劃中時,系統才會提示您升級。)

      只有 Blaze 等級的項目才能使用基於雲端的 API。

    3. 如果尚未啟用基於雲端的 API,請按一下啟用基於雲端的 API

現在您可以為圖像添加標籤了。

1. 準備輸入影像

從您的映像建立FirebaseVisionImage物件。當您使用Bitmap時,圖像標籤器運行速度最快,或者如果您使用camera2 API,則使用 JPEG 格式的media.Image ,如果可能的話,建議您使用這些方式。

  • 若要從media.Image物件建立FirebaseVisionImage物件(例如從裝置的相機擷取影像時),請將media.Image物件和影像的旋轉傳遞給FirebaseVisionImage.fromMediaImage()

    如果您使用CameraX函式庫,則OnImageCapturedListenerImageAnalysis.Analyzer類別會為您計算旋轉值,因此您只需在呼叫FirebaseVisionImage.fromMediaImage()之前將旋轉轉換為 Firebase ML 的ROTATION_常數之一:

    Kotlin+KTX

    private class YourImageAnalyzer : ImageAnalysis.Analyzer {
        private fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) {
            0 -> FirebaseVisionImageMetadata.ROTATION_0
            90 -> FirebaseVisionImageMetadata.ROTATION_90
            180 -> FirebaseVisionImageMetadata.ROTATION_180
            270 -> FirebaseVisionImageMetadata.ROTATION_270
            else -> throw Exception("Rotation must be 0, 90, 180, or 270.")
        }
    
        override fun analyze(imageProxy: ImageProxy?, degrees: Int) {
            val mediaImage = imageProxy?.image
            val imageRotation = degreesToFirebaseRotation(degrees)
            if (mediaImage != null) {
                val image = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation)
                // Pass image to an ML Vision API
                // ...
            }
        }
    }
    

    Java

    private class YourAnalyzer implements ImageAnalysis.Analyzer {
    
        private int degreesToFirebaseRotation(int degrees) {
            switch (degrees) {
                case 0:
                    return FirebaseVisionImageMetadata.ROTATION_0;
                case 90:
                    return FirebaseVisionImageMetadata.ROTATION_90;
                case 180:
                    return FirebaseVisionImageMetadata.ROTATION_180;
                case 270:
                    return FirebaseVisionImageMetadata.ROTATION_270;
                default:
                    throw new IllegalArgumentException(
                            "Rotation must be 0, 90, 180, or 270.");
            }
        }
    
        @Override
        public void analyze(ImageProxy imageProxy, int degrees) {
            if (imageProxy == null || imageProxy.getImage() == null) {
                return;
            }
            Image mediaImage = imageProxy.getImage();
            int rotation = degreesToFirebaseRotation(degrees);
            FirebaseVisionImage image =
                    FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
            // Pass image to an ML Vision API
            // ...
        }
    }
    

    如果您不使用為您提供影像旋轉的相機庫,您可以根據裝置的旋轉和裝置中相機感測器的方向來計算它:

    Kotlin+KTX

    private val ORIENTATIONS = SparseIntArray()
    
    init {
        ORIENTATIONS.append(Surface.ROTATION_0, 90)
        ORIENTATIONS.append(Surface.ROTATION_90, 0)
        ORIENTATIONS.append(Surface.ROTATION_180, 270)
        ORIENTATIONS.append(Surface.ROTATION_270, 180)
    }
    /**
     * Get the angle by which an image must be rotated given the device's current
     * orientation.
     */
    @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
    @Throws(CameraAccessException::class)
    private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int {
        // Get the device's current rotation relative to its "native" orientation.
        // Then, from the ORIENTATIONS table, look up the angle the image must be
        // rotated to compensate for the device's rotation.
        val deviceRotation = activity.windowManager.defaultDisplay.rotation
        var rotationCompensation = ORIENTATIONS.get(deviceRotation)
    
        // On most devices, the sensor orientation is 90 degrees, but for some
        // devices it is 270 degrees. For devices with a sensor orientation of
        // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
        val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
        val sensorOrientation = cameraManager
            .getCameraCharacteristics(cameraId)
            .get(CameraCharacteristics.SENSOR_ORIENTATION)!!
        rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360
    
        // Return the corresponding FirebaseVisionImageMetadata rotation value.
        val result: Int
        when (rotationCompensation) {
            0 -> result = FirebaseVisionImageMetadata.ROTATION_0
            90 -> result = FirebaseVisionImageMetadata.ROTATION_90
            180 -> result = FirebaseVisionImageMetadata.ROTATION_180
            270 -> result = FirebaseVisionImageMetadata.ROTATION_270
            else -> {
                result = FirebaseVisionImageMetadata.ROTATION_0
                Log.e(TAG, "Bad rotation value: $rotationCompensation")
            }
        }
        return result
    }

    Java

    private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
    static {
        ORIENTATIONS.append(Surface.ROTATION_0, 90);
        ORIENTATIONS.append(Surface.ROTATION_90, 0);
        ORIENTATIONS.append(Surface.ROTATION_180, 270);
        ORIENTATIONS.append(Surface.ROTATION_270, 180);
    }
    
    /**
     * Get the angle by which an image must be rotated given the device's current
     * orientation.
     */
    @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
    private int getRotationCompensation(String cameraId, Activity activity, Context context)
            throws CameraAccessException {
        // Get the device's current rotation relative to its "native" orientation.
        // Then, from the ORIENTATIONS table, look up the angle the image must be
        // rotated to compensate for the device's rotation.
        int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
        int rotationCompensation = ORIENTATIONS.get(deviceRotation);
    
        // On most devices, the sensor orientation is 90 degrees, but for some
        // devices it is 270 degrees. For devices with a sensor orientation of
        // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
        CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
        int sensorOrientation = cameraManager
                .getCameraCharacteristics(cameraId)
                .get(CameraCharacteristics.SENSOR_ORIENTATION);
        rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360;
    
        // Return the corresponding FirebaseVisionImageMetadata rotation value.
        int result;
        switch (rotationCompensation) {
            case 0:
                result = FirebaseVisionImageMetadata.ROTATION_0;
                break;
            case 90:
                result = FirebaseVisionImageMetadata.ROTATION_90;
                break;
            case 180:
                result = FirebaseVisionImageMetadata.ROTATION_180;
                break;
            case 270:
                result = FirebaseVisionImageMetadata.ROTATION_270;
                break;
            default:
                result = FirebaseVisionImageMetadata.ROTATION_0;
                Log.e(TAG, "Bad rotation value: " + rotationCompensation);
        }
        return result;
    }

    然後,將media.Image物件和旋轉值傳遞給FirebaseVisionImage.fromMediaImage()

    Kotlin+KTX

    val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)

    Java

    FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
  • 若要從檔案 URI 建立FirebaseVisionImage對象,請將套用上下文和檔案 URI 傳遞給FirebaseVisionImage.fromFilePath() 。當您使用ACTION_GET_CONTENT意圖提示使用者從其圖庫應用程式中選擇影像時,這非常有用。

    Kotlin+KTX

    val image: FirebaseVisionImage
    try {
        image = FirebaseVisionImage.fromFilePath(context, uri)
    } catch (e: IOException) {
        e.printStackTrace()
    }

    Java

    FirebaseVisionImage image;
    try {
        image = FirebaseVisionImage.fromFilePath(context, uri);
    } catch (IOException e) {
        e.printStackTrace();
    }
  • 若要從ByteBuffer或位元組數組建立FirebaseVisionImage對象,請先按照上面針對media.Image輸入所述計算圖像旋轉。

    然後,建立一個FirebaseVisionImageMetadata對象,其中包含圖像的高度、寬度、顏色編碼格式和旋轉:

    Kotlin+KTX

    val metadata = FirebaseVisionImageMetadata.Builder()
        .setWidth(480) // 480x360 is typically sufficient for
        .setHeight(360) // image recognition
        .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
        .setRotation(rotation)
        .build()

    Java

    FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder()
            .setWidth(480)   // 480x360 is typically sufficient for
            .setHeight(360)  // image recognition
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
            .setRotation(rotation)
            .build();

    使用緩衝區或陣列以及元資料物件來建立FirebaseVisionImage物件:

    Kotlin+KTX

    val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata)
    // Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)

    Java

    FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);
    // Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);
  • 要從Bitmap物件建立FirebaseVisionImage物件:

    Kotlin+KTX

    val image = FirebaseVisionImage.fromBitmap(bitmap)

    Java

    FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);
    Bitmap物件表示的影像必須是直立的,不需要額外旋轉。

2. 配置並運行影像標記器

若要標記映像中的對象,請將FirebaseVisionImage物件傳遞給FirebaseVisionImageLabelerprocessImage方法。

  1. 首先,取得FirebaseVisionImageLabeler的實例。

    Kotlin+KTX

    val labeler = FirebaseVision.getInstance().getCloudImageLabeler()
    
    // Or, to set the minimum confidence required:
    // val options = FirebaseVisionCloudImageLabelerOptions.Builder()
    //     .setConfidenceThreshold(0.7f)
    //     .build()
    // val labeler = FirebaseVision.getInstance().getCloudImageLabeler(options)
    

    Java

    FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance()
        .getCloudImageLabeler();
    
    // Or, to set the minimum confidence required:
    // FirebaseVisionCloudImageLabelerOptions options =
    //     new FirebaseVisionCloudImageLabelerOptions.Builder()
    //         .setConfidenceThreshold(0.7f)
    //         .build();
    // FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance()
    //     .getCloudImageLabeler(options);
    

  2. 然後,將影像傳遞給processImage()方法:

    Kotlin+KTX

    labeler.processImage(image)
        .addOnSuccessListener { labels ->
          // Task completed successfully
          // ...
        }
        .addOnFailureListener { e ->
          // Task failed with an exception
          // ...
        }
    

    Java

    labeler.processImage(image)
        .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
          @Override
          public void onSuccess(List<FirebaseVisionImageLabel> labels) {
            // Task completed successfully
            // ...
          }
        })
        .addOnFailureListener(new OnFailureListener() {
          @Override
          public void onFailure(@NonNull Exception e) {
            // Task failed with an exception
            // ...
          }
        });
    

3. 取得標籤物件的信息

如果影像標記操作成功, FirebaseVisionImageLabel物件的清單將傳遞給成功偵聽器。每個FirebaseVisionImageLabel物件都代表影像中標記的內容。對於每個標籤,您可以獲得標籤的文字描述、其知識圖實體 ID (如果可用)以及匹配的置信度分數。例如:

Kotlin+KTX

for (label in labels) {
  val text = label.text
  val entityId = label.entityId
  val confidence = label.confidence
}

Java

for (FirebaseVisionImageLabel label: labels) {
  String text = label.getText();
  String entityId = label.getEntityId();
  float confidence = label.getConfidence();
}

下一步