Google 致力于为黑人社区推动种族平等。查看具体举措

在 Android 上使用 Firebase ML 标记图像

您可以使用 Firebase ML 来标记图像中识别的对象。请参阅概述有关该API的功能的信息。

在你开始之前

  1. 如果你还没有,添加火力地堡到您的Android项目
  2. 使用火力地堡Android的物料清单,声明你的模块(应用程序级)摇篮文件(通常为火力地堡ML视觉的Android库的依赖app/build.gradle )。
    dependencies {
        // Import the BoM for the Firebase platform
        implementation platform('com.google.firebase:firebase-bom:28.4.1')
    
        // Declare the dependency for the Firebase ML Vision library
        // When using the BoM, you don't specify versions in Firebase library dependencies
        implementation 'com.google.firebase:firebase-ml-vision'
    }
    

    通过使用火力地堡Android的物料清单,您的应用程序将始终使用火力地堡的Android库的兼容版本。

    (替代)声明火力地堡库依赖使用物料清单

    如果您选择不使用 Firebase BoM,则必须在其依赖项行中指定每个 Firebase 库版本。

    需要注意的是,如果你在你的应用程序使用多个火力地堡库,我们强烈建议您使用的物料清单管理库版本,以保证所有版本相互兼容。

    dependencies {
        // Declare the dependency for the Firebase ML Vision library
        // When NOT using the BoM, you must specify versions in Firebase library dependencies
        implementation 'com.google.firebase:firebase-ml-vision:24.1.0'
    }
    
  3. 如果您尚未为您的项目启用基于云的 API,请立即启用:

    1. 打开火力地堡ML API页面的火力地堡控制台。
    2. 如果您尚未升级您的项目以大火定价计划,单击升级到这样做。 (仅当您的项目不在 Blaze 计划中时,系统才会提示您升级。)

      只有 Blaze 级别的项目可以使用基于云的 API。

    3. 如果基于云的API尚未启用,单击启用基于云的API。

现在您已准备好标记图像。

1.准备输入图像

创建FirebaseVisionImage从图像中的对象。图像贴标当您使用运行速度最快的Bitmap ,或者,如果您使用camera2 API,一个JPEG格式的media.Image ,这些建议在可能的情况。

  • 要创建FirebaseVisionImage从对象media.Image对象,例如从设备的照相机捕获图像时一样,通过media.Image对象和图像的旋转FirebaseVisionImage.fromMediaImage()

    如果使用CameraX图书馆, OnImageCapturedListenerImageAnalysis.Analyzer类为你计算旋转值,所以你只需要旋转转换成火力地堡ML的一个ROTATION_调用之前常数FirebaseVisionImage.fromMediaImage()

    爪哇

    private class YourAnalyzer implements ImageAnalysis.Analyzer {
    
        private int degreesToFirebaseRotation(int degrees) {
            switch (degrees) {
                case 0:
                    return FirebaseVisionImageMetadata.ROTATION_0;
                case 90:
                    return FirebaseVisionImageMetadata.ROTATION_90;
                case 180:
                    return FirebaseVisionImageMetadata.ROTATION_180;
                case 270:
                    return FirebaseVisionImageMetadata.ROTATION_270;
                default:
                    throw new IllegalArgumentException(
                            "Rotation must be 0, 90, 180, or 270.");
            }
        }
    
        @Override
        public void analyze(ImageProxy imageProxy, int degrees) {
            if (imageProxy == null || imageProxy.getImage() == null) {
                return;
            }
            Image mediaImage = imageProxy.getImage();
            int rotation = degreesToFirebaseRotation(degrees);
            FirebaseVisionImage image =
                    FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
            // Pass image to an ML Vision API
            // ...
        }
    }
    

    科特林+KTX

    private class YourImageAnalyzer : ImageAnalysis.Analyzer {
        private fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) {
            0 -> FirebaseVisionImageMetadata.ROTATION_0
            90 -> FirebaseVisionImageMetadata.ROTATION_90
            180 -> FirebaseVisionImageMetadata.ROTATION_180
            270 -> FirebaseVisionImageMetadata.ROTATION_270
            else -> throw Exception("Rotation must be 0, 90, 180, or 270.")
        }
    
        override fun analyze(imageProxy: ImageProxy?, degrees: Int) {
            val mediaImage = imageProxy?.image
            val imageRotation = degreesToFirebaseRotation(degrees)
            if (mediaImage != null) {
                val image = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation)
                // Pass image to an ML Vision API
                // ...
            }
        }
    }
    

    如果您不使用为您提供图像旋转的相机库,您可以根据设备的旋转和设备中相机传感器的方向来计算它:

    爪哇

    private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
    static {
        ORIENTATIONS.append(Surface.ROTATION_0, 90);
        ORIENTATIONS.append(Surface.ROTATION_90, 0);
        ORIENTATIONS.append(Surface.ROTATION_180, 270);
        ORIENTATIONS.append(Surface.ROTATION_270, 180);
    }
    
    /**
     * Get the angle by which an image must be rotated given the device's current
     * orientation.
     */
    @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
    private int getRotationCompensation(String cameraId, Activity activity, Context context)
            throws CameraAccessException {
        // Get the device's current rotation relative to its "native" orientation.
        // Then, from the ORIENTATIONS table, look up the angle the image must be
        // rotated to compensate for the device's rotation.
        int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
        int rotationCompensation = ORIENTATIONS.get(deviceRotation);
    
        // On most devices, the sensor orientation is 90 degrees, but for some
        // devices it is 270 degrees. For devices with a sensor orientation of
        // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
        CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
        int sensorOrientation = cameraManager
                .getCameraCharacteristics(cameraId)
                .get(CameraCharacteristics.SENSOR_ORIENTATION);
        rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360;
    
        // Return the corresponding FirebaseVisionImageMetadata rotation value.
        int result;
        switch (rotationCompensation) {
            case 0:
                result = FirebaseVisionImageMetadata.ROTATION_0;
                break;
            case 90:
                result = FirebaseVisionImageMetadata.ROTATION_90;
                break;
            case 180:
                result = FirebaseVisionImageMetadata.ROTATION_180;
                break;
            case 270:
                result = FirebaseVisionImageMetadata.ROTATION_270;
                break;
            default:
                result = FirebaseVisionImageMetadata.ROTATION_0;
                Log.e(TAG, "Bad rotation value: " + rotationCompensation);
        }
        return result;
    }

    科特林+KTX

    private val ORIENTATIONS = SparseIntArray()
    
    init {
        ORIENTATIONS.append(Surface.ROTATION_0, 90)
        ORIENTATIONS.append(Surface.ROTATION_90, 0)
        ORIENTATIONS.append(Surface.ROTATION_180, 270)
        ORIENTATIONS.append(Surface.ROTATION_270, 180)
    }
    /**
     * Get the angle by which an image must be rotated given the device's current
     * orientation.
     */
    @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
    @Throws(CameraAccessException::class)
    private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int {
        // Get the device's current rotation relative to its "native" orientation.
        // Then, from the ORIENTATIONS table, look up the angle the image must be
        // rotated to compensate for the device's rotation.
        val deviceRotation = activity.windowManager.defaultDisplay.rotation
        var rotationCompensation = ORIENTATIONS.get(deviceRotation)
    
        // On most devices, the sensor orientation is 90 degrees, but for some
        // devices it is 270 degrees. For devices with a sensor orientation of
        // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
        val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
        val sensorOrientation = cameraManager
                .getCameraCharacteristics(cameraId)
                .get(CameraCharacteristics.SENSOR_ORIENTATION)!!
        rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360
    
        // Return the corresponding FirebaseVisionImageMetadata rotation value.
        val result: Int
        when (rotationCompensation) {
            0 -> result = FirebaseVisionImageMetadata.ROTATION_0
            90 -> result = FirebaseVisionImageMetadata.ROTATION_90
            180 -> result = FirebaseVisionImageMetadata.ROTATION_180
            270 -> result = FirebaseVisionImageMetadata.ROTATION_270
            else -> {
                result = FirebaseVisionImageMetadata.ROTATION_0
                Log.e(TAG, "Bad rotation value: $rotationCompensation")
            }
        }
        return result
    }

    然后,通过media.Image对象和旋转值FirebaseVisionImage.fromMediaImage()

    爪哇

    FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);

    科特林+KTX

    val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)
  • 要创建FirebaseVisionImage从文件URI对象,通过该应用上下文和文件的URI FirebaseVisionImage.fromFilePath()当您使用这是有用的ACTION_GET_CONTENT意图,提示用户选择从他们自己的相册应用程序的图像。

    爪哇

    FirebaseVisionImage image;
    try {
        image = FirebaseVisionImage.fromFilePath(context, uri);
    } catch (IOException e) {
        e.printStackTrace();
    }

    科特林+KTX

    val image: FirebaseVisionImage
    try {
        image = FirebaseVisionImage.fromFilePath(context, uri)
    } catch (e: IOException) {
        e.printStackTrace()
    }
  • 要创建FirebaseVisionImage从一个对象ByteBuffer或一个字节数组,首先是作为上述用于计算所述图像旋转media.Image输入。

    然后,创建一个FirebaseVisionImageMetadata包含图像的高度,宽度对象,颜色编码格式,和旋转:

    爪哇

    FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder()
            .setWidth(480)   // 480x360 is typically sufficient for
            .setHeight(360)  // image recognition
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
            .setRotation(rotation)
            .build();

    科特林+KTX

    val metadata = FirebaseVisionImageMetadata.Builder()
            .setWidth(480) // 480x360 is typically sufficient for
            .setHeight(360) // image recognition
            .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
            .setRotation(rotation)
            .build()

    使用缓冲器或阵列,所述元数据对象,以创建FirebaseVisionImage对象:

    爪哇

    FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);
    // Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);

    科特林+KTX

    val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata)
    // Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)
  • 要创建一个FirebaseVisionImage从对象Bitmap对象:

    爪哇

    FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);

    科特林+KTX

    val image = FirebaseVisionImage.fromBitmap(bitmap)
    由表示的图像Bitmap对象必须是直立,而无需额外的旋转。

2. 配置并运行图像标注器

到标签的图像中的对象,通过FirebaseVisionImage对象到FirebaseVisionImageLabelerprocessImage方法。

  1. 首先,获取的实例FirebaseVisionImageLabeler

    爪哇

    FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance()
        .getCloudImageLabeler();
    
    // Or, to set the minimum confidence required:
    // FirebaseVisionCloudImageLabelerOptions options =
    //     new FirebaseVisionCloudImageLabelerOptions.Builder()
    //         .setConfidenceThreshold(0.7f)
    //         .build();
    // FirebaseVisionImageLabeler labeler = FirebaseVision.getInstance()
    //     .getCloudImageLabeler(options);
    

    科特林+KTX

    val labeler = FirebaseVision.getInstance().getCloudImageLabeler()
    
    // Or, to set the minimum confidence required:
    // val options = FirebaseVisionCloudImageLabelerOptions.Builder()
    //     .setConfidenceThreshold(0.7f)
    //     .build()
    // val labeler = FirebaseVision.getInstance().getCloudImageLabeler(options)
    

  2. 然后,图像传递到processImage()方法:

    爪哇

    labeler.processImage(image)
        .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionImageLabel>>() {
          @Override
          public void onSuccess(List<FirebaseVisionImageLabel> labels) {
            // Task completed successfully
            // ...
          }
        })
        .addOnFailureListener(new OnFailureListener() {
          @Override
          public void onFailure(@NonNull Exception e) {
            // Task failed with an exception
            // ...
          }
        });
    

    科特林+KTX

    labeler.processImage(image)
        .addOnSuccessListener { labels ->
          // Task completed successfully
          // ...
        }
        .addOnFailureListener { e ->
          // Task failed with an exception
          // ...
        }
    

3. 获取标签对象的信息

如果图像标记操作成功,名单FirebaseVisionImageLabel对象将被传递到听者的成功。每个FirebaseVisionImageLabel对象代表的东西,被标记的形象。对于每一个标签,你可以得到标签的文字说明,其知识图的实体ID (如果可用),并且置信度得分的比赛。例如:

爪哇

for (FirebaseVisionImageLabel label: labels) {
  String text = label.getText();
  String entityId = label.getEntityId();
  float confidence = label.getConfidence();
}

科特林+KTX

for (label in labels) {
  val text = label.text
  val entityId = label.entityId
  val confidence = label.confidence
}

下一步