转到控制台

在 Android 上使用机器学习套件识别地标

您可以使用机器学习套件来识别图片中的知名地标。

如需了解此 API 的实际应用示例,请查看 GitHub 上的机器学习套件快速入门示例

准备工作

  1. 将 Firebase 添加到您的 Android 项目(如果尚未添加)。
  2. 请务必在您的项目级 build.gradle 文件的 buildscriptallprojects 部分添加 Google 的 Maven 代码库。
  3. 将 Android 版机器学习套件库的依赖项添加到您的模块(应用级)Gradle 文件(通常为 app/build.gradle):
    dependencies {
      // ...
    
      implementation 'com.google.firebase:firebase-ml-vision:22.0.0'
    }
    
  4. 如果您尚未为项目启用云端 API,请立即完成以下操作:

    1. 打开 Firebase 控制台的机器学习套件 API 页面
    2. 如果您尚未将项目升级到 Blaze 方案,请点击升级以执行此操作。(只有在您的项目未采用 Blaze 方案中时,系统才会提示您进行升级。)

      只有 Blaze 级项目才能使用云端 API。

    3. 如果尚未启用云端 API,请点击启用云端 API

配置地标检测器

默认情况下,Cloud 检测器使用模型的 STABLE 版本并最多返回 10 个结果。如果要更改这两项设置中的任何一项,请使用 FirebaseVisionCloudDetectorOptions 对象指定这两项设置。

例如,如需更改这两项默认设置,请按照以下示例构建 FirebaseVisionCloudDetectorOptions 对象:

Java

FirebaseVisionCloudDetectorOptions options =
        new FirebaseVisionCloudDetectorOptions.Builder()
                .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
                .setMaxResults(15)
                .build();

Kotlin

val options = FirebaseVisionCloudDetectorOptions.Builder()
        .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL)
        .setMaxResults(15)
        .build()

如需使用默认设置,可以在下一步中使用 FirebaseVisionCloudDetectorOptions.DEFAULT

运行地标检测器

如需识别图片中的地标,请基于设备上的以下资源创建一个 FirebaseVisionImage 对象:Bitmapmedia.ImageByteBuffer、字节数组或文件。然后,将 FirebaseVisionImage 对象传递给 FirebaseVisionCloudLandmarkDetectordetectInImage 方法。

  1. 通过图片创建 FirebaseVisionImage 对象。

    • 如需基于 media.Image 对象(例如从设备的相机捕获图片时)创建一个 FirebaseVisionImage 对象,请将 media.Image 对象和图片的旋转角度传递给 FirebaseVisionImage.fromMediaImage()

      如果您使用 CameraX 库,OnImageCapturedListenerImageAnalysis.Analyzer 类会为您计算旋转值,因此您只需在调用 FirebaseVisionImage.fromMediaImage() 之前将旋转角度转换为机器学习套件的 ROTATION_ 常量之一:

      Java

      private class YourAnalyzer implements ImageAnalysis.Analyzer {
      
          private int degreesToFirebaseRotation(int degrees) {
              switch (degrees) {
                  case 0:
                      return FirebaseVisionImageMetadata.ROTATION_0;
                  case 90:
                      return FirebaseVisionImageMetadata.ROTATION_90;
                  case 180:
                      return FirebaseVisionImageMetadata.ROTATION_180;
                  case 270:
                      return FirebaseVisionImageMetadata.ROTATION_270;
                  default:
                      throw new IllegalArgumentException(
                              "Rotation must be 0, 90, 180, or 270.");
              }
          }
      
          @Override
          public void analyze(ImageProxy imageProxy, int degrees) {
              if (imageProxy == null || imageProxy.getImage() == null) {
                  return;
              }
              Image mediaImage = imageProxy.getImage();
              int rotation = degreesToFirebaseRotation(degrees);
              FirebaseVisionImage image =
                      FirebaseVisionImage.fromMediaImage(mediaImage, rotation);
              // Pass image to an ML Kit Vision API
              // ...
          }
      }
      

      Kotlin

      private class YourImageAnalyzer : ImageAnalysis.Analyzer {
          private fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) {
              0 -> FirebaseVisionImageMetadata.ROTATION_0
              90 -> FirebaseVisionImageMetadata.ROTATION_90
              180 -> FirebaseVisionImageMetadata.ROTATION_180
              270 -> FirebaseVisionImageMetadata.ROTATION_270
              else -> throw Exception("Rotation must be 0, 90, 180, or 270.")
          }
      
          override fun analyze(imageProxy: ImageProxy?, degrees: Int) {
              val mediaImage = imageProxy?.image
              val imageRotation = degreesToFirebaseRotation(degrees)
              if (mediaImage != null) {
                  val image = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation)
                  // Pass image to an ML Kit Vision API
                  // ...
              }
          }
      }
      

      如果您不使用可提供图片旋转角度的相机库,则可以根据设备的旋转角度和设备中相机传感器的朝向来计算旋转角度:

      Java

      private static final SparseIntArray ORIENTATIONS = new SparseIntArray();
      static {
          ORIENTATIONS.append(Surface.ROTATION_0, 90);
          ORIENTATIONS.append(Surface.ROTATION_90, 0);
          ORIENTATIONS.append(Surface.ROTATION_180, 270);
          ORIENTATIONS.append(Surface.ROTATION_270, 180);
      }
      
      /**
       * Get the angle by which an image must be rotated given the device's current
       * orientation.
       */
      @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
      private int getRotationCompensation(String cameraId, Activity activity, Context context)
              throws CameraAccessException {
          // Get the device's current rotation relative to its "native" orientation.
          // Then, from the ORIENTATIONS table, look up the angle the image must be
          // rotated to compensate for the device's rotation.
          int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation();
          int rotationCompensation = ORIENTATIONS.get(deviceRotation);
      
          // On most devices, the sensor orientation is 90 degrees, but for some
          // devices it is 270 degrees. For devices with a sensor orientation of
          // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
          CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE);
          int sensorOrientation = cameraManager
                  .getCameraCharacteristics(cameraId)
                  .get(CameraCharacteristics.SENSOR_ORIENTATION);
          rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360;
      
          // Return the corresponding FirebaseVisionImageMetadata rotation value.
          int result;
          switch (rotationCompensation) {
              case 0:
                  result = FirebaseVisionImageMetadata.ROTATION_0;
                  break;
              case 90:
                  result = FirebaseVisionImageMetadata.ROTATION_90;
                  break;
              case 180:
                  result = FirebaseVisionImageMetadata.ROTATION_180;
                  break;
              case 270:
                  result = FirebaseVisionImageMetadata.ROTATION_270;
                  break;
              default:
                  result = FirebaseVisionImageMetadata.ROTATION_0;
                  Log.e(TAG, "Bad rotation value: " + rotationCompensation);
          }
          return result;
      }

      Kotlin

      private val ORIENTATIONS = SparseIntArray()
      
      init {
          ORIENTATIONS.append(Surface.ROTATION_0, 90)
          ORIENTATIONS.append(Surface.ROTATION_90, 0)
          ORIENTATIONS.append(Surface.ROTATION_180, 270)
          ORIENTATIONS.append(Surface.ROTATION_270, 180)
      }/**
       * Get the angle by which an image must be rotated given the device's current
       * orientation.
       */
      @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP)
      @Throws(CameraAccessException::class)
      private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int {
          // Get the device's current rotation relative to its "native" orientation.
          // Then, from the ORIENTATIONS table, look up the angle the image must be
          // rotated to compensate for the device's rotation.
          val deviceRotation = activity.windowManager.defaultDisplay.rotation
          var rotationCompensation = ORIENTATIONS.get(deviceRotation)
      
          // On most devices, the sensor orientation is 90 degrees, but for some
          // devices it is 270 degrees. For devices with a sensor orientation of
          // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees.
          val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager
          val sensorOrientation = cameraManager
                  .getCameraCharacteristics(cameraId)
                  .get(CameraCharacteristics.SENSOR_ORIENTATION)!!
          rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360
      
          // Return the corresponding FirebaseVisionImageMetadata rotation value.
          val result: Int
          when (rotationCompensation) {
              0 -> result = FirebaseVisionImageMetadata.ROTATION_0
              90 -> result = FirebaseVisionImageMetadata.ROTATION_90
              180 -> result = FirebaseVisionImageMetadata.ROTATION_180
              270 -> result = FirebaseVisionImageMetadata.ROTATION_270
              else -> {
                  result = FirebaseVisionImageMetadata.ROTATION_0
                  Log.e(TAG, "Bad rotation value: $rotationCompensation")
              }
          }
          return result
      }c

      然后,将 media.Image 对象及其旋转角度值传递给 FirebaseVisionImage.fromMediaImage()

      Java

      FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation);

      Kotlin

      val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)
    • 如需从文件 URI 创建 FirebaseVisionImage 对象,请将应用上下文和文件 URI 传递给 FirebaseVisionImage.fromFilePath()。如果您使用 ACTION_GET_CONTENT Intent 提示用户从图库中选择图片,则这一选项非常有用。

      Java

      FirebaseVisionImage image;
      try {
          image = FirebaseVisionImage.fromFilePath(context, uri);
      } catch (IOException e) {
          e.printStackTrace();
      }

      Kotlin

      val image: FirebaseVisionImage
      try {
          image = FirebaseVisionImage.fromFilePath(context, uri)
      } catch (e: IOException) {
          e.printStackTrace()
      }
    • 如需从 ByteBuffer 或字节数组创建 FirebaseVisionImage 对象,请首先按上述针对 media.Image 输入的方法计算图片旋转角度。

      然后,创建一个包含图片的高度、宽度、颜色编码格式和旋转角度的 FirebaseVisionImageMetadata 对象:

      Java

      FirebaseVisionImageMetadata metadata = new FirebaseVisionImageMetadata.Builder()
              .setWidth(480)   // 480x360 is typically sufficient for
              .setHeight(360)  // image recognition
              .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
              .setRotation(rotation)
              .build();

      Kotlin

      val metadata = FirebaseVisionImageMetadata.Builder()
              .setWidth(480) // 480x360 is typically sufficient for
              .setHeight(360) // image recognition
              .setFormat(FirebaseVisionImageMetadata.IMAGE_FORMAT_NV21)
              .setRotation(rotation)
              .build()

      使用缓冲区或数组以及元数据对象来创建 FirebaseVisionImage 对象:

      Java

      FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);// Or: FirebaseVisionImage image = FirebaseVisionImage.fromByteArray(byteArray, metadata);

      Kotlin

      val image = FirebaseVisionImage.fromByteBuffer(buffer, metadata)// Or: val image = FirebaseVisionImage.fromByteArray(byteArray, metadata)
    • 如需基于 Bitmap 对象创建一个 FirebaseVisionImage 对象,请使用以下代码:

      Java

      FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);

      Kotlin

      val image = FirebaseVisionImage.fromBitmap(bitmap)
      Bitmap 对象表示的图片必须保持竖直,不需要额外的旋转。

  2. 获取 FirebaseVisionCloudLandmarkDetector 的一个实例:

    Java

    FirebaseVisionCloudLandmarkDetector detector = FirebaseVision.getInstance()
            .getVisionCloudLandmarkDetector();
    // Or, to change the default settings:
    // FirebaseVisionCloudLandmarkDetector detector = FirebaseVision.getInstance()
    //         .getVisionCloudLandmarkDetector(options);

    Kotlin

    val detector = FirebaseVision.getInstance()
            .visionCloudLandmarkDetector
    // Or, to change the default settings:
    // val detector = FirebaseVision.getInstance()
    //         .getVisionCloudLandmarkDetector(options)
  3. 最后,将图片传递给 detectInImage 方法:

    Java

    Task<List<FirebaseVisionCloudLandmark>> result = detector.detectInImage(image)
            .addOnSuccessListener(new OnSuccessListener<List<FirebaseVisionCloudLandmark>>() {
                @Override
                public void onSuccess(List<FirebaseVisionCloudLandmark> firebaseVisionCloudLandmarks) {
                    // Task completed successfully
                    // ...
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // Task failed with an exception
                    // ...
                }
            });

    Kotlin

    val result = detector.detectInImage(image)
            .addOnSuccessListener { firebaseVisionCloudLandmarks ->
                // Task completed successfully
                // ...
            }
            .addOnFailureListener {
                // Task failed with an exception
                // ...
            }

获取识别出的地标的相关信息

如果地标识别操作成功,系统会向成功侦听器传递一组 FirebaseVisionCloudLandmark 对象。每个 FirebaseVisionCloudLandmark 对象代表一个在图片中识别出的地标。对于每个地标,您可以获取它在输入图片中的边界坐标、地标名称、地标的纬度和经度、地标的知识图谱实体 ID(如果有)以及匹配的置信度分数。例如:

Java

for (FirebaseVisionCloudLandmark landmark: firebaseVisionCloudLandmarks) {

    Rect bounds = landmark.getBoundingBox();
    String landmarkName = landmark.getLandmark();
    String entityId = landmark.getEntityId();
    float confidence = landmark.getConfidence();

    // Multiple locations are possible, e.g., the location of the depicted
    // landmark and the location the picture was taken.
    for (FirebaseVisionLatLng loc: landmark.getLocations()) {
        double latitude = loc.getLatitude();
        double longitude = loc.getLongitude();
    }
}

Kotlin

for (landmark in firebaseVisionCloudLandmarks) {

    val bounds = landmark.boundingBox
    val landmarkName = landmark.landmark
    val entityId = landmark.entityId
    val confidence = landmark.confidence

    // Multiple locations are possible, e.g., the location of the depicted
    // landmark and the location the picture was taken.
    for (loc in landmark.locations) {
        val latitude = loc.latitude
        val longitude = loc.longitude
    }
}

后续步骤

在向生产环境中部署使用 Cloud API 的应用之前,您应该采取一些额外步骤来防止未经授权的 API 访问并减轻其造成的影响