Recognize Landmarks with ML Kit on iOS

You can use ML Kit to recognize well-known landmarks in an image.

See the ML Kit quickstart sample on GitHub for an example of this API in use.

Before you begin

  1. If you have not already added Firebase to your app, do so by following the steps in the getting started guide.
  2. Include the ML Kit libraries in your Podfile:
    pod 'Firebase/Core'
    pod 'Firebase/MLVision'
    After you install or update your project's Pods, be sure to open your Xcode project using its .xcworkspace.
  3. In your app, import Firebase:


    import Firebase


    @import Firebase;
  4. If you have not already enabled Cloud-based APIs for your project, do so now:

    1. Open the ML Kit APIs page of the Firebase console.
    2. If you have not already upgraded your project to a Blaze plan, click Upgrade to do so. (You will be prompted to upgrade only if your project isn't on the Blaze plan.)

      Only Blaze-level projects can use Cloud-based APIs.

    3. If Cloud-based APIs aren't already enabled, click Enable Cloud-based APIs.

Configure the landmark detector

By default, the Cloud detector uses the stable version of the model and returns up to 10 results. If you want to change either of these settings, specify them with a VisionCloudDetectorOptions object as in the following example:


let options = VisionCloudDetectorOptions()
options.modelType = .latest
options.maxResults = 20


  FIRVisionCloudDetectorOptions *options =
      [[FIRVisionCloudDetectorOptions alloc] init];
  options.modelType = FIRVisionCloudModelTypeLatest;
  options.maxResults = 20;

In the next step, pass the VisionCloudDetectorOptions object when you create the Cloud detector object.

Run the landmark detector

To recognize landmarks in an image, pass the image as a UIImage or a CMSampleBufferRef to the VisionCloudLandmarkDetector's detect(in:) method:

  1. Get an instance of VisionCloudLandmarkDetector:


    lazy var vision =
    let cloudDetector = vision.cloudLandmarkDetector(options: options)
    // Or, to use the default settings:
    // let cloudDetector = vision.cloudLandmarkDetector()


    FIRVision *vision = [FIRVision vision];
    FIRVisionCloudLandmarkDetector *landmarkDetector = [vision cloudLandmarkDetector];
    // Or, to change the default settings:
    // FIRVisionCloudLandmarkDetector *landmarkDetector =
    //     [vision cloudLandmarkDetectorWithOptions:options];
  2. Create a VisionImage object using a UIImage or a CMSampleBufferRef.

    To use a UIImage:

    1. If necessary, rotate the image so that its imageOrientation property is .up.
    2. Create a VisionImage object using the correctly-rotated UIImage. Do not specify any rotation metadata—the default value, .topLeft, must be used.


      let image = VisionImage(image: uiImage)


      FIRVisionImage *image = [[FIRVisionImage alloc] initWithImage:uiImage];

    To use a CMSampleBufferRef:

    1. Create a VisionImageMetadata object that specifies the orientation of the image data contained in the CMSampleBufferRef buffer.

      For example, if you are using image data captured from the device's back-facing camera:


      let metadata = VisionImageMetadata()
      // Using back-facing camera
      let devicePosition: AVCaptureDevice.Position = .back
      let deviceOrientation = UIDevice.current.orientation
      switch deviceOrientation {
      case .portrait:
          metadata.orientation = devicePosition == .front ? .leftMirrored : .right
      case .landscapeLeft:
          metadata.orientation = devicePosition == .front ? .downMirrored : .up
      case .portraitUpsideDown:
          metadata.orientation = devicePosition == .front ? .rightMirrored : .left
      case .landscapeRight:
          metadata.orientation = devicePosition == .front ? .upMirrored : .down
      case .faceDown, .faceUp, .unknown:
          metadata.orientation = .up


      // Calculate the image orientation
      FIRVisionDetectorImageOrientation orientation;
      // Using front-facing camera
      AVCaptureDevicePosition devicePosition = AVCaptureDevicePositionFront;
      UIDeviceOrientation deviceOrientation = UIDevice.currentDevice.orientation;
      switch (deviceOrientation) {
          case UIDeviceOrientationPortrait:
              if (devicePosition == AVCaptureDevicePositionFront) {
                  orientation = FIRVisionDetectorImageOrientationLeftTop;
              } else {
                  orientation = FIRVisionDetectorImageOrientationRightTop;
          case UIDeviceOrientationLandscapeLeft:
              if (devicePosition == AVCaptureDevicePositionFront) {
                  orientation = FIRVisionDetectorImageOrientationBottomLeft;
              } else {
                  orientation = FIRVisionDetectorImageOrientationTopLeft;
          case UIDeviceOrientationPortraitUpsideDown:
              if (devicePosition == AVCaptureDevicePositionFront) {
                  orientation = FIRVisionDetectorImageOrientationRightBottom;
              } else {
                  orientation = FIRVisionDetectorImageOrientationLeftBottom;
          case UIDeviceOrientationLandscapeRight:
              if (devicePosition == AVCaptureDevicePositionFront) {
                  orientation = FIRVisionDetectorImageOrientationTopRight;
              } else {
                  orientation = FIRVisionDetectorImageOrientationBottomRight;
              orientation = FIRVisionDetectorImageOrientationTopLeft;
      FIRVisionImageMetadata *metadata = [[FIRVisionImageMetadata alloc] init];
      metadata.orientation = orientation;
    2. Create a VisionImage object using the CMSampleBufferRef object and the rotation metadata:


      let image = VisionImage(buffer: bufferRef)
      image.metadata = metadata


      FIRVisionImage *image = [[FIRVisionImage alloc] initWithBuffer:buffer];
      image.metadata = metadata;
  3. Then, pass the image to the detect(in:) method:


    cloudDetector.detect(in: visionImage) { landmarks, error in
      guard error == nil, let landmarks = landmarks, !landmarks.isEmpty else {
        // ...
      // Recognized landmarks
      // ...


    [landmarkDetector detectInImage:image
                         completion:^(NSArray<FIRVisionCloudLandmark *> *landmarks,
                                      NSError *error) {
      if (error != nil) {
      } else if (landmarks != nil) {
        // Got landmarks

Get information about the recognized landmarks

If landmark recognition succeeds, an array of VisionCloudLandmark objects will be passed to the completion handler. From each object, you can get information about a landmark recognized in the image.

For example:


for landmark in landmarks {
  let landmarkDesc = landmark.landmark
  let boundingPoly = landmark.frame
  let entityId = landmark.entityId

  // A landmark can have multiple locations: for example, the location the image
  // was taken, and the location of the landmark depicted.
  for location in landmark.locations {
    let latitude = location.latitude
    let longitude = location.longitude

  let confidence = landmark.confidence


for (FIRVisionCloudLandmark *landmark in landmarks) {
   NSString *landmarkDesc = landmark.landmark;
   CGRect frame = landmark.frame;
   NSString *entityId = landmark.entityId;

   // A landmark can have multiple locations: for example, the location the image
   // was taken, and the location of the landmark depicted.
   for (FIRVisionLatitudeLongitude *location in landmark.locations) {
     double latitude = [location.latitude doubleValue];
     double longitude = [location.longitude doubleValue];

   float confidence = [landmark.confidence floatValue];

Next steps

Before you deploy to production an app that uses a Cloud API, you should take some additional steps to prevent and mitigate the effect of unauthorized API access.

Send feedback about...

Need help? Visit our support page.