在 Apple 平台上使用經過 AutoML 訓練的模型偵測影像中的對象

使用 AutoML Vision Edge 訓練自己的模型後,您可以在應用程式中使用它來偵測影像中的物件。

有兩種方法可以整合從 AutoML Vision Edge 訓練的模型。您可以透過將模型的檔案複製到 Xcode 專案中來捆綁模型,也可以從 Firebase 動態下載它。

模型捆綁選項
捆綁在您的應用程式中
  • 該模型是捆綁包的一部分
  • 即使 Apple 裝置處於離線狀態,該車型也可以立即使用
  • 不需要 Firebase 項目
使用 Firebase 託管
  • 透過將模型上傳到Firebase Machine Learning來託管模型
  • 減少應用程式包大小
  • 模型按需下載
  • 推送模型更新而無需重新發​​布您的應用程式
  • 使用Firebase Remote Config輕鬆進行 A/B 測試
  • 需要 Firebase 項目

在你開始之前

  1. 如果您想下載模型,請確保將Firebase 新增至您的 Apple 專案(如果您尚未這樣做)。捆綁模型時不需要這樣做。

  2. 將 TensorFlow 和 Firebase 庫包含在您的 Podfile 中:

    將模型與您的應用程式捆綁在一起:

    迅速

    pod 'TensorFlowLiteSwift'
    

    Objective-C

    pod 'TensorFlowLiteObjC'
    

    若要從 Firebase 動態下載模型,請新增Firebase/MLModelInterpreter依賴:

    迅速

    pod 'TensorFlowLiteSwift'
    pod 'Firebase/MLModelInterpreter'
    

    Objective-C

    pod 'TensorFlowLiteObjC'
    pod 'Firebase/MLModelInterpreter'
    
  3. 安裝或更新專案的 Pod 後,使用.xcworkspace開啟 Xcode 專案。

1.載入模型

配置本地模型來源

若要將模型與您的應用程式捆綁在一起,請將模型和標籤檔案複製到您的 Xcode 項目,執行此操作時請注意選擇「建立資料夾參考」 。模型檔案和標籤將包含在應用程式包中。

另外,請查看與模型一起建立的tflite_metadata.json檔案。您需要兩個值:

  • 模型的輸入尺寸。預設為 320x320。
  • 模型的最大檢測數。預設為 40。

配置 Firebase 託管的模型來源

若要使用遠端託管模型,請建立一個CustomRemoteModel對象,並指定您在發布模型時為其指定的名稱:

迅速

let remoteModel = CustomRemoteModel(
    name: "your_remote_model"  // The name you assigned in the Google Cloud console.
)

Objective-C

FIRCustomRemoteModel *remoteModel = [[FIRCustomRemoteModel alloc]
                                     initWithName:@"your_remote_model"];

然後,啟動模型下載任務,指定允許下載的條件。如果裝置上沒有模型,或者有更新版本的模型可用,則任務將從 Firebase 非同步下載模型:

迅速

let downloadProgress = ModelManager.modelManager().download(
    remoteModel,
    conditions: ModelDownloadConditions(
        allowsCellularAccess: true,
        allowsBackgroundDownloading: true
    )
)

Objective-C

FIRModelDownloadConditions *conditions =
        [[FIRModelDownloadConditions alloc] initWithAllowsCellularAccess:YES
                                             allowsBackgroundDownloading:YES];
NSProgress *progress = [[FIRModelManager modelManager] downloadModel:remoteModel
                                                          conditions:conditions];

許多應用程式在其初始化程式碼中啟動下載任務,但您可以在需要使用模型之前隨時執行此操作。

從您的模型建立物件偵測器

配置模型來源後,從其中一個建立 TensorFlow Lite Interpreter物件。

如果您只有本地捆綁的模型,只需從模型檔案建立解釋器:

迅速

guard let modelPath = Bundle.main.path(
    forResource: "model",
    ofType: "tflite"
) else {
  print("Failed to load the model file.")
  return true
}
let interpreter = try Interpreter(modelPath: modelPath)
try interpreter.allocateTensors()

Objective-C

NSString *modelPath = [[NSBundle mainBundle] pathForResource:@"model"
                                                      ofType:@"tflite"];

NSError *error;
TFLInterpreter *interpreter = [[TFLInterpreter alloc] initWithModelPath:modelPath
                                                                  error:&error];
if (error != NULL) { return; }

[interpreter allocateTensorsWithError:&error];
if (error != NULL) { return; }

如果您有遠端託管模型,則必須在運行之前檢查它是否已下載。您可以使用模型管理器的isModelDownloaded(remoteModel:)方法檢查模型下載任務的狀態。

儘管您只需在運行解釋器之前確認這一點,但如果您同時擁有遠端託管模型和本地捆綁模型,則在實例化Interpreter時執行此檢查可能是有意義的:如果是,則從遠端模型創建一個解釋器已下載,否則從本機模型下載。

迅速

var modelPath: String?
if ModelManager.modelManager().isModelDownloaded(remoteModel) {
    ModelManager.modelManager().getLatestModelFilePath(remoteModel) { path, error in
        guard error == nil else { return }
        guard let path = path else { return }
        modelPath = path
    }
} else {
    modelPath = Bundle.main.path(
        forResource: "model",
        ofType: "tflite"
    )
}

guard modelPath != nil else { return }
let interpreter = try Interpreter(modelPath: modelPath)
try interpreter.allocateTensors()

Objective-C

__block NSString *modelPath;
if ([[FIRModelManager modelManager] isModelDownloaded:remoteModel]) {
    [[FIRModelManager modelManager] getLatestModelFilePath:remoteModel
                                                completion:^(NSString * _Nullable filePath,
                                                             NSError * _Nullable error) {
        if (error != NULL) { return; }
        if (filePath == NULL) { return; }
        modelPath = filePath;
    }];
} else {
    modelPath = [[NSBundle mainBundle] pathForResource:@"model"
                                                ofType:@"tflite"];
}

NSError *error;
TFLInterpreter *interpreter = [[TFLInterpreter alloc] initWithModelPath:modelPath
                                                                  error:&error];
if (error != NULL) { return; }

[interpreter allocateTensorsWithError:&error];
if (error != NULL) { return; }

如果您只有遠端託管模型,則應停用與模型相關的功能(例如,灰顯或隱藏部分 UI),直到確認模型已下載。

您可以透過將觀察者附加到預設通知中心來取得模型下載狀態。請務必在觀察者區塊中使用對self弱引用,因為下載可能需要一些時間,並且在下載完成時可以釋放原始物件。例如:

迅速

NotificationCenter.default.addObserver(
    forName: .firebaseMLModelDownloadDidSucceed,
    object: nil,
    queue: nil
) { [weak self] notification in
    guard let strongSelf = self,
        let userInfo = notification.userInfo,
        let model = userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue]
            as? RemoteModel,
        model.name == "your_remote_model"
        else { return }
    // The model was downloaded and is available on the device
}

NotificationCenter.default.addObserver(
    forName: .firebaseMLModelDownloadDidFail,
    object: nil,
    queue: nil
) { [weak self] notification in
    guard let strongSelf = self,
        let userInfo = notification.userInfo,
        let model = userInfo[ModelDownloadUserInfoKey.remoteModel.rawValue]
            as? RemoteModel
        else { return }
    let error = userInfo[ModelDownloadUserInfoKey.error.rawValue]
    // ...
}

Objective-C

__weak typeof(self) weakSelf = self;

[NSNotificationCenter.defaultCenter
    addObserverForName:FIRModelDownloadDidSucceedNotification
                object:nil
                 queue:nil
            usingBlock:^(NSNotification *_Nonnull note) {
              if (weakSelf == nil | note.userInfo == nil) {
                return;
              }
              __strong typeof(self) strongSelf = weakSelf;

              FIRRemoteModel *model = note.userInfo[FIRModelDownloadUserInfoKeyRemoteModel];
              if ([model.name isEqualToString:@"your_remote_model"]) {
                // The model was downloaded and is available on the device
              }
            }];

[NSNotificationCenter.defaultCenter
    addObserverForName:FIRModelDownloadDidFailNotification
                object:nil
                 queue:nil
            usingBlock:^(NSNotification *_Nonnull note) {
              if (weakSelf == nil | note.userInfo == nil) {
                return;
              }
              __strong typeof(self) strongSelf = weakSelf;

              NSError *error = note.userInfo[FIRModelDownloadUserInfoKeyError];
            }];

2. 準備輸入影像

接下來,您需要為 TensorFlow Lite 解釋器準備圖像。

  1. 將影像裁剪並縮放至模型的輸入尺寸,如tflite_metadata.json檔案中指定的那樣(預設為 320x320 像素)。您可以使用 Core Image 或第三方函式庫來完成此操作

  2. 將影像資料複製到DataNSData物件):

    迅速

    guard let image: CGImage = // Your input image
    guard let context = CGContext(
      data: nil,
      width: image.width, height: image.height,
      bitsPerComponent: 8, bytesPerRow: image.width * 4,
      space: CGColorSpaceCreateDeviceRGB(),
      bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue
    ) else {
      return nil
    }
    
    context.draw(image, in: CGRect(x: 0, y: 0, width: image.width, height: image.height))
    guard let imageData = context.data else { return nil }
    
    var inputData = Data()
    for row in 0 ..< 320 {    // Model takes 320x320 pixel images as input
      for col in 0 ..< 320 {
        let offset = 4 * (col * context.width + row)
        // (Ignore offset 0, the unused alpha channel)
        var red = imageData.load(fromByteOffset: offset+1, as: UInt8.self)
        var green = imageData.load(fromByteOffset: offset+2, as: UInt8.self)
        var blue = imageData.load(fromByteOffset: offset+3, as: UInt8.self)
    
        inputData.append(&red, count: 1)
        inputData.append(&green, count: 1)
        inputData.append(&blue, count: 1)
      }
    }
    

    Objective-C

    CGImageRef image = // Your input image
    long imageWidth = CGImageGetWidth(image);
    long imageHeight = CGImageGetHeight(image);
    CGContextRef context = CGBitmapContextCreate(nil,
                                                 imageWidth, imageHeight,
                                                 8,
                                                 imageWidth * 4,
                                                 CGColorSpaceCreateDeviceRGB(),
                                                 kCGImageAlphaNoneSkipFirst);
    CGContextDrawImage(context, CGRectMake(0, 0, imageWidth, imageHeight), image);
    UInt8 *imageData = CGBitmapContextGetData(context);
    
    NSMutableData *inputData = [[NSMutableData alloc] initWithCapacity:0];
    
    for (int row = 0; row < 300; row++) {
      for (int col = 0; col < 300; col++) {
        long offset = 4 * (row * imageWidth + col);
        // (Ignore offset 0, the unused alpha channel)
        UInt8 red = imageData[offset+1];
        UInt8 green = imageData[offset+2];
        UInt8 blue = imageData[offset+3];
    
        [inputData appendBytes:&red length:1];
        [inputData appendBytes:&green length:1];
        [inputData appendBytes:&blue length:1];
      }
    }
    

3. 運行物體偵測器

接下來,將準備好的輸入傳遞給解釋器:

迅速

try interpreter.copy(inputData, toInputAt: 0)
try interpreter.invoke()

Objective-C

TFLTensor *input = [interpreter inputTensorAtIndex:0 error:&error];
if (error != nil) { return; }

[input copyData:inputData error:&error];
if (error != nil) { return; }

[interpreter invokeWithError:&error];
if (error != nil) { return; }

4. Get information about detected objects

If object detection succeeds, the model produces as output three arrays of 40 elements (or whatever was specified in the tflite_metadata.json file) each. Each element corresponds to one potential object. The first array is an array of bounding boxes; the second, an array of labels; and the third, an array of confidence values. To get the model outputs:

迅速

var output = try interpreter.output(at: 0)
let boundingBoxes =
    UnsafeMutableBufferPointer<Float32>.allocate(capacity: 4 * 40)
output.data.copyBytes(to: boundingBoxes)

output = try interpreter.output(at: 1)
let labels =
    UnsafeMutableBufferPointer<Float32>.allocate(capacity: 40)
output.data.copyBytes(to: labels)

output = try interpreter.output(at: 2)
let probabilities =
    UnsafeMutableBufferPointer<Float32>.allocate(capacity: 40)
output.data.copyBytes(to: probabilities)

Objective-C

TFLTensor *output = [interpreter outputTensorAtIndex:0 error:&error];
if (error != nil) { return; }
NSData *boundingBoxes = [output dataWithError:&error];
if (error != nil) { return; }

output = [interpreter outputTensorAtIndex:1 error:&error];
if (error != nil) { return; }
NSData *labels = [output dataWithError:&error];
if (error != nil) { return; }

output = [interpreter outputTensorAtIndex:2 error:&error];
if (error != nil) { return; }
NSData *probabilities = [output dataWithError:&error];
if (error != nil) { return; }

Then, you can combine the label outputs with your label dictionary:

迅速

guard let labelPath = Bundle.main.path(
    forResource: "dict",
    ofType: "txt"
) else { return true }
let fileContents = try? String(contentsOfFile: labelPath)
guard let labelText = fileContents?.components(separatedBy: "\n") else { return true }

for i in 0 ..< 40 {
    let top = boundingBoxes[0 * i]
    let left = boundingBoxes[1 * i]
    let bottom = boundingBoxes[2 * i]
    let right = boundingBoxes[3 * i]

    let labelIdx = Int(labels[i])
    let label = labelText[labelIdx]
    let confidence = probabilities[i]

    if confidence > 0.66 {
        print("Object found: \(label) (confidence: \(confidence))")
        print("  Top-left: (\(left),\(top))")
        print("  Bottom-right: (\(right),\(bottom))")
    }
}

Objective-C

NSString *labelPath = [NSBundle.mainBundle pathForResource:@"dict"
                                                    ofType:@"txt"];
NSString *fileContents = [NSString stringWithContentsOfFile:labelPath
                                                   encoding:NSUTF8StringEncoding
                                                      error:&error];
if (error != nil || fileContents == NULL) { return; }
NSArray<NSString*> *labelText = [fileContents componentsSeparatedByString:@"\n"];

for (int i = 0; i < 40; i++) {
    Float32 top, right, bottom, left;
    Float32 labelIdx;
    Float32 confidence;

    [boundingBoxes getBytes:&top range:NSMakeRange(16 * i + 0, 4)];
    [boundingBoxes getBytes:&left range:NSMakeRange(16 * i + 4, 4)];
    [boundingBoxes getBytes:&bottom range:NSMakeRange(16 * i + 8, 4)];
    [boundingBoxes getBytes:&right range:NSMakeRange(16 * i + 12, 4)];

    [labels getBytes:&labelIdx range:NSMakeRange(4 * i, 4)];
    [probabilities getBytes:&confidence range:NSMakeRange(4 * i, 4)];

    if (confidence > 0.5f) {
        NSString *label = labelText[(int)labelIdx];
        NSLog(@"Object detected: %@", label);
        NSLog(@"  Confidence: %f", confidence);
        NSLog(@"  Top-left: (%f,%f)", left, top);
        NSLog(@"  Bottom-right: (%f,%f)", right, bottom);
    }
}

Tips to improve real-time performance

If you want to label images in a real-time application, follow these guidelines to achieve the best framerates:

  • Throttle calls to the detector. If a new video frame becomes available while the detector is running, drop the frame.
  • If you are using the output of the detector to overlay graphics on the input image, first get the result, then render the image and overlay in a single step. By doing so, you render to the display surface only once for each input frame. See the previewOverlayView and FIRDetectionOverlayView classes in the showcase sample app for an example.