有兩種方法可以集成從AutoML Vision Edge訓練的模型:您可以通過將模型放入應用程序的資產文件夾中來捆綁模型,也可以從Firebase動態下載模型。
模型捆綁選項 | |
---|---|
捆綁在您的應用中 |
|
由Firebase託管 |
|
在你開始之前
將ML Kit Android庫的依賴項添加到模塊的應用程序級gradle文件中,該文件通常是
app/build.gradle
:要將模型與您的應用捆綁在一起:
dependencies { // ... // Image labeling feature with bundled automl model implementation 'com.google.mlkit:image-labeling-automl:16.0.0' }
要從Firebase動態下載模型,請添加
linkFirebase
依賴項:dependencies { // ... // Image labeling feature with automl model downloaded // from firebase implementation 'com.google.mlkit:image-labeling-automl:16.0.0' implementation 'com.google.mlkit:linkfirebase:16.0.0' }
如果要下載模型,請確保將Firebase添加到Android項目(如果尚未下載)。捆綁模型時不需要這樣做。
1.加載模型
配置本地模型源
要將模型與您的應用捆綁在一起:
從您從Firebase控制台下載的zip存檔中提取模型及其元數據。我們建議您在下載文件時使用它們,不要進行任何修改(包括文件名)。
將模型及其元數據文件包含在應用包中:
- 如果您的項目中沒有資產文件夾,請通過右鍵單擊
app/
文件夾,然後單擊新建>文件夾>資產文件夾來創建一個。 - 在資產文件夾下創建一個子文件夾以包含模型文件。
- 將文件
model.tflite
,dict.txt
和manifest.json
複製到子文件夾(所有三個文件必須位於同一文件夾中)。
- 如果您的項目中沒有資產文件夾,請通過右鍵單擊
將以下內容添加到應用程序的
build.gradle
文件中,以確保Gradle在構建應用程序時不會壓縮模型文件:android { // ... aaptOptions { noCompress "tflite" } }
模型文件將包含在應用程序包中,並且可以作為原始資產提供給ML Kit。
創建
LocalModel
對象,指定模型清單文件的路徑:爪哇
AutoMLImageLabelerLocalModel localModel = new AutoMLImageLabelerLocalModel.Builder() .setAssetFilePath("manifest.json") // or .setAbsoluteFilePath(absolute file path to manifest file) .build();
科特林
val localModel = AutoMLImageLabelerLocalModel.Builder() .setAssetFilePath("manifest.json") // or .setAbsoluteFilePath(absolute file path to manifest file) .build()
配置Firebase託管的模型源
要使用遠程託管的模型,請創建一個RemoteModel
對象,並指定在發布模型時為其分配的名稱:
爪哇
// Specify the name you assigned in the Firebase console.
AutoMLImageLabelerRemoteModel remoteModel =
new AutoMLImageLabelerRemoteModel.Builder("your_model_name").build();
科特林
// Specify the name you assigned in the Firebase console.
val remoteModel =
AutoMLImageLabelerRemoteModel.Builder("your_model_name").build()
然後,啟動模型下載任務,指定要允許的下載條件。如果模型不在設備上,或者有可用的較新版本的模型,則該任務將從Firebase異步下載模型:
爪哇
DownloadConditions downloadConditions = new DownloadConditions.Builder()
.requireWifi()
.build();
RemoteModelManager.getInstance().download(remoteModel, downloadConditions)
.addOnSuccessListener(new OnSuccessListener<Void>() {
@Override
public void onSuccess(@NonNull Task<Void> task) {
// Success.
}
});
科特林
val downloadConditions = DownloadConditions.Builder()
.requireWifi()
.build()
RemoteModelManager.getInstance().download(remoteModel, downloadConditions)
.addOnSuccessListener {
// Success.
}
許多應用程序都在其初始化代碼中啟動下載任務,但是您可以在需要使用模型之前隨時進行下載。
根據模型創建圖像標籤
配置模型源後,請從其中一個創建ImageLabeler
對象。
如果只有本地捆綁的模型,則只需從AutoMLImageLabelerLocalModel
對象創建一個標籤器,然後配置所需的置信度閾值(請參閱評估模型):
爪哇
AutoMLImageLabelerOptions autoMLImageLabelerOptions =
new AutoMLImageLabelerOptions.Builder(localModel)
.setConfidenceThreshold(0.0f) // Evaluate your model in the Firebase console
// to determine an appropriate value.
.build();
ImageLabeler labeler = ImageLabeling.getClient(autoMLImageLabelerOptions)
科特林
val autoMLImageLabelerOptions = AutoMLImageLabelerOptions.Builder(localModel)
.setConfidenceThreshold(0) // Evaluate your model in the Firebase console
// to determine an appropriate value.
.build()
val labeler = ImageLabeling.getClient(autoMLImageLabelerOptions)
如果您有一個遠程託管的模型,則在運行它之前必須檢查它是否已下載。您可以使用模型管理器的isModelDownloaded()
方法檢查模型下載任務的狀態。
儘管您只需要在運行標籤程序之前進行確認,但是如果您同時擁有遠程託管的模型和本地捆綁的模型,則在實例化圖像標籤程序時執行此檢查可能很有意義:已下載,否則從本地模型下載。
爪哇
RemoteModelManager.getInstance().isModelDownloaded(remoteModel)
.addOnSuccessListener(new OnSuccessListener<Boolean>() {
@Override
public void onSuccess(Boolean isDownloaded) {
AutoMLImageLabelerOptions.Builder optionsBuilder;
if (isDownloaded) {
optionsBuilder = new AutoMLImageLabelerOptions.Builder(remoteModel);
} else {
optionsBuilder = new AutoMLImageLabelerOptions.Builder(localModel);
}
AutoMLImageLabelerOptions options = optionsBuilder
.setConfidenceThreshold(0.0f) // Evaluate your model in the Firebase console
// to determine an appropriate threshold.
.build();
ImageLabeler labeler = ImageLabeling.getClient(options);
}
});
科特林
RemoteModelManager.getInstance().isModelDownloaded(remoteModel)
.addOnSuccessListener { isDownloaded ->
val optionsBuilder =
if (isDownloaded) {
AutoMLImageLabelerOptions.Builder(remoteModel)
} else {
AutoMLImageLabelerOptions.Builder(localModel)
}
// Evaluate your model in the Firebase console to determine an appropriate threshold.
val options = optionsBuilder.setConfidenceThreshold(0.0f).build()
val labeler = ImageLabeling.getClient(options)
}
如果只有遠程託管的模型,則應禁用與模型相關的功能(例如,灰顯或隱藏UI的一部分),直到確認已下載模型為止。您可以通過將偵聽器附加到模型管理器的download()
方法來實現:
爪哇
RemoteModelManager.getInstance().download(remoteModel, conditions)
.addOnSuccessListener(new OnSuccessListener<Void>() {
@Override
public void onSuccess(Void v) {
// Download complete. Depending on your app, you could enable
// the ML feature, or switch from the local model to the remote
// model, etc.
}
});
科特林
RemoteModelManager.getInstance().download(remoteModel, conditions)
.addOnSuccessListener {
// Download complete. Depending on your app, you could enable the ML
// feature, or switch from the local model to the remote model, etc.
}
2.準備輸入圖像
然後,對於要標記的每個圖像,從圖像創建一個InputImage
對象。使用Bitmap
或使用camera2 API時,如果使用YUV_420_888 media.Image
,則圖像media.Image
運行速度最快。您可以從不同的來源創建InputImage
,下面分別說明。
使用media.Image
要從InputImage
對象創建media.Image
對象,例如從設備的相機捕獲圖像時,請將media.Image
對象和圖像的旋轉傳遞給InputImage.fromMediaImage()
。
如果使用CameraX庫,則OnImageCapturedListener
和ImageAnalysis.Analyzer
類將為您計算旋轉值。
爪哇
private class YourAnalyzer implements ImageAnalysis.Analyzer { @Override public void analyze(ImageProxy imageProxy) { if (imageProxy == null || imageProxy.getImage() == null) { return; } Image mediaImage = imageProxy.getImage(); InputImage image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees); // Pass image to an ML Kit Vision API // ... } }
Kotlin + KTX
private class YourImageAnalyzer : ImageAnalysis.Analyzer { override fun analyze(imageProxy: ImageProxy?) { val mediaImage = imageProxy?.image if (mediaImage != null) { val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees) // Pass image to an ML Kit Vision API // ... } } }
如果不使用提供圖像旋轉度的相機庫,則可以根據設備的旋轉度和相機傳感器在設備中的方向來計算圖像:
爪哇
private static final SparseIntArray ORIENTATIONS = new SparseIntArray(); static { ORIENTATIONS.append(Surface.ROTATION_0, 90); ORIENTATIONS.append(Surface.ROTATION_90, 0); ORIENTATIONS.append(Surface.ROTATION_180, 270); ORIENTATIONS.append(Surface.ROTATION_270, 180); } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) private int getRotationCompensation(String cameraId, Activity activity, Context context) throws CameraAccessException { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. int deviceRotation = activity.getWindowManager().getDefaultDisplay().getRotation(); int rotationCompensation = ORIENTATIONS.get(deviceRotation); // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. CameraManager cameraManager = (CameraManager) context.getSystemService(CAMERA_SERVICE); int sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION); rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360; // Return the corresponding FirebaseVisionImageMetadata rotation value. int result; switch (rotationCompensation) { case 0: result = FirebaseVisionImageMetadata.ROTATION_0; break; case 90: result = FirebaseVisionImageMetadata.ROTATION_90; break; case 180: result = FirebaseVisionImageMetadata.ROTATION_180; break; case 270: result = FirebaseVisionImageMetadata.ROTATION_270; break; default: result = FirebaseVisionImageMetadata.ROTATION_0; Log.e(TAG, "Bad rotation value: " + rotationCompensation); } return result; }
Kotlin + KTX
private val ORIENTATIONS = SparseIntArray() init { ORIENTATIONS.append(Surface.ROTATION_0, 90) ORIENTATIONS.append(Surface.ROTATION_90, 0) ORIENTATIONS.append(Surface.ROTATION_180, 270) ORIENTATIONS.append(Surface.ROTATION_270, 180) } /** * Get the angle by which an image must be rotated given the device's current * orientation. */ @RequiresApi(api = Build.VERSION_CODES.LOLLIPOP) @Throws(CameraAccessException::class) private fun getRotationCompensation(cameraId: String, activity: Activity, context: Context): Int { // Get the device's current rotation relative to its "native" orientation. // Then, from the ORIENTATIONS table, look up the angle the image must be // rotated to compensate for the device's rotation. val deviceRotation = activity.windowManager.defaultDisplay.rotation var rotationCompensation = ORIENTATIONS.get(deviceRotation) // On most devices, the sensor orientation is 90 degrees, but for some // devices it is 270 degrees. For devices with a sensor orientation of // 270, rotate the image an additional 180 ((270 + 270) % 360) degrees. val cameraManager = context.getSystemService(CAMERA_SERVICE) as CameraManager val sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION)!! rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360 // Return the corresponding FirebaseVisionImageMetadata rotation value. val result: Int when (rotationCompensation) { 0 -> result = FirebaseVisionImageMetadata.ROTATION_0 90 -> result = FirebaseVisionImageMetadata.ROTATION_90 180 -> result = FirebaseVisionImageMetadata.ROTATION_180 270 -> result = FirebaseVisionImageMetadata.ROTATION_270 else -> { result = FirebaseVisionImageMetadata.ROTATION_0 Log.e(TAG, "Bad rotation value: $rotationCompensation") } } return result }
然後,將media.Image
對象和旋轉度值傳遞給InputImage.fromMediaImage()
:
爪哇
InputImage image = InputImage.fromMediaImage(mediaImage, rotation);
Kotlin + KTX
val image = InputImage.fromMediaImage(mediaImage, rotation)
使用文件URI
要從文件URI創建InputImage
對象,請將應用上下文和文件URI傳遞給InputImage.fromFilePath()
。當您使用ACTION_GET_CONTENT
意圖提示用戶從其圖庫應用中選擇圖像時,此功能很有用。
爪哇
InputImage image; try { image = InputImage.fromFilePath(context, uri); } catch (IOException e) { e.printStackTrace(); }
Kotlin + KTX
val image: InputImage try { image = InputImage.fromFilePath(context, uri) } catch (e: IOException) { e.printStackTrace() }
使用ByteBuffer
或ByteArray
要從ByteBuffer
或ByteArray
創建InputImage
對象,請首先按照先前針對media.Image
輸入的描述計算圖像旋轉度。然後,使用緩衝區或數組以及圖像的高度,寬度,顏色編碼格式和旋轉度創建InputImage
對象:
爪哇
InputImage image = InputImage.fromByteBuffer(byteBuffer, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 );
Kotlin + KTX
val image = InputImage.fromByteBuffer( byteBuffer, /* image width */ 480, /* image height */ 360, rotationDegrees, InputImage.IMAGE_FORMAT_NV21 // or IMAGE_FORMAT_YV12 )
使用Bitmap
要從Bitmap
對象創建InputImage
對象,請進行以下聲明:
爪哇
InputImage image = InputImage.fromBitmap(bitmap, rotationDegree);
Kotlin + KTX
val image = InputImage.fromBitmap(bitmap, 0)
該圖像由Bitmap
像以及旋轉度表示。
3.運行圖像標籤器
要標記image
對象,請將image
對像傳遞給ImageLabeler
的process()
方法。
爪哇
labeler.process(image)
.addOnSuccessListener(new OnSuccessListener<List<ImageLabel>>() {
@Override
public void onSuccess(List<ImageLabel> labels) {
// Task completed successfully
// ...
}
})
.addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
// Task failed with an exception
// ...
}
});
科特林
labeler.process(image)
.addOnSuccessListener { labels ->
// Task completed successfully
// ...
}
.addOnFailureListener { e ->
// Task failed with an exception
// ...
}
4.獲取有關標記對象的信息
如果圖像標記操作成功,ImageLabel
對象的列表傳遞到成功偵聽器。每個ImageLabel
對象代表圖像中標記的對象。您可以獲得每個標籤的文本描述,匹配項的置信度得分和匹配項的索引。例如: 爪哇
for (ImageLabel label : labels) {
String text = label.getText();
float confidence = label.getConfidence();
int index = label.getIndex();
}
科特林
for (label in labels) {
val text = label.text
val confidence = label.confidence
val index = label.index
}
改善實時性能的提示
如果要在實時應用程序中標記圖像,請遵循以下準則以獲得最佳幀率:- 節流閥調用圖像標籤。如果在圖像標籤機運行時有新的視頻幀可用,請放下該幀。有關示例,請參見快速入門示例應用程序中的
VisionProcessorBase
類。 - 如果要使用圖像標籤器的輸出在輸入圖像上疊加圖形,請首先獲取結果,然後一步一步渲染圖像並疊加。這樣,對於每個輸入幀,只渲染一次到顯示表面。有關
CameraSourcePreview
,請參見快速入門示例應用程序中的CameraSourcePreview
和GraphicOverlay
類。 如果使用Camera2 API,請以
ImageFormat.YUV_420_888
格式捕獲圖像。如果使用較舊的Camera API,請以
ImageFormat.NV21
格式捕獲圖像。