CIS 694/EEC 693

Android Sensor Programming

Lecture 26

Wenbing Zhao
Department of Electrical Engineering and Computer Science
Cleveland State University
w.zhao1@csuohio.edu

1/22/2020 Android Sensor Programming 1

Outline
 ML Kit: Machine Learning for Mobile Developers

1/22/2020 Android Sensor Programming 2

ML Kit: Machine Learning for Mobile Developers
 https://developers.google.com/ml-kit/
 Image labeling
 Text recognition
 Face detection
 Barcode scanning
 Landmark detection

1/22/2020 Android Sensor Programming 3

ML Kit: Machine Learning for Mobile Developers
 You will need to register a Google account using your
existing email (any email will be fine)
 Log in to the Firebase console, go to MLKit panel

1/22/2020 Android Sensor Programming 4

ML Kit: Machine Learning for Mobile Developers
 For every app you build, you will have to register with the firebase
 After registration, you will be asked to download a json file google-
services.json to be placed into your project for compilation
 Add Firebase to your Android project
 Add to the root-level build.gradle file:
buildscript {
// ...
dependencies {
// ...
classpath 'com.google.gms:google-services:4.0.1' // google-services plugin
}
}

allprojects {
// ...
repositories {
// ...
google() // Google's Maven repository
}
}

1/22/2020 Android Sensor Programming 5

ML Kit: Machine Learning for Mobile Developers
 Add Firebase to your Android project
 Then, in your module Gradle file (usually the app/build.gradle), add the
apply plugin line at the bottom of the file to enable the Gradle plugin:
dependencies {
// ...
implementation 'com.google.firebase:firebase-core:16.0.1'

// Getting a "Could not find" error? Make sure you have

// added the Google maven respository to your root build.gradle
}

// ADD THIS AT THE BOTTOM

apply plugin: 'com.google.gms.google-services'

 You should also add the dependencies for the Firebase SDKs you want to
use. We recommend starting with com.google.firebase:firebase-core,
which provides Google Analytics for Firebase functionality
 Available firebase libraries:
https://firebase.google.com/docs/android/setup#available_libraries

1/22/2020 Android Sensor Programming 6

ML Kit: Image Labeling
 https://firebase.google.com/docs/ml-kit/label-images
 With ML Kit's image labeling APIs, you can recognize entities in an
image without having to provide any additional contextual metadata,
using either an on-device API or a cloud-based API
 Image labeling gives you insight into the content of images
 When you use the API, you get a list of the entities that were
recognized: people, things, places, activities, etc.
 Each label found comes with a score that indicates the confidence the
ML model has in its relevance
 With this information, you can perform tasks such as automatic
metadata generation and content moderation

1/22/2020 Android Sensor Programming 7

ML Kit: Image Labeling
 The device-based API supports 400+ labels, such as the
following examples: Category Example labels
People Crowd
Selfie
Smile
Activities Dancing
Eating
Surfing
Things Car
Piano
Receipt
Animals Bird
Cat
Dog
Plants Flower
Fruit
Vegetable
Places Beach
Lake
Mountain

1/22/2020 Android Sensor Programming 8

ML Kit: Image Labeling
 The cloud-based API supports 10,000+ labels
Category Example labels Category Example labels

Arts & Sculpture Astronomical Comet

entertainment Musical objects Galaxy
Instrument Star
Dance

Business & Restaurant Colors Red

industrial Factory Green
Airline Blue

Design Floral Drink Coffee

Pattern Tea
Wood Stain Milk

Events Meeting Fictional Santa Claus

Picnic characters Superhero
Vacation Mythical
creature

Food Casserole Home & Laundry

Fruit garden basket
Potato chip Dishwasher
Fountain

Activities Wedding Materials Ceramic

Dancing Textile
Motorsport Fiber

Media Newsprint Modes of Aircraft

Document transport Motorcycle
Sign Subway

Occupations Actor Organisms Plant

Florist Animal
Police Fungus

Organizations Government Places Airport

Club Mountain
College Tent

Technology Robot Things Bicycle

Computer Pipe
Solar panel Doll

1/22/2020 Android Sensor Programming 9

ML Kit: Google Knowledge Graph entity IDs
 In addition the text description of each label that ML Kit
returns, it also returns the label's Google Knowledge
Graph entity ID
 This ID is a string that uniquely identifies the entity
represented by the label, and is the same ID used by the
Knowledge Graph Search API
 https://developers.google.com/knowledge-graph/
 You can use this string to identify an entity across
languages, and independently of the formatting of the text
description

1/22/2020 Android Sensor Programming 10

ML Kit: Google Knowledge Graph entity IDs

1/22/2020 Android Sensor Programming 11

Label Images with ML Kit on Android
 Configuration in Android Studio
 In the project-level build.gradle, add:
dependencies {
classpath 'com.android.tools.build:gradle:3.1.3'
classpath 'com.google.gms:google-services:4.0.1'
}

 In the app-level build.gradle, add:

dependencies {
implementation fileTree(dir: 'libs', include: ['*.jar'])
implementation 'com.android.support:appcompat-v7:27.1.1'
implementation 'com.android.support.constraint:constraint-layout:1.1.2’
implementation 'com.google.firebase:firebase-core:16.0.1'
implementation 'com.google.firebase:firebase-ml-vision:16.0.0'
implementation 'com.google.firebase:firebase-ml-vision-image-label-model:15.0.0'
testImplementation 'junit:junit:4.12'
androidTestImplementation 'com.android.support.test:runner:1.0.2'
androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
}

apply plugin: 'com.google.gms.google-services'

1/22/2020 Android Sensor Programming 12

 Label Images with ML Kit on Android
 Configuration in Android Studio
 If you use the on-device API, configure your app to automatically
download the ML model to the device after your app is installed from
the Play Store
 To do so, add the following declaration to your app's
AndroidManifest.xml file:
<application ...>
...
<meta-data
android:name="com.google.firebase.ml.vision.DEPENDENCIES"
android:value="label" />
<!-- To use multiple models: android:value="label,model2,model3" -->
</application>

 If you do not enable install-time model downloads, the model will be

downloaded the first time you run the on-device detector. Requests
you make before the download has completed will produce no results

1/22/2020 Android Sensor Programming 13

 Register with Firebase
 Go to https://console.firebase.google.com/u/0/project/mlkit-
aa998/overview
 Click “Add another app” link, then “Add Firebase to your Android app

1/22/2020 Android Sensor Programming 14

 Register with Firebase
 Go to https://console.firebase.google.com/u/0/project/mlkit-
aa998/overview
 Click “Add another app” link, then “Add Firebase to your Android app

1/22/2020 Android Sensor Programming 15

 Register with Firebase
 Then, follow the steps as prompted

1/22/2020 Android Sensor Programming 16

 On-device image labeling
 Configure the image labeler
 By default, the on-device image labeler returns at most 10 labels for an
image. If you want to change this setting, create a
FirebaseVisionLabelDetectorOptions object as in the following example
FirebaseVisionLabelDetectorOptions options =
new FirebaseVisionLabelDetectorOptions.Builder()
.setConfidenceThreshold(0.8f).build();

 Run the image labeler

 Create a FirebaseVisionImage object from a Bitmap object
FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);

 You can also create a FirebaseVisionImage from a file, from a ByteBuffer or byte
array, or a media.Image taken by the device’s camera

1/22/2020 Android Sensor Programming 17

 On-device image labeling
 Run the image labeler
 Get an instance of FirebaseVisionLabelDetector:
FirebaseVisionLabelDetector detector = FirebaseVision.getInstance().getVisionLabelDetector();
// Or, to set the minimum confidence required:
FirebaseVisionLabelDetector detector = FirebaseVision.getInstance().getVisionLabelDetector(options);

 Finally, pass the image to the detectInImage method:

Task<List<FirebaseVisionLabel>> result = detector.detectInImage(image)
.addOnSuccessListener(
new OnSuccessListener<List<FirebaseVisionLabel>>() {
@Override
public void onSuccess(List<FirebaseVisionLabel> labels) {
// Task completed successfully
// ...
}
})
.addOnFailureListener(
new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
// Task failed with an exception
// ...
}
});

1/22/2020 Android Sensor Programming 18

 On-device image labeling
 Get information about labeled objects
 If the image labeling operation succeeds, a list of FirebaseVisionLabel objects will be
passed to the success listener
 Each FirebaseVisionLabel object represents something that was labeled in the image.
 For each label, you can get the label's text description, its Knowledge Graph entity ID
(if available), and the confidence score of the match

for (FirebaseVisionLabel label: labels) {

String text = label.getLabel();
String entityId = label.getEntityId();
float confidence = label.getConfidence();
}

1/22/2020 Android Sensor Programming 19

 A complete on-device image labeling app
 Preparation: find an interesting image and copy it to res/mipmap:

1/22/2020 Android Sensor Programming 20

 <?xml version="1.0" encoding="utf-8"?>

A complete on- <RelativeLayout

xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"

device image android:layout_width="match_parent"

android:layout_height="match_parent"
tools:context=".MainActivity">

labeling app <Button

 Setup the layout for the
activity as shown in the
previous slide, and here:
</RelativeLayout>
android:text="Lable Image"
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:layout_centerHorizontal="true"
android:id="@+id/btnLabelImage" />
<ImageSwitcher
android:id="@+id/imageSwitcher"
android:layout_width="match_parent"
android:layout_height="250dp"
android:layout_alignParentStart="true"
android:layout_alignParentTop="true"
android:layout_marginTop="50dp" />
<TextView
android:id="@+id/txtLabels"
android:layout_width="match_parent"
android:layout_height="172dp"
android:layout_alignParentBottom="true"
android:layout_alignParentEnd="true"
android:layout_below="@+id/imageSwitcher"
android:layout_centerHorizontal="true"
android:layout_marginTop="5dp"
android:textColor="#ffffff"
android:background="#000000" />

1/22/2020 Android Sensor Programming 21

 A complete on-device image labeling app
 Populate MainActivity
import android.support.v7.app.AppCompatActivity;
import android.os.Bundle;
import android.graphics.BitmapFactory;
import android.graphics.Bitmap;
import java.util.List;
import com.google.android.gms.tasks.OnSuccessListener;
import com.google.android.gms.tasks.OnFailureListener;
import android.support.annotation.NonNull;
import android.widget.ImageSwitcher;
import android.support.v7.app.ActionBar;
import android.widget.TextView;
import android.widget.ViewSwitcher;
import android.widget.ImageView;
import android.view.View;
import android.widget.Button;

import com.google.android.gms.tasks.Task;
import com.google.firebase.ml.vision.FirebaseVision;
import com.google.firebase.ml.vision.common.FirebaseVisionImage;
import com.google.firebase.ml.vision.label.FirebaseVisionLabel;
import com.google.firebase.ml.vision.label.FirebaseVisionLabelDetector;

1/22/2020 Android Sensor Programming 22

 A complete on-device image labeling app
 Populate MainActivity
public class MainActivity extends AppCompatActivity {
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);

ImageSwitcher imgSwitcher = (ImageSwitcher) findViewById(R.id.imageSwitcher);

imgSwitcher.setFactory(new ViewSwitcher.ViewFactory() {
@Override
public View makeView() {
ImageView myView = new ImageView(getApplicationContext());
myView.setScaleType(ImageView.ScaleType.FIT_CENTER);
myView.setLayoutParams(new ImageSwitcher.LayoutParams(
ActionBar.LayoutParams.WRAP_CONTENT,
ActionBar.LayoutParams.WRAP_CONTENT));
return myView;
}
});
imgSwitcher.setImageResource(R.mipmap.windows);
Button btnLabelImage = (Button) findViewById(R.id.btnLabelImage);

1/22/2020 Android Sensor Programming 23

 Populate MainActivity
btnLabelImage.setOnClickListener(new View.OnClickListener() {
@Override
public void onClick(View v) {
Bitmap bm = BitmapFactory.decodeResource(getResources(), R.mipmap.windows);
FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bm);
FirebaseVisionLabelDetector detector = FirebaseVision.getInstance().getVisionLabelDetector();
Task<List<FirebaseVisionLabel>> result = detector.detectInImage(image).addOnSuccessListener(
new OnSuccessListener<List<FirebaseVisionLabel>>() {
@Override
public void onSuccess(List<FirebaseVisionLabel> labels) {
// Task completed successfully
for (FirebaseVisionLabel label: labels) {
String text = label.getLabel();
String entityId = label.getEntityId();
float confidence = label.getConfidence();
TextView tv = (TextView) findViewById(R.id.txtLabels);
tv.append(text+"; entity id="+entityId+"; confidence="+confidence+"\n");
}
}
}).addOnFailureListener(new OnFailureListener() {
@Override
public void onFailure(@NonNull Exception e) {
// Task failed with an exception
// ...
}
});
}
});
}
}

1/22/2020 Android Sensor Programming 24

A complete on-device image labeling app

1/22/2020 Android Sensor Programming 25

Homework#29
 Build the on-device image labeling app

1/22/2020 Android Sensor Programming 26