Hot Chips 33, August 24, 2021

New Value Creation by Nano-Tactile

Sensor Chip Exceeding our Fingertip
Discrimination Ability
Hidekuni TAKAO and Kazuki WATATANI
Kagawa University, JAPAN
Kazutami ARIMOTO
Okayama Prefectural University, JAPAN
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 2

Texture Discrimination and Quantification

New Value Creation in Future Applications

Quality Control Health Care Remote Treatment H/M Interface

High-Resolution Information Processing

Tactile Signal
Exceeds
• Signal Modeling Human’s
• Deep Learning
• Texture Knowledge Sense of
Touch?
INPUT

(2015-2021)
(2020-2026)
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 3
Our Sense of Touch
Distributed Receptors under the Elastic Skin Structure

Meissner’s Corpuscles

Merkel Cells

Ruffini endings

Pacinian Corpuscles

Corpuscles in Fingertip Skin

（Darian-Smith. 1984）

All the Nerve Cells Sense “Vibration”

• Different Depth (Spatial Resolution)
• Different Speed of Response
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 4

How Do We Feel the Surface Texture?

Sweep Roughness

Friction

Central
Nervous
Brain
System
Peripheral
Nerve
Network
Fingertip
Sense
Distributions of …..
• Surface Texture
Fingertip
• Frictional Force Sense
• Hardness
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 5

Tactile Sensors at Present

• Flexible Sensors (ex. Organic Semiconductors) for Wide Area Body Skin
Conductive Rubber

Texture

M. Cheng, Y. Yang et al. M. Shikida, K. Sato et al. Pressure Profile Systems Inc.,

National Taiwan Univ., Taiwan Nagoya University, Japan U.S.A
• Silicon-MEMS Sensors for Fingertip Like Sensation Piezoresistive
Capacitive Metal Strain-gauge

M. Sohgawa, M Noda et al.

N. Sato, K. Machida et al. Osaka University, Japan O. Paul, P. Ruther et al.
NTT Corporation, Japan Kyoto Institute of Tech., Japan IMTEK, Univ. of Freiburg, Germany

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 6

We Use Silicon as Texture Sensor Material

Silicon as an Electron Device Material
• Ideal for Integrated Circuits
• Suitable for Strain Sensor Intel
• Multifunctional Integration 4004

Silicon as a Mechanical Device Material

• Ideal Mechanical Deformation
• Supple Structure by Micromachining
• Faster Response and Motion

Tactile Sensor Made of Silicon can Realize;

• High Spatial Resolution (~1µm or less)
• High Time Resolution (~2kHz or more)
• High Signal Resolution (~10µN or less)
KAGAWA TAKAO
UNIVERSITY LABOLATORY
“Basic Nano-Tactile Sensor”
7

Surface
Shape
Friction
Contactor tip

Reference plane 50µm

Contactor Surface Shape

Sticking out Frictional Force Detector
from the Plane Detectors

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 8

Similarity: Single Fingerprint Sensor

Fingerprint Contactor Tip

Meissner’s 
Detection
Corpuscle
Structures

“Model of Fingerprint of Human”

T. Maeno et al. JSME C, Nano-Tactile Sensor Device
Vol. 71, No.701, pp. 245-250, 2005 Kagawa University CREST Project

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 9

Similarity: Single Fingerprint Sensor

Nano Tactile Sensor Chips on Wafer

Friction Sensor

Shape Sensor

Micro Roughness
Vsup Strain
圧縮応力
Sensor
R - ΔR
Vout high
Contactor R

Friction Wave Friction R

Strain
R + ΔR
引張応力
Force Sensor Vout low
GND

0.1mm Micro
Roughness
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 10

Similarity: Single Fingerprint Sensor

Nano Tactile Sensor Chips on Wafer
Rough Fine
AFM
Tactile Texture
Average Pitch of Fingerprint
100

Minimum Force Resolution (µN)

Takao 2018 400 µm
Human Skin’s Sensitivity Limit
~10 µN

101
Wu 2018 Takao 2021
Takao 2017

Micro Roughness 102
Takao 2018
KES Kato
Tech. Co.

Zone2: Micro-Texture Sensing

103
Contactor Shogawa
Takao 2015 1mN Threshold
Shimoyama 2016
2013 BioTac Zone1: Macro-Tactile Sensing
104 Tanaka 2018

Friction Wave
Fujitsu
2015 Wang 2019

104 102 100 10-2

Surface Spatial Resolution (µm)
0.1mm
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 11

“Soft Cloth” Measurement

Device chip Cloth sample

Surface shape

Local friction force

11
Sensor chip

Plain Stich Cloth

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 12

Tactile Waveforms on Plain Stich

Contactor

Sweeping direction
Plain Stich
Surface Shape

Micro-area Frictional force

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 13

Tactile Sense Obtainable in Submicron!

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 14

Tactile Waveform on Human Hair

1. Before Shampoo 2. After Shampoo 3. Wash by a Soap

Covered by sebum Covered by Shampoo Seriously Damaged

3D optical image 100µm 100µm 100µm

3D optical image 3D optical image
Friction μ

Distance [µm] Distance [µm] Distance [µm]

The same hair sample is used
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 15

Tactile Waveform on Human Skin

Barn Skin Groove

Younger Skin Aged Skin

Groove Barns

Micro Surface Shape of Skin

Outputs

Surface Friction

Scanned Distance [mm]

KAGAWA TAKAO
UNIVERSITY LABOLATORY
1st-Gen.“Touch-Feeling Scanner”
16

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 17

Structures of Paper Surface

（カラー画像） （カラー画像）

High Quality Paper Low Quality Paper

(Smooth and Low Friction) (Rough and Connective)

• Difference of surface appearance is unexpectedly small.

• Nano-Tactile Sensor can distinguish the small difference.

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 Touch Feeling Data of PAPERs
18

Glossy Photo Paper Matte Photo Paper Kent Paper

3000 4000 3000 4000 3000 4000

2500 Shape 3500 2500 Shape 3500 2500 Shape 3500

表⾯形状信号[count]
表⾯形状信号[count]
表⾯形状信号[count]

摩擦⼒信号[count]
摩擦⼒信号[count]

摩擦⼒信号[count]
2000 3000 2000 3000 2000 3000

1500 2500 1500 2500 1500 2500

1000 2000 1000 2000 1000 2000

500
Friction
1500 500
Friction 1500 500
Friction 1500

0 1000 0 1000 0 1000

5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0
⾛査距離[mm] ⾛査距離[mm] ⾛査距離[mm]

High Grade Paper Middle Grade Paper Low Grade Paper

3000 4000 3000 4000 3000 4000

2500 Shape 3500 2500 Shape 3500 2500 Shape 3500

表⾯形状信号[count]
表⾯形状信号[count]

表⾯形状信号[count]

摩擦⼒信号[count]
摩擦⼒信号[count]

摩擦⼒信号[count]
2000 3000 2000 3000 2000 3000

1500 2500 1500 2500 1500 2500

1000 2000 1000 2000 1000 2000

Friction Friction
500 1500 500 1500 500 1500

0 1000 0 1000 0
Friction 1000
5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0
⾛査距離[mm] ⾛査距離[mm] ⾛査距離[mm]

Japanese Traditional Paper Painting Paper Kraft Paper

3000 4000 3000 4000 3000 4000

2500 3500 2500 Shape 3500 2500 Shape 3500

表⾯形状信号[count]

表⾯形状信号[count]

表⾯形状信号[count]
摩擦⼒信号[count]

摩擦⼒信号[count]
2000 3000 2000 3000

摩擦⼒信号[count]
2000 3000

1500
Shape 2500 1500 2500 1500 2500

1000 2000 1000 2000 1000 2000

Friction
500 1500

Friction
500 1500

Friction
500 1500

0
5.0 5.5 6.0 6.5
KAGAWA
7.0 7.5 8.0 8.5 9.0 9.5 10.0
1000 0
5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0
1000
0
5.0 5.5 6.0 6.5 7.0
TAKAO
7.5 8.0 8.5 9.0 9.5 10.0
1000

⾛査距離[mm] ⾛査距離[mm]
UNIVERSITY LABOLATORY ⾛査距離[mm]
 19

Very Small Difference (Tissue Papers)

A Nepi-Nepi E ShiKou

B Premium Soft F Lotion Ufu

A. Nepia’s Nepi-Nepi (0.2JPY/Sheet)

C Japan Premium G Zeitaku Hoshitu

B. Nepia’s Premium Soft (0.3JPY/Sheet)

D Hana-Celebrity

Friction
Shape
C. Nepia’s Japan Premium (1.0JPY/Sheet)

D. Nepia’s Hana-Celebrity (2.0JPY/Sheet) F. Elleair’s Lotion Ufu (2.0JPY/Sheet)

KAGAWA TAKAO
E. Kleenex’s Shikou (5.0JPY/Sheet)
UNIVERSITY G. Elleair’s Zeitaku-Hoshitsu (1.5JPY/Sheet)
LABOLATORY
 20

Neural Network for Tactile Discrimination

Training Data Input Layer Activation Functions
Intermediate: ReLu Function
Convolution 1 Output: Softmax Function
Pooling 1 Loss: Cross Entropy Function
Optimization: Adam Optimizer
Convolution 2
Trained Machine
Pooling 2

Convolution 3 Evaluation
7Class Tissue Papers
Pooling 3

Convolution 4

Pooling 4

Fully Connected Test Data (Untrained)

Labels Output Confusion Matrix

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 21

Confusion Matrix (Deep Learning)

100

Correction Rate [%]

80 Correct Answer

80.4%
60

40

20 Training Test
0
0 100 Epoch 200 300

Output Probability
A B C D E F G
A. Nepia’s Nepi-Nepi 0.73 0.15 0.01 0.04 0.08 0.00 0.00
Evaluation Input

B. Nepia’s Premium Soft 0.23 0.76 0.00 0.00 0.00 0.00 0.01

C. Nepia’s Japan Premium 0.01 0.01 0.86 0.03 0.08 0.00 0.00

D. Nepia’s Hana-Celebrity 0.01 0.01 0.01 0.84 0.08 0.04 0.02

E. Kleenex’s Shikou 0.03 0.03 0.16 0.07 0.59 0.14 0.01

F. Elleair’s lotion Ufu 0.00 0.00 0.01 0.12 0.09 0.74 0.04 Ave. Compliance
G. Elleair’s Zeitaku-Hoshitsu 0.00 0.00 0.00 0.07 0.00 0.00 0.93
78.0%
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 22

Confused Matrix (Human’s Finger)

1 Learning and Training for 20min.

2 “Vision Removal” with Eye Mask

3 Discrimination Test with Samples

Average Age: 22

Output Probability
A B C D E F G
A. Nepia’s Nepi-Nepi 0.56 0.25 0.00 0.13 0.00 0.06 0.00
Evaluation Input

B. Nepia’s Premium Soft 0.67 0.13 0.00 0.13 0.00 0.07 0.00

C. Nepia’s Japan Premium 0.00 0.23 0.23 0.08 0.08 0.23 0.15

D. Nepia’s Hana-Celebrity 0.07 0.07 0.21 0.29 0.00 0.07 0.29

E. Kleenex’s Shikou 0.00 0.00 0.07 0.07 0.71 0.00 0.14

F. Elleair’s lotion Ufu 0.00 0.20 0.20 0.20 0.07 0.07 0.27 Correct Answer
G. Elleair’s Zeitaku-Hoshitsu 0.00 0.39 0.00 0.08 0.15 0.23 0.15 31%
KAGAWA The Larger the Data Volume, the Higher the  TAKAO
UNIVERSITY Correct Percentage of Discrimination in DNN LABOLATORY
Trained Sensor vs. Human
23

Deep learning
Human

Accuracy [％]

Deep Learning Human

80% 31%
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 24

Similarity: Multiple Fingerprint Sensor

Fingerprint Contactor Tips 500[µm]

Meissner’s 
Corpuscle
Detection
“Model of Fingerprint of Human” Structures
T. Maeno et al. JSME C, Nano-Tactile Sensor Array
Vol. 71, No.701, pp. 245-250, 2005 The Latest Device realized in 2018
H. Takao et al., IEEE MEMS2019
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 25

2nd Generation Sensor Chip (6 Array)

Electrodes
Surface Shape Detectors
Friction Detectors

“Rounded” Array Contactors

S1 M1 L1 L2 M2 S2
Reference line
500µm 150µm100µm 50µm
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 26

2nd Generation Sensor Chip (6 Array)

500[µm] Spring units

500[µm]

Curvature like fingertip

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 27

Distributions of Tactile Information

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 28

Distributions of Tactile Information

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 29

Directional Touch Feeling Discrimination

Surface Shape Friction Force

0
1mm
the Stiches

Position (mm)
Crossing

Position (mm)
1

1
Stitch

2
140 3

3
Scanning Yarn

4
0 1 2 3 4 0 1 2 3 4

0
1mm 0 -1
(µm) (mN)
Along the

1
Stiches

Position (mm)
Position (mm)
2

2
3

3
Scanning
4

4
0 1 2 3 4 0 1 2 3 4
Scanning Position (mm) Scanning Position (mm)
KAGAWA TAKAO
UNIVERSITY Feature in Friction is Detected LABOLATORY
 30

“Elasticity” and Its Distribution

Hardness Electrodes Surface Shape Sensor
Recognition
Friction Sensor

Flattened

Harder Material

Curved

Softer Material Initial Curvature

KAGAWA TAKAO
UNIVERSITY LABOLATORY
100µm-Resolution Elasticity Distribution
31

Non-woven

050 10

25 Surface Shape
8
50 0

‐25
6

100
‐50

Slip friction
‐75 4
4

150
‐100 Slipping Friction 3

(mN)
‐125
2
2

1
‐150
0
0

‐175

0.4
Hardness Distribution
Elasticity
(mN/µm)

0.4

0.3
0.3

0.2
0.2

0.1
0.1

0 0
10 11 12 13 14 15 16 17 18 19 20

0 1 2 3 4 5 6 7 8 9 10
Position of sample surface (mm)
KAGAWA TAKAO
UNIVERSITY LABOLATORY
 32

2nd-Gen.“Touch-Feeling Scanner”

A Hard Blister on
Palm Skin

Surface Shape

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 33

Additional Information: Skin Hardness

350 0.2
300 Mole Protruding Part 0.15
Surface Hight

Skin Hardness
表⾯硬さ [mN/um]
250 0.1
Skin Hardness
[um]

Skin Mole (Soft)

[mN/um]
200 0.05
[um]
表⾯形状

150 0
100 Surface Shape ‐0.05
50 ‐0.1
0 ‐0.15
‐50 ‐0.2
0 5 10 15 20 25 30 35 40 45 50
⾛査距離
Scanning Position
[mm] [mm]

350 0.2
300 Blister Protruding Part 0.15

Hardness
Surface Hight

表⾯硬さ [mN/um]
250 0.1
Skin Hardness Skin Blister (Hard)
[um][um]

[mN/um]
200 0.05
表⾯形状

150 0
100 Surface Shape ‐0.05

Skin
50 ‐0.1
0 ‐0.15
‐50 ‐0.2
0 5 10 15 20 25 30 35 40 45 50
⾛査距離
Scanning Position
[mm] [mm]

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 34

Conclusions
Nano-Tactile Sensor Chip realizes 0.5µm-spatial
resolution and below 50µN-force resolution that
are the Highest Performances among tactile
sensors.

Combination of Deep N.N. with The Nano-Tactile

Sensor Chip has distinguished 7-types of Tissue
Papers at 80% accuracy (Human is around 30%).

System performance can be more enhanced by

Visualized Hardness Distribution at 100µm spatial
resolution.

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 35

“Nano-Tactile Sensor” Will Come in 2022

TOUCHENCE Inc. JAPAN http://touchence.jp/en/
“Shokac Probe” Sensor http://touchence.jp/en/products/probe.html

KAGAWA TAKAO
UNIVERSITY LABOLATORY
 36

Acknowledgements
This research has been partially supported by
the competitive funding programs in JAPAN
• JST CREST (Grant Number JPMJCR1531)
• JST CREST (Grant Number JPMJCR20C2)
• JST A-STEP StageⅡProgram
• JSPS KAKENHI (Scientific Research (A)17H01271)

The presenters thank to their cooperation;

• Prof. Toru Shimizu (Toyo University, JAPAN)
• Prof. Kinya Kumazawa (JAPN Inst. Invention and Innovation)
• Prof. Yusaku Maeda (National Institute of Tech. Kagawa)
• All the Staff with Micro-Nano Structure Device Integrated
Research Center in Kagawa University, JAPAN
• All the Staff and Students with Takao CREST Team
KAGAWA TAKAO
UNIVERSITY LABOLATORY