1 6단계 autoML

Bojan Tunguz (Jan 23, 2020), “Six Levels of Auto ML”, medium.com

자율주행

Level 0: Automated system issues warnings and may momentarily intervene but has no sustained vehicle control.
Level 1 (“hands on”): The driver and the automated system share control of the vehicle. Examples are Adaptive Cruise Control and Parking Assistance
Level 2 (“hands off”): The automated system takes full control of the vehicle (accelerating, braking, and steering).
Level 3 (“eyes off”): The driver can safely turn their attention away from the driving tasks, e.g. the driver can text or watch a movie.
Level 4 (“mind off”): As level 3, but no driver attention is ever required for safety, e.g. the driver may safely go to sleep or leave the driver’s seat.
Level 5 (“steering wheel optional”): No human intervention is required at all. An example would be a robotic taxi.

autoML

Level 0: No automation. You code your own ML algorithms. From scratch. In C++.

Level 1: Use of high-level algorithm APIs. Sklearn, Keras, Pandas, H2O, XGBoost, etc.

Level 2: Automatic hyperparameter tuning and ensembling. Basic model selection.

Level 3: Automatic (technical) feature engineering and feature selection, technical data augmentation, GUI.

Level 4: Automatic domain and problem specific feature engineering, data augmentation, and data integration.

Level 5: Full ML Automation. Ability to come up with super-human strategies for solving hard ML problems without any input or guidance. Fully conversational interaction with the human user.

1.1 MLOps

MLOPS 25% 규칙
- 데이터 25%
- 비즈니스 25%
- 모형 25%
- DevOps 25%

1.2 모형 공장

1.3 개념과 데이터 Drift

Concept Drift
Label Drift
Feature Drift

1.4 Drift 유형

1.5 Drift 탐지

Lu, Jie, et al. “Learning under concept drift: A review.” IEEE Transactions on Knowledge and Data Engineering 31.12 (2018): 2346-2363. 다운로드

2 AutoKeras

AutoKeras는 Texas A&M 대학에서 개발한 Keras 에 기반한 AutoML 시스템이다. AutoKeras 의 목표는 모든 사람에게 기계학습을 쉽게 할 수 있게 하는 것으로 일종의 기계학습 민주화 운동으로 볼 수 있다.

Luis Sobrecueva (May 21, 2021), “Automated Machine Learning with AutoKeras: Deep learning made accessible for everyone with just few lines of coding”, Packt Publishing

2.1 설치

pip install git+https://github.com/keras-team/keras-tuner.git
pip install autokeras

import numpy as np
import tensorflow as tf
from tensorflow.keras.datasets import mnist
from tensorflow.python.keras.utils.data_utils import Sequence
import autokeras as ak

###### GPU 활성화 코드 ##############
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "2"
config = tf.compat.v1.ConfigProto(log_device_placement=True)
config.gpu_options.allow_growth = True
sess = tf.compat.v1.Session(config=config)
#####################################

(x_train, y_train), (x_test, y_test) = mnist.load_data()
print(x_train.shape)  # (60000, 28, 28)
print(y_train.shape)  # (60000,)
print(y_train[:3])  # array([7, 2, 1], dtype=uint8)


# Initialize the image classifier.
clf = ak.ImageClassifier(overwrite=True, 
                         max_trials=1,
                         distribution_strategy=tf.distribute.MirroredStrategy())
# Feed the image classifier with training data.
clf.fit(x_train, 
        y_train, 
        # Split the training data and use the last 15% as validation data.
        validation_split=0.15,
        epochs=5)

Epoch 10/10

   1/1500 [..............................] - ETA: 16s - loss: 4.0384e-04 - accuracy: 1.0000
   7/1500 [..............................] - ETA: 12s - loss: 0.0262 - accuracy: 0.9911    
  11/1500 [..............................] - ETA: 15s - loss: 0.0275 - accuracy: 0.9915
  18/1500 [..............................] - ETA: 13s - loss: 0.0526 - accuracy: 0.9913
  24/1500 [..............................] - ETA: 13s - loss: 0.0443 - accuracy: 0.9909
  30/1500 [..............................] - ETA: 13s - loss: 0.0413 - accuracy: 0.9906
  33/1500 [..............................] - ETA: 23s - loss: 0.0490 - accuracy: 0.9896
  39/1500 [..............................] - ETA: 22s - loss: 0.0451 - accuracy: 0.9904
  45/1500 [..............................] - ETA: 20s - loss: 0.0448 - accuracy: 0.9896
  50/1500 [>.............................] - ETA: 20s - loss: 0.0413 - accuracy: 0.9900
  54/1500 [>.............................] - ETA: 19s - loss: 0.0407 - accuracy: 0.9902
  61/1500 [>.............................] - ETA: 18s - loss: 0.0390 - accuracy: 0.9908
  67/1500 [>.............................] - ETA: 18s - loss: 0.0379 - accuracy: 0.9911
  74/1500 [>.............................] - ETA: 17s - loss: 0.0359 - accuracy: 0.9916
  79/1500 [>.............................] - ETA: 17s - loss: 0.0345 - accuracy: 0.9917
  86/1500 [>.............................] - ETA: 16s - loss: 0.0369 - accuracy: 0.9909
  93/1500 [>.............................] - ETA: 16s - loss: 0.0367 - accuracy: 0.9903
 100/1500 [=>............................] - ETA: 15s - loss: 0.0348 - accuracy: 0.9906
 104/1500 [=>............................] - ETA: 15s - loss: 0.0340 - accuracy: 0.9910
 111/1500 [=>............................] - ETA: 15s - loss: 0.0322 - accuracy: 0.9916
 116/1500 [=>............................] - ETA: 15s - loss: 0.0333 - accuracy: 0.9911
 123/1500 [=>............................] - ETA: 14s - loss: 0.0329 - accuracy: 0.9909
 126/1500 [=>............................] - ETA: 24s - loss: 0.0346 - accuracy: 0.9903
 133/1500 [=>............................] - ETA: 23s - loss: 0.0340 - accuracy: 0.9906
 140/1500 [=>............................] - ETA: 23s - loss: 0.0354 - accuracy: 0.9902
 143/1500 [=>............................] - ETA: 23s - loss: 0.0350 - accuracy: 0.9902
 149/1500 [=>............................] - ETA: 22s - loss: 0.0348 - accuracy: 0.9901
 156/1500 [==>...........................] - ETA: 21s - loss: 0.0346 - accuracy: 0.9898
 163/1500 [==>...........................] - ETA: 21s - loss: 0.0343 - accuracy: 0.9898
 167/1500 [==>...........................] - ETA: 21s - loss: 0.0353 - accuracy: 0.9891
 174/1500 [==>...........................] - ETA: 20s - loss: 0.0359 - accuracy: 0.9885
 181/1500 [==>...........................] - ETA: 20s - loss: 0.0356 - accuracy: 0.9886
 187/1500 [==>...........................] - ETA: 19s - loss: 0.0359 - accuracy: 0.9880
 191/1500 [==>...........................] - ETA: 19s - loss: 0.0357 - accuracy: 0.9881
 198/1500 [==>...........................] - ETA: 19s - loss: 0.0351 - accuracy: 0.9883
 205/1500 [===>..........................] - ETA: 18s - loss: 0.0346 - accuracy: 0.9886
 212/1500 [===>..........................] - ETA: 18s - loss: 0.0346 - accuracy: 0.9885
 215/1500 [===>..........................] - ETA: 18s - loss: 0.0342 - accuracy: 0.9887
 222/1500 [===>..........................] - ETA: 17s - loss: 0.0343 - accuracy: 0.9887
 229/1500 [===>..........................] - ETA: 17s - loss: 0.0346 - accuracy: 0.9887
 236/1500 [===>..........................] - ETA: 17s - loss: 0.0349 - accuracy: 0.9887
 240/1500 [===>..........................] - ETA: 17s - loss: 0.0346 - accuracy: 0.9888
 247/1500 [===>..........................] - ETA: 16s - loss: 0.0351 - accuracy: 0.9886
 254/1500 [====>.........................] - ETA: 16s - loss: 0.0350 - accuracy: 0.9886
 261/1500 [====>.........................] - ETA: 16s - loss: 0.0359 - accuracy: 0.9885
 265/1500 [====>.........................] - ETA: 16s - loss: 0.0357 - accuracy: 0.9886
 272/1500 [====>.........................] - ETA: 16s - loss: 0.0353 - accuracy: 0.9887
 279/1500 [====>.........................] - ETA: 15s - loss: 0.0363 - accuracy: 0.9886
 284/1500 [====>.........................] - ETA: 15s - loss: 0.0357 - accuracy: 0.9888
 291/1500 [====>.........................] - ETA: 15s - loss: 0.0352 - accuracy: 0.9889
 298/1500 [====>.........................] - ETA: 15s - loss: 0.0358 - accuracy: 0.9888
 305/1500 [=====>........................] - ETA: 15s - loss: 0.0356 - accuracy: 0.9887
 310/1500 [=====>........................] - ETA: 15s - loss: 0.0351 - accuracy: 0.9889
 317/1500 [=====>........................] - ETA: 14s - loss: 0.0346 - accuracy: 0.9890
 324/1500 [=====>........................] - ETA: 14s - loss: 0.0346 - accuracy: 0.9889
 331/1500 [=====>........................] - ETA: 14s - loss: 0.0341 - accuracy: 0.9890
 335/1500 [=====>........................] - ETA: 14s - loss: 0.0337 - accuracy: 0.9892
 342/1500 [=====>........................] - ETA: 14s - loss: 0.0338 - accuracy: 0.9893
 348/1500 [=====>........................] - ETA: 14s - loss: 0.0350 - accuracy: 0.9890
 354/1500 [======>.......................] - ETA: 13s - loss: 0.0348 - accuracy: 0.9891
 358/1500 [======>.......................] - ETA: 13s - loss: 0.0346 - accuracy: 0.9892
 365/1500 [======>.......................] - ETA: 13s - loss: 0.0343 - accuracy: 0.9893
 372/1500 [======>.......................] - ETA: 13s - loss: 0.0345 - accuracy: 0.9892
 379/1500 [======>.......................] - ETA: 13s - loss: 0.0343 - accuracy: 0.9891
 383/1500 [======>.......................] - ETA: 13s - loss: 0.0342 - accuracy: 0.9891
 390/1500 [======>.......................] - ETA: 13s - loss: 0.0339 - accuracy: 0.9893
 397/1500 [======>.......................] - ETA: 12s - loss: 0.0339 - accuracy: 0.9891
 404/1500 [=======>......................] - ETA: 12s - loss: 0.0336 - accuracy: 0.9892
 408/1500 [=======>......................] - ETA: 12s - loss: 0.0334 - accuracy: 0.9892
 414/1500 [=======>......................] - ETA: 12s - loss: 0.0333 - accuracy: 0.9893
 420/1500 [=======>......................] - ETA: 12s - loss: 0.0332 - accuracy: 0.9892
 427/1500 [=======>......................] - ETA: 12s - loss: 0.0330 - accuracy: 0.9892
 434/1500 [=======>......................] - ETA: 12s - loss: 0.0327 - accuracy: 0.9892
 441/1500 [=======>......................] - ETA: 12s - loss: 0.0330 - accuracy: 0.9890
 448/1500 [=======>......................] - ETA: 12s - loss: 0.0328 - accuracy: 0.9890
 453/1500 [========>.....................] - ETA: 11s - loss: 0.0326 - accuracy: 0.9891
 460/1500 [========>.....................] - ETA: 11s - loss: 0.0325 - accuracy: 0.9891
 467/1500 [========>.....................] - ETA: 11s - loss: 0.0327 - accuracy: 0.9890
 474/1500 [========>.....................] - ETA: 11s - loss: 0.0325 - accuracy: 0.9891
 478/1500 [========>.....................] - ETA: 11s - loss: 0.0324 - accuracy: 0.9891
 484/1500 [========>.....................] - ETA: 11s - loss: 0.0326 - accuracy: 0.9891
 491/1500 [========>.....................] - ETA: 11s - loss: 0.0326 - accuracy: 0.9891
 498/1500 [========>.....................] - ETA: 11s - loss: 0.0329 - accuracy: 0.9890
 503/1500 [=========>....................] - ETA: 11s - loss: 0.0329 - accuracy: 0.9889
 510/1500 [=========>....................] - ETA: 11s - loss: 0.0327 - accuracy: 0.9889
 517/1500 [=========>....................] - ETA: 10s - loss: 0.0324 - accuracy: 0.9890
 523/1500 [=========>....................] - ETA: 10s - loss: 0.0325 - accuracy: 0.9891
 529/1500 [=========>....................] - ETA: 10s - loss: 0.0324 - accuracy: 0.9891
 536/1500 [=========>....................] - ETA: 10s - loss: 0.0323 - accuracy: 0.9891
 543/1500 [=========>....................] - ETA: 10s - loss: 0.0323 - accuracy: 0.9892
 547/1500 [=========>....................] - ETA: 10s - loss: 0.0322 - accuracy: 0.9892
 554/1500 [==========>...................] - ETA: 10s - loss: 0.0320 - accuracy: 0.9892
 561/1500 [==========>...................] - ETA: 10s - loss: 0.0317 - accuracy: 0.9894
 568/1500 [==========>...................] - ETA: 10s - loss: 0.0320 - accuracy: 0.9892
 572/1500 [==========>...................] - ETA: 10s - loss: 0.0320 - accuracy: 0.9891
 579/1500 [==========>...................] - ETA: 10s - loss: 0.0320 - accuracy: 0.9891
 586/1500 [==========>...................] - ETA: 9s - loss: 0.0319 - accuracy: 0.9891 
 593/1500 [==========>...................] - ETA: 9s - loss: 0.0317 - accuracy: 0.9892
 597/1500 [==========>...................] - ETA: 9s - loss: 0.0315 - accuracy: 0.9893
 604/1500 [===========>..................] - ETA: 9s - loss: 0.0317 - accuracy: 0.9893
 611/1500 [===========>..................] - ETA: 9s - loss: 0.0318 - accuracy: 0.9893
 617/1500 [===========>..................] - ETA: 9s - loss: 0.0317 - accuracy: 0.9893
 624/1500 [===========>..................] - ETA: 9s - loss: 0.0314 - accuracy: 0.9894
 631/1500 [===========>..................] - ETA: 9s - loss: 0.0323 - accuracy: 0.9893
 638/1500 [===========>..................] - ETA: 9s - loss: 0.0324 - accuracy: 0.9893
 642/1500 [===========>..................] - ETA: 9s - loss: 0.0323 - accuracy: 0.9893
 649/1500 [===========>..................] - ETA: 9s - loss: 0.0322 - accuracy: 0.9894
 656/1500 [============>.................] - ETA: 8s - loss: 0.0322 - accuracy: 0.9894
 663/1500 [============>.................] - ETA: 8s - loss: 0.0324 - accuracy: 0.9894
 665/1500 [============>.................] - ETA: 9s - loss: 0.0323 - accuracy: 0.9894
 671/1500 [============>.................] - ETA: 9s - loss: 0.0325 - accuracy: 0.9894
 677/1500 [============>.................] - ETA: 9s - loss: 0.0324 - accuracy: 0.9894
 681/1500 [============>.................] - ETA: 8s - loss: 0.0323 - accuracy: 0.9895
 688/1500 [============>.................] - ETA: 8s - loss: 0.0324 - accuracy: 0.9894
 695/1500 [============>.................] - ETA: 8s - loss: 0.0323 - accuracy: 0.9894
 702/1500 [=============>................] - ETA: 8s - loss: 0.0322 - accuracy: 0.9894
 706/1500 [=============>................] - ETA: 8s - loss: 0.0323 - accuracy: 0.9893
 713/1500 [=============>................] - ETA: 8s - loss: 0.0326 - accuracy: 0.9892
 720/1500 [=============>................] - ETA: 8s - loss: 0.0323 - accuracy: 0.9892
 727/1500 [=============>................] - ETA: 8s - loss: 0.0322 - accuracy: 0.9893
 731/1500 [=============>................] - ETA: 8s - loss: 0.0320 - accuracy: 0.9893
 738/1500 [=============>................] - ETA: 8s - loss: 0.0319 - accuracy: 0.9894
 745/1500 [=============>................] - ETA: 8s - loss: 0.0318 - accuracy: 0.9893
 752/1500 [==============>...............] - ETA: 8s - loss: 0.0317 - accuracy: 0.9894
 756/1500 [==============>...............] - ETA: 8s - loss: 0.0316 - accuracy: 0.9894
 763/1500 [==============>...............] - ETA: 7s - loss: 0.0314 - accuracy: 0.9895
 770/1500 [==============>...............] - ETA: 7s - loss: 0.0314 - accuracy: 0.9895
 777/1500 [==============>...............] - ETA: 7s - loss: 0.0317 - accuracy: 0.9895
 782/1500 [==============>...............] - ETA: 7s - loss: 0.0316 - accuracy: 0.9896
 789/1500 [==============>...............] - ETA: 7s - loss: 0.0315 - accuracy: 0.9895
 795/1500 [==============>...............] - ETA: 7s - loss: 0.0314 - accuracy: 0.9895
 801/1500 [===============>..............] - ETA: 7s - loss: 0.0315 - accuracy: 0.9895
 808/1500 [===============>..............] - ETA: 7s - loss: 0.0315 - accuracy: 0.9896
 815/1500 [===============>..............] - ETA: 7s - loss: 0.0316 - accuracy: 0.9896
 822/1500 [===============>..............] - ETA: 7s - loss: 0.0314 - accuracy: 0.9897
 826/1500 [===============>..............] - ETA: 7s - loss: 0.0315 - accuracy: 0.9896
 832/1500 [===============>..............] - ETA: 7s - loss: 0.0318 - accuracy: 0.9894
 838/1500 [===============>..............] - ETA: 6s - loss: 0.0323 - accuracy: 0.9893
 845/1500 [===============>..............] - ETA: 6s - loss: 0.0326 - accuracy: 0.9893
 849/1500 [===============>..............] - ETA: 6s - loss: 0.0325 - accuracy: 0.9894
 856/1500 [================>.............] - ETA: 6s - loss: 0.0325 - accuracy: 0.9894
 862/1500 [================>.............] - ETA: 6s - loss: 0.0327 - accuracy: 0.9894
 869/1500 [================>.............] - ETA: 6s - loss: 0.0326 - accuracy: 0.9895
 873/1500 [================>.............] - ETA: 6s - loss: 0.0326 - accuracy: 0.9895
 879/1500 [================>.............] - ETA: 6s - loss: 0.0324 - accuracy: 0.9895
 886/1500 [================>.............] - ETA: 6s - loss: 0.0325 - accuracy: 0.9895
 893/1500 [================>.............] - ETA: 6s - loss: 0.0323 - accuracy: 0.9896
 897/1500 [================>.............] - ETA: 6s - loss: 0.0323 - accuracy: 0.9897
 904/1500 [=================>............] - ETA: 6s - loss: 0.0322 - accuracy: 0.9897
 911/1500 [=================>............] - ETA: 6s - loss: 0.0322 - accuracy: 0.9896
 917/1500 [=================>............] - ETA: 6s - loss: 0.0326 - accuracy: 0.9895
 923/1500 [=================>............] - ETA: 6s - loss: 0.0325 - accuracy: 0.9896
 929/1500 [=================>............] - ETA: 5s - loss: 0.0324 - accuracy: 0.9896
 936/1500 [=================>............] - ETA: 5s - loss: 0.0324 - accuracy: 0.9895
 941/1500 [=================>............] - ETA: 5s - loss: 0.0325 - accuracy: 0.9895
 947/1500 [=================>............] - ETA: 5s - loss: 0.0326 - accuracy: 0.9895
 954/1500 [==================>...........] - ETA: 5s - loss: 0.0326 - accuracy: 0.9895
 961/1500 [==================>...........] - ETA: 5s - loss: 0.0324 - accuracy: 0.9896
 965/1500 [==================>...........] - ETA: 5s - loss: 0.0324 - accuracy: 0.9896
 972/1500 [==================>...........] - ETA: 5s - loss: 0.0324 - accuracy: 0.9896
 979/1500 [==================>...........] - ETA: 5s - loss: 0.0323 - accuracy: 0.9896
 986/1500 [==================>...........] - ETA: 5s - loss: 0.0321 - accuracy: 0.9897
 989/1500 [==================>...........] - ETA: 5s - loss: 0.0321 - accuracy: 0.9897
 995/1500 [==================>...........] - ETA: 5s - loss: 0.0322 - accuracy: 0.9897
1002/1500 [===================>..........] - ETA: 5s - loss: 0.0323 - accuracy: 0.9896
1008/1500 [===================>..........] - ETA: 5s - loss: 0.0322 - accuracy: 0.9897
1012/1500 [===================>..........] - ETA: 5s - loss: 0.0324 - accuracy: 0.9897
1019/1500 [===================>..........] - ETA: 4s - loss: 0.0323 - accuracy: 0.9897
1025/1500 [===================>..........] - ETA: 4s - loss: 0.0324 - accuracy: 0.9897
1031/1500 [===================>..........] - ETA: 4s - loss: 0.0323 - accuracy: 0.9897
1035/1500 [===================>..........] - ETA: 4s - loss: 0.0321 - accuracy: 0.9897
1042/1500 [===================>..........] - ETA: 4s - loss: 0.0320 - accuracy: 0.9898
1048/1500 [===================>..........] - ETA: 4s - loss: 0.0321 - accuracy: 0.9897
1054/1500 [====================>.........] - ETA: 4s - loss: 0.0320 - accuracy: 0.9898
1060/1500 [====================>.........] - ETA: 4s - loss: 0.0319 - accuracy: 0.9898
1067/1500 [====================>.........] - ETA: 4s - loss: 0.0317 - accuracy: 0.9899
1074/1500 [====================>.........] - ETA: 4s - loss: 0.0315 - accuracy: 0.9899
1078/1500 [====================>.........] - ETA: 4s - loss: 0.0318 - accuracy: 0.9899
1085/1500 [====================>.........] - ETA: 4s - loss: 0.0318 - accuracy: 0.9899
1092/1500 [====================>.........] - ETA: 4s - loss: 0.0320 - accuracy: 0.9898
1099/1500 [====================>.........] - ETA: 4s - loss: 0.0319 - accuracy: 0.9898
1104/1500 [=====================>........] - ETA: 4s - loss: 0.0320 - accuracy: 0.9898
1111/1500 [=====================>........] - ETA: 3s - loss: 0.0322 - accuracy: 0.9898
1117/1500 [=====================>........] - ETA: 3s - loss: 0.0323 - accuracy: 0.9898
1124/1500 [=====================>........] - ETA: 3s - loss: 0.0322 - accuracy: 0.9898
1127/1500 [=====================>........] - ETA: 3s - loss: 0.0322 - accuracy: 0.9899
1134/1500 [=====================>........] - ETA: 3s - loss: 0.0322 - accuracy: 0.9899
1141/1500 [=====================>........] - ETA: 3s - loss: 0.0320 - accuracy: 0.9899
1148/1500 [=====================>........] - ETA: 3s - loss: 0.0320 - accuracy: 0.9899
1155/1500 [======================>.......] - ETA: 3s - loss: 0.0319 - accuracy: 0.9899
1162/1500 [======================>.......] - ETA: 3s - loss: 0.0318 - accuracy: 0.9900
1168/1500 [======================>.......] - ETA: 3s - loss: 0.0320 - accuracy: 0.9900
1172/1500 [======================>.......] - ETA: 3s - loss: 0.0321 - accuracy: 0.9900
1179/1500 [======================>.......] - ETA: 3s - loss: 0.0320 - accuracy: 0.9900
1186/1500 [======================>.......] - ETA: 3s - loss: 0.0320 - accuracy: 0.9900
1192/1500 [======================>.......] - ETA: 3s - loss: 0.0319 - accuracy: 0.9900
1196/1500 [======================>.......] - ETA: 3s - loss: 0.0319 - accuracy: 0.9900
1203/1500 [=======================>......] - ETA: 3s - loss: 0.0318 - accuracy: 0.9901
1209/1500 [=======================>......] - ETA: 2s - loss: 0.0318 - accuracy: 0.9900
1215/1500 [=======================>......] - ETA: 2s - loss: 0.0317 - accuracy: 0.9901
1217/1500 [=======================>......] - ETA: 2s - loss: 0.0318 - accuracy: 0.9900
1223/1500 [=======================>......] - ETA: 2s - loss: 0.0318 - accuracy: 0.9900
1229/1500 [=======================>......] - ETA: 2s - loss: 0.0318 - accuracy: 0.9900
1234/1500 [=======================>......] - ETA: 2s - loss: 0.0319 - accuracy: 0.9899
1237/1500 [=======================>......] - ETA: 2s - loss: 0.0319 - accuracy: 0.9899
1243/1500 [=======================>......] - ETA: 2s - loss: 0.0320 - accuracy: 0.9899
1249/1500 [=======================>......] - ETA: 2s - loss: 0.0319 - accuracy: 0.9899
1255/1500 [========================>.....] - ETA: 2s - loss: 0.0320 - accuracy: 0.9899
1259/1500 [========================>.....] - ETA: 2s - loss: 0.0319 - accuracy: 0.9899
1266/1500 [========================>.....] - ETA: 2s - loss: 0.0319 - accuracy: 0.9900
1273/1500 [========================>.....] - ETA: 2s - loss: 0.0318 - accuracy: 0.9899
1281/1500 [========================>.....] - ETA: 2s - loss: 0.0318 - accuracy: 0.9899
1286/1500 [========================>.....] - ETA: 2s - loss: 0.0316 - accuracy: 0.9899
1293/1500 [========================>.....] - ETA: 2s - loss: 0.0317 - accuracy: 0.9899
1300/1500 [=========================>....] - ETA: 2s - loss: 0.0318 - accuracy: 0.9898
1307/1500 [=========================>....] - ETA: 1s - loss: 0.0317 - accuracy: 0.9898
1311/1500 [=========================>....] - ETA: 1s - loss: 0.0319 - accuracy: 0.9898
1318/1500 [=========================>....] - ETA: 1s - loss: 0.0320 - accuracy: 0.9898
1325/1500 [=========================>....] - ETA: 1s - loss: 0.0319 - accuracy: 0.9898
1331/1500 [=========================>....] - ETA: 1s - loss: 0.0321 - accuracy: 0.9897
1334/1500 [=========================>....] - ETA: 1s - loss: 0.0320 - accuracy: 0.9898
1341/1500 [=========================>....] - ETA: 1s - loss: 0.0322 - accuracy: 0.9898
1348/1500 [=========================>....] - ETA: 1s - loss: 0.0322 - accuracy: 0.9898
1355/1500 [==========================>...] - ETA: 1s - loss: 0.0320 - accuracy: 0.9898
1359/1500 [==========================>...] - ETA: 1s - loss: 0.0321 - accuracy: 0.9898
1366/1500 [==========================>...] - ETA: 1s - loss: 0.0320 - accuracy: 0.9898
1373/1500 [==========================>...] - ETA: 1s - loss: 0.0320 - accuracy: 0.9898
1380/1500 [==========================>...] - ETA: 1s - loss: 0.0320 - accuracy: 0.9898
1383/1500 [==========================>...] - ETA: 1s - loss: 0.0319 - accuracy: 0.9898
1389/1500 [==========================>...] - ETA: 1s - loss: 0.0319 - accuracy: 0.9898
1396/1500 [==========================>...] - ETA: 1s - loss: 0.0318 - accuracy: 0.9898
1402/1500 [===========================>..] - ETA: 0s - loss: 0.0318 - accuracy: 0.9898
1404/1500 [===========================>..] - ETA: 0s - loss: 0.0317 - accuracy: 0.9899
1411/1500 [===========================>..] - ETA: 0s - loss: 0.0318 - accuracy: 0.9898
1419/1500 [===========================>..] - ETA: 0s - loss: 0.0317 - accuracy: 0.9898
1426/1500 [===========================>..] - ETA: 0s - loss: 0.0317 - accuracy: 0.9898
1432/1500 [===========================>..] - ETA: 0s - loss: 0.0318 - accuracy: 0.9897
1439/1500 [===========================>..] - ETA: 0s - loss: 0.0319 - accuracy: 0.9897
1446/1500 [===========================>..] - ETA: 0s - loss: 0.0320 - accuracy: 0.9897
1450/1500 [============================>.] - ETA: 0s - loss: 0.0320 - accuracy: 0.9897
1457/1500 [============================>.] - ETA: 0s - loss: 0.0320 - accuracy: 0.9896
1463/1500 [============================>.] - ETA: 0s - loss: 0.0320 - accuracy: 0.9896
1470/1500 [============================>.] - ETA: 0s - loss: 0.0320 - accuracy: 0.9896
1474/1500 [============================>.] - ETA: 0s - loss: 0.0320 - accuracy: 0.9897
1481/1500 [============================>.] - ETA: 0s - loss: 0.0321 - accuracy: 0.9896
1488/1500 [============================>.] - ETA: 0s - loss: 0.0322 - accuracy: 0.9896
1495/1500 [============================>.] - ETA: 0s - loss: 0.0322 - accuracy: 0.9895
1500/1500 [==============================] - ETA: 0s - loss: 0.0322 - accuracy: 0.9895
1500/1500 [==============================] - 17s 11ms/step - loss: 0.0322 - accuracy: 0.9895 - val_loss: 0.0423 - val_accuracy: 0.9895

# Predict with the best model.
predicted_y = clf.predict(x_test)
print(predicted_y)

#   1/313 [..............................] - ETA: 6:49
#  16/313 [>.............................] - ETA: 1s  
#  24/313 [=>............................] - ETA: 1s
#  38/313 [==>...........................] - ETA: 1s
#  53/313 [====>.........................] - ETA: 1s
#  68/313 [=====>........................] - ETA: 1s
#  80/313 [======>.......................] - ETA: 1s
#  89/313 [=======>......................] - ETA: 1s
# 103/313 [========>.....................] - ETA: 0s
# 112/313 [=========>....................] - ETA: 0s
# 125/313 [==========>...................] - ETA: 0s
# 138/313 [============>.................] - ETA: 0s
# 151/313 [=============>................] - ETA: 0s
# 155/313 [=============>................] - ETA: 0s
# 159/313 [==============>...............] - ETA: 0s
# 171/313 [===============>..............] - ETA: 0s
# 186/313 [================>.............] - ETA: 0s
# 194/313 [=================>............] - ETA: 0s
# 209/313 [===================>..........] - ETA: 0s
# 222/313 [====================>.........] - ETA: 0s
# 237/313 [=====================>........] - ETA: 0s
# 246/313 [======================>.......] - ETA: 0s
# 259/313 [=======================>......] - ETA: 0s
# 272/313 [=========================>....] - ETA: 0s
# 285/313 [==========================>...] - ETA: 0s
# 288/313 [==========================>...] - ETA: 0s
# 301/313 [===========================>..] - ETA: 0s
# 313/313 [==============================] - ETA: 0s
# 313/313 [==============================] - 3s 5ms/step
# 
#   1/313 [..............................] - ETA: 29s
#   6/313 [..............................] - ETA: 3s 
#  20/313 [>.............................] - ETA: 1s
#  33/313 [==>...........................] - ETA: 1s
#  45/313 [===>..........................] - ETA: 1s
#  49/313 [===>..........................] - ETA: 1s
#  61/313 [====>.........................] - ETA: 1s
#  76/313 [======>.......................] - ETA: 1s
#  92/313 [=======>......................] - ETA: 1s
# 101/313 [========>.....................] - ETA: 1s
# 117/313 [==========>...................] - ETA: 0s
# 132/313 [===========>..................] - ETA: 0s
# 142/313 [============>.................] - ETA: 0s
# 149/313 [=============>................] - ETA: 0s
# 163/313 [==============>...............] - ETA: 0s
# 177/313 [===============>..............] - ETA: 0s
# 187/313 [================>.............] - ETA: 0s
# 193/313 [=================>............] - ETA: 0s
# 208/313 [==================>...........] - ETA: 0s
# 223/313 [====================>.........] - ETA: 0s
# 232/313 [=====================>........] - ETA: 0s
# 240/313 [======================>.......] - ETA: 0s
# 255/313 [=======================>......] - ETA: 0s
# 269/313 [========================>.....] - ETA: 0s
# 283/313 [==========================>...] - ETA: 0s
# 292/313 [==========================>...] - ETA: 0s
# 306/313 [============================>.] - ETA: 0s
# 313/313 [==============================] - 1s 4ms/step
# >>> print(predicted_y)
# [['7']
#  ['2']
#  ['1']
#  ...
#  ['4']
#  ['5']
#  ['6']]
# Evaluate the best model with testing data.

print(clf.evaluate(x_test, y_test))

#   1/313 [..............................] - ETA: 1:53 - loss: 0.0070 - accuracy: 1.0000
#   9/313 [..............................] - ETA: 2s - loss: 0.0132 - accuracy: 0.9965  
#  14/313 [>.............................] - ETA: 2s - loss: 0.0262 - accuracy: 0.9911
#  19/313 [>.............................] - ETA: 2s - loss: 0.0245 - accuracy: 0.9901
#  24/313 [=>............................] - ETA: 2s - loss: 0.0259 - accuracy: 0.9883
#  33/313 [==>...........................] - ETA: 2s - loss: 0.0433 - accuracy: 0.9839
#  38/313 [==>...........................] - ETA: 2s - loss: 0.0437 - accuracy: 0.9844
#  46/313 [===>..........................] - ETA: 2s - loss: 0.0526 - accuracy: 0.9810
#  55/313 [====>.........................] - ETA: 2s - loss: 0.0545 - accuracy: 0.9807
#  64/313 [=====>........................] - ETA: 1s - loss: 0.0542 - accuracy: 0.9814
#  69/313 [=====>........................] - ETA: 1s - loss: 0.0599 - accuracy: 0.9805
#  75/313 [======>.......................] - ETA: 1s - loss: 0.0585 - accuracy: 0.9808
#  84/313 [=======>......................] - ETA: 1s - loss: 0.0589 - accuracy: 0.9807
#  93/313 [=======>......................] - ETA: 1s - loss: 0.0588 - accuracy: 0.9802
#  99/313 [========>.....................] - ETA: 1s - loss: 0.0576 - accuracy: 0.9804
# 107/313 [=========>....................] - ETA: 1s - loss: 0.0553 - accuracy: 0.9813
# 115/313 [==========>...................] - ETA: 1s - loss: 0.0556 - accuracy: 0.9818
# 123/313 [==========>...................] - ETA: 1s - loss: 0.0561 - accuracy: 0.9817
# 128/313 [===========>..................] - ETA: 1s - loss: 0.0552 - accuracy: 0.9819
# 137/313 [============>.................] - ETA: 1s - loss: 0.0541 - accuracy: 0.9822
# 145/313 [============>.................] - ETA: 1s - loss: 0.0536 - accuracy: 0.9823
# 153/313 [=============>................] - ETA: 1s - loss: 0.0541 - accuracy: 0.9820
# 155/313 [=============>................] - ETA: 1s - loss: 0.0538 - accuracy: 0.9821
# 163/313 [==============>...............] - ETA: 1s - loss: 0.0513 - accuracy: 0.9829
# 171/313 [===============>..............] - ETA: 1s - loss: 0.0492 - accuracy: 0.9836
# 178/313 [================>.............] - ETA: 1s - loss: 0.0474 - accuracy: 0.9842
# 180/313 [================>.............] - ETA: 1s - loss: 0.0474 - accuracy: 0.9842
# 188/313 [=================>............] - ETA: 0s - loss: 0.0460 - accuracy: 0.9845
# 197/313 [=================>............] - ETA: 0s - loss: 0.0446 - accuracy: 0.9849
# 204/313 [==================>...........] - ETA: 0s - loss: 0.0431 - accuracy: 0.9854
# 212/313 [===================>..........] - ETA: 0s - loss: 0.0440 - accuracy: 0.9851
# 220/313 [====================>.........] - ETA: 0s - loss: 0.0425 - accuracy: 0.9857
# 228/313 [====================>.........] - ETA: 0s - loss: 0.0410 - accuracy: 0.9862
# 234/313 [=====================>........] - ETA: 0s - loss: 0.0400 - accuracy: 0.9865
# 242/313 [======================>.......] - ETA: 0s - loss: 0.0387 - accuracy: 0.9870
# 251/313 [=======================>......] - ETA: 0s - loss: 0.0375 - accuracy: 0.9874
# 254/313 [=======================>......] - ETA: 0s - loss: 0.0378 - accuracy: 0.9873
# 257/313 [=======================>......] - ETA: 0s - loss: 0.0374 - accuracy: 0.9875
# 265/313 [========================>.....] - ETA: 0s - loss: 0.0365 - accuracy: 0.9877
# 273/313 [=========================>....] - ETA: 0s - loss: 0.0355 - accuracy: 0.9881
# 278/313 [=========================>....] - ETA: 0s - loss: 0.0348 - accuracy: 0.9883
# 283/313 [==========================>...] - ETA: 0s - loss: 0.0355 - accuracy: 0.9882
# 291/313 [==========================>...] - ETA: 0s - loss: 0.0346 - accuracy: 0.9885
# 299/313 [===========================>..] - ETA: 0s - loss: 0.0339 - accuracy: 0.9887
# 307/313 [============================>.] - ETA: 0s - loss: 0.0362 - accuracy: 0.9879
# 312/313 [============================>.] - ETA: 0s - loss: 0.0362 - accuracy: 0.9879
# 313/313 [==============================] - 3s 8ms/step - loss: 0.0362 - accuracy: 0.9879
# [0.0361783429980278, 0.9879000186920166]

2.2 모형 저장

mnist_auto_model = clf.export_model()

mnist_auto_model.save("data/mnist_auto_model.h5")

2.3 Inference

from tensorflow.keras.models import load_model
mnist_inference_model = load_model("data/mnist_auto_model.h5")

print(mnist_inference_model.predict(x_test[:10]))

# 1/1 [==============================] - ETA: 0s
# 1/1 [==============================] - 3s 3s/step
# [[1.32050326e-09 5.58818547e-10 3.78649474e-07 9.99164172e-07
#   1.33203268e-10 2.78588975e-11 5.63584436e-16 9.99998569e-01
#   3.92891764e-09 2.32371775e-08]
#  [3.08622106e-07 4.29323836e-06 9.99992967e-01 2.70977765e-08
#   1.83834836e-09 4.31682412e-10 1.94188556e-06 6.18551694e-12
#   5.03049534e-07 3.12175737e-11]
#  [7.60133787e-08 9.99809682e-01 1.49333109e-05 2.55621785e-07
#   1.34683374e-04 2.41513249e-06 2.98452250e-07 2.57335432e-05
#   1.18708749e-05 3.97069293e-08]
#  [9.99997616e-01 3.81995867e-12 8.80706295e-07 2.07579753e-09
#   6.43973819e-10 2.07098960e-08 1.32341120e-06 1.70947601e-09
#   1.69439502e-08 2.69762097e-07]
#  [3.89613888e-08 1.29475808e-09 1.52856661e-08 1.29996272e-08
#   9.99991417e-01 2.09018178e-10 5.77958437e-09 1.13876794e-07
#   3.00419210e-08 8.33169815e-06]
#  [5.08510603e-08 9.99519467e-01 9.68662516e-06 5.38548832e-08
#   1.75294699e-04 2.23298940e-07 1.94389855e-08 2.77489919e-04
#   1.75310124e-05 1.79468174e-07]
#  [6.19284788e-12 2.30456476e-06 1.10171030e-07 3.57055230e-09
#   9.99865294e-01 5.86728781e-08 3.17030707e-11 4.35810307e-06
#   1.24242419e-04 3.70814996e-06]
#  [9.81296034e-12 7.29698924e-09 1.23314339e-06 1.13969929e-06
#   1.19719491e-03 6.22080261e-05 2.93067370e-09 3.59439731e-08
#   6.32558251e-03 9.92412508e-01]
#  [1.11055797e-05 3.99178468e-11 8.96933550e-09 7.30776932e-08
#   8.18820172e-06 9.72040772e-01 2.64719762e-02 5.06327380e-09
#   1.46416645e-03 3.62182686e-06]
#  [4.99363679e-08 2.78283840e-10 1.06951381e-08 4.76083045e-07
#   1.60303130e-03 4.39877361e-08 5.24907071e-12 8.03113682e-04
#   1.69744788e-04 9.97423530e-01]]

2.4 모형 시각화

from tensorflow.keras.utils import plot_model
plot_model(mnist_inference_model,
           show_shapes=True, 
           show_layer_names=True, 
           to_file='assets/automation/mnist_autokeras.png')

20분만에 끝내는 딥러닝

딥러닝 모형 개발 자동화

한국 R 사용자회