I need val losses in the output file log.json

[YusukeImadegawa]

Now, I'm trying mmpretrain for classification.
I'd like to know how to get val losses as log.json. This code is a part of my config below.
I could get train acc and loss but now only val acc...

val_evaluator = dict(type='Accuracy', topk=(1, ))

optim_wrapper = dict(
    optimizer=dict(type='SGD', lr=0.005, momentum=0.9, weight_decay=0.0001))

param_scheduler = dict(type='StepLR', step_size=6, gamma=0.1)

train_cfg = dict(by_epoch=True, max_epochs=5, val_interval=1) 
default_hooks = dict(
    timer=dict(type='IterTimerHook'),
    logger=dict(type='LoggerHook', interval=10),
    param_scheduler=dict(type='ParamSchedulerHook'),
    checkpoint=dict(type='CheckpointHook', interval=1, max_keep_ckpts=3),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    visualization=dict(type='VisualizationHook', enable=False))

randomness = dict(seed=0, diff_rank_seed=True) 

[Ezra-Yu]

Very sorry that this feature is not available right now.

[YusukeImadegawa]

Thanks for your help. I got it done.

%%writefile /content/mmpretrain/tools/analysis_tools/analyze_logs.py
### I added the code below
def plot_phase_valntrain(metric, train_logs, val_logs, curve_label):
    train_xs = np.array([log['step'] for log in train_logs])
    train_ys = np.array([log[metric] for log in train_logs])
    val_xs = np.array([log['step'] for log in val_logs])
    val_ys = np.array([log[metric] for log in val_logs])

    if 'epoch' in train_logs[0]:
        scale_factor = train_logs[-1]['step'] / train_logs[-1]['epoch']
        plt.xlabel('Epochs')
    else:
        plt.xlabel('Iters')
    plt.plot(train_xs, train_ys, label=curve_label, linewidth=0.75)
    plt.plot(val_xs, val_ys, label=curve_label, linewidth=0.75)

---

def plot_curve_helper(log_dicts, metrics, args, legend):
    num_metrics = len(metrics)
    for i, log_dict in enumerate(log_dicts):
        for j, key in enumerate(metrics):
            json_log = args.json_logs[i]
            print(f'plot curve of {json_log}, metric is {key}')
            curve_label = legend[i * num_metrics + j]

            train_keys = {} if len(log_dict['train']) == 0 else set(
                log_dict['train'][0].keys()) - {'step', 'epoch'}
            val_keys = {} if len(log_dict['val']) == 0 else set(
                log_dict['val'][0].keys()) - {'step'}

            ### I modified the code below
            if key in val_keys and key in train_keys:
                plot_phase_valntrain(key, log_dict['val'], curve_label) # you need to define
            elif key in val_keys:
                plot_phase_val(key, log_dict['val'], curve_label)
            elif key in train_keys:
                plot_phase_train(key, log_dict['train'], curve_label)
            else:
                raise ValueError(
                    f'Invalid key "{key}", please choose from '
                    f'{set.union(set(train_keys), set(val_keys))}.')
            plt.legend()

---

and then i tried this code below.

 !python tools/analysis_tools/analyze_logs.py \
  plot_curve my_result.json \
  --keys accuracy_top-1 accuracy/top1 \
  --legend Train Val \
  --out work_dirs/vgg16_b32x8_imagenet_building/results_accuracy.jpg

[Ezra-Yu]

Have you compared the log data to see if the images are correct? At the beginning VAl>TRAIN doesn't look reasonable

[YusukeImadegawa]

Thanks, --legend Val Train is not correct in this pic (--legend Train Val). but Accuracy itself is not good actually.
cuz, I got error "The model and loaded state dict do not match exactly" when i run train.py.

size mismatch for classifier.6.weight: copying a param with shape torch.Size([1000, 4096]) from checkpoint, the shape in current model is torch.Size([10, 4096]).
size mismatch for classifier.6.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([10]).

Sorry for asking so many questions. please check my config below.

_base_ = [
    '../_base_/models/vgg16.py', 
    '../_base_/datasets/imagenet_bs256_davit_224.py',
    '../_base_/schedules/imagenet_bs256_epochstep.py',
    '../_base_/default_runtime.py'
]

model = dict(
    type='ImageClassifier', 
    backbone=dict(
        init_cfg=dict(
            type='Pretrained',
            checkpoint='https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_batch256_imagenet_20210208-db26f1a5.pth',
            prefix='backbone',
        ),
        frozen_stages=2,
        num_classes=10,
    ),
    head=dict(topk=(1, ), cal_acc=True), 
)

data_preprocessor = dict(
    mean=[124.508, 116.050, 106.438], 
    std=[58.577, 57.310, 57.437], 
    to_rgb=True,
)

train_pipeline = [
    dict(type='LoadImageFromFile'), 
    dict(type='RandomResizedCrop', size=224, backend='pillow'), 
    dict(type='RandomFlip', flip_prob=0.5, direction='horizontal'), 
    dict(type='ImageToTensor', keys=['img']),
    dict(type='ToTensor', keys=['gt_label']), 
    dict(type='Collect', keys=['img', 'gt_label']) 
]

val_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='Resize', size=(256, -1), backend='pillow'), 
    dict(type='CenterCrop', crop_size=224),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='Collect', keys=['img'])
]

test_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='Resize', size=(256, -1), backend='pillow'),
    dict(type='CenterCrop', crop_size=224),
    dict(type='ImageToTensor', keys=['img']),
    dict(type='Collect', keys=['img'])
]

data_root = '/content/drive/MyDrive/MMClassification/my_dataset'
train_dataloader = dict(
    batch_size=32,
    num_workers=2, 
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        data_prefix='train/',
        ann_file='meta/trainQ1.txt',
        classes=[0,1,2,3,4,5,6,7,8,9],
    ))

val_dataloader = dict(
    batch_size=32,
    num_workers=2, 
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        data_prefix='val/',
        ann_file='meta/valQ1.txt',
        classes=[0,1,2,3,4,5,6,7,8,9],
    ))

test_dataloader = dict(
    batch_size=32,
    num_workers=2, 
    dataset=dict(
        type='CustomDataset',
        data_root=data_root,
        data_prefix='test/',
        ann_file='meta/testQ1.txt',
        classes=[0,1,2,3,4,5,6,7,8,9],
    ))

val_evaluator = dict(type='Accuracy', topk=(1, ))

# optimizer hyper-parameters
optim_wrapper = dict(
    optimizer=dict(type='SGD', lr=0.005, momentum=0.9, weight_decay=0.0001))
# learning policy
param_scheduler = dict(type='StepLR', step_size=6, gamma=0.1)

# runtime settings
train_cfg = dict(by_epoch=True, max_epochs=100, val_interval=1) 
default_hooks = dict(
    timer=dict(type='IterTimerHook'),
    logger=dict(type='LoggerHook', interval=10),
    param_scheduler=dict(type='ParamSchedulerHook'),
    checkpoint=dict(type='CheckpointHook', interval=1, max_keep_ckpts=3),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    visualization=dict(type='VisualizationHook', enable=False))

randomness = dict(seed=0, diff_rank_seed=True)

+----------------------------+----------------------+-----------+--------------+
| module | #parameters or shape | #flops | #activations |
+----------------------------+----------------------+-----------+--------------+
| backbone | 0.134G | 15.466G | 13.556M |
| features | 14.715M | 15.347G | 13.548M |
| features.0.conv | 1.792K | 86.704M | 3.211M |
| features.0.conv.weight | (64, 3, 3, 3) | | |
| features.0.conv.bias | (64,) | | |
| features.1.conv | 36.928K | 1.85G | 3.211M |
| features.1.conv.weight | (64, 64, 3, 3) | | |
| features.1.conv.bias | (64,) | | |
| features.3.conv | 73.856K | 0.925G | 1.606M |
| features.3.conv.weight | (128, 64, 3, 3) | | |
| features.3.conv.bias | (128,) | | |
| features.4.conv | 0.148M | 1.85G | 1.606M |
| features.4.conv.weight | (128, 128, 3, 3) | | |
| features.4.conv.bias | (128,) | | |
| features.6.conv | 0.295M | 0.925G | 0.803M |
| features.6.conv.weight | (256, 128, 3, 3) | | |
| features.6.conv.bias | (256,) | | |
| features.7.conv | 0.59M | 1.85G | 0.803M |
| features.7.conv.weight | (256, 256, 3, 3) | | |
| features.7.conv.bias | (256,) | | |
| features.8.conv | 0.59M | 1.85G | 0.803M |
| features.8.conv.weight | (256, 256, 3, 3) | | |
| features.8.conv.bias | (256,) | | |
| features.10.conv | 1.18M | 0.925G | 0.401M |
| features.10.conv.weight | (512, 256, 3, 3) | | |
| features.10.conv.bias | (512,) | | |
| features.11.conv | 2.36M | 1.85G | 0.401M |
| features.11.conv.weight | (512, 512, 3, 3) | | |
| features.11.conv.bias | (512,) | | |
| features.12.conv | 2.36M | 1.85G | 0.401M |
| features.12.conv.weight | (512, 512, 3, 3) | | |
| features.12.conv.bias | (512,) | | |
| features.14.conv | 2.36M | 0.462G | 0.1M |
| features.14.conv.weight | (512, 512, 3, 3) | | |
| features.14.conv.bias | (512,) | | |
| features.15.conv | 2.36M | 0.462G | 0.1M |
| features.15.conv.weight | (512, 512, 3, 3) | | |
| features.15.conv.bias | (512,) | | |
| features.16.conv | 2.36M | 0.462G | 0.1M |
| features.16.conv.weight | (512, 512, 3, 3) | | |
| features.16.conv.bias | (512,) | | |
| classifier | 0.12G | 0.12G | 8.202K |
| classifier.0 | 0.103G | 0.103G | 4.096K |
| classifier.0.weight | (4096, 25088) | | |
| classifier.0.bias | (4096,) | | |
| classifier.3 | 16.781M | 16.777M | 4.096K |
| classifier.3.weight | (4096, 4096) | | |
| classifier.3.bias | (4096,) | | |
| classifier.6 | 40.97K | 40.96K | 10 |
| classifier.6.weight | (10, 4096) | | |
| classifier.6.bias | (10,) | | |
+----------------------------+----------------------+-----------+--------------+
Input shape: (3, 224, 224)
Flops: 15.466G
Params: 0.134G
Activation: 13.556M

[Ezra-Yu]

It looks correct that you are loading the number of categories of the pre-trained model (1000) versus the number of categories of your classification (10). So the final classification header cannot directly load the pre-trained weights, but the default initialization method used

[YusukeImadegawa]

I made the following fixes to avoid the error. but same error still happens. How can i modify config?
Error 1 : train.py

size mismatch for classifier.6.weight: copying a param with shape torch.Size([1000, 4096]) from checkpoint, the shape in current model is torch.Size([10, 4096]).
size mismatch for classifier.6.bias: copying a param with shape torch.Size([1000]) from checkpoint, the shape in current model is torch.Size([10]).
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x10 and 1024x10)

Error 2 : get_flops.py

Unsupported operator aten::max_pool2d encountered 5 time(s)
The following submodules of the model were never called during the trace of the graph. They may be unused, or they were accessed by direct calls to .forward() or via other python methods. In the latter case they will have zeros for statistics, though their statistics will still contribute to their parent calling module.
data_preprocessor, head, head.fc, head.loss_module

I'd like to fine-tune VGG16 pretrained by ImageNet to classify into 10 classes. sorry for bothering you.

_base_ = [
    '../_base_/models/vgg16.py',
    '../_base_/datasets/imagenet_bs256_davit_224.py', 
    '../_base_/schedules/imagenet_bs256_epochstep.py',
    '../_base_/default_runtime.py'
]

model = dict(
    type='ImageClassifier', 
    backbone=dict(
        type='VGG', ###
        depth=16, ###
        init_cfg=dict( 
            type='Pretrained',
            checkpoint='https://download.openmmlab.com/mmclassification/v0/vgg/vgg16_batch256_imagenet_20210208-db26f1a5.pth',
            prefix='backbone',
        ),
        frozen_stages=2, 
        num_classes=10,
    ),
    head=dict( 
        type='LinearClsHead',
        num_classes=10, ###
        in_channels=1024, ###
        topk=(1, ), 
        cal_acc=True), 
)