Release

Author: imantdaunhawer

LICENSE

@@ -0,0 +1,25 @@
+MIT License
+
+Copyright (c) 2021 Imant Daunhawer
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
+Note that individual files (e.g., fid_score.py, inception.py, etc.) derive from
+other projects that were publicly accessible at the time of writing and might
+have their own licensing.

README.md

@@ -0,0 +1,67 @@
+# Disentangling Multimodal Variational Autoencoder
+
+Official code to supplement the paper [Self-supervised Disentanglement of
+Modality-specific and Shared Factors Improves Multimodal Generative
+Models](https://mds.inf.ethz.ch/fileadmin/user_upload/gcpr_daunhawer_camera_ready.pdf)
+published at [GCPR
+2020](https://link.springer.com/chapter/10.1007/978-3-030-71278-5_33). This
+repository contains a pytorch implementation of the disentangling multimodal
+variational autoencoder (DMVAE) and the code to run the experiments from our
+paper.
+
+## Installation
+
+```bash
+# set up environment
+$ conda env create -f environment.yml  # install dependencies
+$ conda activate dmvae                 # activate environment
+```
+
+## Paired MNIST experiment
+```bash
+$ cd mmmnist
+$ ./run_jobs                     # create dataset and run experiment
+$ tensorboard --logdir runs/tmp  # monitor training
+```
+
+## MNIST/SVHN experiment
+```bash
+$ cd mnist_svhn
+$ python make_mnist_svhn.py      # create dataset
+$ ./run_jobs                     # run experiment
+$ tensorboard --logdir runs/tmp  # monitor training
+```
+
+## Post-hoc analysis
+
+The tensorboard logs contain a lot of metrics (likelihood values,
+classification accuracies, etc.), but not the complete evaluation; for
+instance, they do not include the coherence values nor the the unconditionally
+generated samples and FID values with ex-post density estimation. To compute
+these, run the post-hoc analysis using the script `post_hoc_analysis.py` or,
+more conveniently, using the bash script `post_hoc_analysis_batch` as follows:
+```
+$ ./post_hoc_analysis_batch <path_to_experiment> <logdir>
+```
+where `path_to_experiment` is the directory of the experiment (e.g.,
+`$PWD/mmmnist`) and `logdir` denotes directory with the logfiles for the
+respective experiment (e.g., `$PWD/mmmnist/runs/tmp/version_x`). Results from
+the post-hoc analysis are saved to the respective `logdir`.  There, you will
+find quantitative results in `results.txt` and qualitative results in the form
+of png images.
+
+## BibTeX
+
+If you find this project useful, please cite our paper:
+```bibtex
+@article{daunhawer2020dmvae,
+  author    = {Imant Daunhawer and
+               Thomas M. Sutter and
+               Ricards Marcinkevics and
+               Julia E. Vogt},
+  title     = {Self-supervised Disentanglement of Modality-Specific and Shared Factors
+               Improves Multimodal Generative Models},
+  booktitle = {German Conference on Pattern Recognition},
+  year      = {2020},
+}
+```

abstract_getters.py

@@ -0,0 +1,46 @@
+class AbstractGetters:
+    """
+    This abstract class defines getter methods that need to be implemented for every multimodal dataset separately.
+    """
+    def get_encs_decs(self, flags, liks):
+        """
+        Getter for lists with encoders and decoders for all modalities.
+
+        Args:
+            flags: argparse.Namespace with input arguments.
+            liks: List with likelihoods for every modality.
+
+        Returns:
+            Lists with newly initialized encoders and decoders for all modalities.
+        """
+        raise NotImplementedError
+
+    def get_img_to_digit_clfs(self, flags):
+        """
+        Getter for the list with pre-trained image-to-digit classifiers.
+
+        Args:
+            flags: argparse.Namespace with input arguments.
+
+        Returns:
+            A list with pre-trained image-to-digit classifiers for all modalities.
+        """
+        raise NotImplementedError
+
+    def get_data_loaders(self, batch_size, num_modalities, num_workers, shuffle=True, device="cuda",
+                         random_noise=False):
+        """
+        Getter for train and test set DataLoaders.
+
+        Args:
+            batch_size: Batch size to use when loading data.
+            num_modalities: Number of modalities.
+            num_workers: How many subprocesses to use for data loading.
+            shuffle: Flag identifying whether to shuffle the data.
+            device: Which device to use for storing tensors, "cuda" (by default) or "cpu".
+            random_noise: Flag identifying whether to augment images with Gaussian white noise.
+
+        Returns:
+            DataLoader for training and test sets.
+        """
+        raise NotImplementedError

environment.yml

@@ -0,0 +1,53 @@
+name: dmvae
+dependencies:
+  - pip=19.1=py36_0
+  - python=3.6.3=h6c0c0dc_5
+  - pip:
+    - backcall==0.1.0
+    - certifi==2019.11.28
+    - chardet==3.0.4
+    - cycler==0.10.0
+    - decorator==4.4.1
+    - idna==2.8
+    - ipdb==0.12.3
+    - ipython==7.12.0
+    - ipython-genutils==0.2.0
+    - jedi==0.16.0
+    - jsonpatch==1.25
+    - jsonpointer==2.0
+    - kiwisolver==1.1.0
+    - matplotlib==3.1.3
+    - numpy==1.18.1
+    - opencv-python==4.2.0.32
+    - pandas==1.0.1
+    - parso==0.6.1
+    - pexpect==4.8.0
+    - pickleshare==0.7.5
+    - Pillow==7.0.0
+    - prompt-toolkit==3.0.3
+    - protobuf==3.11.3
+    - ptyprocess==0.6.0
+    - Pygments==2.5.2
+    - pyparsing==2.4.6
+    - python-dateutil==2.8.1
+    - pytz==2019.3
+    - pyzmq==18.1.1
+    - requests==2.22.0
+    - scipy==1.3.3
+    - six==1.14.0
+    - tensorflow
+    - tensorboardX==2.0
+    - torch==1.4.0
+    - torchfile==0.1.0
+    - torchnet==0.0.4
+    - torchvision==0.5.0
+    - tornado==6.0.3
+    - tqdm==4.42.1
+    - traitlets==4.3.3
+    - urllib3==1.25.8
+    - visdom==0.1.8.9
+    - wcwidth==0.1.8
+    - websocket-client==0.57.0
+    - dtw==1.4.0
+    - fastdtw==0.3.4
+    - scikit-learn==0.22.2