Robustness of Contrastive Learning on Multilingual Font Style Classification Using Various Contrastive Loss Functions

Abstract

Font is a crucial design aspect, however, classifying fonts is challenging compared with that of other natural objects, as fonts differ from images. This paper presents the application of contrastive learning in font style classification. We conducted various experiments to demonstrate the robustness of contrastive image representation learning. First, we built a multilingual synthetic dataset for Chinese, English, and Korean fonts. Next, we trained the model using various contrastive loss functions, i.e., normalized temperature scaled cross-entropy loss, triplet loss, and supervised contrastive loss. We made some explicit changes to the approach of applying contrastive learning in the domain of font style classification by not applying any image augmentation. We compared the results with those of a fully supervised approach and achieved comparable results using contrastive learning with fewer annotated images and a smaller number of training epochs. In addition, we also evaluated the effect of applying different contrastive loss functions on training.

1. Introduction

The increasing use of computers and digital typography has resulted in a large number of available typefaces, providing users with many options for selecting typefaces for various purposes. With the increasing amount of digital text, there are several situations where it may be helpful to recognize the font or identify the resemblance between two fonts that are included in a collection of fonts. In these cases, it is essential to be able to distinguish between fonts that are similar to one another. For instance, if a user has an image containing text, they may want to determine the typeface used in the image. In addition, a user may wish to discover a font comparable to the font used in the image, because using the font used in the image may be prohibitively expensive or may not be an option in a specific application. However, classifying fonts or identifying similar fonts has become increasingly difficult for several reasons, such as the open-ended nature of font classes and extremely sensitive and characteristic-dependent differences among the fonts such as the different weights, strokes, slopes, etc. Additionally, the numerous online font repositories make this challenge even more challenging.

Computer vision models have traditionally been trained through supervised learning. This means that people examined the images and labeled them with various labels so that the model could learn the patterns of these labels [1,2,3,4,5,6]. For instance, a human annotator adds a class label to an image or draws bounding boxes around the objects in the images. However, as everyone who has dealt with labeling jobs knows, creating a suitable training dataset requires considerable effort as the annotation cost per image is very high. Additionally, existing generative adversarial network (GAN)-based methods applied in the font domain focus on font generation rather than classification [7,8,9]. Although these methods use classifiers to learn different styles, they primarily focus on font generation rather than style classification.

Self-supervised methods, in contrast, do not require any labels created by humans. As the name suggests, the model learns to supervise itself. In computer vision, the most typical technique to represent this self-supervision is to take various cropped images or apply different augmentations on images before feeding the modified inputs through the model. Even if the image has the same visual information but does not appear identical, Chen et al. [10] presented a contrastive learning framework based on metric learning [11,12,13]; they trained the model so that these images still contain the same visual information, i.e., the same item, even though they appear different. This results in the model learning a comparable latent representation (an output vector) for identical items in a label-free environment. Later, Khosla et al. [14] extended Chen’s work [10] in a supervised setting and used class labels to draw more positive images instead of using only one positive image. In self-supervised contrastive learning, for every anchor image, there is only one positive image (i.e., an augmented image) in the entire batch. However, as demonstrated in [10,14], it cannot be directly applied to the font domain by simply applying augmentations [15,16] on training images. As shown in Figure 1, flipping a font image horizontally or vertically results in different image contents, i.e., flipping ‘R’ horizontally results in an undefined character. Because of this reason, the application of contrastive learning in the font domain, in general, and font style classification in particular, has remained largely underexplored.

On unlabeled data, contrastive learning learns generic representations of images, which may subsequently be fine-tuned with a limited number of labeled images to achieve acceptable performance for a specified classification problem, in this case, font style classification. The generic representations are learned by maximizing the agreement between multiple transformed views of the same image, while simultaneously reducing the agreement between transformed views of different images. When the parameters of a neural network are updated using this contrastive aim, comparable view representations “attract” each other while noncorresponding view representations “repel” each other. Therefore, the model learns the similarities between two inputs instead of learning how to classify the input images. Therefore, we used Normalized Temperature-scaled Cross-Entropy Loss (NT-Xent) [17], Triplet Loss [18], and Supervised Contrastive Loss functions [14].

Another important problem in font style classification is the ability to learn new styles over time. The traditional approach of fine-tuning [19,20,21] a fully supervised classifier has limitations in terms of generalization to new classes, resulting in the need for separate classifiers for old and new classes, which can be ineffective when a few training samples [22,23] of new classes are used. However, contrastive learning, allows for continued learning of new representations and features without sacrificing the model’s ability to classify previously learned classes. By fine-tuning [24,25,26] the pretrained model in the pretext task using few-shot images [27], the model can learn new representations that encode useful features of new input data, thereby improving its ability to classify new classes in the downstream tasks. This approach leads to more efficient and effective learning, as well as more robust and adaptive models that can continue to improve over time.

In this paper, we present a study on the use of contrastive-based learning techniques to learn font style characteristics in feature spaces for Chinese, English, and Korean characters. We experimented with different contrastive loss functions and compared them with a baseline label-based fully supervised classifier. We also proposed a triplet dataset sampling technique to select hard-negative and hard-positive images for font images. We achieved high accuracy in downstream tasks and demonstrated that the proposed approach is effective for font style recognition, regardless of the base encoder used.

2. Related Works

The success of Convolutional Neural Networks (CNNs) is that they can learn from labeled data end-to-end. Deep CNNs have been successfully applied to many problems in Document Image Analysis [28,29] which include complete image classification [30,31], image preprocessing [32], character recognition [33], and classifying font classes into predefined font classes trained on labeled data [34].

2.1. Convolutional Neural Networks for Font Classification

Chris Tensmeyer et al. [34] presented a CNN-based framework to classify text lines into fonts. They used Optical Character Recognition (OCR) to recognize the text in documents. Because handling multiple text fonts is a difficult task for OCR, they used a specialist OCR to recognize text based on labeled text lines. They trained a CNN to classify small patches into font classes. However, at the inference time, they densely extracted patches and averaged the individual predictions of each patch to get the predicted font. Although it achieved a prediction accuracy of 98.6% on the King Fahd University Arabic Font Database (KAFD) for 40 typefaces in four styles and ten different sizes, the work is limited to predicting and classifying the Arabic fonts on which the model is trained on, it was also trained on labeled data. On the Classification of Latin Medieval Manuscripts (CLaMM) dataset, the edge performance of this model is significantly lower, with an accuracy rate of 86.6%. This study also demonstrates that CNNs are capable of correctly classifying fonts. By varying the characteristics of the images that were fed into CNNs, they demonstrated the features that were used for classification.

In contrast, we trained our model in a self-supervised environment. The model was trained in two stages. During the first stage, the model learns the underlying mapping in a self-supervised manner based on contrastive learning, which is referred to as Pretext training. During the second stage, termed the downstream task, the model was fine-tuned using a small number of labels to train only the classifier, whereas the base network remained frozen.

2.2. Contrastive Learning

Our study is based on the existing literature [10,14,17,18]. Chen et al., in their work, proposed a framework that focuses on self-supervised representation learning by leveraging data augmentation to train the model on learning similarities among images using N-pair loss or NT-Xent loss. Similar to the work of Chen et al. [10], Khosla et al. [14] proposed a loss function for supervised learning that incorporates label information. The loss function is designed to pull normalized embeddings from the same class close together while keeping embeddings from different classes farther apart. The key difference in this loss function is the use of many positives per anchor and many negatives rather than just a single positive, as in self-supervised contrastive learning. This loss function uses class labels to draw positive samples of the same class as the anchor. However, these studies cannot be directly extended to the font domain by simply applying augmentations to obtain positive views of the same image or using labels to train the classifier via cross-entropy loss. Inspired by triplet loss, another contrastive-based loss that uses metric learning, we used labels to guide the choice of the positive image instead of just applying the augmentation on the font image.

3. Dataset

We used printed fonts to create the training dataset. We downloaded Chinese fonts from freechinesefont.com, which has 187 Chinese fonts. We downloaded and filtered 65 fonts with all the desired 1000 commonly used Chinese characters. For English, we used Google Fonts, which has 1455 different font families. We downloaded and filtered 1000 fonts supporting all 52 capital and lower-case English characters. For Korean fonts, we downloaded font files from Naver and used 70 Korean fonts to create our dataset; ee used 500 commonly used Korean characters. We used Python code to generate training images of 224 × 224 pixel size. Therefore, we generated 65 × 1000, 1000 × 52, and 70 × 500 training images for the Chinese, English, and Korean characters, respectively.

3.1. Triplet Dataset Sampling

The triplet loss function was initially presented in [16] and has since grown in popularity as a contrastive-based learning technique. A triplet-based loss function requires three images: an anchor image, a positive image that is a member of the same class as the anchor, and a negative image. This suggests that dissimilar pairs should be some margin away from similar ones. Figure 2 visualizes the triplet loss learning objective. Initially, the authors used the Euclidian distance to measure the distance, but we used cosine similarity in our study. Unlike the contrastive loss function, where we can randomly select an image sample and apply transformations to obtain two augmented images. Font images are typically grayscale and provide limited information, such as strokes, thickness, and serifs, which differentiate one font style from another. Applying augmentation in this scenario will result in different contents of the image, as illustrated in Figure 1. For this reason, here, we chose the hard positive and hard negative of an anchor in such a way that the distance between the anchor and negative is smaller than the distance between the anchor and positive. This process is known as triplet mining.

In our test case, as shown in Figure 3, suppose we select a random image $x_i^a$, that belongs to style class m, with character class i, and we select a positive image $x_j^p$ from the same class m but with different character content, j from the anchor. However, in the case of the negative image $x_i^n$, the image should belong to another class but have the same character content i as the anchor, as font images are grayscale, and very little information is available within the training images, such as strokes, thickness, serifs, etc., which separate one font style from another. It is a crucial aspect of triplet mining to select the hard-negative image of the anchor, where the distance between the anchor and the selected negative image should be the shortest compared to the remaining images. Therefore, we chose images with identical character labels from different style classes as the negative images. We followed the same triplet selection strategy for Chinese and English data loaders.

3.2. Self-Supervised and Supervised Contrastive Data Sampling

Contrastive image representation learning aims to capture the visual similarities between images by embedding them in a common feature space. Previous works have conducted this by applying image augmentations to the original image. However, font images are different in nature compared with natural images, and applying augmentations on font images may lead to varying contents of font images. As shown in Figure 1, it can be observed that a flipped or cropped squirrel is still a squirrel, but that is not true in case of the font images; therefore applying augmentation on them will lead to a different image content. Due to this reason, as shown in Figure 4, we did not apply any image augmentations to train the model using self-supervised and supervised contrastive loss functions. Because fonts are readily accessible and have associated labels, we only utilized these labels during data preprocessing at the time of data loading; similar to triplet mining for positive images, we arbitrarily selected different images from the same class, as discussed in the previous section.

4. Training Details

This section discusses the several experiments that we conducted. We demonstrate the efficacy of using contrastive learning to learn font style characteristics in the feature space, in contrast to training a model that classifies based on labels. In addition, we demonstrate the impact of several contrastive loss functions, such as NT-Xent, triplet loss, and supervised contrastive loss.

4.1. Contrastive Learning Framework

The basic idea behind contrastive learning is to learn a representation space in which examples of the same class or category are brought closer together in the feature space; in contrast, those belonging to different classes are pushed farther apart. We achieved this by training the model on a pretext task that involves comparing the representations of different views of the same input, i.e., two-character images belonging to the same font style but with different character contents. By carrying this out, the model learns to identify the similarities and differences between the different views, which in turn allows it to learn meaningful representations of the data.

4.1.1. Pretext Training

As defined in [6], contrastive learning is a two-step training framework: the pretext and the downstream task. We primarily used ResNet-18 [3] as our base encoder with output dimensions of 100 for the embedding vector for pretext representation learning. We trained the base encoder with the Adam optimizer [25] using decay with a learning rate of 0.004 and standard batch size of 256. We also used VGG-19 [2] as a base encoder for pretext training to countercheck the effect of contrastive learning on the achieved results, whether they resulted from using contrastive learning; these results were consistent. Therefore, it is the ability of contrastive-based learning to learn better representation results with such high accuracy irrespective of the base encoder used. Another reason to use ResNet-18 or VGG-19 as our base encoders is that we used them as our baseline for the fully supervised classifier. However, it is worth noting that the choice of the base encoder depends on the specific task at hand, and it is possible that a different architecture may perform better on another task. Moreover, ResNet-18 is less likely to overfit the training data and more likely to perform well on new data; therefore it was used as the base encoder in our experiments to exploit its superior generalization capabilities and ensure a fair comparison between different contrastive loss functions and baseline classifiers.

4.1.2. Downstream Task

We removed the projection head for the downstream tasks and attached a classification layer with 65, 100, and 70 output dimensions for Chinese, English, and Korean datasets, respectively. The classifier was then trained in three distinct settings with a batch size of 256 and optimized using SGD with momentum [23] with a momentum parameter of 0.9. We trained the classifiers using the labeled dataset for 50 epochs in the first configuration for all the languages. However, we trained the classifiers using 50% and 10% ot the dataset labeled images in the second and third settings, respectively. It is important to note that only the classifiers were fine-tuned during the downstream task, and the base encoder was not trained again.

4.2. Hardware and Software

We trained the deep neural network models on a single machine comprising an Intel Core i9 Processor with 16 cores clocked at 5.2 GHz, an NVIDIA RTX 3080 Ti GPU, and 32 GB of RAM. We used PyTorch 1.11 as our deep learning framework and Python 3.6 and CUDA 10.2 on an Ubuntu 20.04 LTS operating system. To preprocess our data, we applied normalization techniques to ensure optimal performance during training. During pretext training, we used the Adam optimizer with a learning rate of 0.004 for 50 epochs and a batch size of 256. Based on the existing literature [35,36] on contrastive learning, it has been stated that there is a positive correlation between increasing batch size and the quality of embeddings generated by the model. Specifically, larger batch sizes result in more discriminative and robust embeddings, faster convergence, and more efficient training for contrastive learning models. However, due to limited computing power availability, we used a batch size of 256 throughout our experiments.

5. Results and Evaluation

The results are presented in

Table 1. Comparison of training and test accuracies of classifiers trained on NT-Xent, triplet, and supervised contrastive approaches with a fully supervised classifier used as a baseline.

Language	Method	Epochs	Labeled Dataset Size	Training Accuracy	Test Accuracy
Chinese	Supervised	500	65,000	99.90	99.38
	NT-Xent	50/50	65,000 (100%)	99.23	98.67
			32,500 (50%)	99.00	98.58
			6500 (10%)	98.34	98.13
	Triplet	50/50	65,000 (100%)	99.78	99.11
			32,500 (50%)	99.63	98.98
			6500 (10%)	98.93	98.27
	Supervised Contrastive	50/50	65,000 (100%)	99.82	99.31
			32,500 (50%)	99.53	99.14
			6500 (10%)	98.45	98.00
English	Supervised	500	52,000	100	99.95
	NT-Xent	50/50	52,000 (100%)	99.35	98.10
			26,000 (50%)	99.15	98.37
			5200 (10%)	98.90	98.45
	Triplet	50/50	52,000 (100%)	99.85	99.17
			26,000 (50%)	98.68	98.40
			5200 (10%)	99.33	99.20
	Supervised Contrastive	50/50	52,000 (100%)	99.91	99.73
			26,000 (50%)	99.53	99.07
			5200 (10%)	98.10	97.80
Korean	Supervised	500	35,000	100	100
	NT-Xent	50/50	35,000 (100%)	99.35	99.00
			17,500 (50%)	99.15	98.13
			3500 (10%)	98.90	98.05
	Triplet	50/50	35,000 (100%)	99.85	99.17
			17,500 (50%)	99.53	99.07
			3500 (10%)	99.10	98.95
	Supervised Contrastive	50/50	35,000 (100%)	100	99.76
			17,500 (50%)	99.68	99.40
			3500 (10%)	98.33	98.10

. The classifier trained on a supervised contrastive model’s downstream task showed better test accuracy. The classifier achieved the highest test accuracies of 99.76%, 99.73%, and 99.31% for Korean, English, and Chinese fonts, respectively, with supervised contrastive loss when trained using 100% of the images in the labeled dataset. Notably, the supervised contrastive classifier achieved such high accuracy in only 50 epochs, which is significantly fewer epochs than the fully supervised classifier, which was trained for 500 epochs. When trained with only 10% of the labeled images, the classifier achieved a test accuracy of 98.1%, 98%, and 97.8% for Korean, Chinese, and English font styles, respectively, in fewer training iterations than a fully supervised classifier trained on the entire dataset. These results demonstrate the effectiveness of using contrastive learning for font-style classification in scenarios where labeled data are scarce, which can achieve high accuracy with few labeled images.

Figure 5 shows the t-SNE Projections of Chinese, English, and Korean embeddings vectors models trained using NT-Xent, Triplet loss and supervised contrastive loss. In the pretext part, it shows the effect of utilizing various contrastive loss functions on training the model. For this, we randomly selected ten fonts from each language, i.e., Chinese, English, and Korean, and plotted the embeddings of 10 characters in each font. The model can group all the font images belonging to the same font style irrespective of the character content of the images. Moreover, if we closely inspect all the visualizations of embedding vectors for NT-Xent, Triplet loss, and Supervised Contrastive loss functions, we will notice that supervised contrastive loss projections are the most segregated of the three. Supervised contrastive training produces better clusters for image representations because it is trained using labels; this explains why supervised contrastive loss is superior to the other loss functions. Contrary to training the model using labels, the triplet loss function also produced well-separated projections, while the NT-Xent loss function produced less separated projections. Overall, the results suggest that the supervised contrastive loss function is superior to the other loss functions in this font classification task.

To further establish the effectiveness of training our model with a contrastive loss function for learning individual font styles, rather than relying on labels and cross-entropy loss for classification, we conducted an experiment to recommend similar font styles based on input font images. Figure 6 shows the recommended similar images for query images in Chinese, English, and Korean font styles on the seen and unseen images during training. After training the models of all languages, i.e., Chinese, English, and Korean, we removed the projection layer and passed all the images through the encoder to get the embeddings for the entire training dataset and stored them into separate pickle files. At the test time, we passed two images of each language, one seen and one unseen, and calculated the distance using the cosine similarity from embedding vectors of training images. Subsequently, we selected the three images closest to query images based on a similarity value, as shown against all query images in Figure 6. It can be observed that the model precisely recommends the same style for seen style and the closest style images for unseen styles. This indicates that the contrastive-based approach is effective in learning the semantics of individual font styles and can be used to recommend similar font styles to users.

The results presented in

Table 2. Accuracy comparison of contrastive learning and fully supervised methods for few-shot fine-tuning for new styles.

Fine-Tuning Method	N-Shots	Training Accuracy	Test Accuracy
Fully Supervised	50/500	89.0	35.0
	100/500	95.6	53.0
	250/500	98.9	74.4
Contrastive based	50/500	93.0	82.0
	100/500	98.3	89.6
	250/500	99.1	96.3

demonstrate the effectiveness of using contrastive learning for few-shot fine-tuning of a pretrained model on new font styles. Compared with the fully supervised method, the contrastive-based approach achieved higher training and test accuracies for all the numbers of shots considered. For instance, when only 50 shots were used for fine-tuning, the contrastive-based method achieved a training accuracy of 93.0% and a test accuracy of 82.0%. In contrast, the fully supervised method only achieved a training accuracy of only 89.0% and a test accuracy of only 35.0%. As the number of shots increased, the contrastive-based method continued to outperform the fully supervised method, achieving test accuracy of 96.3% with 250 shots, compared with 74.4% for the fully supervised method. These results highlight the potential of contrastive learning in improving the robustness and adaptability of font style classification models, particularly when learning new styles with limited data.

6. Discussion

In this paper, we demonstrated the application of contrastive-based learning in the domain of font style classification and effect of various contrastive-based techniques to learn font style characteristics in feature spaces for Chinese, English, and Korean characters. We achieved results comparable to those of a label-based fully supervised approach to font style classification. Experiments revealed that the pretext training base architecture had no noticeable impact on training efficacy. By combining our findings with metric-based learning on visual representations, we also demonstrated the effectiveness of contrastive learning in a font style recommender system to identify related font styles or fonts with the closest pixel dimensions.

This study opens up possibilities for future work in the font domain, where unsupervised contrastive learning can be used to learn generic representations of fonts and be fine-tuned with a limited number of labeled images to achieve acceptable performance in font-based problems. Our work can also be extended to other scripts, such as Arabic and Devanagari. Overall, our study demonstrates that self-supervised contrastive learning can be a powerful tool for font style recognition, particularly when labeled datasets are limited or expensive to create.