Data Matrix Based Low Cost Autonomous Detection of Medicine Packages

Abstract

Counterfeit medicine is still a crucial problem for healthcare systems, having a huge impact in worldwide health and economy. Medicine packages can be traced from the moment of their production until they are delivered to the costumers through the use of Data Matrix codes, unique identifiers that can validate their authenticity. Currently, many practitioners at hospital pharmacies have to manually scan such codes one by one, a very repetitive and burdensome task. In this paper, a system which can simultaneously scan multiple Data Matrix codes and autonomously introduce them into an authentication database is proposed for the Hospital Pharmacy of the Centro Hospitalar de Vila Nova de Gaia/Espinho, E.P.E. Relevant features are its low cost and its seamless integration in their infrastructure. The results of the experiments were encouraging, and with upgrades such as real-time feedback of the code’s validation and increased robustness of the hardware system, it is expected that the system can be used as a real support to the pharmacists.

1. Introduction

Healthcare systems continuously seek for innovative treatments to overcome the unmet health needs as well as the unforeseeable issues, such as COVID-19, that appear without prediction. Medicines are becoming more crucial in the global health coverage and, due to the high complexity and cost of their development and manufacturing, the pharmaceutical price is increasing, which is reflected in the life quality of the end user [1,2]. According to the World Health Organization (WHO), the counterfeit medicines are problematic, which is evidenced by their entry in the pharmaceutical supply chain, has been negatively affecting the worldwide health. In low and middle income countries, in the time period between 2007 and 2016, approximately 10.5% of the pharmaceutical products were substandard or falsified [3]. Worldwide, between 2013 and 2017, 42% of all the reported counterfeit medicines were from Africa whilst 21% were from America and 21% from Europe [4]. The accounting firm PwC states that 1–30% of drugs in circulation are fake and a high number of people die annually from toxic counterfeit pharmaceuticals [5]. Actually, in 2015, a study related with substandard drugs to fight the malaria in sub-Saharan Africa estimated more than 122,000 deaths per year of children under the age of five [6]. The profitable business associated with a large number of stakeholders belonging to the distribution network (illustrated in Figure 1) increases the difficulty of controlling the market of fraudulently mislabeled medicines [7].

The legislation applied in the Directive 2011/62/EU of the European Parliament and of the Council, and by the Commission Delegated Regulation (EU) 2016/161, looks forward to eradicate the counterfeit medicines by implementing a unique identifier which guarantees medication packaging authenticity. The regulation is set for the usage of a two-dimensional (2D) barcode—a machine-readable Data Matrix, which follows the format defined by the International Organization for Standardisation/International Electrotechnical Commission standard (‘ISO/IEC’) 16022:2006 [8,9,10]. The National Medicine Verification Organization is the entity which allows the trace of medicine from the production to the patient, ensuring its integrity [11], and holds the national repository where the data of the medicine packages circulating in the Portuguese territory resides. The national stakeholders will establish a connection with the repository to perform the verification and deactivation operations of the unique identifier of the Data Matrix. To evaluate the authenticity of the medicine, a comparison is performed between the information in the unique identifier registered by the entity responsible for uploading data to the European Medicines Verification System and the one in the medicine package [12]. This procedure is required in Portugal since the 9th of February 2019, according to the Decree-Law n.º 128/2013 [13].

Regarding the supply chain, at the pharmacy level, the medicine products are commonly packaged in three different levels: tertiary for logistics purposes (from the wholesaler to the pharmacy), secondary for inventory (performed within the pharmacy) and primary level, where the operator has direct contact with the product (e.g., vials or pills) as illustrated in Figure 2. The medicines can be traced through a bar-code which identifies the product when electronically scanned [14]. To ensure the interoperability of the pharmaceutical products verification repositories, it is mandatory to adopt a harmonized coding approach. The GS1 (Global Standards One) Data Matrix is embraced as the recommended standard, consisting in a unique identifier (product code—Global Trade Item Number (GTIN), serial number, batch number, expire date and, if required, the national registration number) and an anti-tampering device [15].

There are actually several mathematical approaches to detect matrix-type markers, such as [17,18]. Some of the approaches are based on deep learning theory to solve the problem of barcode detection [19]. These algorithms are useful to detect the required patterns in image. The description of the markers generation can be found in [20].

Currently, some commercial solutions can achieve pharmaceutical product traceability through the Data Matrix reading. Funcode technology developed a QR Code/Data Matrix (2D Barcode) reader whose features comprise the scanning, with a Raspberry Pi, of tiny, moving and multiple barcodes using multiple cameras. The algorithm is robust to conditions such as non-uniform lighting or blurred and distorted codes. Its main applications include industrial automation, pharmaceutical products and consumer goods labels, intelligent logistics center, package tracking, automated tracking of product, among others [21]. Scandit presents a mobile barcode scanning application robust to different 1D or 2D symbologies such as Data Matrix. MatrixScan allows to locate, track and decode multiple barcodes, while displaying feedback in the device’s screen as an Augmented Reality-overlay, to identify which have already been scanned. The software is able to handle barcodes in challenging conditions such as glare and metal reflections, low light and shadows, damaged, torn, and blurry barcodes, or skewed angles. Some applications in the healthcare sector emcompass shelf management, search-and-find packages, patient verification and medication tracking [22]. In the pharmaceutical/medical industry, Cognex resorts to image-based fixed-mount barcode readers (including Data Matrix symbology) to ensure the simultaneous track-and-trace of multiple medical devices on high-speed production lines, multiples sides, and even extreme angles. The algorithm is able to efficiently scan blurred, damaged or low contrast barcodes. Additional applications include the automation of inspection and distribution of primary (e.g., vials or pre-filled syringes) and secondary (e.g., vaccine boxes) packages. Cognex supplies guidance to other markets such as logistics, life sciences and aerospace [23].

To the best of the authors’ knowledge, there is still not a established solution to be applied at the national hospital pharmacies. In this work, a hospital pharmacy scenario is focused, more precisely the pharmacy department of the Centro Hospitalar de Vila Nova de Gaia/Espinho, E.P.E., Portugal, where, on average, more than 27,000 medicine packages are handled per month. Due to the lack of human resources, the action of having each medicine be manually validated at the secondary level by an operator by means of a scanner is burdensome, time-consuming and not the foremost task for the operator. To prioritize some medicine packages, criteria such as their medical significance or cost might be applied. Moreover, the aforementioned action is monotonous, tedious, frequent and highly repetitive, due to the recurrent lifting and transferring of packages, and repeated hand/wrist motions to place the scan in the correct position and press the button to detect the Data Matrix code [24]. The addressed cases are common physical work-related risk factors that can cause upper limb musculoskeletal disorders to the operator [25], and therefore should be avoided. To include all the information of medicine products in a pharmaceutical system, hence enabling their traceability, as well as to contribute to the operator’s ergonomy, a low cost solution for automating the simultaneous scanning of multiple medicine packages based on Data Matrix codes is proposed in this work. One important requirement was that the system should be easily integrated in the infrastructure of the Hospital Pharmacy.

The remainder of this paper is organized as follows. Section 2 presents the proposed Methodology addressing the hardware, the system architecture and the Software employed. Section 3 stresses the experimental results performed in hospital pharmacy facilities. Finally, Section 4 outlines the work with conclusions, comprising functionalities that may be interesting to incorporate in the system in the near future.

2. Proposed Methodology

Nowadays, the introduction of medicine packages into the authentication database is performed through the manual scan of their unique Data matrix, one by one by the user, becoming a repetitive and time-consuming process. To improve the users’ ergonomy as well as to redirect them to other tasks, the proposed methodology addresses the simultaneous and autonomous acquisition of multiple codes. The present system is initiated by the user that presses a button and, after, the remaining process will automatically be executed. An image of the working area is acquired and the system can, autonomously, identify, extract the information of all the Data matrix codes contained in that area, and send it to the verification system database.

2.1. System Architecture

The structure of the developed system is presented in Figure 3a,b (frontal and bottom-up views, respectively), comprising a camera to acquire the images as well as a lighting system, in order to avoid being constrained by shadows or lack of brightness.

The acquired image can contain several Data matrix codes restricted by the size of the work area, which is 200 mm × 200 mm. The structure was assembled using MakerBeam small aluminum profiles and additive manufacturing (3D printing) parts. The prototype is a ready-to-go system since it has all the components installed and is also equipped with a power supply. It only requires to be connected by USB (Universal Serial Bus) to the computer (in this case study, from the hospital pharmacy) and the electrical grid. An illustrative image acquired by the camera is presented in Figure 4.

The system architecture schematic is presented in Figure 5. The proposed system is connected to the computer and it will be identified as a keyboard, the same process as the Data Matrix reader used in the current implementation of the presented case study. The Pro Micro Arduino board, composed of an ATmega32U4, is configured to be recognized as a keyboard by the MVO (Medicines Verification System) verification computer. The identified Data Matrix codes are processed by the Raspberry Pi library $libdmtx$ (https://github.com/dmtx/libdmtx, accessed on 25 September 2022) and transferred one by one through a serial port to the Arduino that further sends it to the computer (using the defined protocol). The used integrated development environment was Lazarus, into the raspbian operating system. Moreover, the Arduino board is also used to perform an Human-Machine Interface (HMI) to the user, by one RGB led, one push-button, and also to control the lighting system (with 8+8 Watt LED panels). The RGB led allows the user to infer the state of the system from the LED color, while the push-button is used to start the acquisition process of a new image containing a Data Matrix, both presented in Figure 6. Regarding the feedback from the MVO to the computer, the connection has not been implemented yet.

The serial port communication between Arduino and Raspberry uses the channel library developed by the authors that encapsulate several data that will be decoded at the target. The image is captured with a 12 Megapixels camera connected to the Raspberry pi camera BUS attached with an 8–50 mm C-Mount Zoom Lens. The illumination system allows having a lens aperture that enables a depth of field capable of detecting Data Matrix codes in boxes at a distance of 15–25 cm. The HMI is also composed of a monitor that gives information to the user about the Data Matrix detection (Data Matrix codes are bordered by green squares) and also the total number of decoded codes, as illustrated in Figure 7.

2.2. Data Matrix

Data Matrix codes are two-dimensional symbols that hold a dense pattern of data with built-in error correction. Special scanners were designed to read them; however, high-resolution cameras can be used to acquire multiple Data Matrix images simultaneously. If the resolution is enough to identify each point of the Data Matrix, a library can be applied to decode its data. In the developed system, the $libdmtx$ library is used to scan the Data Matrix codes in the medicines’ boxes. $libdmtx$ is an open-source software for reading and writing Data Matrix barcodes on Linux, Unix, OS X, Windows, and mobile devices. According to [26], several commercial scanners can be used. The software developer (for the hospital management) published the Data Formatting protocol and settings that should be used for the configuration of each matrix scanner (Datalogic Quickscan QD2403, Zebra DS2278, Honeywell HH660, 1450G, and 7580G). Several validation tests were executed with different medicine boxes. For example, Figure 8a presents the acquired image and Figure 8b the respective $libdmtx$ outputs that allowed to verify its operation (with the proper lighting system). Figure 8d exhibits the Data Matrix detected by the library, which are enclosed in green squares, and Figure 8e shows the extracted codes according to the defined protocol where each line corresponds to the decoded matrix code, as it can be read by a matrix code scanner, i.e., the output of the $libdmtx$ library.

3. Experiments and Results

The developed prototype was installed and tested at the Hospital Pharmacy of the Centro Hospitalar de Vila Nova de Gaia/Espinho, E.P.E., by technicians which perform the medicine packages registration. The proposed prototype is a stand-alone system that can be placed alongside with the conventional scanner, replacing it. A space of 300 mm × 300 mm in a table is enough to accommodate the developed reader (with a height of 400 mm). It requires to be supplied by 110 V or 230 V and it owns all the required hardware and software to introduce by USB the decodes matrix codes present on the boxes into the hospital system. The developed prototype was connected, by a USB link, to the same computer to which the original Data Matrix scanner is connected. The computer is used to send the verification codes to the central database. Instead of using the original Data Matrix scanner, the proposed system was assessed by reading the codes, decoding, and introducing them in the database by USB. Figure 9 shows the prototype installed and tested at the hospital pharmaceutical facilities. The right monitor shows the developed application and the Data Matrix identified. The left one shows the hospital computer receiving the codes to validate.

The system is launched when the user presses the button and the lighting system is turned on for two seconds to stabilize the camera and capture the image. The LED turns red and the $libdmtx$ is used to extract and decode each Data Matrix. Each code is introduced into the hospital computer (as a keyboard) using the same settings and protocol as the original scanner was programmed. The acquired image and its Data Matrix codes identified and delimited by green squares, allow the user to visually validate the task.

Preliminary tests were executed in order to analyze the time and the robustness of the Data Matrix code detection system. According to the commonly performed process, a technician takes between half a second and one second to scan each medicine box. In contrast, the proposed system takes about two seconds to stabilize the image and an average of one second per box. Meanwhile, while the scanning is being performed, the technician can be redirected to do another task. The proposed system has the advantage to scan multiple codes at the same time. To assess the system robustness, the scanning process was performed on boxes covered by reflector plastic as well as rounded bottles, having achieved successful results, as demonstrated by Figure 10.

The integration with the hospital software was tested and validated. Instead of using the original scanner to shoot Data Matrix codes one by one, the proposed system acquires one image and introduces several Data matrix codes sequentially without the intervention of the technician. The obtained feedback from four technicians (from different shifts) that used the prototype was very positive since they mentioned that the system allows them to do other tasks while introducing the codes increasing the productivity. Moreover, they pointed out some future work possibilities, such as using a conveyor belt to move the boxes in front of the camera in a FIFO (First In, First Out) cue type. Moreover, increasing the detection area was also pointed. As a statistical result, detection robustness was done with real medicine boxes. In total, about two hundred medicine boxes were placed in front of the camera (within 15 images) and 90% of Data Matrix codes were correctly identified, using the proposed system. The remaining codes had slight defects, such as being torn (as illustrated in Figure 11) or their color being too grey.

In order to evaluate the illumination interference, a test was performed with different light conditions. The first test was executed turning the embedded illumination off. In this case (with a natural illuminance of 150 Lux), the extracting capacity was decreased to 18%, as presented in Figure 12 where from eleven matrix codes, only two of them were extracted.

More tests were executed to evaluate the interference of the lighting.

Table 1. Extracting result ratio for different illumination conditions.

	Natural Illuminance (Lux)	Measured Illuminance (Lux)	Extracting Ratio Result (%)
LEDS off	5	5	0 (0 of 11)
LEDS off	90	90	0 (0 of 11)
LEDS off	150	150	18 (2 of 11)
LEDS on	90	1985	100 (11 of 11)
LEDS on	440	2320	100 (11 of 11)
midrule LEDS on	1040	3200	100 (11 of 11)
LEDS on (1 panel)	0	980	100 (11 of 11)
LEDS on (2 panels)	0	1880	100 (11 of 11)

summarizes the obtained ratios for data matrix extractions with different lightings. The first three tests were evaluated with both LEDS panels off for an Illuminance of 5, 90 and 150 Lux. The proposed system is very dependent on the illumination conditions. As it can be observed, one LED panel is enough but brings some reflexes that are reduced with two LED panels assembled as previously addressed. In these cases (one and two LED panels on), the ratio of detection for six image samples were 100% even when changing the external illumination from 0 to 1000 Lux.

In order to improve the work, there are two main points that might be pointed out (to solve a limitation of the prototype), namely increasing the size of the acquired image, and a more powerful computational performance could be included to speed up the detection of matrix codes.

4. Conclusions and Future Work

The counterfeit medicine problem is still negatively affecting worldwide health. The Data Matrix code, a unique identifier that guarantees medication packaging authenticity, allows to trace a medicine from its production until its delivery to the patient. In the hospital pharmaceutical industry, at the secondary level, for the inventory task, the scanning of the medicine packages to introduce them into the management system is performed manually by the user through a Data Matrix reader. This paper proposes a low cost system to control medicine packages, acquiring a set of medicine boxes simultaneously and autonomously introducing them into the authentication database. A proof of concept was performed by installing the proposed system in a hospital pharmacy and testing it with real medicine. The achieved results were very encouraging, and it is expected that, with the addition of extra functionalities, the system can be used as a real support for the pharmacist work. The proposed prototype, works as a standalone system, that can be added to any validation system by connecting an USB port. It owns an embedded illumination system that eliminated the exterior lighting conditions. It could reach 90% of extraction ratio for real conditions and real medicines. Nevertheless, there are some limitations such as the small acquisition area and the processing time for extraction of the matrix codes.

As future work, feedback from the MVO database will be received through an ethernet port and the system will project the codes accordingly. For this purpose, a projection mapping will be used to assist the technician in the authentication operation. This task will be performed as soon as the information from the MVO is received. To improve the system flexibility and to avoid the user being forced to put the packages face with code upside, new cameras in distinct orientations will be integrated as well as the size of the work area will be increased. Additionally, including a conveyor belt in order to increase the amount of medicine packages which are autonomously scanned, and a robotic manipulator to pick the counterfeit medicine boxes, are improvements to consider. Finally, in order to improve the detection ratio, a machine learning based vision system could be added to detect matrix codes. Then, it would be possible to map the matrix codes and detect the missed ones improving their quality (e.g., rotation, histogram equalization, among others) and sent them again to the $libdmtx$ library. In this way, it would be possible to improve the detection and extraction.