How Do Users Respond to Mass Vaccination Centers? A Cross-Sectional Study Using Natural Language Processing on Online Reviews to Explore User Experience and Satisfaction with COVID-19 Vaccination Centers

Abstract

To reach large groups of vaccine recipients, several high-income countries introduced mass vaccination centers for COVID-19. Understanding user experiences of these novel structures can help optimize their design and increase patient satisfaction and vaccine uptake. This study drew on user online reviews of vaccination centers to assess user experience and identify its key determinants over time, by sentiment, and by interaction. Machine learning methods were used to analyze Google reviews of six COVID-19 mass vaccination centers in Berlin from December 2020 to December 2021. 3647 user online reviews were included in the analysis. Of these, 89% (3261/3647) were positive according to user rating (four to five of five stars). A total of 85% (2740/3647) of all reviews contained text. Topic modeling of the reviews containing text identified five optimally latent topics, and keyword extraction identified 47 salient keywords. The most important themes were organization, friendliness/responsiveness, and patient flow/wait time. Key interactions for users of vaccination centers included waiting, scheduling, transit, and the vaccination itself. Keywords connected to scheduling and efficiency, such as “appointment” and “wait”, were most prominent in negative reviews. Over time, the average rating score decreased from 4.7 to 4.1, and waiting and duration became more salient keywords. Overall, mass vaccination centers appear to be positively perceived, yet users became more critical over the one-year period of the pandemic vaccination campaign observed. The study shows that online reviews can provide real-time insights into newly set-up infrastructures, and policymakers should consider their use to monitor the population’s response over time.

1. Introduction

In the wake of large-scale COVID-19 vaccination campaigns starting in December 2020, numerous countries across the globe opted to use mass vaccination centers as a key pillar of their vaccine roll-outs [1]. A vaccination center is “a location, normally used for non-healthcare activities, set up for high-volume and high-speed vaccinations during infectious disease emergencies” [2]. In Germany, as in many other high-income countries, mass vaccination centers were a novel construct, as vaccinations are usually administered in outpatient practices.

Understanding how users experience these novel structures can provide valuable insights. Positive user experience of healthcare is not just an inherent goal by itself [3], well-designed vaccination services can also contribute to higher vaccine uptake [4]: they can increase perceived acceptability and lower adherence barriers to vaccines [5,6], satisfied users appear more likely to comply with the vaccination schedule (i.e., get a follow-up shot) [7], and recounted personal experiences could serve as a cue to action for others to get vaccinated, according to the health belief model [8,9]. In the context of a pandemic, with the objective of getting large proportions of the population vaccinated [10], positive user experience can be a crucial driver for a successful mass vaccination campaign.

Patient online expressions (e.g., via tweets or review platforms such as Google) are a readily available resource for user feedback. Their availability has grown exponentially in the past decade [11] and they have proven to provide valuable insights into patient experience and satisfaction [12,13,14,15,16,17,18,19] and also towards vaccines and vaccinations, in the context of measles or more recently COVID-19 [20,21,22,23,24,25,26,27,28,29]. Vaccination experiences or temporary pandemic response structures within or outside of the context of COVID-19, however, have not yet been studied using patient online expressions. Overall, studies on user experiences of mass vaccination centers are still limited.

Using Google Maps online reviews, we therefore aimed to understand user experience of vaccination centers in Germany. We further aimed to identify key determinants of their experience, which may assist center operators, healthcare providers, and policymakers in optimizing vaccination center design and vaccination processes. We found that the vaccination centers included in our study were perceived positively overall, yet users became more critical as the vaccination campaigns lasted, in particular with regards to efficiency. Policymakers should carefully monitor user experience and its determinants over time. Online reviews can help provide real-time insights for this.

2. Materials and Methods

Online reviews of vaccination centers in Berlin, Germany, were analyzed using natural language processing (NLP). Review texts were analyzed overall, by sentiment, and by time period. A framework for vaccination user experience was developed and key interactions for positive user experience within the framework were identified. The study followed the STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) [30] reporting guideline for cross-sectional studies. It was deemed exempt from informed consent as it used publicly available data.

2.1. Data Collection

The German capital Berlin was chosen as a study site since online reviews were available for all its six vaccination centers. This provided a large sample and allowed the team to study an entire federal state. Furthermore, Berlin hosts a diverse population, promising heterogeneous reviewers. Vaccination centers accounted for approximately 40% of all COVID vaccinations administered in Berlin between December 2020 and December 2021 [31]. The Appendix A ([32,33,34,35,36,37,38,39,40,41]) provides more details .

Google Maps reviews for all six COVID vaccination centers were retrieved for 12 consecutive months, from 27 December 2020 (start of the vaccination campaign) to 26 December 2021. A dataset containing 3797 online reviews was extracted from publicly available Google Maps reviews sites (Appendix B). Online reviews are provided voluntarily and contain, at a minimum, a star rating ranging from one star to five stars, where one star is the lowest score and five stars is the highest score. In addition, reviews can contain a free text comment. Among other items, the dataset extracted included the following: a review rating ranging from one star to five stars; a timestamp detailing date and time when the review was submitted; a review text. Reviews dating before the opening and after the closing of a site were excluded. For one site (a festival arena) that did not have a dedicated review sub-site, reviews that clearly related to the event location rather than to the vaccination center were removed. Non-English reviews were translated to English using Google Translate [42].

2.2. Analysis

2.2.1. Key Themes

An analysis of key themes and sentiments was conducted in Python 3.9.9 [43] on all reviews containing text using validated machine learning algorithms: topic modeling with Latent Dirichlet Allocation (LDA) and keyword extraction with Bidirectional Encoder Representations from Transformers (BERT).

LDA is a probabilistic topic modeling algorithm. It extracts word clusters called “topics” that are distributed across texts and connected as latent constructs based on their relative frequency and location [44]. Review texts were pre-processed for LDA using the NLTK Python library [45]. The list of stop words, i.e., words to exclude, was amended to also exclude words such as “COVID” and “vaccine” that did not add meaning (Figure S1). LDA was then run on the entire text using the Gensim Python library [46]. To refine the model parameters and ensure the topics are interpretable and meaningful, the model was evaluated based on topic coherence (C_v measure) and the intertopic distance map. The C_v coherence score was calculated for 2 to 15 topics to determine the optimal number of topics. The Dirichlet hyperparameters α and η were set to default [46]. For topic labeling, each topic’s salient terms were examined based on term frequency (λ = 1.0) and relevance (λ = 0.6) as suggested by Sievert and Shirley [47], and sample reviews were inspected. The weight parameter λ ranged from 0 to 1. A smaller λ indicates rare but exclusive terms for the topic, whereas a larger λ features frequent but not necessarily exclusive words [25].

For validation and a more granular look into the reviews overall and sub-groups, automated keyword extraction was used. Keyword extraction identifies single salient words based on semantics. The recent NLP model keyBERT [48,49,50] was used, which can identify more than one keyword in each text. For example, in “Super organized and very friendly staff.”, keyBERT identified “staff”, “organized”, and “friendly” as keywords. Text pre-processing was not needed, but as with topic modeling, additional stop words were added. Synonyms were then aggregated through lemmatization and manual review (e.g., “employees”, “staff”, and “helper” were grouped to “staff”; “appointment” and “appointments” to “appointment”; Table S1). To ensure the relevance of the results, only words with high frequency were considered in the analysis. Keywords identified in at least 1% (28/2740) of reviews analyzed were then grouped and sorted to a user experience framework (see Section 2.2.2). Sub-set analyses were conducted by sentiment and time period. All extractions were validated by one human investigator on a random sub-sample of 30 reviews.

2.2.2. Key Interactions

A vaccination center user experience framework was developed to summarize the findings and draw actionable conclusions (Appendix C). User experience was defined as the combination of a vaccine recipient’s user journey through the vaccination center and the key determinants of their experience at the vaccination center. User journey phases were defined according to the German Ministry of Health’s handbook for vaccination centers [37]. Key determinants of user experience, such as accessibility, hygiene, and duration, were identified through policy documents on vaccination center planning [37] and case studies of vaccination centers [38,39], as well as existing Patient Reported Experience Measures (PREMs) frameworks [40,41]. These determinants were clustered into 3 themes: staff, process/management, and location. Cost was omitted from the framework, as COVID-19 vaccinations were freely available in Germany.

2.2.3. Timeline

To understand user responses during different phases of the vaccination campaign, seven time periods were defined based on population eligibility and the setup of alternative vaccination structures over time (Figure 1). In the beginning, vaccinations were solely offered in vaccination centers based on appointments and priority groups. As the campaign progressed, eligible groups were gradually expanded, and other structures (e.g., outpatient practices) could offer vaccines. From June 2021 (period 4) onwards, the prioritization of certain population groups was abandoned, and vaccinations were offered to anyone above the age of 12 in vaccination centers. From July 2021 (period 5) onwards, walk-ins without appointment were introduced [36], and the first vaccination centers closed. September 2021 (period 6) marked the start of the booster campaign for elderly, and November 2021 (period 7) marked the start of the booster campaign for all.

2.2.4. Satisfaction

To understand which themes elicit satisfaction or dissatisfaction, keyword analysis was conducted separately on positive and negative reviews. Users directly expressed their sentiment through review ratings ranging from one to five stars. To ensure that the review rating corresponded to the review content, a Pearson correlation coefficient between the sentiment of the review text and the star rating was calculated using the NLTK and SciPy Python libraries [45,51]. Since the Pearson coefficient showed a statistically significant correlation, the rating score was used for further analysis. A review was deemed positive if it received four or five stars, neutral if it received three stars, and negative if it received one or two stars. Users leaving a positive review were assumed to be satisfied, and those leaving a negative review were considered dissatisfied [14].

3. Results

After exclusion, the final dataset contained 3647 reviews. Of these, 907 (25%) solely contained a rating, and 2740 (75%) contained rating and text (Figure 2A). The average length of a review text was 33 words in English, with a minimum of 1 and a maximum of 544 words.

3.1. Key Themes

Topic modeling with LDA identified 5 key topics. Keyword extraction identified 47 keywords.

For topic modeling, five topics received the highest coherence score (0.486). The five topics (Figure 3, Table S2) covered the vaccination process, location, and staff. Topics 1 and 4 contained comparatively narrow terms. Topics 2, 3, and 5 were more convoluted. Topic 1 was related to scheduling and wait time, and Topic 4 was related to staff friendliness and overall organization. Topic 2 contained terms related to efficiency and duration (“quick”, “start”, and “finish”), vaccination effects, and personal protective equipment (“mask” and “ffp”). An inspection showed that some users reported the effects they experienced shortly after their vaccination, e.g., “[...] I tolerated the vaccine without side effects, only the shoulder hurt a little, like sore muscles. [...]”. Reviews mentioning masks discussed subjects ranging from complaints about security staff not wearing masks to the provision of masks at the centers. Topic 3 broadly dealt with the vaccination center location, specifically its accessibility (“parking”, “welcome”, and “entrance”), uniqueness (“effort”, “event”, and “concert”), and the intuitiveness of the process exemplified through the term “uncomplicated”. Several users commented on the vaccination being a special event, but also on the fact that a location housed events before becoming a vaccination center. Topic 5 again encompassed site access (“bus”, “shuttle”, and “entrance”), but also registration and documentation (“certificate”, “code”, and “digital”). Based on the marginal topic distribution (Figure 3A), the two most salient topics were 1 and 4. Topic 5, regarding accessibility and documentation, was also quite prominent.

Keyword extraction identified 47 keywords overall that featured in at least 1% (28/2740) of reviews containing text. The top 30 keywords by frequency are displayed in Figure 4. Four terms were featured in more than 10% (274/2740) of reviews: “organization” (45%, 1227/2740), “friendliness” (34%, 940/2740), “staff” (34%, 940/2740), and “appointment” (17%, 455/2740). Overall, most terms were related to staff and process, few to location, and some to undefined dimensions, such as “people” (5%, 132/2740) and “user” (2%, 48/2740) (Figure S2).

3.2. Key Interactions

The 47 keywords were mapped against the user experience framework. Overall, more keywords were related to key determinants of the experience rather than moments in the vaccination center user journey (Figure 5). All three key determinant themes (staff, process/management, and location) were mentioned in reviews. Terms related to overall organization (52%, 1426/2740) and friendliness/responsiveness (49%, 1331/2740) were more frequent than those related to patient flow/wait time (25%, 688/2740), duration/efficiency (18%, 497/2740), and accessibility (12%, 321/2740). Intuitiveness, information/education, and hygiene/infection prevention and control were seldom or not featured.

There were five key moments in the user journey that were frequently mentioned: scheduling, transit, arrival, waiting, and the vaccination itself. Most keywords were related to the visit itself, while only a few were concerning experiences outside of the vaccination center pre- or post-visit. Aside from scheduling, other administrative processes like registration and documentation were not salient. Neither were adverse events or continuous monitoring of vaccination effects.

3.3. Timeline

The number of reviews varied over time (Figure 2A). In period 2 (when more—also younger—individuals gained access to vaccinations), the number of reviews steeply increased. There was a noticeable drop in reviews after period 5. The top keywords by time period largely overlapped with overall keywords (Figure 2C). Organization and staff were salient in all periods. The three terms “organization”, “staff”, and “friendliness” consistently dominated the top three keywords across all periods until period 7. Other words related to friendliness, e.g., “helpful”, “nice”, and “thanks”, were salient in all periods but became less prevalent over time. Terms related to waiting and duration, such as “wait”, “quick”, “queue”, and particularly the term “appointment”, became gradually more salient from period 2 onwards. “Taxi”, “accompany”, and “support” are only featured among the top 10 keywords in period 1.

3.4. Satisfaction

Overall, 3261 (89%) of the 3647 reviews received a positive rating, 250 (6%) received a negative rating, and 136 (4%) received a neutral rating (Figure 2B). The rating distribution was strongly unimodal, with 2886 (79%) five-star ratings.

Over time, ratings stayed relatively constant with little variance until period 3. In period 4, average ratings started to decrease. Similar to the overall keywords, the keywords identified in positive (N = 2426) and negative (N = 223) reviews containing text were mostly linked to organization, staff, and appointments (Figure 4B).

Positive reviews focused on organization, staff, friendliness, and competence. Negative reviews featured several keywords related to waiting and duration. Specific staff were salient in negative reviews, namely “doctor” (9%, 20/223) and “security” (7%, 15/223) (Figure 3B). An inspection of reviews illustrates this: “outside the security is unfriendly and self-absorbed, inside everyone is very nice. [...]”. The military, which supported center operations, was not featured in negative reviews but appeared in the positive reviews. Furthermore, negative reviews contained several keywords not salient in the reviews overall. These were linked to the weather (“rain” (2%, 5/223), “cold” (5%, 12/223), and “outside” (2%, 4/223)), accessibility (“parking” (5%, 12/223)), general “chaos” (3%, 7/223) and “german” (3%, 7/223). For example, one review reads: “[...] I stood outside in the cold for 1 h at 2 degrees and wind. [...]”.

4. Discussion

4.1. Principal Results

This study had three principal findings. First, the overall reception of vaccination centers was strongly positive. Second, the most important themes identified in the online reviews were wait time, overall organization, and friendliness, while the most important moments in the user journey were scheduling, transit, arrival, waiting, and vaccination. Third, efficiency/duration and wait time/patient flow were leading drivers for dissatisfaction, and their prevalence increased over the vaccination campaign as satisfaction overall decreased.

Many online review-based studies in healthcare [11,24,28,52] and consumer research [53,54,55] observed a positive rating tendency. The vaccination center online reviews were extremely positively skewed compared to those of studies in other settings however [11]. Two survey-based studies of COVID-19 vaccination experience in Saudi Arabia and Saxony (Germany) [56,57] found similarly high overall satisfaction scores of above 90% for vaccination centers. Aside from actual experiences or self-selection, this positive skew could be linked to the gratitude and hope associated with the COVID vaccination [58], specifically in the beginning of the vaccination campaign. These feelings could also explain the satisfaction decrease over time; at the start, COVID vaccines were rare, and users were likely more appreciative of the opportunity to get vaccinated, especially early target populations that were more exposed and vulnerable to the virus. As the campaign lasted, the vaccine became more of a commodity. User expectations with regards to the vaccination process probably increased, while recipients became less eager and dependent on vaccines. This seems to fit with the timeline analysis: once the prioritization was abandoned in period 4 and vaccines were offered to anyone aged 12 and above, the rating scores dropped more noticeably (Figure 2B). Additionally, after period 5, only two centers stayed open, and, in period 7, the time between initial and booster vaccination was suddenly decreased, leading to “chaos” [59] around Christmas 2021. Both put pressure on scheduling and impacted accessibility.

Two studies previously evaluated English language tweets on COVID-19 vaccination after the vaccine rollout, irrespective of geographical location, and also found that scheduling and appointments were prevalent themes [25,28]. A systematic review found that important themes in patient online reviews generally included physicians’ demeanor, staff friendliness, time spent with patients, ease of scheduling, wait time, and cost [11]. While staff interaction and appointment management were reflected in the vaccination center reviews, costs were irrelevant in the context of our study. With regards to time spent with patients, vaccination center users seemed more focused on efficiency (i.e., little time spent at the vaccination site) and the competence and friendliness of staff rather than ample exposure time to doctors. A survey-based study comparing vaccination centers and GP offices in Saxony, a more rural German region, found that wait times were lower at mass vaccination centers than at GP offices [56].

Topic modeling provided five topics with broader terms that could not always be distinctly sorted to a single determinant, whereas keyword extraction provided keywords more focused on distinct determinants. Both methods identified similar principal themes overall, which increases confidence in the results. Furthermore, the spheres indicating the topics in the intertopic distance map for LDA topic modeling (Figure 3) do not overlap, which can be understood as a characteristic of good quality. Yet, it is noteworthy that keyword extraction did not determine terms related to side effects or documentation (e.g., “certificate” and “code”) as salient. They were featured, however, in Topics 2 and 5 of topic modeling. “Side effects” were also a prevalent topic identified in analyses of tweets related to COVID-19 vaccine rollout [25,28].

Studies cite a range of critical themes in negative online reviews: discordant expectations (education, support, and promises) and sub-optimal communication and quality of care (management, organization, staff, and equipment) [12,60,61]. These were only partly reflected in our findings. As vaccination centers are single-purpose facilities under special circumstances, user priorities appear different from those of traditional healthcare delivery. Huangfu and colleagues [28] also found that appointments played a major role in negative tweets about COVID-19 vaccine rollout. In our study, scheduling, wait time, patient flow, and duration became more salient as the vaccination campaign progressed. In 2020, Volpp and colleagues highlighted the need to reduce these “hassle factors” as crucial for driving COVID vaccine uptake [10]. Dysfunctional scheduling in particular has been a common criticism from the beginning of the German vaccine roll-out [62]. Scheduling, which influences wait time, and choice of location, which influences accessibility, lie outside of the vaccination site operators’ control, however. Both are managed by the public administration. To address these critical user experience themes, cross-stakeholder collaboration is necessary.

4.2. Limitations

Our study has three key limitations. First, it is important to note that online reviews are voluntary and predominantly anonymous. Online reviews hence suffer from self-selection bias, rely on user sincerity and certain groups appear more prone to providing them, i.e., this limits representativeness. A survey of patients in Germany showed that younger, female, more educated, and chronically ill people were more likely to use patient-review websites [63]. The reviews used in this study also lacked information on reviewer characteristics and motives. Furthermore, some reviews may not be left by actual service users [64]. A manual examination showed that some reviews were written by companions, e.g., “I went to the vaccination with my 90-year-old mother today. […]”. As the objective was to understand general population response, this does not tarnish the results.

Second, although this study used a large sample covering an entire federal state in Germany over different phases of the COVID-19 vaccination campaign, it studied only one geographical area. Berlin may be particular as vaccination centers were central to the state’s vaccine roll-out and some reviews mirror local specificities, e.g., the existence of language barriers. Furthermore, experiences from a well-communicated city-state with a balanced distribution of centers may not translate to rural areas. A study showed that the share of people able to use public transport was higher and the median travel time to vaccination centers in Berlin was lower than they were in other German federal states [65]. It would hence be interesting to see future studies apply the methodology to other geographical areas.

Third, the text quality, specifically related to emojis and translations, was challenging. A total of 258 reviews used emojis. An inclusion was attempted using demoji [66] but produced misleading results in thematic analysis (e.g., light, face, skin, and syringe). Colloquial language, spelling mistakes, and wordplay were not always perfectly translated to English. Some, but not all, translation errors, e.g., “snake” for the German word “Schlange”, which means both queue and snake, could be corrected at the stage of keyword merging. Despite machine learning techniques having made tremendous progress, this highlights some of the currently persisting shortcomings of computer-aided analysis. Improved algorithms for the translation and the analysis of non-English texts and emojis are needed.

To our knowledge, this is the first study to systematically assess vaccination centers from a user’s perspective in Germany using online reviews and the first to assess pandemic response structures using NLP. It lays the foundation for future research in this area and contributes to pandemic response planning. Aside from addressing the limitations above, future research should include a comparison to other vaccination structures, e.g., physicians’ offices, and other single-purpose healthcare structures, e.g., test centers, and vaccination experiences outside of a global pandemic. It should also contrast online reviews with traditional survey results to further gauge their potential.

5. Conclusions

Positive user experience seems a worthwhile investment for decision-makers and operators during a pandemic, where adherence to services and societal cohesion are essential. Overall, the results of the study with regards to vaccination centers as a pandemic mass vaccination structure are encouraging: User receptions of vaccination centers were very positive, and smooth processes and friendly staff were highly valued. As the mass vaccination campaign lasted, however, efficiency and time management needed to be monitored to ensure long-term satisfaction. Online reviews provided useful, free, and real-time feedback. This makes them an attractive, early recognition “armchair epidemiology” tool [67] for evaluating novel structures during crises, where circumstances continuously change. Their use in pandemic response should therefore be strengthened—encouraging user reviews, establishing units in public administration capable of their analysis, and promoting an iterative design mindset.