# Entropic Bounds on the Average Length of Codes with a Space

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

## 2. Relations between One-to-One Codes and Prefix Codes Ending with a Space

**Lemma**

**1.**

**Proof.**

**Lemma**

**2.**

**Proof.**

**Lemma**

**3.**

**Proof.**

- $C\left({s}_{i}\right)=D\left({s}_{i}\right)$ and $C\left({s}_{j}\right)=D\left({s}_{j}\right)$: then $D\left({s}_{i}\right)\ne D\left({s}_{j}\right)$ since $C\left({s}_{i}\right)\ne C\left({s}_{j}\right)$;
- $C\left({s}_{i}\right)=D\left({s}_{i}\right)\bigsqcup $ and $C\left({s}_{j}\right)=D\left({s}_{j}\right)\bigsqcup $: then $D\left({s}_{i}\right)\ne D\left({s}_{j}\right)$ since $C\left({s}_{i}\right)$ is not a prefix of $C\left({s}_{j}\right)$ and vice versa;
- $C\left({s}_{i}\right)=D\left({s}_{i}\right)\bigsqcup $ and $C\left({s}_{j}\right)=D\left({s}_{j}\right)$: then $D\left({s}_{i}\right)\ne D\left({s}_{j}\right)$ since $C\left({s}_{j}\right)$ is not a prefix of $C\left({s}_{i}\right)$;
- $C\left({s}_{i}\right)=D\left({s}_{i}\right)$ and $C\left({s}_{j}\right)=D\left({s}_{j}\right)\bigsqcup $: then $D\left({s}_{i}\right)\ne D\left({s}_{j}\right)$ since $C\left({s}_{i}\right)$ is not a prefix of $C\left({s}_{j}\right)$.

**Lemma**

**4.**

**Proof.**

## 3. Lower Bounds on the Average Length

**Lemma**

**5.**

**Proof.**

**Lemma**

**6.**

- 1.
- If $0<{p}_{1}\le 0.5$,$$\begin{array}{cc}\hfill {L}_{\u03f5}\ge & {H}_{k}\left(\mathbf{p}\right)-({H}_{k}\left(\mathbf{p}\right)-{p}_{1}{log}_{k}\frac{1}{{p}_{1}}+(1-{p}_{1}){log}_{k}(k-1))\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& {log}_{k}\left(1+\frac{1}{{H}_{k}\left(\mathbf{p}\right)-{p}_{1}{log}_{k}\frac{1}{{p}_{1}}+(1-{p}_{1}){log}_{k}(k-1)}\right)\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& -{log}_{k}({H}_{k}\left(\mathbf{p}\right)-{p}_{1}{log}_{k}\frac{1}{{p}_{1}}+(1-{p}_{1}){log}_{k}(k-1)+1)\hfill \\ & -{log}_{k}\left(1-{\left(\frac{1}{kn}\right)}^{{log}_{k}\left(1+\frac{1}{1-{p}_{1}}\right)}\right),\hfill \end{array}$$
- 2.
- if $0.5<{p}_{1}\le 1$$$\begin{array}{cc}\hfill {L}_{\u03f5}\ge & {H}_{k}\left(\mathbf{p}\right)-({H}_{k}\left(\mathbf{p}\right)-{\mathcal{H}}_{k}\left({p}_{1}\right)+(1-{p}_{1})(1+{log}_{k}(k-1)))\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& {log}_{k}\left(1+\frac{1}{{H}_{k}\left(\mathbf{p}\right)-{\mathcal{H}}_{k}\left({p}_{1}\right)+(1-{p}_{1})(1+{log}_{k}(k-1))}\right)\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& -{log}_{k}({H}_{k}\left(\mathbf{p}\right)-{\mathcal{H}}_{k}\left({p}_{1}\right)+(1-{p}_{1})(1+{log}_{k}(k-1))+1)\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& -{log}_{k}\left(1-{\left(\frac{1}{kn}\right)}^{{log}_{k}\left(1+\frac{1}{1-{p}_{1}}\right)}\right),\hfill \end{array}$$where ${\mathcal{H}}_{k}\left({p}_{1}\right)=-{p}_{1}{log}_{k}{p}_{1}-(1-{p}_{1}){log}_{k}(1-{p}_{1})$.

**Proof.**

**Theorem**

**1.**

**Proof.**

**Theorem**

**2.**

- 1.
- If $0<{p}_{1}\le 0.5$:$$\begin{array}{cc}\hfill L\left(C\right)\ge & {H}_{k}\left(\mathbf{p}\right)-({H}_{k}\left(\mathbf{p}\right)-{p}_{1}{log}_{k}\frac{1}{{p}_{1}}+(1-{p}_{1}){log}_{k}(k-1))\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& {log}_{k}\left(1+\frac{1}{{H}_{k}\left(\mathbf{p}\right)-{p}_{1}{log}_{k}\frac{1}{{p}_{1}}+(1-{p}_{1}){log}_{k}(k-1)}\right)\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& -{log}_{k}({H}_{k}\left(\mathbf{p}\right)-{p}_{1}{log}_{k}\frac{1}{{p}_{1}}+(1-{p}_{1}){log}_{k}(k-1)+1)\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& -{log}_{k}\left(1-{\left(\frac{1}{kn}\right)}^{{log}_{k}\left(1+\frac{1}{1-{p}_{1}}\right)}\right)+\sum _{i=1}^{\lceil \frac{n}{k}\rceil -1}{p}_{n-i+1}+\sum _{i=1}^{\lfloor {log}_{k}\lceil \frac{n-1}{k}\rceil \rfloor}{p}_{\frac{{k}^{i}-1}{k-1}}.\hfill \end{array}$$
- 2.
- If $0.5<{p}_{1}\le 1$:$$\begin{array}{cc}\hfill L\left(C\right)\ge & {H}_{k}\left(\mathbf{p}\right)-({H}_{k}\left(\mathbf{p}\right)-{\mathcal{H}}_{k}\left({p}_{1}\right)+(1-{p}_{1})(1+{log}_{k}(k-1)))\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& {log}_{k}\left(1+\frac{1}{{H}_{k}\left(\mathbf{p}\right)-{\mathcal{H}}_{k}\left({p}_{1}\right)+(1-{p}_{1})(1+{log}_{k}(k-1))}\right)\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& -{log}_{k}({H}_{k}\left(\mathbf{p}\right)-{\mathcal{H}}_{k}\left({p}_{1}\right)+(1-{p}_{1})(1+{log}_{k}(k-1))+1)\hfill \\ \hfill \phantom{\rule{1.em}{0ex}}& -{log}_{k}\left(1-{\left(\frac{1}{kn}\right)}^{{log}_{k}\left(1+\frac{1}{1-{p}_{1}}\right)}\right)+\sum _{i=1}^{\lceil \frac{n}{k}\rceil -1}{p}_{n-i+1}+\sum _{i=1}^{\lfloor {log}_{k}\lceil \frac{n-1}{k}\rceil \rfloor}{p}_{\frac{{k}^{i}-1}{k-1}}.\hfill \end{array}$$

**Proof.**

## 4. Upper Bounds on the Average Length

**Lemma**

**7.**

**Proof.**

**Lemma**

**8.**

**Proof.**

**Theorem**

**3.**

**Proof.**

**Theorem**

**4.**

**Proof.**

**Remark**

**1.**

## 5. Concluding Remarks

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Daniels, P.T.; Bright, W. (Eds.) The World’s Writing Systems; Oxford University Press: Oxford, UK, 1996. [Google Scholar]
- Wikipedia: Writing Systems without Word Boundaries. Available online: https://en.wikipedia.org/wiki/Category:Writing_systems_without_word_boundaries (accessed on 25 March 2024).
- Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; Wiley-Interscience: Hoboken, NJ, USA, 2006. [Google Scholar]
- Alon, N.; Orlitsky, A. A lower bound on the expected length of one-to-one codes. IEEE Trans. Inf. Theory
**1994**, 40, 1670–1672. [Google Scholar] [CrossRef] - Blundo, C.; De Prisco, R. New bounds on the expected length of one-to-one codes. IEEE Trans. Inf. Theory
**1996**, 42, 246–250. [Google Scholar] [CrossRef] - Jaynes, E. Note on unique decipherability. Ire Trans. Inf. Theory
**1959**, 5, 98–102. [Google Scholar] [CrossRef] - Shannon, C.E. The Mathematical Theory of Communication; University of Illinois Press: Urbana, IL, USA, 1949. [Google Scholar]
- Courtade, T.; Verdú, S. Cumulant generating function of codeword lengths in optimal lossless compression. IEEE Int. Symp. Inf. Theory
**2014**, 2494–2498. [Google Scholar] - Kontoyiannis, I.; Verdú, S. Optimal lossless data compression: Non-asymptotics and asymptotics. IEEE Trans. Inf. Theory
**2014**, 60, 777–795. [Google Scholar] [CrossRef] - Kosut, O.; Sankar, L. Asymptotics and non-asymptotics for universal Fixed-to-Variable source coding. IEEE Trans. Inf. Theory
**2017**, 63, 3757–3772. [Google Scholar] [CrossRef] - Szpankowski, W.; Verdu, S. Minimum expected length of Fixed-to-Variable lossless compression without prefix constraints. IEEE Trans. Inf. Theory
**2011**, 57, 4017–4025. [Google Scholar] [CrossRef] - Knuth, D. The Art of Computer Programming, Volume 3, 2nd ed.; Sorting and Searching; Addison Wesley Longman Publishing Co., Inc.: Boston, MA, USA, 1998. [Google Scholar]
- Wyner, A. An upper bound on the entropy series. Inf. Control
**1972**, 20, 176–181. [Google Scholar] [CrossRef] - Chan, S.-L.; Golin, M.J. A dynamic programming algorithm for constructing optimal 1-ended binary prefix-free codes. IEEE Trans. Inf. Theory
**2000**, 46, 1637–1644. [Google Scholar] [CrossRef]

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bruno, R.; Vaccaro, U.
Entropic Bounds on the Average Length of Codes with a Space. *Entropy* **2024**, *26*, 283.
https://doi.org/10.3390/e26040283

**AMA Style**

Bruno R, Vaccaro U.
Entropic Bounds on the Average Length of Codes with a Space. *Entropy*. 2024; 26(4):283.
https://doi.org/10.3390/e26040283

**Chicago/Turabian Style**

Bruno, Roberto, and Ugo Vaccaro.
2024. "Entropic Bounds on the Average Length of Codes with a Space" *Entropy* 26, no. 4: 283.
https://doi.org/10.3390/e26040283