.png)
.png)
ENG
In our previous articles, we demonstrated how this tool overcomes the limitations of traditional full-text search and makes information from digital libraries more accessible. In this article, we will examine the tool from a more technical perspective: how we selected the data, what technical and content-related limitations we overcame, and what challenges semantic search presents. This opens up new possibilities while placing higher demands on data quality, linguistic analysis, and how results are processed.
Paměť novin is a prototype tool for semantic search in historical texts, which we recently made available for testing.
Semantic search understands the meaning of a query, handles OCR inaccuracies or archaic language, and thus speeds up the research process.
From the perspective of an average user, Newspaper Memory represents a revolutionary way to search through historical periodicals – you simply ask a question in natural language, without needing to know period-specific terminology or the exact spelling of terms. However, a closer look from a professional perspective reveals a number of interesting connections, decisions, and challenges that accompanied the creation of this tool.
For semantic search to be truly useful, it was necessary to carefully select a suitable data corpus.
We built it with a focus on achieving the highest possible level of consistency and reliability in search results. Recognizing the challenges of working with historical documents, we carefully selected sources with high historical and research value. Digitized newspapers proved to be the ideal choice because they:
We placed an emphasis on content diversity – our selection includes not only national dailies and weeklies, but also regional press and thematic periodicals aimed at various social strata. This approach allowed us to create a wide-reaching tool that offers a deeper understanding of the historical context.
Although we are currently working with only a sample of data, we ensured that it represents various types of periodicals and provides the most comprehensive picture possible of the social events of that time. For the purposes of prototype development, we have processed 500,000 pages (65,000 issues) from 25 Czech periodicals published between 1880 and 1914. These titles are freely available in five different digital libraries using the Kramerius system.
In our case, data integrity played a key role in the selection process – thematic (newspapers), temporal (1880–1914), and geographical (Czech-language newspapers from our territory).
If the content were thematically or typologically inconsistent, for example, if news articles were mixed with fiction or scientific texts with popular articles, it would lead to distorted results and make it difficult for the user to navigate. Similar problems would be caused by including documents from historical periods without direct chronological continuity, or from geographically distant regions without clear boundaries.
Imagine if we combined historical newspaper articles with the book War with the Newts by Karel Čapek into a single file. The system might then return reports on real geopolitical events alongside fictional accounts of the rise of intelligent newts, which would lead to confusion and erroneous interpretations.
Although we eliminated major issues caused by inconsistent content when selecting data for the Newspaper Memory project – working with language models introduces new types of challengesthat must be taken into account.
Even a thematically and chronologically coherent dataset does not guarantee that a language model will always evaluate context accurately. Especially in cases of overly general queries, a lack of clearly relevant documents, or ambiguous key terms, the model may combine texts that are only loosely related or entirely unrelated when generating an answer. Such an answer may seem convincing, but upon closer inspection, it is inaccurate or misleading.
For example, the situation in Europe in 1898 was tense—great powers were competing for influence in colonies, military buildups were underway, and tensions were rising in the Balkans. Theoretically, this atmosphere could retrospectively resemble the events of 1914, when the First World War broke out. However, if an algorithm were to work with texts from these two periods without awareness of the chronological context, it could, when asked more general questions, create the false impression that a diplomatic rift from 1898 was a direct cause of the war in 1914.
Similarly, it could be misleading to connect newspaper articles from different regions, such as large cities and rural areas, without taking into account local differences in mentality, language, or socioeconomic conditions. This inconsistency could lead to an inaccurate picture of the society of that time. For this reason, we have not yet included so-called expatriate periodicals of Czechs living abroad, which were published extensively, particularly in the USA, in the current dataset.
Even though the data in our application is not mixed randomly and was selected carefully, it cannot be entirely ruled out that incorrect connections of information may occur in certain cases, especially with queries that are not sufficiently specific. Therefore, it is always necessary to evaluate answers critically and check which source articles they are based on.
We faced technical limitations when selecting data. It was crucial that the documents contained a text layer in ALTO format and at the same time supported the IIIF protocol. Without these elements, it would not be possible to process the data correctly or offer it to users.
For this reason, we had to exclude some significant titles right from the start, such as Národní listy, which would otherwise represent a very valuable historical resource. Although these newspapers are digitized, they lack high-quality, structured text transcripts, meaning they cannot be analyzed, broken down into logical units, or integrated into the system.
To effectively utilize semantic search, it was necessary to divide each page into smaller partsthat are semantically related.
If we had left the page as one continuous text, the results would have been too general, as individual articles can cover different topics. The system would not have been able to determine exactly what corresponds to the query. On the other hand, if we had divided the text too finely, for example by individual sentences, the context would have been lost and the answers would have seemed fragmented.
We therefore sought a balanced approachthat would allow us to preserve both meaning and readability. Using structural information contained in the ALTO format, we divided the text into logical blocks of approximately 500 to 1000 characters.
During this process, we combined various rules – we identified the beginnings of articles based on font type and indentation, recognized paragraphs and column breaks, and worked with the closure of text blocks. The resulting parts correspond to individual articles or their segments.
The result was an average of 21 sections per newspaper page. Given the total volume of processed data, this means more than 10 million individually indexed text segments. Each of these sections contains a basic set of metadata:
This makes it possible to link the found texts to their original sources, filter by date and periodical title, and also display the found segment directly in the digital library, where the article is zoomed in and highlighted.
Data prepared in this way forms the foundation upon which semantic search in the Newspaper Memory can function effectively. It allows the system to better understand the meaning of a query, maintain context while offering accurate and localizable results.
While testing the Newspaper Memory tool, we encountered several topics that deserve deeper consideration. Working with diverse data and a wide range of user queries—across languages, genres, and regions—showed that even with advanced technologies, there are limits and room for further development.
One of the key challenges is handling multilingual queries. The system supports various languages, which is a significant advantage as users are not limited to Czech. At the same time, however, this flexibility brings certain consequences. Results may vary slightly depending on how the query is phrased.
For example:
It is important to note that in both cases, the search was conducted within the same collection of Czech-language periodicals. The difference lies not in the content of the data, but in the way the language model understands and contextualizes the query. Multilingualism is therefore not just a matter of translation, but also of the cultural framework in which the question is asked.
Another observation is the influence of query specificity on the output. The system is highly sensitive to phrasing – the more general the query, the more general the response. However, users can choose whether they want to search across all titles, within a specific time interval, or only in one particular periodical. This choice significantly affects the results.
For example, the query "Did people complain about the youth?" returns, when a general dataset is selected, answers that summarize articles from various periods and regions with occasional specific examples, often describing moral decay, noise, or the bad manners of young people. However, if we apply a filter for the periodical Plzeňský obzor, the outputs become more precise – displaying articles that specifically describe, for instance, particular conflicts between youths and authorities or alcoholism and the negative influence of newspapers.
The ability to influence the scope of the answer in this way is an advantage, but it also places emphasis on the correct formulation of the query and understanding the principleson which the system operates.
We assumed that semantic search would work primarily with meaning, regardless of the text's language. However, testing showed that both the query language and the source document language have a fundamental impact on the results.
If a user searched for information in German periodicals, they had to formulate the query in German—especially regarding local specifics. For example, a query about the situation of German merchants in Brno around 1910 had to include the name Brünn, not Brno. In this case, the system likely recognized that it was the same city, but it did not include articles from German newspapers in the response, as it assigned them lower relevance. Consequently, the user could not find such articles in the selection of sources.
This phenomenon shows that linguistic nuances, especially regarding proper names and geographical terms, still play a significant role. In the future, it will be necessary to address this sensitivity, for example, by using multilingual name maps or intelligent query rewrites.
Another challenge is posed by specific genres found in historical press – for example, feuilletons, serialized novels, or advertisements. These texts often repeat, have a distinctive style , and sometimes take up a significant portion of the page. As a result, they can overshadow more informative articles.
A typical example is a query about the most popular product, e.g., which beer was the most popular. The answer may be influenced by which brands had the budget for extensive advertising at the time. The results, therefore, do not reflect the actual preferences of the population, but rather the brewery's financial resources and the resulting marketing space in the press.
For this reason, it will be necessary in the future to consider whether to exclude these types of texts in certain cases, or at least label them separately so they do not interfere with the analysis.
Newspaper Memory is currently just a prototype demonstrating the possibilities of semantic search in historical documents. It contains a limited sample of data, specifically 25 titles of Czech periodicals from the years 1880 to 1914. Furthermore, some of these titles have not been preserved in their entirety. Some are missing volumes that have not yet been digitized, while others experienced interruptions or ceased publication during certain periods.
Technically, it is possible to create a similar tool using other datasets, but historical newspapers have proven to be an exceptionally suitable medium that we believe is worth continuing to work with.
In the future, we can expect digital libraries to continuously add new data and improve the quality of existing content. This will involve not only more accurate OCR transcriptions but also the addition of text layers in ALTO format and making scan images accessible via the IIIF protocol for other valuable resources. This will enable even more precise search results and provide broader access to digitized archives.
We are also interested in how semantic search could be applied in other areas. What if, for example, a tool like Paměť novin included the complete corpus of Czech newspaper production – from the beginning of the 19th century to the end of the 20th century? Such a scope would open up new research possibilities, such as tracking changes in social discourse, the evolution of language, and the transformation of stereotypes over time.
This would make it possible to study how Paměť novin is formed, how certain narratives repeat or how the press reacted to key moments in modern history.
Such a vision is still a long way off, but technologies like Paměť novin can gradually pave the way for it. For it to become a reality, a number of other aspects will need to be addressed – not only technical, but also legal. In particular, the issue of copyright for a portion of newspaper production will play a crucial role and will require a systematic solution. Paths exist, but they require a broader interdisciplinary discussion that goes beyond the scope of this text.
In the next phases of development, we want to include personalization of the user environment, taking into account individual user preferences, and the ability to combine different types of queries. Our goal is for Paměť novin to become not just a research tool, but also a reliable daily assistant – for researchers, teachers, students, and the general public.
Paměť novin is not just a tool for searching historical texts, but also a space for further research and experimentation. Testing has shown that despite advanced technology, challenges still exist – from working with multilingual sources and the correct interpretation of queries to the genre differences of historical texts. We are therefore faced with questions of how to better adapt the results to the needs of researchers, how to make data analysis more efficient, and what new technologies can help push the possibilities of searching even further.
In the future, we plan to:
We would be happy if you shared your experiences with us – what worked for you, what you missed, and what would help you in your daily work with historical documents.
👉 If you would like to see how this technology works in practice, try out Paměť novin at www.pametnovin.cz. You can now also follow Paměť novin on social media, specifically on Facebook and Instagram.
👉 Would you like similar search capabilities for your own institution and data? Get in touch with us and we will be happy to discuss the possibilities with you, no strings attached.
We will contact you within 2 business days.
