DeepSeek – How Does an LLM Make the News?

First, let us provide some background. DeepSeek was founded in China in 2023, and although it is a relatively new player in the market, R1 is not its first publicly available model. In November 2023, the company released its first proprietary large language model, DeepSeek Coder, which was essentially a copy of Meta’s publicly released Llama model. This was followed by a series of LLMs specialized for different tasks; however, the company did not achieve a major breakthrough until the release of R1.

DeepSeek R1: The Champion

The DeepSeek R1 model is essentially a further-trained version of DeepSeek V3. This new version has been specifically optimized for logical reasoning, coding, and solving mathematical problems, using the so-called Chain-of-Thought technique. This means that the R1 model essentially discusses the individual steps with itself and provides feedback to itself, increasing the likelihood of arriving at the correct conclusions for a given problem.

However, this alone does not explain DeepSeek R1’s meteoric rise. Let us look at the reasons:

• China is currently subject to US restrictions on GPU exports, significantly limiting its options for acquiring and using hardware. To overcome these constraints, DeepSeek researchers employed several optimization techniques. As a result, they achieved better performance than the Llama models—which had previously been among the leading openly available models—while using ten times fewer training resources.

• DeepSeek R1 competes with OpenAI’s latest comparable models and even outperforms them on certain benchmarks. (Although the competition is close, and the performance improvement is not dramatic.)

• R1 is an open model that anyone can download and use. It is also available alongside several smaller, lower-performance and less hardware-intensive so-called Distilled models developed by the company. In addition, DeepSeek offers API-based access to its models at significantly lower prices than OpenAI.

Technological Innovations

In terms of architecture, DeepSeek V3 and R1 do not differ significantly from the models that have dominated the field so far, all of which are based on the so-called Transformer architecture. The structural innovations they employ are designed to increase text-generation speed and reduce resource consumption.

The real breakthrough lies in the methods used to train the models, which reduced the required computing capacity to a fraction of previous levels. These included a combination of low-level modifications to code running directly on the hardware and techniques that enabled the models to learn efficiently from far fewer example data points.

In addition, by using the finished R1 model as a kind of teacher LLM, DeepSeek created a series of smaller models based on previously existing open models and further trained them for reasoning, mathematical, and coding tasks.

From a strictly technical perspective, the following techniques were used:

• Multi-Headed Latent Attention: a version of the Attention mechanism operating in a lower-dimensional vector space, reducing the size of the model.

• DeepSeekMoE: a Mixture-of-Experts method that allows only a small proportion of the model’s weights to be activated during text generation.

• Multi-Token Prediction: the model generates multiple tokens simultaneously, improving training performance and enabling speculative generation.

• FP8 Mixed-Precision Training: certain parts of the model use lower-precision parameters, reducing model size and accelerating training.

• Near Full Computation-Communication Overlap: an optimized framework was written for the GPUs used for training in a language operating at a lower level than CUDA, making distributed training across multiple GPUs significantly more efficient.

• Large-Scale Reinforcement Learning and Supervised Fine-Tuning: during R1’s training, the model was trained using a combination of predefined reasoning chains and methods where only the final conclusion was provided, with generated outputs rewarded based on evaluation. This required significantly less data than previous methods.

Using DeepSeek R1

To run the full R1 model, hardware capable of holding all 671 trillion parameters in memory is required, representing approximately 1,543 GB of RAM/VRAM data—or roughly 16 NVIDIA A100 GPUs. Quantized, or reduced-precision, models can operate with less memory, but their performance is also lower. The same applies to the aforementioned Distilled models, which deliver outstanding performance within their respective categories but cannot compete with the larger models.

If this amount of memory is unavailable, using DeepSeek’s API is another option. However, this comes with data security risks, as DeepSeek reserves the right to use data uploaded to its system. Registration is possible via a Google Account or a Chinese email address/phone number, after which both the V3 and R1 models can be tested manually free of charge.

Pricing for API usage is highly competitive compared with OpenAI:

An Invitation to Dance – First-Hand Experience

• Hungarian language capabilities: Both models tested can communicate in Hungarian. V3 and R1 both respond in sophisticated language with virtually no spelling errors, similarly to ChatGPT 4o.

• Mathematics: When dealing with mathematical problems, the models are capable of more complex proofs. However, they tend not to contradict the user, meaning they may attempt to prove false statements instead of refuting them.

• Complex coding tasks: More complex coding problems can already pose difficulties. In our tests, the models occasionally generated incorrect code or lacked knowledge of certain functions in specific libraries.

• Censorship: The information provided is censored in relation to certain sensitive topics.

Consequences

DeepSeek R1 will probably hold the title of “best LLM” only briefly, as several companies already claim that their solutions outperform R1. However, the significance of the model does not lie in maintaining a long-term leading position, nor necessarily in being an openly available model capable of competing with paid services—although this is also an important aspect, as it redirects attention toward the open-source community. In the future, other companies may also take steps in this direction, which could contribute to the long-term democratization of AI. The true value of the R1 model lies in the results achieved in optimizing the training process.

Until now, almost exclusively large corporations such as Google, OpenAI, Meta, and Anthropic have been able to afford training their own LLMs. However, the technology demonstrated by DeepSeek opens the door to players with significantly less computing capacity, including smaller companies and universities. The combination of lower resource requirements and reduced costs enabled by optimized models suggests that a new category may soon emerge: companies and startups that were previously unable to work with large AI models may now be able to develop their own solutions and enter the industry’s competitive landscape.

Although we are still talking about costs in the millions of dollars, developments of this kind demonstrate that there are still untapped opportunities for optimization that could fundamentally reshape the position of LLMs within the industry and redefine market norms.

And while the field of AI has already been evolving continuously, if industry trends continue in this direction and development costs keep falling at this pace, even more significant leaps may be expected in the coming years, with AI applications becoming an increasingly integral part of everyday business operations.

Efficiency vs. Convenience

Another increasingly important aspect of AI development is worth highlighting: the difference between the Eastern and Western worlds when it comes to technological competition. While the Western world often prioritizes wellbeing and convenience—for example, through workplace wellbeing programs such as the four-day workweek—China and other Eastern countries are willing to make considerable sacrifices in the interest of efficiency and rapid development. The competition is therefore clearly shifting toward efficiency and effectiveness, which may ultimately determine the long-term evolution of the global AI market.

What Does All This Mean for the Hungarian Market?

Until now, training proprietary LLMs specifically designed for the Hungarian language has only been conceivable within very limited constraints for domestic players. The innovations introduced by DeepSeek could change this and create opportunities for Hungarian companies to pursue similar developments. The publicity generated by DeepSeek may also attract the attention of larger players that have not previously been active in the LLM market but may now decide that investing in these technologies is worthwhile.