Artificial intelligence is becoming an indispensable part of modern business, but it also brings new legal and technical risks. One of the most well-known phenomena in the operation of AI systems is the occurrence of so-called hallucinations, i.e. situations in which the system confidently generates incorrect, fabricated, or illogical information while presenting it as factual.

Hallucinations do not occur due to intent or consciousness of the algorithm, but as a consequence of the way AI models function. AI models do not know facts; instead, they predict the most probable sequence of words based on prior patterns. In practice, this can lead to serious consequences. For example, lawyers in the United States, relying on an AI tool, submitted court filings citing non-existent judgments, which led to disciplinary proceedings and damage to their professional reputation.

Legal Framework and Nature of Liability

An AI system’s hallucination is not conscious deception, but it can cause harm to a user who relies on an inaccurate output. The question arises: who bears responsibility?

Depending on the contractual relationship, three approaches are possible:

• Contractual liability of the service provider, if the system fails to meet expected technical and professional standards.
• Tort liability, if the damage is caused by negligence in the development, testing, or use of the system.
• Limited liability of the user, in cases where the provider clearly warned of the risks, and the user relied on the output as final information without verification.

Most commercial AI platforms explicitly exclude liability for the accuracy of generated information in their terms of service. However, such clauses do not fully release the service provider from responsibility, especially if the provider knowingly presented the system as more reliable than it actually is or failed to provide adequate warnings to the user.

EU AI Act – A New Standard of Liability

The European Union adopted the AI Act, which entered into force on 1 August 2024, making it the first jurisdiction in the world with a comprehensive legal framework for artificial intelligence.

The regulation introduces a regulatory approach based on risk assessment and risk level, under which AI service providers, particularly those operating high-risk systems,  are required to ensure data traceability, testing, monitoring, and clear warnings about potential errors.

• Prohibited practices apply as of February 2025;
• Rules for general-purpose AI entered into force in August 2025;
• Rules for high-risk systems will apply from August 2027.

This law significantly affects contractual relationships between AI companies and users, as it establishes minimum standards of transparency and due care expected from service providers.

Contractual Aspects and Practice

AI service agreements increasingly include specific provisions on:

• limitation of the provider’s liability for inaccurate outputs (hallucinations),
• the user’s obligation to verify generated information,
• accuracy standards and ethical rules,
• procedures for compensation or correction of errors.

For lawyers and companies integrating AI into their work, it is crucial that these clauses are clearly formulated and tailored to the purpose of the system. Particular attention is needed for services that may cause significant financial or reputational harm, such as legal advice, medical diagnostics, or financial decision-making.

Conclusion

AI hallucinations do not mean that the technology is unusable, but rather that it must be used responsibly and with an understanding of its limitations. An AI service provider cannot completely absolve itself of responsibility through a simple disclaimer clause; instead, it must demonstrate that it applied reasonable due care and ensured transparency in how the system operates.

In an era in which machines write, analyze, and propose decisions, the law must ensure a balance between innovation and accountability. A reliable AI system is not one that never makes mistakes, but one that knows how to warn when it might be wrong.

This article is for informational purposes only and does not constitute legal advice. Should you require additional information, please feel free to contact us.