Consider referencing RFC 9839 to restrict problematic Unicode characters in RDF 1.2 literals and IRIs

[domel]

While RDF 1.2 already builds on Unicode and related IRI specifications, it currently does not explicitly restrict the use of so-called problematic Unicode characters. These include, among others:

• Surrogates (e.g., unpaired UTF-16 surrogates, which must not appear in UTF-8),
• Legacy control codes (e.g., C0/C1 controls, which have no clear semantics in RDF literals),
• Noncharacters (e.g., U+7FFFF and others that Unicode designates as permanently invalid).

The recently published RFC 9839 provides a clear and concise framework for identifying these problematic classes and offers subsets of Unicode suitable for use in text fields of protocols and data formats.

Why this matters for RDF

1. Literals: RDF literals may currently contain any Unicode string, but if problematic characters are present, different parsers or libraries may handle them inconsistently. For example, ordering, regex filtering, or equality comparisons in SPARQL may yield nondeterministic results.

2. IRIs: While IRIs already exclude some code points, RFC 9839 could serve as a consistent additional reference for implementers to avoid interoperability issues with surrogate code points or noncharacters.

3. Serializations: RDF serializations such as Turtle, N-Triples, and RDF/XML inherit restrictions from their host grammars (e.g., XML partially excludes certain controls, Turtle allows escaping). However, these rules are uneven. Aligning them with RFC 9839 would provide a coherent policy across serializations and improve robustness of RDF processing.

Proposal

• Reference RFC 9839 in RDF 1.2 (possibly in the section on Literals and IRIs) as a recommended baseline for excluding problematic Unicode characters.
• Clarify how this applies across serializations (Turtle, N-Triples, RDF/XML), so that implementers consistently reject or sanitize problematic code points.
• Provide non-normative guidance that RDF data should conform to one of the RFC 9839 subsets (e.g., Scalars) to ensure interoperability.

Benefits

• Increased interoperability between RDF implementations,
• Simplified parser behavior and clearer error handling,
• Prevention of subtle bugs due to inconsistent Unicode handling,
• Alignment of RDF 1.2 with current IETF best practices for Unicode in protocols and data formats.

[pfps]

As far as I can tell RDF already prohibits surrogates everywhere.

Prohibiting C0 controls would prohibit tab and newline, among others, which doesn't seem like a good idea.

Noncharacters might be disallowed by I don't see any strong reason for this, so inertia says not to make a change.

As far as I can tell, there is no ambiguity in the RDF specs as to which characters are allowed in RDF. The "Strings in RDF" section of Concepts https://www.w3.org/TR/rdf12-concepts/#rdf-strings is quite specific. Serializing RDF graphs in any of the concrete syntaxes for DF may have to escape some characters, of course, but that should be well defined.

I note that it appears that xsd:string currently prohibits C0 controls except tab and newline and also prohibits surrogate characters.

The text from https://www.w3.org/TR/xmlschema11-2/ is

The ·value space· of string is the set of finite-length sequences of characters (as defined in [XML 1.0 (Second Edition)]) that ·match· the Char production from [XML 1.0 (Second Edition)]. A character is an atomic unit of communication; it is not further specified except to note that every character has a corresponding Universal Character Set code point, which is an integer.

From https://www.w3.org/TR/xml11/#NT-Char

Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

There is an encouragement to avoid some problematic characters there.

As far as I can tell this does not change in any more-recent version of the documents.

It might be a good idea to provide some "encouragement to avoid" problematic character but any prohibition at this stage seems like a bad idea to me.

[afs]

In RDF Concepts currently:

Literals:
The lexical form is an RDF String - which is unicode scalars.

IRIs:
IRI Grammar - in particular, ucschar which excludes surrogates. iprivate is the area for private unicode characters.

Noncharacters (e.g., U+7FFFF and others that Unicode designates as permanently invalid).

The Unicode name is confusing. They are not invalid.

https://www.unicode.org/versions/Unicode16.0.0/core-spec/chapter-23/#G12612

"""
Applications are free to use any of these noncharacter code points internally. They have no standard interpretation when exchanged outside the context of internal use. However, they are not illegal in interchange, nor does their presence cause Unicode text to be ill-formed.
"""

[domel]

Interesting discussion in the context of CBOR, JSON, YAML etc. https://www.tbray.org/ongoing/When/202x/2025/08/14/RFC9839

[domel]

@pfps Thanks for the detailed feedback. I agree with several of your points:

• RDF Concepts is indeed explicit about the character repertoire, and surrogates are already disallowed by definition of Unicode scalar values.
• Tabs, newlines, and carriage return are clearly necessary in practice, so prohibiting all C0 controls would not be viable.
• XSD string (via XML’s Char production) already imposes restrictions that overlap with the problem space, especially around surrogates.

My motivation in filing this issue was not to argue that the current RDF specs are ambiguous - they are clear. Rather, RFC 9839 provides a compact and more recent articulation of which Unicode characters tend to cause interoperability and security issues across protocols. Having RDF point to such a document could serve as non-normative guidance, especially for implementers working with Turtle, N-Triples, or RDF/XML serializers and parsers.

For example:

• While surrogates are already prohibited, RFC 9839 provides a succinct reference for explaining why.
• Noncharacters (e.g., U+7FFFF) are not forbidden by the RDF specs, but implementations may treat them inconsistently - RFC 9839 could be cited to encourage avoidance, without making it a normative prohibition.
• Guidance like this could reduce the likelihood of bugs in SPARQL processing (e.g., regexes, sorting, equality) where invisible or undefined characters create non-deterministic outcomes.

So the suggestion is not necessarily to introduce new prohibitions into RDF 1.2, but rather to consider whether a pointer to RFC 9839 as "good practice" would help implementers align on Unicode handling beyond the formal minimum.

This would be especially useful because RDF serializations each inherit slightly different constraints (e.g., XML vs Turtle). A common reference might help close some of those gaps.

[pfps]

My motivation in filing this issue was not to argue that the current RDF specs are ambiguous - they are clear. Rather, RFC 9839 provides a compact and more recent articulation of which Unicode characters tend to cause interoperability and security issues across protocols. Having RDF point to such a document could serve as non-normative guidance, especially for implementers working with Turtle, N-Triples, or RDF/XML serializers and parsers.

RFC 9839 only calls out a few of the security issues of Unicode. In my view the common indistinguishability of different Unicode sequences is much more problematic from a security standpoint.

Suppose that RDF disallowed some legal IRIs. That would mean that some IRIs that correspond to potentially existing web domains and documents could not be used in RDF. That seems to me to be a situation to be avoided.

[aphillips]

I added the i18n-tracker label to make this visible to the I18N WG.

(chair hat off)

I did a review of 9839 when it was an I-D. I think the materials in the RFC are pretty clear and a rather good explanation of some specific issues found with certain Unicode code points, while noting that there are issues that it doesn't cover. The guidance provided by the resulting RFC doesn't seem to provide any additional normative needs for RDF* specs (or their foundations, such as IRI, XML, etc.).

I18N participants will no doubt be curious to follow how this conversation proceeds, particularly as it relates to how we document user guidance. Our "magnum opus" (CharMod: String Matching) on this topic is more focused on identifier definitions in specs.

[afs]

Guidance like this could reduce the likelihood of bugs in SPARQL processing (e.g., regexes, sorting, equality) where invisible or undefined characters create non-deterministic outcomes.

Example of a non-deterministic outcome allowed by a specification please.

The advice should be to the content creators. It might appear in the Security Considerations section.

Clarify how this applies across serializations (Turtle, N-Triples, RDF/XML), so that implementers consistently reject or sanitize problematic code points.

Parsers, serializers and SPARQL processors are intermediate systems. They do not know the original (producer) intent of the data nor the application (consumer) usage of the data. Intermediate systems should not alter data (and which would break canonicalization and signing).

[domel]

@afs Let me try to clarify.

On the "non-deterministic outcomes": a concrete example is when a literal contains an unpaired surrogate or a noncharacter. Some Unicode libraries treat these as illegal and reject them, others silently replace them with U+FFFD, and some allow them through unchanged. If two conformant SPARQL implementations rely on different Unicode libraries, then FILTER(?x = "…") or ORDER BY can yield different results. That is the kind of implementation divergence I had in mind — not that the RDF spec itself is ambiguous, but that it leaves enough room for platform/library differences that outcomes may vary in practice.

I completely agree that RDF processors should not silently alter data, as this would indeed break canonicalization and signing. My suggestion was not to require alteration, but rather to provide guidance that content creators avoid problematic code points in the first place. Placing this in the Security Considerations section, as you suggest, could be a good compromise. It would warn data producers about the risks without changing the normative behavior of parsers, serializers, or SPARQL engines.

So perhaps the useful takeaway is:

• No change to normative constraints in RDF 1.2,
• But add a note in Security Considerations pointing to RFC 9839 as a reference, especially for data producers, to encourage avoiding problematic characters.

This way, we acknowledge existing spec clarity, preserve canonicalization/signing guarantees, but still help implementers and data creators reduce avoidable interoperability risks.

[domel]

@pfps I agree with you that confusables and indistinguishable Unicode sequences (e.g., homoglyph attacks) represent a broader and in many cases more serious security concern than the relatively narrow classes called out by RFC 9839. That said, I see these as complementary issues: one is about semantic indistinguishability, the other about syntactic instability across platforms and libraries.

My suggestion is not to tighten the normative definition of IRIs or literals, but to consider pointing to RFC 9839 as non-normative guidance for content creators and implementers. The intent is not to forbid IRIs or literals that are otherwise valid, but rather to highlight classes of code points that are often mishandled by software stacks and that can lead to interoperability or robustness problems.

In short:

• I fully agree that confusable sequences are a critical issue.
• RFC 9839 doesn’t solve that, but it does provide a useful and concise articulation of another class of problems.
• Referencing it in a non-normative way (e.g., Security Considerations) could help RDF implementers and data producers avoid pitfalls without changing what is allowed in RDF.

This way, we avoid disallowing valid IRIs, but still provide helpful contemporary guidance to those working with RDF tooling.

[afs]

Referencing it in a non-normative way (e.g., Security Considerations) could help RDF implementers and data producers avoid pitfalls without changing what is allowed in RDF.

Reasonable. It won't take much.

Security Considerations already has text about appearance issues.

Surrogates are illegal in RDF 1.2 and SPARQL 1.2, whether written in the input bytes or as escape sequences.

Some Unicode libraries treat these [an unpaired surrogate or a noncharacter -ed] as illegal and reject them, others silently replace them with U+FFFD, and some allow them through unchanged.

I'm still uncomfortable with broad-brush statements that something may be wrong in systems outside the remit of this WG.

What is the scale of the problem? Is it because software hasn't been updated or has there been an intentional decision on a particular policy and current versions have this behavior? -- especially with regard to surrogates as RDF Concepts has already forbidden them.

https://www.w3.org/TR/rdf12-concepts/#rdf-strings
https://www.w3.org/TR/rdf12-turtle/#sec-escapes

[domel]

Regarding scale: from my experience, the inconsistent handling of noncharacters (and historically, surrogates) stems from differences in Unicode library policies across programming languages. Some libraries reject, some replace with U+FFFD, and others pass them through.

Would it be useful if I drafted a short PR to RDF Concepts to extend the Security Considerations section with a pointer to RFC 9839 (framed as non-normative guidance)? That might be the minimal change needed while addressing the concerns raised here.