ID_Start and ID_Continue specification and regular expression

[flip111]

I'm looking at this

openCypher/grammar/basic-grammar.xml

Lines 718 to 740 in a514465

	<production name="IdentifierStart" rr:inline="true" oc:lexer="true">
	<description>
	Based on the unicode identifier and pattern syntax
	(http://www.unicode.org/reports/tr31/)
	And extended with a few characters.
	</description>
	<alt>
	<character set="ID_Start"/>
	<character set="Pc"/> <!-- Punctuation connectors (underscores) -->
	</alt>
	</production>
	<production name="IdentifierPart" rr:inline="true" oc:lexer="true">
	<description>
	Based on the unicode identifier and pattern syntax
	(http://www.unicode.org/reports/tr31/)
	And extended with a few characters.
	</description>
	<alt>
	<character set="ID_Continue"/>
	<character set="Sc"/> <!-- Currency symbols -->
	</alt>
	</production>

well actually the EBNF version that can be downloaded from the website (The legacy one).

How should i implement this? When you look for unicode ID_Start there a jungle of documentation. But i only need the regex of this. Fortunately unicode has made a tool for this https://unicode.org/cldr/utility/regex.jsp?a=%5B%3AID_Start%3A%5D&b=

That leaves the question what about And extended with a few characters?

So perhaps to clear up this section of the spec some of the following could be done:

1. Specify the "extended few characters"
2. Give regular expression in the comments.
3. Links to the unicode tools.

I think the unicode tool gives the regular expression in some standard regex format, but there can be other engines like PCRE which can define other shorthand classes which could be used instead. Therefor it can be useful to include regular expressions in comments.

[Mats-SX]

I believe it is the Pc and Sc classes that are referred to when it says a few characters. So it's ID_Start plus Pc for IdentifierStart and ID_Continue plus Sc for IdentifierPart.

You can check this code for the model's definition of these constructs (derived from Unicode):

openCypher/tools/grammar/src/main/java/org/opencypher/grammar/CharacterSet.java

Lines 310 to 314 in 4a2896e

	ID_Start( union( Ll.set, Lm.set, Lo.set, Lt.set, Lu.set, Nl.set, Other_ID_Start.set )
	.except( Pattern_Syntax.set ).except( Pattern_White_Space.set ) ),
	/** Characters allowed in an identifier. http://unicode.org/reports/tr31/ */
	ID_Continue( union( ID_Start.set, Mn.set, Mc.set, Nd.set, Pc.set, Other_ID_Continue.set )
	.except( Pattern_Syntax.set ).except( Pattern_White_Space.set ) ),;

[flip111]

Will it be acceptable to change anything in the comments of the xml ?

[Mats-SX]

Sure, but please note that in order to accept contributions there is a CLA that must be signed.

[flip111]

Ah ok, thank you for the notice. I won't sign anything so i won't make PR's here. If this isn't important for anyone else, this issue can be closed. Hopefully for other people they will be able to find this issue to get the information that is not (yet) in the xml ..

[Mats-SX]

I'll keep it open for a bit to see if someone (like me) decides to improve the comment, and will close at that point (or before if nothing happens).