Syntax errors (Python)
Deephaven Persistent Queries are programs or "scripts" written in either Python or Groovy. As with any programming language, the interpreter has specific syntax rules around quotes, valid tokens, and sometimes even whitespace. Within your Deephaven query, the select, update, update_view, view, where, and other operations use Formulas. The Deephaven engine interprets formulas to produce efficient Java code; however, formulas may also have syntax errors.
A treatment of Python syntax is beyond the scope of this guide. However, during development, you are likely to write code with misplaced whitespace or an unknown variable. Diagnosing a syntax error revolves around reading the stack trace. For example, the following notebook assigns a value to the variable named val but then references value:
val = 2
if value > 1:
print(value)
else:
print("value is too small")
When executed from a Code Studio or Notebook, the snippet produces the following traceback:
r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'NameError'>
Value: name 'value' is not defined
Line: 3
Namespace: <module>
File: <string>
Traceback (most recent call last):
File "<string>", line 3, in <module>
The type of the Python exception is a NameError occuring on line 3 (where value is first used). Similarly, the else statement may be incorrect, as in the following snippet:
value = 2
if value > 1:
print(value)
else:
print("value is too small")
This produces a traceback referencing that line number:
r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'SyntaxError'>
Value: invalid syntax (<string>, line 5)
Line: <not available>
Namespace: <not available>
File: <not available>
Traceback (most recent call last):
When executed as part of a Persistent Query, the traceback is the same, but the Java stack trace that wraps it is slightly different:
io.deephaven.enterprise.controller.exception.ScriptException: java.lang.RuntimeException: Error in Python interpreter:
Type: <class 'SyntaxError'>
Value: invalid syntax (<string>, line 5)
Line: <not available>
Namespace: <not available>
File: <not available>
Traceback (most recent call last):
Use the line number and Python's error message to examine your script and find the syntax error. Using an editor that offers syntax highlighting (such as the Deephaven Code Studio, Visual Studio Code, or PyCharm) is often effective in identifying incorrect syntax, such as missing braces and quotes or other simple errors.
Formula syntax errors
Deephaven formulas and filters use a Java-like syntax to express operations on table columns, variables from your session, and globally imported functions. As in any programming language, proper syntax is necessary. In the following example, the Python syntax is correct, but the closing parenthesis inside the formula is missing:
from deephaven import empty_table
x = empty_table(1).update("X=2 * (3 + 4")
This results in a traceback of:
1 r-Scheduler-Serial-1 | .c.ConsoleServiceGrpcImpl | Error running script: java.lang.RuntimeException: Error in Python interpreter:
2 Type: <class 'deephaven.dherror.DHError'>
3 Value: table update operation failed. : Invalid expression 2 * (3 + 4
4 Traceback (most recent call last):
5 File "/usr/illumon/coreplus/venv/latest/lib/python3.10/site-packages/deephaven/table.py", line 994, in update
6 return Table(j_table=self.j_table.update(*formulas))
7 RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: 2 * (3 + 4
8 at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:201)
9 at io.deephaven.engine.table.impl.select.SwitchColumn.initDef(SwitchColumn.java:64)
10 at io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.createContext(SelectAndViewAnalyzer.java:128)
11 at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$34(QueryTable.java:1527)
12 at io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
13 at io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$35(QueryTable.java:1509)
14 at io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3646)
15 at io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1508)
16 at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1487)
17 at io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:98)
18 at io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
19 at org.jpy.PyLib.executeCode(Native Method)
20 at org.jpy.PyObject.executeCode(PyObject.java:138)
21 at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
22 at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
23 at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
24 at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
25 at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
26 at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
27 at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
28 at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
29 at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
30 at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
31 at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
32 at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
33 at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
34 at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
35 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
36 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
37 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
38 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
39 at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
40 at java.base/java.lang.Thread.run(Thread.java:840)
41 caused by io.deephaven.engine.table.impl.lang.QueryLanguageParser$QueryLanguageParseException:
42 Having trouble with the following expression:
43 Full expression : 2 * (3 + 4
44 Expression having trouble :
45 Exception type : java.lang.IllegalArgumentException
46 Exception message : Invalid expression 2 * (3 + 4: [Encountered unexpected token:<EOF>
47 at line 1, column 10.
48 Was expecting one of:
49 "!="
50 "%"
51 "%="
52 "&"
53 "&&"
54 "&="
55 ")"
56 "*"
57 "*="
58 "+"
59 "+="
60 "-"
61 "-="
62 "->"
63 "/"
64 "/="
65 "::"
66 "<"
67 "<<="
68 "<="
69 "="
70 "=="
71 ">"
72 ">="
73 ">>="
74 ">>>="
75 "?"
76 "^"
77 "^="
78 "instanceof"
79 "|"
80 "|="
81 "||"
82 Problem stacktrace :
83 com.github.javaparser.GeneratedJavaParser.generateParseException(GeneratedJavaParser.java:13708)
84 com.github.javaparser.GeneratedJavaParser.jj_consume_token(GeneratedJavaParser.java:13554)
85 com.github.javaparser.GeneratedJavaParser.PrimaryPrefix(GeneratedJavaParser.java:4760)
86 com.github.javaparser.GeneratedJavaParser.PrimaryExpression(GeneratedJavaParser.java:4548)
87 com.github.javaparser.GeneratedJavaParser.PostfixExpression(GeneratedJavaParser.java:4412)
88 com.github.javaparser.GeneratedJavaParser.UnaryExpressionNotPlusMinus(GeneratedJavaParser.java:4382)
89 com.github.javaparser.GeneratedJavaParser.UnaryExpression(GeneratedJavaParser.java:4255)
90 com.github.javaparser.GeneratedJavaParser.MultiplicativeExpression(GeneratedJavaParser.java:4152)
91 com.github.javaparser.GeneratedJavaParser.AdditiveExpression(GeneratedJavaParser.java:4070)
92 com.github.javaparser.GeneratedJavaParser.ShiftExpression(GeneratedJavaParser.java:4025)
93 com.github.javaparser.GeneratedJavaParser.RelationalExpression(GeneratedJavaParser.java:3965)
94 com.github.javaparser.GeneratedJavaParser.InstanceOfExpression(GeneratedJavaParser.java:3884)
95 com.github.javaparser.GeneratedJavaParser.EqualityExpression(GeneratedJavaParser.java:3816)
96 com.github.javaparser.GeneratedJavaParser.AndExpression(GeneratedJavaParser.java:3771)
97 com.github.javaparser.GeneratedJavaParser.ExclusiveOrExpression(GeneratedJavaParser.java:3731)
98 com.github.javaparser.GeneratedJavaParser.InclusiveOrExpression(GeneratedJavaParser.java:3691)
99 com.github.javaparser.GeneratedJavaParser.ConditionalAndExpression(GeneratedJavaParser.java:3651)
100 com.github.javaparser.GeneratedJavaParser.ConditionalOrExpression(GeneratedJavaParser.java:3606)
101 com.github.javaparser.GeneratedJavaParser.ConditionalExpression(GeneratedJavaParser.java:3574)
102 com.github.javaparser.GeneratedJavaParser.Expression(GeneratedJavaParser.java:3387)
103 com.github.javaparser.GeneratedJavaParser.ExpressionParseStart(GeneratedJavaParser.java:8192)
104 com.github.javaparser.JavaParser.parse(JavaParser.java:123)
105 io.deephaven.engine.table.impl.lang.JavaExpressionParser.parseExpression(JavaExpressionParser.java:27)
106 io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:311)
107 io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:212)
108 io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:240)
109 io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:122)
110 io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:181)
111 io.deephaven.engine.table.impl.select.SwitchColumn.initDef(SwitchColumn.java:64)
112 io.deephaven.engine.table.impl.select.analyzers.SelectAndViewAnalyzer.createContext(SelectAndViewAnalyzer.java:128)
113 io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$34(QueryTable.java:1527)
114 io.deephaven.engine.table.impl.perf.QueryPerformanceRecorder.withNugget(QueryPerformanceRecorder.java:369)
115 io.deephaven.engine.table.impl.QueryTable.lambda$selectOrUpdate$35(QueryTable.java:1509)
116 io.deephaven.engine.table.impl.QueryTable.memoizeResult(QueryTable.java:3646)
117 io.deephaven.engine.table.impl.QueryTable.selectOrUpdate(QueryTable.java:1508)
118 io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:1487)
119 io.deephaven.engine.table.impl.QueryTable.update(QueryTable.java:98)
120 io.deephaven.api.TableOperationsDefaults.update(TableOperationsDefaults.java:94)
121 org.jpy.PyLib.executeCode(Native Method)
122 org.jpy.PyObject.executeCode(PyObject.java:138)
123 io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
124 io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
125 io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
126 io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
127 io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
128 io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
129 io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
130 io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
131 io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
132 io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
133 io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
134 io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
135 io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
136 io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
137 java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
138 java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
139 java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
140 java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
141 io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
142 java.base/java.lang.Thread.run(Thread.java:840)]
143 at io.deephaven.engine.table.impl.lang.JavaExpressionParser.parseExpression(JavaExpressionParser.java:30)
144 at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:311)
145 at io.deephaven.engine.table.impl.lang.QueryLanguageParser.<init>(QueryLanguageParser.java:212)
146 at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:240)
147 at io.deephaven.engine.table.impl.select.codegen.FormulaAnalyzer.parseFormula(FormulaAnalyzer.java:122)
148 at io.deephaven.engine.table.impl.select.DhFormulaColumn.initDef(DhFormulaColumn.java:181)
149 ... 32 more
150 Line: 996
151 Namespace: update
152 File: /usr/illumon/coreplus/venv/latest/lib/python3.10/site-packages/deephaven/table.py
153 Traceback (most recent call last):
154 File "<string>", line 3, in <module>
155 File "/usr/illumon/coreplus/venv/latest/lib/python3.10/site-packages/deephaven/table.py", line 996, in update
156 at org.jpy.PyLib.executeCode(Native Method)
157 at org.jpy.PyObject.executeCode(PyObject.java:138)
158 at io.deephaven.engine.util.PythonEvaluatorJpy.evalScript(PythonEvaluatorJpy.java:73)
159 at io.deephaven.integrations.python.PythonDeephavenSession.lambda$evaluate$1(PythonDeephavenSession.java:205)
160 at io.deephaven.util.locks.FunctionalLock.doLockedInterruptibly(FunctionalLock.java:51)
161 at io.deephaven.integrations.python.PythonDeephavenSession.evaluate(PythonDeephavenSession.java:205)
162 at io.deephaven.engine.util.AbstractScriptSession.lambda$evaluateScript$0(AbstractScriptSession.java:165)
163 at io.deephaven.engine.context.ExecutionContext.lambda$apply$0(ExecutionContext.java:196)
164 at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:207)
165 at io.deephaven.engine.context.ExecutionContext.apply(ExecutionContext.java:195)
166 at io.deephaven.engine.util.AbstractScriptSession.evaluateScript(AbstractScriptSession.java:165)
167 at io.deephaven.engine.util.DelegatingScriptSession.evaluateScript(DelegatingScriptSession.java:72)
168 at io.deephaven.engine.util.ScriptSession.evaluateScript(ScriptSession.java:75)
169 at io.deephaven.server.console.ConsoleServiceGrpcImpl.lambda$executeCommand$4(ConsoleServiceGrpcImpl.java:193)
170 at io.deephaven.server.session.SessionState$ExportBuilder.lambda$submit$2(SessionState.java:1537)
171 at io.deephaven.server.session.SessionState$ExportObject.doExport(SessionState.java:995)
172 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
173 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
174 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
175 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
176 at io.deephaven.server.runner.scheduler.SchedulerModule$ThreadFactory.lambda$newThread$0(SchedulerModule.java:100)
177 at java.base/java.lang.Thread.run(Thread.java:840)
The summary information is on lines 3 and 4, indicating an error with the formula 2 * 3 ( + 4.
...
3 Value: table update operation failed. : Invalid expression 2 * (3 + 4
...
7 RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: 2 * (3 + 4
The traceback and stack traces indicate where the error occurred, but the relevant details begin on line 41 and continue until line 81:
41 caused by io.deephaven.engine.table.impl.lang.QueryLanguageParser$QueryLanguageParseException:
42 Having trouble with the following expression:
43 Full expression : 2 * (3 + 4
44 Expression having trouble :
45 Exception type : java.lang.IllegalArgumentException
46 Exception message : Invalid expression 2 * (3 + 4: [Encountered unexpected token:<EOF>
47 at line 1, column 10.
...
In this case, the message indicates that the <EOF> or end of file token was encountered at column 10, but another token was expected. Many options are listed, including the correct closing parenthesis on line 55. In practice, with so many options, you must carefully examine the formula around the indicated position to determine what syntax is incorrect.
Invalid operator overloads
You may also have errors related to operators. The Deephaven formula parser converts operators to method calls to permit users to naturally express mathematic operations between various types unsupported by Java (for example, Deephaven permits you to add two BigIntegers together using the plus (+) operator). This is similar to operator overloading in many programming languages. In the example below, the formula attempts to multiply an integer and a string, which is not supported by Deephaven formulas:
from deephaven import empty_table
x = empty_table(1).update("X=2 * `abc`")
The following error is produced:
RuntimeError: io.deephaven.engine.table.impl.select.FormulaCompilationException: Formula compilation error for: 2 * `abc`
...
Having trouble with the following expression:
Full expression : 2 * "abc"
Expression having trouble :
Exception type : io.deephaven.engine.table.impl.lang.QueryLanguageParser$ParserResolutionFailure
Exception message : Cannot find method multiply(int, java.lang.String)
...
The relevant detail is that no method is named multiply for int and Strings. The following operators may also be present in your stack trace, corresponding to an operator used in your formula:
| Operator | Method |
|---|---|
+ | plus |
- | minus |
* | multiply |
/ | divide |
% | remainder |
= or == | eq |
< | less |
<= | lessEquals |
> | greater |
>= | greaterEquals |
^ | xor |
| | binaryOr |
^ | binaryAnd |
|| | or |
&& | and |
Invalid use of in
The in keyword can only be used in a standalone expression and cannot be combined with other operators or formula elements (even if surrounded in parenthesis). For example, this is valid syntax:
from deephaven import empty_table
a = [1, 2, 3]
x = empty_table(10).update("X=ii").where("X in a")
However, in this example in is combined with another expression:
from deephaven import empty_table
a = [1, 2, 3]
x = empty_table(10).update("X=ii").where("X % 2 == 0 && X in a")
This results in an error:
Having trouble with the following expression:
Full expression : X % 2 == 0 && X in a
Expression having trouble :
Exception type : java.lang.IllegalArgumentException
Exception message : Invalid expression X % 2 == 0 && X in a: [Encountered unexpected token: "in" <IDENTIFIER>
at line 1, column 17.
When the in operator begins the combined expression, then an alternative error is produced indicating that the expression is not a valid long value:
RuntimeError: java.lang.IllegalArgumentException: Failed to convert literal value <a && X % 2 == 0> for column "X" of type long
...
caused by java.lang.NumberFormatException: For input string: "a && X % 2 == 0"