Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ Furthermore, the fact that a given RFC has been accepted and is
'active' implies nothing about what priority is assigned to its
implementation, nor whether anybody is currently working on it.

Modifications to active RFC's can be done in followup PR's. We strive
Modifications to active RFCs can be done in followup PRs. We strive
to write each RFC in a manner that it will reflect the final design of
the feature; but the nature of the process means that we cannot expect
every merged RFC to actually reflect what the end result will be at
Expand All @@ -129,7 +129,7 @@ If you are interested in working on the implementation for an 'active'
RFC, but cannot determine if someone else is already working on it,
feel free to ask (e.g. by leaving a comment on the associated issue).

## Reviewing RFC's
## Reviewing RFCs

Members of the core team will attempt to review some set of open RFC
pull requests on a regular basis. If a core team member believes an RFC PR is ready to be accepted into active status, they can approve the PR using GitHub's review feature to signal their approval of the RFC.
Expand Down
16 changes: 8 additions & 8 deletions fedb/api-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

# Summary

HTTP interface is one of the most friendly interface and many developers like using http interface to develop their program. REST API also help us to show demo simply. So we want to support uing http api to access FEDB.
HTTP interface is one of the most friendly interface and many developers like using http interface to develop their program. REST API also help us to show demo simply. So we want to support using http api to access FEDB.

# Detailed Design

Expand All @@ -30,10 +30,10 @@ HTTP interface is one of the most friendly interface and many developers like us
JSON does not have a built-in type for date/timestamp values. We store the date value as a String in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format and store the timestamp as a Number, e.g '2021-05-06', 1620471840256.

### HTTP Status
API Server will return 200 in most case and store the error in the code of reponse's json.
API Server will return 200 in most case and store the error in the code of response's json.

### Put
reqeust url: http://ip:port/dbs/{db_name}/tables/{table_name}
request url: http://ip:port/dbs/{db_name}/tables/{table_name}
http method: PUT
request body:
```
Expand Down Expand Up @@ -64,7 +64,7 @@ response:
Note: Put multi records to ApiServer is not supported. If put multi records in value array, the error will be return.

### Execute Procedure
reqeust url: http://ip:port/dbs/{db_name}/procedures/{procedure_name}
request url: http://ip:port/dbs/{db_name}/procedures/{procedure_name}
http method: POST
request body:
```
Expand All @@ -91,7 +91,7 @@ response:
|0|ok|
|-1|execute procedure failed|

Note: If there is no common col, the reponse json will not contain common_cols_data field
Note: If there is no common col, the response json will not contain common_cols_data field

### Get Procedure
request url: http://ip:port/dbs/{db_name}/procedures/{procedure_name}
Expand All @@ -117,11 +117,11 @@ response:
|0|ok|
|-1|procedure not found|

Note: If there is no common col, the reponse json will not contain input_common_cols/output_common_cols field
Note: If there is no common col, the response json will not contain input_common_cols/output_common_cols field

## Server design

We use [brpc HTTP Service](https://github.com/apache/incubator-brpc/blob/master/docs/en/http_service.md) framework to tackle the http request. So what need to do is forwarding the reques to FEDB server with fedb sdk in brpc http service implementation.
We use [brpc HTTP Service](https://github.com/apache/incubator-brpc/blob/master/docs/en/http_service.md) framework to tackle the http request. So what need to do is forwarding the request to FEDB server with fedb sdk in brpc http service implementation.

1. Design proto file.
```
Expand Down Expand Up @@ -177,5 +177,5 @@ For complete document, refer to [here](https://postgrest.org/en/stable/api.html#

This method make API Interfaces more complicated.

# Adoption stratege
# Adoption strategy
If we implement this proposal, there is no impact on existing interfaces.
22 changes: 11 additions & 11 deletions hybridse/using-zetasql.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,13 @@ We are going to use ZetaSQL as our new SQL front-end. The document is trying to

# Motivation

HybridSE is LLVM-based SQL engine written in C++. It aims at supporting standard ANSI SQL and some new SQL traits at the same time. However, as a newly engine, HybridSE only supports a limited set of SQL syntaxs currently and its syntax error message is unfriendly. For example, the syntax error message can't tell the location of wrong expression.
HybridSE is LLVM-based SQL engine written in C++. It aims at supporting standard ANSI SQL and some new SQL traits at the same time. However, as a newly engine, HybridSE only supports a limited set of SQL syntaxes currently and its syntax error message is unfriendly. For example, the syntax error message can't tell the location of wrong expression.

ZetaSQL is a SQL Analyzer Framework from Google. We are considering using it as HybridSE's SQL parser and analyzer for following reasons:

- ZetaSQL has been used for BigQuery, Spanner and DataflowSQL by Google
- ZetaSQL is a C++ SQL parser, which can be easily integrate into HybridSE
- ZetaSQL supports standard ANSI SQL and provides clearly error messge.
- ZetaSQL supports standard ANSI SQL and provides clearly error message.



Expand All @@ -40,21 +40,21 @@ We do some survey on ZetaSQL. ZetaSQL provides many APIs and services.

![zetasql analyzer workflow](./images/image-zetasql-workflow.png)

The figure above demonstrate the zetasql's analyze workflow. But we **only** use its parser module, since It can fullfill our requirements.
The figure above demonstrate the zetasql's analyze workflow. But we **only** use its parser module, since It can fulfill our requirements.

## Do not use ZetaSQL analyzer

We won't use zetasql analyer in this vesion for the following reasons:
We won't use zetasql analyzer in this version for the following reasons:

- HybridSE already has its statement validation module. We use `SchemaContext` and `ExprPass` to validate column, expression and functions. ZetaSQL analyzer might not meet our need, since it would be difficult to validate our udf and udaf. A lot works will be done with few benefits.
- Analyzer outputs AnalyzerOutput which mainly contains ResolvedStatement. It would be much harder to convert `ResolvedExpression` comparing to convert `ASTExpression`. For instance:
- `ExprNode` and `ASTExpression` both use `BinaryExpression` to represent operator like `+`, `-`, `*`, `/`, `MOD`, but `ResolvedExpression` use `FunctionCallNode` instead.
- `ResolvedExpression` do not have `CastNode`, `CaseWhenNode`
- ZetaSQL can validate SQL statement when we register catalog information to zetasql. It would be better if we use analyzer after refactor hyrbidse `Catalog`.
- ZetaSQL can validate SQL statement when we register catalog information to zetasql. It would be better if we use analyzer after refactor hybridse `Catalog`.

## Build and Test zetasql parser

In order to build and test in different sysmtem, we can simplify build and test packages.
In order to build and test in different system, we can simplify build and test packages.

Check [Discussion link](https://github.com/4paradigm/HybridSE/discussions/42) for more details.

Expand Down Expand Up @@ -129,7 +129,7 @@ https://github.com/jingchen2222/zetasql/pull/3/files#diff-7ebc691e7f3f59ec4bc171

### Use Zetasql as cmake dependency

Although zetasql is a bazel-based project and have its dependencies in the mean time, We'll use zetasql in a cmake proeject as following steps (Checkout [zetasql-sample](https://github.com/aceforeverd/zetasql-sample) to see the details.):
Although zetasql is a bazel-based project and have its dependencies in the mean time, We'll use zetasql in a cmake project as following steps (Checkout [zetasql-sample](https://github.com/aceforeverd/zetasql-sample) to see the details.):

- Prepare headers
- find every header file and install to a include directory
Expand Down Expand Up @@ -171,7 +171,7 @@ We will keep using `ExprNode` in hybridse. This will introduce least changes in

**Cons**:

We hava `ExprNode` and `ASTExpression` in hybridse at the same time. This is redundant.
We have `ExprNode` and `ASTExpression` in hybridse at the same time. This is redundant.

### Convert `ASTExpression` to `ExprNode`

Expand Down Expand Up @@ -294,7 +294,7 @@ frame_unit:
### Create SQL

```SQL
# This is syntax decription from zetasql fro create table statement
# This is syntax description from zetasql for create table statement
create_table_statement:
"CREATE" opt_or_replace opt_create_scope "TABLE" opt_if_not_exists
maybe_dashed_path_expression opt_table_element_list
Expand Down Expand Up @@ -398,7 +398,7 @@ table_constraint_definition:
) OPTIONS REPLICANUM = 3, PARTITIONNUM = 8, LEADER = endpoint1, LEADER = endpoint1, FOLLOWER = endpoint2, FOLLOWER = endpoint3, FOLLOWER = endpoint3;
```

- **We sugguest design distribution options in standard SQL style.**
- **We suggest design distribution options in standard SQL style.**

- **We will implement `table_options` syntax considering compatibility.**

Expand Down Expand Up @@ -649,7 +649,7 @@ If the tips above don't help, you can get more help from [slack channel](https:/

### Plan Error Tips

We will discuss plan error tips in the furture works.
We will discuss plan error tips in the future works.

## Test

Expand Down
22 changes: 11 additions & 11 deletions style-guide/code-convention.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@

[motivation]: motivation

Archive better code quality by:
Achieve better code quality by:

- Enforce well-known code convention like Google's Style to main programing languages
- Enforce well-known code convention like Google's Style to main programming languages
- Use linter check tools and bootstrap those tools in automatic CICD check

# Detailed design
Expand Down Expand Up @@ -43,39 +43,39 @@ Data-Serialization language like yaml/json, should follow:
- max line length: 120 character
- prefer double quote (json excluded)

## Suggested linter and formater
## Suggested linter and formatter

By default, appropriate linters should setup in CICD to enforce convention in place like Pull Request. Formater is usually used in local development to quickly resolve some style issues. There is no need to use exactly same tools and configuration, any tools respect [overview](#overview) rules should suffice.
By default, appropriate linters should setup in CICD to enforce convention in place like Pull Request. Formatter is usually used in local development to quickly resolve some style issues. There is no need to use exactly same tools and configuration, any tools respect [overview](#overview) rules should suffice.

| language | linter | linter config | formater | format config |
| language | linter | linter config | formatter | format config |
| --------- | ---------------------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------------ |
| cpp | [cpplint](https://github.com/cpplint/cpplint) | default config | [clang-format](https://clang.llvm.org/docs/ClangFormat.html) | [.clang-format](https://github.com/4paradigm/HybridSE/blob/main/.clang-format) |
| java | [checkstyle](https://checkstyle.sourceforge.io/) | [style.xml](https://github.com/4paradigm/HybridSE/blob/main/java/style_checks.xml) | [eclipse.jdt](https://github.com/eclipse/eclipse.jdt.core) | [eclipse-formater.xml](https://github.com/4paradigm/HybridSE/blob/main/java/eclipse-formatter.xml) |
| java | [checkstyle](https://checkstyle.sourceforge.io/) | [style.xml](https://github.com/4paradigm/HybridSE/blob/main/java/style_checks.xml) | [eclipse.jdt](https://github.com/eclipse/eclipse.jdt.core) | [eclipse-formatter.xml](https://github.com/4paradigm/HybridSE/blob/main/java/eclipse-formatter.xml) |
| scala | [scalastyle](http://www.scalastyle.org/) | [scala_style.xml](https://github.com/4paradigm/NativeSpark/blob/main/native-spark/scala_style.xml) | [scalafmt](https://scalameta.org/scalafmt) | |
| python | [pylint](https://www.pylint.org/) | [pylintrc](https://github.com/4paradigm/HybridSE/blob/main/pylintrc) | [yapf](https://github.com/google/yapf) | default with google style |
| yaml/json | - | - | [prettier](https://prettier.io/) | [prettierrc](https://github.com/4paradigm/HybridSE/blob/main/.prettierrc.yml) |
| shell | [shellcheck](https://github.com/koalaman/shellcheck) | default | [shfmt](https://github.com/mvdan/sh) | default with indednt = 4 |
| shell | [shellcheck](https://github.com/koalaman/shellcheck) | default | [shfmt](https://github.com/mvdan/sh) | default with indent = 4 |

formater do not always solve lint error, dig hard to the lint rule.
formatter do not always solve lint error, dig hard to the lint rule.

### Cpp

[style-cpp]: style-cpp

- linter: cpplint, with `linelength=120`
- formater: clang-format and [.clang-format](https://github.com/4paradigm/HybridSE/blob/main/.clang-format) based on Google style config
- formatter: clang-format and [.clang-format](https://github.com/4paradigm/HybridSE/blob/main/.clang-format) based on Google style config

### java

- linter: checkstyle
- use template [google_checks.xml](https://github.com/checkstyle/checkstyle/blob/master/src/main/resources/google_checks.xml)
- checkstyle maven plugin: <https://maven.apache.org/plugins/maven-checkstyle-plugin/>
- formater: [spotless](https://github.com/diffplug/spotless/tree/master/plugin-maven) provide eclipse.jdt integration.
- formatter: [spotless](https://github.com/diffplug/spotless/tree/master/plugin-maven) provide eclipse.jdt integration.

### Scala

- linter: scalastyle with [maven plugin](http://www.scalastyle.org/maven.html)
- formater: scalafmt with [spotless](https://github.com/diffplug/spotless/tree/master/plugin-maven)
- formatter: scalafmt with [spotless](https://github.com/diffplug/spotless/tree/master/plugin-maven)

### Python

Expand Down
16 changes: 8 additions & 8 deletions style-guide/commit-convention.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Convention for PR/Issue/Commits on GitHub

[motivation]: motivation

To standardize workflows in GitHub. Create a clean and mantainable code space.
To standardize workflows in GitHub. Create a clean and maintainable code space.

# Guide-level explanation

Expand All @@ -24,9 +24,9 @@ To standardize workflows in GitHub. Create a clean and mantainable code space.

1. commit pushed to default branch (merge commit exclude) must follow [conventional commit](https://www.conventionalcommits.org/en/v1.0.0/)
2. every commit pushed must sign with gpg, and shows verified signature in GitHub page
3. commiter user: use a valid GitHub username to commit as much as you can
3. committer user: use a valid GitHub username to commit as much as you can
4. force push is prohibited in most case, it should use only for security patch
5. idealy, we encourage conventional commit in every related branch, also name branch with similar style, e.g `fix/resolve-core-dump`
5. ideally, we encourage conventional commit in every related branch, also name branch with similar style, e.g `fix/resolve-core-dump`

rule 1 and 2 is enforced.

Expand All @@ -36,7 +36,7 @@ rule 1 and 2 is enforced.

Follow the issue template defined in each project, which should at least have

- bug report tempate: e.g [bug report](https://github.com/4paradigm/fedb/blob/main/.github/ISSUE_TEMPLATE/bug_report.md)
- bug report template: e.g [bug report](https://github.com/4paradigm/fedb/blob/main/.github/ISSUE_TEMPLATE/bug_report.md)
- feature request template

## Pull Request
Expand Down Expand Up @@ -67,15 +67,15 @@ Follow the issue template defined in each project, which should at least have
- use [semantic versioning](https://semver.org)
- a semantic version tag should start with a `v` prefix, e.g `v1.0.1`
- semantic version tags (start with `v` prefix) is immutable, force push is prohibited.
- release asserts should provide DIGESTS{.asc,} file as well, which contains hashes and gpg sign
- release assets should provide DIGESTS{.asc,} file as well, which contains hashes and gpg sign

# Reference-level explanation

[reference-level-explanation]: #reference-level-explanation

## Conventional Commits Reference

[conventional-commits-ref]: #coventional-commits-ref
[conventional-commits-ref]: #conventional-commits-ref

Complete rules available in

Expand Down Expand Up @@ -107,7 +107,7 @@ Convention commit define `<type>` of follow types:
'perf', // A code change that improves performance
'refactor', // A code change that neither fixes a bug nor adds a feature
'revert', // Reverts a previous commit
'style', // Changes that do not affect the meaning of the code (white-space, formating)
'style', // Changes that do not affect the meaning of the code (white-space, formatting)
'test' // Adding missing tests or correcting existing tests
```

Expand Down Expand Up @@ -135,7 +135,7 @@ Enforce conventional style locally, use

Detail usage also available in #23

Enforce conventional style remotely, use GitHub action to archive it in PR
Enforce conventional style remotely, use GitHub action to achieve it in PR

- [pr name lint action](https://github.com/JulienKode/pull-request-name-linter-action)
- [commit lint action](https://github.com/wagoid/commitlint-github-action)